LLM as a Judge Solution

Okay, let's take a look at the code for our scorer first. The schema is the bit I want to examine first. We are providing some feedback here:

A short feedback message about the answer.

And then a score with a z.enum of A, B, C, or D:

This then gets mapped into a score down below so we're confident we can return a number here:

Inside the messages above, that's where we're passing in all the good stuff, where we pass in the chain of thought paper and a text part:

The answer you are evaluating is:

The original question posed was:

We could wrap these in XML tags, but I found that wasn't quite necessary. In Evalite then, we are hitting 100%, getting lots of quotes here from this output. And if we click into here we can see the metadata for the attribution scorer:

The answer has been accurately identified by the paper, including accurate quotes and source citations.

If we click into the next one down below, again we can see that it says:

All claims are backed up by the paper.

So using Evalite and the AI SDK we are able to give us an assistant that can check the outputs of our system for us, which gives us really good signal especially if something goes wrong. In the exercises coming up we're going to put these into practice and do a bit of comparison between deterministic and LLM as a judge scorers.

So nice work and I will see you in the next one.