Deterministic Eval Solution

All right, let's check the results of our evals. We can see that Tell me about the TypeScript 5.8 release adds this output, which includes at least one markdown link and is the correct length. That looks pretty good to me, let's go and check the code.

Inside our system prompt, we are now passing in the links, rendering them as XML:

<links>
  ${links.map((link) => `<link>${link.title}: ${link.url}</link>`).join("\n")}
</links>

We are adding a rules section here saying:

Answer the question extremely succinctly. ALWAYS include relevant links in your answer.

And I've also provided some XML <markdown-link-example> tags for markdown link examples:

<markdown-link-example>
I really like [this website about cakes](https://www.cakes.com).
</markdown-link-example>
<markdown-link-example>
For more information, check out [this piece of reference material](https://www.cakes.com).
</markdown-link-example>

I found that without these explicit examples it would do really weird things with markdown links.

Then we do the ask down the bottom:

Answer the question, with relevant links. Reply only with the answer.

So that's good, let's now check what our scorers look like. We have the Includes Markdown Links scorer which checks whether markdown links have been found:

{
  name: 'Includes Markdown Links',
  scorer: ({ input, output, expected }) => {
    const markdownLinksFound =
      output.match(/\[.*?\]\((.*?)\)/g) ?? [];

    return markdownLinksFound.      

I'm just going to sort of trust that this regex is good. Knowing regexes it probably isn't but it seems to work okay for now. We check if the markdownLinksFound.length is greater than zero. If it is, we return 1.

Then we go down to Output length which is extremely simple:

{
  name: 'Output length',
  scorer: ({ input, output, expected }) => {
    return output.length < 500 ? 1 : 0;
  },
}

If it's less than 500, we return 1. Otherwise, we return 0.

These deterministic evals are really, really great when you have deterministic criteria you're judging your outputs on. In other words, output length, whether it includes a certain type of character, whether it includes references, lists, all sorts. It's great, especially for checking output formats, checking that certain XML tags have been included.

And using these, you can get a really strong signal on how well your system is performing.

They are also free to run which is unlike other types of evals that we'll look at soon.

Nice work and I will see you in the next one.