AIHeroOkay, we now have a generateText call inside the task. We're using gemini-2.0-flash-lite. We're providing a simple prompt with an ask and with an output format and passing in the question.
When we run this, we'll notice a few things. First of all, we have a localhost dev server running on 3006.
If we follow the link here we can see the history of all the runs I've done and a bunch of results.
We can see the output for the capital of France question was Paris, Berlin for Germany and Rome for Italy. And our score at the end is showing:
100%
We also have this repeated inside our CLI here. This is a watch server too, so if I change this to Milan for instance, and run it again, we can see that it correctly returned Rome, but because our input is incorrect, we got:
0%
in the score.
If I want to, I can click on a row of a result to look at more information. I can see how long the trace took and when it was run, as well as a detailed breakdown of all of the inputs, expected and outputs. This will become more important to us as our evals become more complex.
So that's the basics of Evalite. We have data, which is the data points that we're putting through our system.
We have the task, which is the thing that tests our system. And then we have the scorers, which provide a bit of background information to help us look at the outputs.
Nice work, and I will see you in the next one.
task: async (input) => {const capitalResult = await generateText({model: google('gemini-2.0-flash-lite'),prompt: `You are a helpful assistant that can answer questions about the capital of countries.<question>${input}</question>Answer the question.Reply only with the capital of the country.`,});return capitalResult.text;},
data: () => [{input: 'What is the capital of France?',expected: 'Paris',},{input: 'What is the capital of Germany?',expected: 'Berlin',},{input: 'What is the capital of Italy?',expected: 'Rome',},],
scorers: [{name: 'includes',scorer: ({ input, output, expected }) => {return output.includes(expected!) ? 1 : 0;},},],
