Model Router Solution

All right, I'm going to ask my system a classic one. How many Rs are there in the word strawberry? We can see we get an answer from the AI and we have a model: advanced.

In the logs too, we can see it took about 500 milliseconds to complete and we got a 1 indicating we should use the advanced model.

If I ask it, how are you, then it replies with the basic model and just gives a kind of stock response.

And again we can see that the result was a 0 indicating use the basic one and it took about half a second. Inside the code we can see that I've given it the BASIC_MODEL to do the calculation.

const ADVANCED_MODEL = google("gemini-2.0-flash")
const BASIC_MODEL = google("gemini-2.0-flash-lite")

In the system prompt I've given it some task context.

You will need to determine whether to use an advanced model or a basic model, based on the question being asked.

In the rules we can see that we have

If the question is about something trivial, use the basic model.

And here's where the strawberry thing came in.

If the question involves any kind of counting or math, use the advanced model.

system: `
  You are a model router. Your job is to figure out whether to use an advanced model or a basic model.

  You will be given a conversation history. You will need to determine whether to use an advanced model or a basic model, based on the question being asked.

  <rules>
    - If the question is about something trivial, use the basic model.
    - If the question involves any kind of counting or math, use the advanced model.
  </rules>

  

We've got that nice output format at the end.

Return a single number: 0 or 1.
Return 0 to choose the basic model.
Return 1 to choose the advanced model.

Further down we can see we have a modelSelected where if the model opts into the advanced model then we choose advanced.

const modelSelected =
  modelRouterResult.text.trim() === "1"
    ? "advanced"
    : // Note that the fallback is the basic model
      "basic"

It's worth noting that the fallback here is the basic model. The only way we can get the advanced model is if the model router returns exactly what we expect. Because it could return 1, or it could return 0, or it could return something completely random. And if it returns something completely random, then we fall back to the basic model. Finally, let's look at the message metadata below.

Inside here, we're waiting for the start chunk, and we're sending back the model that was selected. If we run our query again and open up the stream we can see that the message metadata has been sent alongside the start part. So that means it's available from the very start of the stream.

If for whatever reason we wanted to delay sending the metadata until the end, then we could change it to finish. But for this use case, start makes sense to me.

Model routers like this are a fantastic optimization on an existing system. They let you cut down costs and also provide a better experience. Because if you can use a cheaper model, then you pretty much always should because the response will be faster. And just like with guardrails, the model router is really easy to eval. You build up a set of basic queries and a set of advanced queries, and you just check that the right result comes back.

Nice work, and I'll see you in the next one.