AIHeroThis is a funny exercise to show a solution for because I really want you to think about your solution and how good it is. What I noticed when I was implementing this is that scorers weren't really helping me very much. I was able to get some outputs that I was happy with just by visually checking them.
"React Query Updates" versus "React Query Recent Changes", "Electric Car Research" versus "Electric Car Comparison". And so I didn't feel the need to introduce scorers here myself.
The bulk of my work ended up going into my prompt.
Find the most concise title that captures the essence of the conversation. Titles should be at most 30 characters.
I added this rule too:
If there are acronyms used in the conversation, use them in the title.
That was because I noticed this edge case here, "FAANG Company Preparing for FAANG Interviews" - without this rule it was doing something strange. Further down I added this extra rule:
Since all items in the list of conversations do not use the word chat, conversation, or discussion.
That was because I was noticing that it would say "TypeScript versus JavaScript discussion", "plant issues discussion". Whereas I wanted something that was a bit more visually interesting when you scan down it. And finally, of course, I had a bit of output formatting:
Generate a title for the conversation. Return only the title.
I hope what this tells you about evals is that they're mostly a way of pushing a lot of data through your system to see what happens. We might think of them as unit tests, but they're really more like a playground and the scorers are really there just to give you some signal. But I'm very, very keen to see your solutions to this problem.
So go into the Discord and post them now. That would be awesome!
Thank you so much and I will see you in the next one.
evalite("Chat Title Generation", {data: () => dataForEvalite,task: async (input) => {const result = await generateText({model: google("gemini-2.0-flash-lite"),prompt: `You are a helpful assistant that can generate titles for conversations.<conversation-history>${input}</conversation-history>Find the most concise title that captures the essence of the conversation.Titles should be at most 30 characters.Titles should be formatted in sentence case, with capital letters at the start of each word. Do not provide a period at the end.Use no punctuation or emojis.If there are acronyms used in the conversation, use them in the title.Use formal language in the title, like 'troubleshooting', 'discussion', 'support', 'options', 'research', etc.Since all items in the list are conversations, do not use the word 'chat', 'conversation' or 'discussion' in the title - it's implied by the UI.The title will be used for organizing conversations in a chat application.Generate a title for the conversation.Return only the title.`,})return result.text},scorers: [],})
