This is going to be a multi-part series written as I put together my lecture for Stanford CS 25: Transformers United. Part one includes chapter 0, kickstart, as well.
I’ve created a HuggingFace collection of all the artifacts in the above figure and the resulting lecture. This slides are here.
https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
As in every post in this series, let’s look at a few prompts and how the models respond. I recommend you compare these to future posts (I’ll include the ChatGPT reference in all of them).
Question (MT Bench question 101, reasoning): Imagine you are participating in a race with a group of people. If you have just overtaken the second person, what’s your current position? Where is the person you just overtook?
Qwen 1.5 72B Chat
Cohere Command R Plus
ChatGPT4
If you've just overtaken the second person in the race, your current position is now second place. The person you overtook, who was previously in second, is now in third place.
Question (MT Bench question 116, math): x+y = 4z, x*y = 4z^2, express x-y in z
Qwen 1.5 72B Chat
Cohere Command R Plus