This is going to be a multi-part series written as I put together my lecture for Stanford CS 25: Transformers United. Part one includes chapter 0, kickstart, as well.
I’ve created a HuggingFace collection of all the artifacts in the above figure and the resulting lecture. This slides are here.
Here the models start to move up a bit on the leaderboard, with the likes of Llama 2 chat and Guanaco, though many of the models in this era were not added to ChatBotArena. The Arena wasn’t as big of an operation then, so many popular models never made it on. From here on, the LMSYS rankings are almost always available.
As in every post in this series, let’s look at a few prompts and how the models respond. I recommend you compare these to future posts (I’ll include the ChatGPT reference in all of them).
Question (MT Bench question 101, reasoning): Imagine you are participating in a race with a group of people. If you have just overtaken the second person, what’s your current position? Where is the person you just overtook?
Dolly v2 12b
If you have just overtaken the second person, the second person is behind you. The person you just overtook is somewhere in front of you, since you just overtook someone.
Vicuna 7b v1.3
If I have just overtaken the second person, then I am currently in second place. The person I just overtook would be in third place.
ChatGPT4
If you've just overtaken the second person in the race, your current position is now second place. The person you overtook, who was previously in second, is now in third place.