This is going to be a multi-part series written as I put together my lecture for Stanford CS 25: Transformers United. Part one includes chapter 0, kickstart, as well.
I’ve created a HuggingFace collection of all the artifacts in the above figure and the resulting lecture. This slides are here.
https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
As in every post in this series, let’s look at a few prompts and how the models respond. I recommend you compare these to future posts (I’ll include the ChatGPT reference in all of them).
Question (MT Bench question 101, reasoning): Imagine you are participating in a race with a group of people. If you have just overtaken the second person, what’s your current position? Where is the person you just overtook?
Yi 34B Chat
If you have just overtaken the second person in a race, your current position is third. This is because you have moved up one place by overtaking the person who was previously in second place. The person you just overtook is now in fourth place, one position behind you. The first place remains unchanged, occupied by the person who was leading the race before you made your move.
Tulu 2 70B DPO
As an AI, I do not have a physical form and therefore, I do not participate in races. However, I can give you the answer based on the given scenario.
If you have just overtaken the second person, it means you were initially in the third position (assuming that the group consists of at least three people including yourself). Therefore, your current position would be the second position, and the person you just overtook would be in the third position now.
It's important to note that this answer assumes that there are no other racers in front of the person you just overtook. If there are, then the answer would be slightly different, depending on how many racers are ahead.
ChatGPT4
If you've just overtaken the second person in the race, your current position is now second place. The person you overtook, who was previously in second, is now in third place.
Question (MT Bench question 116, math): x+y = 4z, x*y = 4z^2, express x-y in z