Jeff Dean, the head of AI at Google, recently unveiled the latest addition to their AI offerings, a new Gemini model, under the Gemini Pro family. This new model is distinctively tagged with the suffix ‘scale’, hinting at its advanced capabilities and features.
The introduction of the Gemini updates has reportedly enhanced Bard. According to Dean, these updates have elevated Bard to a much superior level, both in terms of performance and the range of capabilities, compared to its initial launch in March.
While Dean did not provide explicit details on what the ‘scale’ suffix entails, the terminology suggests an expansion or a more advanced version of the previously released Pro model, which was not even able to beat GPT 3.5 based on benchmarks.
Google’s Gemini Pro model is the middle tier in the company’s Gemini offerings. It starts with Gemini Nano, which powers consumer devices like smartphones including the Pixel 8 while Gemini Ultra is said to be more sophisticated and will reportedly feature as a paid tier in Bard.
The new Pro model has swiftly ascended to a commendable second place in the neutral Chatbot Arena benchmark. This placement is significant, as it positions the Pro model ahead of two iterations of the GPT-4 model: 0314, released in March 2023, and 0613 from the summer of the same year. However, it still trails behind the GPT-4 Turbo, introduced in November 2023.
Chatbot Arena has adopted the Elo rating system, a method renowned in the realms of chess and e-sports, to critically assess and compare the performance of various language models. Within the Arena, a diverse range of language models are tested against each other. These contests are unique in that they are both anonymous and randomly selected, ensuring an unbiased and equitable comparison.
The AI chatbots are then ranked based on user ratings, so results may vary compared to synthetic benchmarks. However, it is worth mentioning that the new Gemini model has only been rated about 3000 times so far while GPT 4 models have been ranked over 30,000 times.
Either way, this is still a respectable achievement from Google and deserves praise, which only leaves users curious about the capabilities of Gemini Ultra.
Via: The Decoder
