Gemma 2 or Llama 3 is the best open source model?

At I/O 2024, Google announced its next line of Gemma 2 models, and now the company has finally released the lightweight models under an open source license. The new Gemma 2 27B model is said to be very promising, outperforming some of the larger models like the Llama 3 70B and Qwen 1.5 32B. So to test this claim, let's compare the Gemma 2 and Llama 3 - two of the top open source models available today.

Creative writing

First, let’s test how well Gemma 2 and Llama 3 do when it comes to creative writing. The author of the article asked both models to write a short story about the relationship between the moon and the sun. Both did very well, but Google’s Gemma 2 model stood out with its interesting prose and good story.

On the other hand, the Llama 3 feels a bit dull and robotic. Google has always been good at generating text with its Gemini models, and the smaller Gemma 2 27B is no exception.

Winning Option: Gemma 2

Multilingual Testing

In the next round, let’s see how well both models handle languages ​​other than English. Since Google advertises that Gemma 2 is good at understanding multiple languages, I compared it to Meta’s Llama 3 model. I asked both models to translate a text in Hindi. Both Gemma 2 and Llama 3 performed very well.

The author also tried another language, Bengali, and the models performed similarly well. At least for Indian languages, it can be said that Gemma 2 and Llama 3 trained well on a large corpus. However, Gemma 2 27B is almost 2.5 times smaller than Llama 3 70B, which makes it even more impressive.

Winning Options: Gemma 2 and Llama 3

Test your reasoning

While the Gemma 2 and Llama 3 aren't the smartest models out there, they can perform some common reasoning tests just as well as much larger models. In the previous comparison between the Llama 3 and GPT-4 , Meta's 70B model was impressive because it demonstrated quite good intelligence even at its smaller size.

In this round, Llama 3 beat Gemma 2 by a wide margin. Llama 3 answered 2 out of 3 questions correctly while Gemma 2 struggled to get even one correct. Gemma 2 simply wasn’t trained to solve complex reasoning questions.

Llama 3, on the other hand, has a solid reasoning foundation, most likely derived from the token dataset. Despite its small size — at least compared to trillion-parameter models like GPT-4 — it demonstrates more than a decent level of intelligence. Ultimately, using more tokens to train the model actually results in a more robust model.

Winning Option: Llama 3

Follow the instructions

In the next round, the authors asked Gemma 2 and Llama 3 to generate 10 words ending in “NPU”. And Llama 3 achieved 10/10 correct answers. In contrast, Gemma 2 only generated 7 correct sentences out of 10. In many previous releases, Google models including Gemini did not follow the user instructions well. And the same trend continued with Gemma 2.

Following user instructions is very important for AI models. It ensures reliability and generates accurate responses to what you have instructed. In terms of safety as well, it helps keep the model grounded to better comply with safety protocols.

Winning Option: Llama 3

Find information

Both Gemma 2 and Llama 3 have a context length of 8K tokens. The author added a huge block of text, sourced directly from the book Pride and Prejudice, containing over 17,000 characters and 3.8K tokens. As usual, the author placed a random quote somewhere in the text and asked both models to find it.

Gemma 2 quickly found the information and pointed out that the statement was inserted randomly. Llama 3 also found it and suggested that the statement seemed out of place. Regarding long context memory, although limited to 8K tokens, both models are quite strong in this regard.

Note the author ran this test on HuggingChat (web) because meta.ai refused to run this prompt, most likely due to copyright content.

Winning Options: Gemma 2 and Llama 3

Check for hallucinations

Smaller models tend to suffer from AI hallucinations due to limited training data, often making up information when the model encounters unfamiliar topics. So the author threw in a made-up country name to test whether Gemma 2 and Llama 3 would hallucinate. And surprisingly, they didn’t, which means both Google and Meta have a pretty good foundation for their models.

The author also posed another (false) question to test the validity of the models, but again, they did not hallucinate. By the way, the author tested Llama 3 on HuggingChat while meta.ai browsed the internet for current information on related topics.

Winning Options: Gemma 2 and Llama 3

Conclude

While Google's Gemma 2 27B model doesn't do well on reasoning tests, it's capable of a number of other tasks. It's great at creative writing, supports multiple languages, has good memory, and best of all, it doesn't cause hallucinations like previous models.

Llama 3 is better, of course, but it's also a significantly larger model, trained on 70 billion parameters. Developers will find the Gemma 2 27B model useful for many use cases. And for inference, the Gemma 2 9B is also available.

Additionally, users should check out the Gemini 1.5 Flash, which is again a much smaller model and also supports multi-modal input. Not to mention, it is extremely fast and efficient.