Llama 3 or GPT-4 is better?

Llama 3 and GPT-4 are two of the most advanced large language models (LLMs) available to the public. Let's see which LLM is better by comparing both models in terms of multimodality, context length, performance, and cost.

Table of Contents

• What is GPT-4?
• What is Llama 3?
• Multimodal
• Context length
• Efficiency
• Price
• Accessibility
• GPT-4 or Llama 3 which is better?

What is GPT-4?

GPT-4 is the latest large language model (LLM) developed by OpenAI. It builds on the foundation of the older GPT-3 models while using different training and optimization techniques using a much larger dataset. This has significantly increased the parameter size of GPT-4, which is rumored to have a total of 1.7 trillion parameters from its smaller expert models. With the new training, optimizations, and larger number of parameters, GPT-4 offers improvements in reasoning, problem solving, understanding context, and better handling of nuanced instructions.

There are currently 3 variations of the model:

• GPT-4 : An evolution from GPT-3 with significant improvements in speed, accuracy, and knowledge base.
• GPT-4 Turbo : An optimized version of GPT-4, designed to deliver faster performance while reducing operating costs.
• GPT-4o (Omni) : Extends the capabilities of GPT-4 by integrating multimodal inputs and outputs, including text, images, and audio.

You can now access all three GPT-4 models by subscribing to OpenAI's API service, interacting with ChatGPT, or through services like Descript, Perplexity AI, and many other ancillary services from Microsoft.

What is Llama 3?

Llama 3 is an open-source LLM developed by Meta AI (the parent company of Facebook, Instagram, and WhatsApp), trained using a combination of supervised tuning, sampling, and policy optimization on a diverse dataset, including millions of human annotations. For example, its training focuses on high-quality prompts and priority rankings, creating a flexible and capable AI model.

You can access Llama 3 through Meta AI, its Generative AI chatbot. Alternatively, you can run LLM locally on your computer by downloading Llama 3 models and loading them through Ollama, Open WebUI, or LM Studio.

Multimodal

The release of GPT-4o finally brought initial information that GPT-4 has multimodal capabilities. You can now access these multimodal features by interacting with ChatGPT using the GPT-4o model. As of June 2024, GPT-4o does not have any built-in way to generate video and audio. However, it is capable of generating text and images based on video and audio inputs.

Llama 3 is also planning to provide a multimodal model for the upcoming Llama 3 400B. It will most likely incorporate similar technologies with CLIP (Contrast Language-Imager Pre-Training) to generate images using Zero-shot Learning techniques. But since the Llama 400B is still in training, the only way for the 8B and 70B models to generate images is to use extensions like LLaVa, Visual-LLaMA, and LLaMA-VID. As of now, Llama 3 is purely a language-based model that can take text, images, and audio as input to generate text.

Context length

Context length refers to the amount of text a model can process at once. This is an important factor when considering the capabilities of an LLM because it determines the amount of context the model can work with when interacting with a user. Generally, a higher context length makes an LLM better because it provides a higher level of coherence, continuity, and can reduce repetition errors during interactions.

The Llama 3 models have an effective context length of 8,000 tokens (about 6,400 words). This means that the Llama 3 model will have a context memory of about 6,400 words during the interaction. Any words that exceed the 8,000 token limit will be forgotten and will not provide any additional context during the interaction.

In contrast, GPT-4 currently supports significantly larger context lengths of 32,000 tokens (about 25,600 words) for ChatGPT users and 128,000 tokens (about 102,400 words) for those using the API endpoint. This gives the GPT-4 model an advantage in managing extended conversations and the ability to read long documents or even entire books.

Efficiency

Let's compare performance by looking at Meta AI's April 18, 2024 Llama 3 benchmark report and OpenAI's May 14, 2024 GPT-4 GitHub report. Here are the results:

Here's what each criterion measures:

• MMLU (Massive Multitask Language Understanding) : Assesses the model's ability to understand and answer questions on a variety of academic topics.
• GPTQA (General Purpose Question Answering) : Assesses the model's ability to answer real-world questions in an open domain
• MATH : Test the model's ability to solve problems.
• HumanEval : Measures the model's ability to generate correct code based on given human programming prompts.
• DROP (Discrete Reasoning Over Paragraphs) : Evaluates the model's ability to perform discrete reasoning and answer questions based on text passages.

Recent benchmarks highlight the performance differences between the GPT-4 and Llama 3 models. While the Llama 3 8B model appears to be significantly behind, the 70B and 400B models perform lower but similar to both the GPT-4o and GPT-4 Turbo models in academic and general knowledge, reading comprehension, reasoning and logic, and coding. However, no Llama 3 model has yet reached the performance of GPT-4 in pure mathematics.

Price

Cost is an important factor for many users. OpenAI's GPT-4o model is available for free to all ChatGPT users with a limit of 16 messages every 3 hours. If you need more, you'll need to subscribe to ChatGPT Plus for $20/month to expand GPT-4o's message limit to 80, as well as get access to other GPT-4 models.

On the other hand, both the Llama 3 8B and 70B models are open source and free, which can be a significant advantage for developers and researchers looking for a cost-effective solution without compromising on performance.

Accessibility

GPT-4 models are widely accessible through OpenAI’s Generative AI chatbot ChatGPT and through its API. You can also use GPT-4 on Microsoft Copilot, which is a way to use GPT-4 for free . This wide availability ensures that users can easily leverage its capabilities in different use cases. In contrast, Llama 3 is an open-source project, which provides model flexibility and encourages broader experimentation and collaboration within the AI ​​community. This open-access approach can democratize AI technology, making it available to a wider audience.

While both models are available, GPT-4 is much easier to use because it is integrated into popular productivity tools and services. On the other hand, Llama 3 is primarily integrated into research and business platforms like Amazon Bedrock, Ollama, and DataBricks (with the exception of Meta AI chat support), which doesn’t appeal to a larger market of non-technical users.

GPT-4 or Llama 3 which is better?

So which LLM is better? GPT-4 is the better LLM. GPT-4 excels at multimodality with advanced capabilities for handling text, image, and audio input, while similar features of Llama 3 are still under development. GPT-4 also offers much larger context lengths and better performance, and is widely accessible through popular tools and services, making GPT-4 more user-friendly.

However, it is important to emphasize that the Llama 3 models have performed very well for a free and open source project. As such, Llama 3 remains a prominent LLM, favored by researchers and businesses for its free and open source nature, while also offering impressive performance, flexibility, and reliable security features. While the general consumer may not immediately find a use for Llama 3, it remains the most viable option for many researchers and businesses.

In summary, while GPT-4 stands out for its advanced multi-modal capabilities, greater context length, and seamless integration into widely used tools, Llama 3 offers a valuable alternative with its open-source nature, allowing for greater customization and cost savings. So, in terms of applications, GPT-4 is ideal for those looking for ease of use and comprehensive features in one model, while Llama 3 is well suited for developers and researchers looking for flexibility and adaptability.