Arena Prompt Hub Leaderboard

🦍 Gorilla X LMSYS Chatbot Arena

Agent Arena

Agent Arena

Nithik Yekollu Arth Bohra Ashwin Chirumamilla Kai Wen Sai Kolasani Wei-Lin Chiang Anastasios Angelopoulos Joseph Gonzalez Ion Stoica Shishir G. Patil

Agent Arena introductory image

Agent Arena: Evaluating and Comparing LLM Agents Across Models, Tools, and Frameworks

Introduction

With the growing interest in Large Language Model (LLM) agents, there is a need for a unified and systematic way to evaluate agents.

LLM Agents are being used across a diverse set of use-cases, from search and code generation to complex tasks like finance and research. We take the view that LLM agents consist of three components - LLM models (e.g GPT-4, Claude, Llama 3.1), frameworks (LangChain, LlamaIndex, CrewAI, etc), and tools (code interpreters, APIs like Brave Search or Yahoo Finance). For example, an agent to summarize an earnings report, might be powered by a GPT-4o model, use PDFReader as tool to read pdf of earnings reports, and be orchestrated by langchain! Agent-arena captures and ranks user-preferences for agents as a unit, and for each of the three sub-components, providing insights to model-developers, tool-developers, and more critically, users of LLM agents!

Agents come with many nuances in model and framework evaluation. For example, let's say I wanted to build a financial assistant that retrieves the top performing stocks of the week.

❓What model should I use? One model have been trained on far more financial data 💸, while another may excel in reasoning ♟️ and computation ➗.
❓ And what about frameworks? One platform might have more API integrations but another might index the internet better.
❓ What tools should I use? Do I need tools that return stock prices 📈 or APIs that can return news 📰 about the market for this specific use-case.

As this example illustrates, there is much to think about when designing an agentic workflow - and this is only one use-case out of potentially dozens in the financial domain alone. Different use-cases will call for different combinations of models, tools, and frameworks.

We are delighted to release 🤖 Agent Arena, an interactive sandbox where users can compare, visualize, and rate agentic workflows personalized to their needs. Agent Arena allows users to choose from a combinations of tasks, LLM providers, frameworks, tools, etc and also vote on their performance. We enable users to see how different agents perform against each other in a structured and systematic way. By doing this, we believe that users can make more informed decisions regarding their agentic stack. Further, with the Agent Arena we wish to showcase the shortcomings and impressive advacements of the state of agents!

Agent Arena also consists of live leaderboard and ranking of LLM models, frameworks, and tools grouped by domain. Additionally, we believe these rankings can help inform model, tooling, and framework developers, helping them understand where they stand on various use-cases and how they can improve. Also recognizing that user-vote based elections are affected by selection-bias, as a new feature, Agent Arena also includes a Prompt Hub, where you can subscribe to specific prompt-experts and see their invidual opinions on various tasks. You can also publish your set of prompts!

This blog post will look into the key elements of Agent Arena, including the definition of agents, the novel ranking algorithm, model tuning, examples of agent use cases, and a roadmap for future developments. And saving the best for the last, along with this blog, we are also releasing 2,000 real-world, pair-wise agent battles, and user preferences!! We'll continue to periodically release more battle data!

Quick Links:

🦜 What are Agents?

In Agent Arena, agents are defined as entities that can perform complex tasks by leveraging various subcomponents. We define each agent to be made up to three components - LLM models, tools, and frameworks. The agents we consider are sourced from established frameworks like LangChain, LlamaIndex, CrewAI, Composio, and assistants provided by OpenAI and Anthropic. Each of these agents may display characteristics such as chain-of-thought reasoning, tool use, and function calling, which enable them to execute complex tasks efficiently. For the platform, we utilized models that support function calling and tool use, which are critical aspects for LLM agents.

For example, LangChain and LlamaIndex agents come equipped with specific toolkits that enhance their problem-solving capabilities. OpenAI's assistants, such as code interpreters and file processing models, also qualify as agents due to their demonstrated ability to interpret code, process files, and call external functions. Anthropic's agents are integrated with external tools, and similar examples from other frameworks further enhance their utility for specific tasks.

The Agent Arena Platform

Executor Flow

A high-level overview of agent comparisons based on user goals, models, frameworks, and performance metrics like execution time and ELO

At its core, Agent Arena allows for goal-based agent comparisons. On a high level, users will first input a task that they want to accomplish. Then, an LLM automatically assign relevant agents based on the task. These agents are then tasked with completing the goal, with the agent's actions and chain of thought being streamed to the user in real-time. Once the agents have completed the task, the user can compare the outputs side-by-side and vote on which agent performed better.

The evaluation process includes voting on agent performance, with users assessing which agent met the task's requirements more effectively. This user-driven evaluation contributes to an evolving leaderboard system, which ranks agents based on their relative performance across multiple tasks and competitions. This comparison is not limited to the agents as a whole but extends to the individual components (i.e., LLM models, tools, and frameworks) that comprise each agent.

In the sections below, we will look into the core components of Agent Arena, including the router system, execution, evaluation and ranking mechanisms, leaderboard, and prompt hub. We will also explore some example tasks and applications that can be performed on the platform.

The Router: Agent Matching and Task Assignment

A central element of Agent Arena is its router system, which is powered by GPT-4o currently. We plan to cycle between all models, and also judge each model's ability to route prompts to the most relevant agents! The router's primary function is to match users' specified goals with the most suitable agents available on the platform.

The router operates by analyzing the user's input (the goal or task) and selects two agents that are optimally suited to complete that task. This selection process factors in the agents' historical performance across similar tasks, as well as their configurations in terms of models, tools, and frameworks.

For example, a user might provide the following: input("Tell me about whats going on in NVIDIA in the last week.") The router would then select two suitable options given the available agents and the leaderboard ELOs. For this use-case, the router might select the agent agent_a = Agent(model="GPT-4o", tools=["Yahoo Finance", "Matplotlib"], framework="Langchain") to analyze the stock information about NVIDIA. On the other side, to compare against Agent A, the router might select the combination: agent_b = Agent(model="Claude", tools=["Yahoo News"], framework="CrewAI") to observe the goal from the perspective of news.

This comparison is fruitful because it allows the platform and the user to understand the nuances in the agents' capabilities and the different ways they can approach the same task. Then, they themselves can vote for which style they like better.

Evaluation and Ranking System

Agent Arena employs a comprehensive ranking system that evaluates agents based on their performance in head-to-head comparisons. The leaderboard ranks agents not only based on their overall performance but also by breaking down the performance of individual components such as LLM models, tools, and frameworks. The ranking process is informed by both user evaluations and an ELO-based rating system, commonly used in competitive ranking environments, where agent performance is dynamically adjusted after each task or comparison.

The rating system in Agent Arena is designed to reflect the cumulative performance of agents across a wide range of tasks, taking into account factors such as:

The leaderboards analyzing the subcomponents of the agents

  • Model performance: Evaluating the effectiveness of the underlying LLM models (e.g., GPT-4, Claude, Llama 3.1).
  • Tool efficiency: Ranking the tools agents use to complete tasks (e.g., code interpreters, APIs like Brave Search or Yahoo Finance).
  • Framework functionality: Assessing the broader frameworks that support agents, such as LangChain, LlamaIndex, and CrewAI.

Check out the latest rankings for each category on our leaderboard: Agent Arena Leaderboard.

⚖️ Evaluating Agents with the Extended Bradley-Terry Model

The Extended Bradley-Terry Model

Agent Arena uses the Bradley-Terry extension, which allows us to compare different agents based on their subcomponents, including tools, models, and frameworks. Instead of just evaluating the agents atomically, we also assess the performance of each individual subcomponent. This allows us to more accurately pinpoint where an agent's strength lies. For example, our first agent could be a combination of LangChain, Brave-Search, and GPT-4o-2024-08-06, while the second agent could be LlamaIndex, Wikipedia, and Claude-3-5-Sonnet-20240620. Therefore, we propose the following observation model for the Extended Bradley-Terry Model. Given P_1,

For each battle \(i \in [n]\), let we have a a prompt and two agents, encoded as the following:

  • Agent A: The first agent being compared with an elo of E_A and with the subcomponents (A_T, A_M, A_F)
  • Agent B: The second agent being compared, having an elo of E_B and with the subcomponents (B_T, B_M, B_F)
  • Y_i: Outcome of the battle (1 for win, 0 for loss)

🧑‍💻 Example: LangChain Brave-Search Agent vs. LlamaIndex Wikipedia Agent

Let's walk through an example to illustrate how the Extended Bradley-Terry Model works in practice. Take the following agents and their subcomponents:
  • Agent_A is the Langchain Brave-Search Agent, using the following subcomponents: { Brave-Search (A_T), LangChain (A_F), and GPT-4o-2024-08-06 (A_M) } and an Elo of 1600.
  • Agent B is the LlamaIndex Wikipedia Agent, with the subcomponents: {Wikipedia (B_T), LlamaIndex (B_F), and Claude-3-5-Sonnet-20240620 (B_M)} and an Elo of 1500.

In a traditional Elo system, we would have calculated the probability of the Brave-Search agent winning 64% of the time. Then given the actual outcome of the battle, Y_1, assuming that the Brave-Search agent wins, the new rating of the agents would be 1601.44 and 1498.56, respectively.

In the Bradley-Terry model, however, we calculate the ratings by minimizing the following loss function given the real-world battle outcome: Y_i:

\[L = -\sum_{i=1}^{n} Y_i \cdot log(\frac{1}{1 + e^{E_A - E_B}}) + (1 - Y_i) \cdot log(\frac{1}{1 + e^{E_B - E_A}})\]

Finally, to get a holistic evaluation of an agent, we combine all its subcomponents into a single analysis. Instead of treating each subcomponent as an isolated entity, we consider their interaction within the broader agent architecture. For each battle, we build a design matrix X that represents all the subcomponents involved. Here, let's assume that A, the Brave-Search agent, wins the battle; in that case the design matrix would look like this:

\[X = [ +log(A_T), +log(A_M), +log(A_F), -log(B_T), -log(B_M), -log(B_F) ]\]

This allows us to evaluate the collective contribution of the subcomponents (tools, models, frameworks) in a single calculation. We then apply logistic regression with L2 regularization to control for overfitting and confounding effects caused by frequent pairings.

By using this combined approach, Agent Arena ensures more accurate rankings across agents and their subcomponents. 🔄 This method provides clearer insights into each agent's performance and contributions, preventing the bias that can occur from frequent pairings or overused configurations.

🎉 As a result, our system generates a real-time, continuously updating leaderboard that not only reflects the agents' overall performance but also their specific subcomponent strengths. 🏆

Check out our live leaderboards for agents, tools, models, and frameworks here!

The Prompt Hub

The Agent Arena also comes with a prompt hub that has over 1000+ tasks that have been tested and verified to work on the platform. Users will be able to search for similar use cases as theirs and observe how different prompts are executed and perform. Furthermore, the platform also enables users to post their prompts to the community. This public view of prompts that are being evaluates through agent arena provides strong infrastructure and data for future anayltics for future agent development and evaluation.

The prompt hub featuring registered users in the arena

🏠 Prompt Hub Overview

The prompt hub is a way for users to interact with other users and see a unique view of the individual and domain specific use cases that users demand with agents. This is a great way to see user activity at a granular level and see what specifically users are using agents to do and how to prioritize future agent development.

View, like, and dislike individual user prompts

🧑‍💻 Individual User View

Additionaly, users can provide feedback to other users on their individual prompts through the prompt hub by liking and disliking individual prompts. This provides an additional data point for future for prompt analytics to potentially evaluate domain-specific performance of various agents in the arena.

💼 Case Studies

Your choice of model, framework, and tools will often differ greatly depending on domain applications and use cases. Domain-specific agent developer will need to find the optimal combination of these factors to maximize performance. The vision of the future is that eventually, agents will become accurate enough to the point where we will allow them to make informed and perhaps critical decisions without the need for a human in the loop. While there's ways to go, here are a few industries that could get shaken up by agents:

Agent Analyst Example

Example flow of LLM agents providing projections and insights on GE stock prices based on relevant earnings and competitors

Real-World Agent Workflows: Interesting User Scenarios

In the following section, we showcase some of the most interesting real-world examples from the Agent Arena. These examples represent diverse user scenarios where agents were tasked with solving specific challenges, highlighting the variety of models, frameworks, and tools employed. Each prompt illustrates the agents' thought process, execution, and areas for improvement, offering insights for both developers and users.

Education & Personalized Tutoring 📚

"Generate a step-by-step solution and explanation for this high school physics problem: A 5 kg object is dropped from a height of 10 meters. How long does it take to hit the ground?"

+ Prompt Execution

In education-focused scenarios, LLM agents have the potential to offer rich, step-by-step explanations that guide students through complex problems, such as physics calculations. In the example of determining how long it takes for a 5 kg object to fall from 10 meters, the agents approached the problem using basic equations of motion. However, while Agent A (anthropic calculator tool, claude-3-opus-20240229) provided a thorough breakdown of the solution, the simplicity of its approach highlighted a need for more nuanced handling of kinematics, such as adaptive responses that dynamically adjust based on user queries. Meanwhile, Agent B (langchain Wolfram Alpha, claude-3-haiku-20240307) leveraged Wolfram Alpha but struggled with obtaining relevant data from the tool, indicating gaps in API integration that hinder real-time computational accuracy. These cases show opportunities for fine-tuning the agents' interaction with APIs and frameworks, ensuring that agents not only retrieve correct data but also process and apply it efficiently to real-world scenarios. Improving the fluidity and depth of these calculations, especially when leveraging multiple APIs, can bring enhanced precision and adaptability in educational contexts, enriching the learning experience and making the agent more capable of handling varied educational queries.

Business Data Analytics 💸

"Given this .csv file of last year's sales data, generate insights of what products to scale up."

+ Prompt Execution

In business data analytics, LLM agents can offer valuable insights by processing large datasets, such as CSV files containing sales data, to uncover trends and make strategic recommendations. In this example, Agent A (sql agent plotter langchain, gpt-4o-2024-05-13) struggled with an error, misinterpreting the CSV file as a SQLite database, which highlights limitations in the agent’s error-handling capabilities and its adaptability to different file formats. Although the agent attempted to switch tools and correct the process, it was clear that a more seamless integration between SQL and file-processing tools was needed to maintain workflow fluidity. Meanwhile, Agent B (langchain Pandas DataFrame, gpt-4o-2024-08-06) effectively analyzed the sales data, identifying top-performing products like “Laptop” and “Smartphone” based on sales revenue, and suggested scaling up "Headphones" and "Keyboard" due to high sales volume. However, Agent B could benefit from deeper contextual understanding by linking sales patterns with external factors such as seasonality or promotions. These examples underscore the need for agents to better handle complex datasets, enhance error resilience, and offer more context-aware analysis, especially when switching between tools or working with diverse data formats. Improving these areas would significantly enhance the agent’s ability to deliver more robust, actionable insights, particularly in complex business scenarios.

Social Media Management / Content Creation 📸

"Schedule daily Instagram posts for a week, promoting our upcoming sale using relevant hashtags and current influencer trends."

+ Prompt Execution

In this example, the task of scheduling daily Instagram posts for a week using relevant hashtags and influencer trends isn't fully realized by the agents due to their more generalist nature. Both agents—langchain google-serper search agent (gemini-1.5-pro-001) and langchain You.com Search (gpt-4o-mini-2024-07-18)—attempted to craft content and suggest hashtags but lacked the specific capabilities necessary for handling nuanced social media scheduling tasks. The gemini-1.5-pro-001 agent looped through asking for more information about the sale, while the You.com agent focused on general suggestions for a week-long post schedule without real-time engagement insights or adaptive content creation based on platform-specific trends.

For instance, the gemini-1.5-pro-001 agent's output repeatedly asked for input on the sale details, which indicates a limitation in context handling. Additionally, the response structure failed to account for Instagram's unique features, such as optimal posting times or Story integration. Meanwhile, the gpt-4o-mini-2024-07-18 agent provided a decent post schedule but didn't fully leverage influencer data or real-time trends to inform its content suggestions.

These agents, while functional, demonstrate that more specialized frameworks tailored for social media platforms are needed. Frameworks integrating direct connections to platforms like Instagram, Twitter, or Facebook, and incorporating up-to-date social media engagement analytics, would enable agents to generate more precise, platform-specific recommendations. Moreover, agents could benefit from handling tasks like scheduling posts directly, rather than only generating content ideas, making them far more effective in streamlining social media strategies.

Medical Diagnostics 👩‍⚕️

"Given this file containing a patient’s medical history, cross-reference it with recent research papers to recommend the most up-to-date treatment options for chronic migraines."

+ Prompt Execution

Manually sifting through medical research is not only time-consuming but also leaves room for oversight, particularly when the data spans numerous clinical trials and evolving treatment protocols. LLM agents offer a powerful alternative, adept at combing through extensive medical databases, pulling relevant findings, and connecting them to specific patient conditions. In cases like chronic migraines, an agent can swiftly gather recent studies on effective treatments, such as erenumab, a preventive therapy, and radiofrequency ablation (RFA), which offers long-term relief for headache pain. For example, in this instance, the langchain PubMed Biomedical Literature Tool (gpt-4o-mini-2024-07-18) agent successfully retrieved relevant research, presenting concrete treatment options based on the latest findings. In contrast, the ArXiv Article Fetcher (claude-3-haiku-20240307) struggled due to ArXiv’s focus on pre-prints, not clinical treatments. Despite this, the fallback recommendations to check more appropriate medical journals like JAMA show how agents can adapt when limitations arise. Enhancing integration with specialized databases and refining multi-step query handling could unlock even more potential, allowing these agents to provide faster, more accurate, and contextually relevant medical recommendations, ultimately pushing the boundaries of how automated systems can support healthcare decisions.

Sports Data Analytics 🏀

"Predict the odds of the Denver Nuggets winning the NBA championship, given individual player statistics, team performance trends, and recent trade news."

+ Prompt Execution

In sports data analytics, LLM agents are tasked with analyzing player statistics, team performance, and trade news to predict outcomes, like the Denver Nuggets' chances of winning the NBA championship. Agent A (langchain google-serper search agent, open-mixtral-8x22b) gathered relevant data but lacked the depth of analysis needed for a precise prediction, missing key insights like player injuries or competition from other teams. Similarly, Agent B (anthropic web page reader, claude-3-5-sonnet-20240620) emphasized the importance of real-time data but couldn't provide a detailed, data-driven prediction due to the absence of current season statistics.

Both agents highlight the need for better integration of real-time data sources and more advanced statistical modeling. Improving how these agents handle multi-step reasoning and predictive analytics would significantly enhance their ability to deliver accurate, actionable insights, making them more useful for teams, analysts, and sports bettors who depend on such forecasts for decision-making.

Trip Planning ✈️

"Plan a day trip to Carmel-by-the-Sea from San Francisco. Optimize the itinerary by choosing the most fuel-efficient routes with the most sights to see."

+ Prompt Execution

In travel planning, LLM agents have the potential to craft detailed itineraries by optimizing routes and suggesting relevant stops along the way. In the example of planning a day trip from San Francisco to Carmel-by-the-Sea, Agent A (crewai AI Crew for Trip Planning, gemini-1.5-pro-002) and Agent B (langchain google-serper search agent, open-mixtral-8x22b) both suggested fuel-efficient routes and popular attractions. However, while Agent B efficiently identified major landmarks and provided a clear route, the responses lacked dynamic adjustments based on real-time conditions like traffic or road closures.

Additionally, Agent A struggled with more seamless transitions between tools, which affected the ability to fully integrate relevant trip data. These examples point to areas where enhancing real-time API integration and improving the agents' adaptability to changing travel conditions could provide more tailored and accurate trip plans. Better handling of dynamic factors like current road conditions or user-specific preferences would result in richer, more relevant travel experiences.

Next Steps and Project Roadmap

We have an exciting roadmap ahead for Agent Arena, with several initiatives planned to both enhance and expand the platform's capabilities. We envision that the agent arena will become a central hub for both agent developers and providers.

For developers and users interested in building/using agents, the platform will be a sandbox for them to perfect their agentic stack, with the right providers and frameworks tailored to their use-cases.

By providing a systematic way to run agents, compare them against each other, view advanced analytics for providers based on their use-case, and even view the prompts of similar users, we hope to deliver value to the agent-building community.

To reach this vision, we have laid out a comprehensive roadmap of feature development and improvement. The general theme of these changes will be to improve the personalization of the arena to individual users along with expanding the available analytics.

📈 Increasing the Number of LLM & Framework Providers on the Platform

One of the primary goals of the Agent Arena is to show users all of the combinations of agents that they can build, so they can definitely know which options are the best suited for their use-cases. While we currently offer the main providers in each category, we hope to expand our selection to include more niche providers that are specialized in certain tasks.

🧑‍💻 Incorporating User Personalization

In order to make the platform as useful as possible, we want to ensure that users are met with specific recommendations on the latest releases and agents that are best suited for their use-cases. This will involve us learning their preferences in their providers and output formats, enabling us to then recommend the best agents for them.

🪜 Enabling multi-turn prompts

Most agentic tasks involve multiple steps of reasoning and action from the agent. This requires keeping track of the state of the context of the task. For example, take the following task:

Task: "Search for the top 5 performing stocks this year in the S&P 500 and then find the latest news about them."

This task requires the agent to first find the top 5 stocks, keep it somewhere in backend 'memory',and then call another set of individual tools to find the latest news about them. This is a multi-turn prompt, and other examples can start to involve 5+ steps. We plan on releasing this feature in the upcoming few months for users.

🏋️‍♀️ Expanding the Capabilities of the Platform

The current implementation of the platform has left several domains of agent use-cases unexplored. More specifically, we hope to start integrating with APIs like Jira, Github, GSuite and other tools to enable users to actually run agents on their personal data. While this will involve a lot of security and privacy considerations, we believe this is a critical step in making the platform more useful to users.

📊 Improving the Recommendation Algorithm

Based on user preferences and the providers/frameworks they like, we plan on improving the routing of goals to more relevant agents for the user. Additionally, we will include two different modes of routing: one that is more exploratory and one that is more focused on the user's preferences.

Conclusion

Agent Arena is a platform to evaluate and compare LLM agents. By offering a comprehensive ranking system and tools to test agents from various frameworks, the platform allows users to make informed decisions about the best models and tools for their specific needs. With continuous improvements and expansions planned, Agent Arena is set to play a pivotal role in shaping the future of LLM agent evaluation.

We invite researchers, developers, and AI enthusiasts to explore Agent Arena, contribute to its growth, and help shape the future of agent-based AI systems. Together, we can push the boundaries of what's possible with LLM agents and unlock new potentials in AI-driven problem-solving.

We hope you enjoyed this blog post. We would love to hear from you on Discord, Twitter (#GorillaLLM), and GitHub.

Citation

If you would like to cite Agent Arena: 

@inproceedings{agent-arena,
            title={Agent Arena},
            author={Nithik Yekollu and Arth Bohra and Ashwin Chirumamilla and Kai Wen and Sai Kolasani
                    Wei-Lin Chiang and Anastasios Angelopoulos and Joseph E. Gonzalez and
                    Ion Stoica and Shishir G. Patil},
            year={2024},
            howpublished={\url{https://gorilla.cs.berkeley.edu/blogs/14_agent_arena.html}},
}