See discussions, stats, and author profiles for this publication at: [Link]

net/publication/386083531

A Comparative Study of AI Agent Orchestration Frameworks

Article · November 2024

CITATIONS READS
0 109

1 author:

Kiumarse Zamanian
Accel Experts
9 PUBLICATIONS 113 CITATIONS

SEE PROFILE

All content following this page was uploaded by Kiumarse Zamanian on 24 November 2024.

The user has requested enhancement of the downloaded file.

 A Comparative Study of AI Agent Orchestration Frameworks
M. Kiumarse Zamanian, PhD

kzamanian@[Link]

November 23, 2024

Abstract
Over the past two years, Generative Artificial Intelligence (GenAI) and particularly conversational
chatbots, like ChatGPT and Gemini, have become popular companions of millions of people worldwide for
doing research, generating content, or simply having fun. As I wrote in my previous article in June, we
have entered a new era of human-computer collaboration where routine tasks are planned and performed
automatically by groups of autonomous AI agents that use Large Language Models (LLM) and predefined
functions. This is a shift from asking a chatbot to propose a trip itinerary to having a group of AI agents
research the best deals, make the reservations for flights, hotels, car rentals, etc., and add them to
calendars. So, this is moving from having a copilot to delegating the job to an autopilot.

Humans have been programming computers to perform complex tasks for eight decades, but now, a new
paradigm allows humans to use natural languages to tell a computer what they need. The computer will
figure out how to do the necessary tasks to get the job done, like a director asking her team to hold a
product launch meeting in two weeks to brief essential clients and partners. Asking computers to “just do
it” is easier said than done. It requires a new computing paradigm with its technology stack and
governance rules that ensure accuracy, reliability, legal safety, and ethical/responsible behavior while
keeping humans in the loop. This article presents AI agents’ technology stack and reviews several
multi-agent orchestration frameworks.

Technology Stack for Creating AI Agents

Prominent thought leaders and technology giants worldwide claim AI agents are ushering in the fifth
industrial revolution for human-computer collaboration. New companies, frameworks, and tools are
announced daily for building agentic systems in different domains. Despite the hype and many riding on
the AI agent bandwagon, there is a lot of rigorous research and solid products that make this field exciting
and worth trying out.

Over the past 25 years, there has been much progress in standardizing software components and
adopting service-oriented architecture (SOA), which has made valuable applications like commerce and
travel readily available over the Internet to billions of people worldwide. We are witnessing a new
generation of agentic applications with “intelligent” components (AI agents) that can learn, plan, and get
things done independently or in collaboration with humans.

Unlike GenAI chatbots, AI agents require state management like ordinary application programs to retain
the history of messages, events, and data used to execute multiple LLM calls in a loop, call external
functions, and pass results to other agents. Although the technology stack for agentic applications differs
from the SaaS stack, they share some common principles, particularly for standard protocols to manage
data and function calling interoperability.

© 2024 M. Kiumarse Zamanian Page 1

Here are the components of the technology stack for AI agents, starting from the bottom layer to the final
layer on top:

1. Foundation models and external storage

2. Memory and tools for agents to use
3. Agent frameworks
4. Services for hosting, serving, and observing agents
5. Domain-specific agents
6. Multi-agent orchestration frameworks

The number of companies with offerings in these layers continues to grow. For example, in the first layer,
Google, Meta, Mistral, and OpenAI offer LLMs and inference engines, while Milvus and Pinecone have
robust vector databases to store embeddings and persistent memory. The second layer is essential for
retaining the state with the history of messages, events, etc., and MemGPT and LangMem are significant
players in this area. Standard tool libraries, e.g., Composio, for agents to call are also in the second layer.
It’s important to note that all agents use the OpenAI Jason schema to call different tools. This facilitates
compatibility across different agent frameworks.

Agent frameworks are essential for managing an agent’s state and context window structure and
communications between agents. Depending on the type of agentic application, e.g., conversational
chatbot or workflow automation, one must select the proper agent framework. Over the past two years,
many robust frameworks have emerged from CrewAI, LangGraph, Microsoft AutoGen, LlamaIndex,
Amazon Bedrock, IBM Bee, Letta, and AutoGPT. Some frameworks, like LangGraph, Amazon Bedrock,
and Letta, also provide the hosting environment for AI agents to operate.

Domain-specific AI agents perform specific tasks, like customer support, launching marketing campaigns,
or writing software, and can be created faster and more reliably using agent frameworks. Standard
protocols used in some agent frameworks make orchestrating multiple agents created in different
frameworks easier. Like standards-based integration of enterprise software applications, these standards
allow combining the best available agents to build a multi-agent system without vendor lock-in.

Several important AI agent frameworks with multi-agent orchestration capabilities are reviewed next.

LangGraph
LangGraph Studio is a platform designed for developing and orchestrating multi-agent systems with a
strong emphasis on control, usability, collaboration, and performance monitoring. The platform is built on
LangChain's foundation, which allows it to remain model-agnostic and support a range of open and
closed LLMs. LangChain provides a robust infrastructure for building applications that leverage LLMs with
tools like chains, memory, and agent capabilities.

LangGraph's flexible framework supports diverse control flows, such as single-agent, multi-agent,
hierarchical, and sequential, and robustly handles realistic, complex scenarios. Reliability can be ensured
with easy-to-add moderation and quality loops that prevent agents from veering off course.

LangGraph’s visual graph editor offers an intuitive drag-and-drop interface for designing and connecting
agent workflows. This allows developers to easily visualize how components interact and modify

© 2024 M. Kiumarse Zamanian Page 2

workflows while instantly observing the impact of changes. The platform also includes real-time
debugging tools, enabling users to inspect agent states, step through execution paths, and adjust
behavior dynamically, enhancing development speed and accuracy.

LangGraph Studio emphasizes team collaboration and supports real-time project editing and sharing. It
integrates seamlessly with LangSmith, a project management tool, to provide shared spaces for version
control, documentation, and project files. This makes it particularly effective for teams working on complex
AI projects.

LangGraph Studio incorporates built-in tools to log and analyze agent performance over time and under
varying conditions for monitoring and optimization. These insights are invaluable for debugging and
improving system reliability.

What differentiates LangGraph Studio from other multi-agent platforms is its graph-centric design. The
visual graph editor and stateful debugging tools streamline creating and maintaining complex multi-agent
systems. Additionally, the focus on collaborative features and close integration with LangSmith positions it
as a comprehensive tool for individual developers and larger AI teams.

To quickly experiment with and refine workflows, Langflow offers an intuitive drag-and-drop interface that
makes it an excellent choice for content creation, customer experience mapping, or any application that
requires iterative design. Langflow 1.1 was recently launched, and it has a new agent component
designed to support complex orchestration with built-in model selection, chat memory, and traceable
intermediate steps for reasoning and tool-calling actions.

With the introduction of the Tool Mode in Langflow 1.1, any component can be repurposed as a toolset for
agents. Whether a built-in calculator or a custom component, one can decide which fields agents can
auto-fill by turning on the Tool Mode at the field level. This granular control enables assigning specific
actions to agents while retaining manual input where needed. Furthermore, Tool Mode allows agents to
call other agents as tools, creating a multi-agent system where they can interact and build upon each
other. This recursive orchestration enables multi-layered, dynamic problem-solving, where agents can
compose complex workflows by calling one another in sequence or nested formations. With a library of
pre-built templates categorized by use case and methodology, one can jump-start a project by choosing
from templates for assistants, Q&A, coding, or content generation.

CrewAI
CrewAI is designed to create and orchestrate multi-agent AI systems using an intuitive graphical user
interface (GUI). Its primary strength is enabling users to define, deploy, and manage AI agents in
collaborative and efficient workflows. CrewAI excels in multi-agent orchestration, offering robust support
for task management, inter-agent communication, and error handling.

CrewAI uses LangChain as a foundational framework for its implementation. It builds on LangChain’s
modular framework to seamlessly integrate agents, tools, and memory systems, allowing for more
sophisticated orchestration and collaboration between multiple agents. By leveraging LangChain, CrewAI
benefits from its extensibility, support for external APIs, and established mechanisms for tool invocation
while enhancing it with a user-friendly graphical interface, advanced guardrails, and cooperative
multi-agent workflows tailored for diverse use cases.

© 2024 M. Kiumarse Zamanian Page 3

CrewAI’s key features include:

Role Playing: CrewAI allows users to assign specialized roles to agents, tailoring their behavior to specific
personas or expertise areas. For example, one agent might act as a technical expert, another as a project
manager, and a third as a data analyst, collaborating effectively on shared objectives.

Memory Management: CrewAI supports three memory types for enhanced contextual understanding and
performance:

● Short-term memory for session-specific tasks

● Long-term memory to retain insights across sessions
● Shared memory for seamless inter-agent data exchange

Tools Integration: Through API integrations, agents can utilize pre-built or custom tools to perform
complex tasks, such as web searches, data retrieval, or domain-specific operations.

Interoperability: Using the OpenAI JSON schema for function calls, CrewAI standardizes agent
interactions with tools, ensuring clarity, input validation, and compatibility.

Task Decomposition: CrewAI improves efficiency by breaking large tasks into subtasks and distributing
them among agents for structured and accurate execution.

Guardrails: Reliability is enhanced through mechanisms that:

● Detect and recover from errors

● Validate outputs to reduce inaccuracies
● Prevent infinite loops with logic checks

Cooperation: Agents collaborate flexibly by:

● Delegating tasks step-by-step (serial workflows)

● Working simultaneously on independent tasks (parallel workflows)
● Operating hierarchically, with higher-level agents managing sub-agents

Graphical User Interface (GUI): CrewAI’s intuitive GUI enables:

● Agent creation and configuration: Using drag-and-drop tools for role and task design
● Workflow monitoring: Visualizing agent interactions and task progress
● Performance fine-tuning: Adjusting settings and troubleshooting via an interactive dashboard

These features make CrewAI a powerful solution for orchestrating AI agents in dynamic, multi-faceted
applications.

Coginzant’s Neuro AI Platform

The Cognizant Neuro AI Platform is a multi-agent orchestration system designed to streamline creating,
deploying, and managing AI solutions across diverse industries. Built on the LangChain framework, the

© 2024 M. Kiumarse Zamanian Page 4

platform maintains flexibility in model selection, supporting large-scale and niche requirements across
enterprises.

Cognizant Neuro AI Platform’s capabilities are structured into four key steps, each powered by
specialized, pre-configured agents:

Opportunity Finder: This agent identifies AI use cases by analyzing industry-specific needs. It allows
users to define their problems or goals, leveraging its knowledge base to propose relevant AI-driven
solutions across healthcare, finance, and agriculture industries. By entering a company name,
Opportunity Finder Agent generates a list of potential decision optimization use cases, including improved
revenue streams and cost savings.

Scoping Agent: Scoping Agent leverages generative AI and data analysis to identify relevant data
categories and success metrics for a chosen use case. For example, this agent defines each AI solution's
contexts, actions, and outcomes.

Data Generator: This agent creates synthetic data to simulate real-world scenarios and test AI
applications before full deployment. It supports generating and preparing data streams tailored to the
specific use case, ensuring robust testing environments.

Model Orchestrator: At the heart of the platform, this feature provides a drag-and-drop interface to
coordinate and implement AI models. It manages communication between multiple agents, such as
context agents or outcome mappers, ensuring seamless collaboration to construct a functional AI
solution. The orchestrator supports various AI models and enables LLM-agnostic operations, making it
flexible for enterprises using open-source and proprietary models.

Cognizant Neuro AI Platform’s key features and benefits include:

Industry-Specific Configurations: The platform offers templates for applications like fraud prevention,
inventory management, crop optimization, and customer retention. These pre-built solutions speed up
deployment and shorten the time needed to realize value.

Agent Collaboration: Agents in the platform communicate dynamically, sharing expertise to craft solutions
tailored to specific use cases. This inter-agent collaboration enhances the platform's adaptability and
efficiency.

Business User Focus: The platform is designed for non-technical users, enabling business leaders to
identify, prioritize, and scale AI use cases without relying heavily on data scientists.

Flexibility and Scalability: The platform supports complex workflows by integrating prescriptive
decision-making models beyond traditional predictive analytics.

GUI for Easy Interaction: The platform’s intuitive graphical interface empowers business leaders and
domain experts to interact seamlessly with its multi-agent orchestration capabilities. Users can:

● Specify business challenges and explore use cases through the Opportunity Finder
● Refine use cases and evaluate their impact during the Scoping phase
● Test scenarios with synthetic data generated in the Data Generator phase

© 2024 M. Kiumarse Zamanian Page 5

 ● Oversee AI model and agent orchestration via the Model Orchestrator, managing task
breakdowns and execution hierarchies

Cognizant Neuro AI Platform is differentiated by its

● Business-Centric Design: Unlike developer-focused platforms, this platform caters to business

leaders with an intuitive GUI and simplified workflows.
● End-to-end Industry Focus: It addresses the entire AI lifecycle with industry-specific templates for
streamlined deployment.
● Synthetic Data for Testing: A standout feature is its emphasis on synthetic data generation for
rigorous testing.

Cognizant Neuro AI’s GUI-first approach and focus on real-world utility empower non-technical users to
lead AI initiatives, making it a strong contender in AI orchestration.

Microsoft Magentic-One
Microsoft Magentic-One is an advanced multi-agent system designed to solve complex, open-ended
tasks by leveraging a collaborative and adaptive architecture. It is implemented using the Microsoft
AutoGen framework, which supports creating and deploying multi-agent systems. AutoGen provides the
necessary tools for orchestrating the interactions between Magentic-One's agents and ensures
modularity, scalability, and flexibility. Magentic-One utilizes AutoGen to integrate various large and small
language models, enabling it to be model-agnostic and adaptable to specific performance and cost
requirements. Magentic-One is model-agnostic, defaulting to GPT-4o but supporting heterogeneous
LLMs, allowing cost and performance optimization flexibility based on different use cases.

Magentic-One operates on a multi-agent system with a central Orchestrator agent that oversees task
execution. The Orchestrator starts by formulating a plan to address the task, recording essential
information and assumptions in a Task Ledger. This ledger acts as a roadmap for the task. As the plan
progresses, the Orchestrator maintains a Progress Ledger, which evaluates the current status and
determines whether the task is completed. If not, the Orchestrator assigns specific subtasks to other
specialized agents within the system. Once a subtask is completed, the Orchestrator updates the
Progress Ledger and assigns additional tasks until the overall goal is achieved. If the Orchestrator detects
a lack of progress over multiple steps, it revisits the Task Ledger, adjusts the plan, and restarts the
process.

This framework ensures adaptability and efficiency by breaking tasks into manageable steps and
dynamically responding to challenges. The Orchestrator’s workflow is divided into two interconnected
loops: an outer loop for updating the Task Ledger with new strategies and an inner loop for updating the
Progress Ledger with ongoing task status and assigning subtasks. This dual-loop mechanism enables
effective coordination among agents and ensures task completion.

Magentic-One includes these specialized agents:

● WebSurfer: for navigating and interacting with web content

● FileSurfer: for managing local files and directories
● Coder: for code generation and artifact creation
● ComputerTerminal: for executing programs and managing environments

© 2024 M. Kiumarse Zamanian Page 6

What differentiates Magentic-One is its robust task reflection framework, seamless agent collaboration,
and multimodal adaptability, making it well-suited for dynamic real-world challenges. It enables
enterprises to tackle diverse problems—from software development to data analysis—by dynamically
orchestrating agents that autonomously plan, execute, and adapt. Its modularity, scalability, and flexibility
ensure its effectiveness across industries.

Amazon Web Services (AWS) Multi-Agent Orchestrator

The recently released AWS Multi-Agent Orchestrator framework facilitates complex collaborations of
multiple specialized agents by intelligently routing queries to the most suitable agent while preserving
context. While offering pre-built components for rapid deployment, it has the flexibility to create custom
agents and integrate new features as needed. The orchestrator’s universal deployment capabilities allow
it to run in different environments, from AWS Lambda to local or cloud platforms. Here are its key features
and capabilities:

Intent Classification: The Classifier is at the framework's core, which functions as the system's
orchestrator. It intelligently routes user requests to the most suitable agent based on:

● The request's nature

● Agent descriptions
● Conversation history
● Session context

Agents process requests independently, focusing on their specific tasks, while the Classifier maintains a
global overview, ensuring efficient and accurate responses.

Flexible Agent Deployment: The framework supports pre-built agents tailored for various tasks:

● Bedrock LLM Agent: Integrates Amazon Bedrock models with Guardrails and tool use
● Lex Bot Agent: Connects to Amazon Lex for conversational interfaces
● Lambda Agent: Links to other services like Amazon SageMaker
● Chain Agent: Executes tasks sequentially, enabling agent collaboration
● Comprehend Filter Agent: Analyzes and filters the content using Amazon Comprehend for
sentiment, PII, and toxicity

This framework's extensible architecture allows users to create custom agents for unique services or
systems, ensuring adaptability to diverse use cases.

Routing Patterns: The Orchestrator optimizes performance and cost with advanced routing:

● Routes simple queries to cost-effective models

● Directs complex tasks to specialized models
● Supports multi-lingual requests seamlessly

Monitoring and Analysis: Built-in logging and monitoring tools provide insights into:

● Agent interactions and classifier decisions

© 2024 M. Kiumarse Zamanian Page 7

 ● Raw and processed outputs
● Execution timings

The framework has an Agent Overlap Analysis tool to enhance system optimization by identifying
redundant roles and ensuring distinct agent functionalities.

Memory Management: The framework maintains conversation history across agents to ensure coherent
interactions. It tracks interactions through unique identifiers for users, sessions, and agents, preserving
context and enabling coherent conversations. It supports three types of storage: in-memory, DynamoDB,
and custom.

Language Support: Provides flexibility in language choice by supporting both Python and TypeScript.

Flexible Response Handling: Accommodates streaming and non-streaming responses, enabling

smooth interactions or discrete responses as required.

AWS Multi-Agent Orchestrator’s versatile deployment options, cost-efficient routing, and robust
monitoring make multi-agent orchestration accessible and pave the way for more efficient and intelligent
AI solutions.

Comparison of Multi-agent Orchestration Frameworks

The table below compares the multi-agent orchestration frameworks reviewed above.

Platform Strengths Areas to improve

LangGraph - It is best for technical teams - Focused primarily on
needing a graph-centric design developers; less accessible to
and advanced debugging tools. non-technical users.
- Visual graph editor with - Limited emphasis on
drag-and-drop workflow design industry-specific pre-built
for intuitive multi-agent system templates.
development.
- Real-time debugging and state
inspection enhance
development accuracy.
- Seamless collaboration
features are available via
integration with LangSmith for
project management.
- Robust control flows for
complex, hierarchical, or
sequential agent tasks.

© 2024 M. Kiumarse Zamanian Page 8

CrewAI - Excels in robust task - Primarily relies on LangChain,
decomposition and guardrails which might limit customization
with user-friendly orchestration beyond its framework.
for dynamic tasks - Industry-specific pre-built
- Strong multi-agent templates not emphasized.
orchestration with robust - The level of customization and
role-based collaboration. flexibility can make CrewAI
- Intuitive GUI for easy agent more complex to learn and use
setup and workflow compared to simpler
management. frameworks.
- Advanced guardrails ensure
reliability (error detection,
hallucination validation).
- Supports task decomposition
for efficient execution.
- Strong Community Support: A
large, active community
provides valuable resources,
tutorials, and support.

Cognizant Nuero AI - Prioritizes non-technical users - Lacks developer-focused

with business-specific debugging tools and
templates and synthetic data customizability.
capabilities, which are ideal for - Limited dynamic task
industries. decomposition and role-based
- Business-centric design collaboration compared to
accessible to non-technical others.
users.
- Comprehensive end-to-end
industry focus with pre-built
templates.
- Emphasis on synthetic data
generation for testing.
- GUI enables exploration,
refinement, and orchestration of
multi-agent workflows.

© 2024 M. Kiumarse Zamanian Page 9

 Microsoft Magentic-One - Strong adaptability for - Heavily reliant on the
open-ended tasks, focusing on orchestrator model, which may
modularity and task reflection add complexity.
for iterative improvement. - Technical expertise is required
- Modular, scalable system for effective customization and
suitable for dynamic, use.
open-ended tasks.
- Strong task reflection with
adaptive outer and inner loops
for task management.
- Model-agnostic framework
supporting diverse LLMs for
cost-performance optimization.
- Includes specialized agents
(e.g., WebSurfer, Coder).

AWS Multi-Agent - Centralized control and - While the framework offers

Orchestrator intelligent routing allow the flexibility, the degree of
framework to efficiently customization, particularly for
distribute tasks among the central classifier and agent
specialized agents, optimizing behavior, might be limited.
performance and accuracy. - As a relatively new offering,
- The central classifier offers a this framework might need
holistic view of the agent more maturity and extensive
system, enabling efficient task community support than more
distribution and coordination. established frameworks.
- Seamless integration with
other AWS services (e.g.,
Bedrock, Lex, Lambda)
streamlines development and
deployment.
- The LLM-powered classifier
ensures that tasks are routed to
the most suitable agent,
optimizing performance and
accuracy.
- Comprehensive logging and
analysis tools provide valuable
insights into agent behavior and
system performance.

Governance for Agent Collaboration

AI agents play a significant role in the new generation of intelligent systems and require clear and
trustworthy guidelines to operate safely and responsibly. Eric Broda has recently introduced the Agentic
Mesh framework for creating a collaborative ecosystem of autonomous AI agents. Each agent in this

© 2024 M. Kiumarse Zamanian Page 10

framework has a clear purpose, is accountable to a human owner, and operates within defined
boundaries. Agents can discover and interact with each other, leveraging generative AI for intelligent
collaboration. The following Agentic Mesh principles ensure collaborative AI agents function within a
governance framework:

● Discoverability: Agents can find and connect with relevant counterparts

● Observability: Agent behavior and performance are monitored
● Interoperability: Agents communicate using standardized protocols
● Certifiability: Agents are verified to ensure compliance
● Operability: Tools are provided for agent management and system stability
● Economic Vitality: Incentives are in place to foster innovation and growth

Adhering to these principles, the Agentic Mesh unlocks AI agents' potential to drive innovation and solve
complex problems safely and responsibly.

Summary
The rise of generative AI has ushered in a new era of human-computer collaboration. We're transitioning
from simple task automation to the collaboration of autonomous AI agents capable of complex
problem-solving and decision-making.

This shift demands a robust technology stack, including foundation models, memory systems, agent
frameworks, and orchestration tools. Multi-agent orchestration frameworks facilitate creating and
managing AI agents working together to automate complex tasks.

Areas for future exploration include:

● Advanced Orchestration: Developing more sophisticated mechanisms for coordinating complex

agent interactions, including dynamic task allocation and resource optimization.
● Ethical Considerations: Establishing clear guidelines for responsible AI agent development and
deployment, addressing issues like bias, fairness, and transparency.
● Human-Agent Collaboration: Enhancing human-agent collaboration by designing intuitive
interfaces and seamless communication channels.
● Marketplaces: Sharing and monetizing certified orchestrations of AI agents that perform various
tasks for different industries.
● Hybrid Intelligence: Exploring the potential of combining human and AI capabilities to achieve
superior performance and creativity.

By addressing these areas, researchers and developers can unlock the full potential of AI agents and
create a future where humans and machines work together harmoniously to solve complex challenges.

References
Cognitive Architectures for Language Agents:
[Link]

The AI Agents Stack:

[Link]

© 2024 M. Kiumarse Zamanian Page 11

View publication stats

LangGraph:
[Link]
[Link]
Langflow:
[Link]

Comparison of LangChain, LangGraph, Langflow, and LangSmith:

[Link]
[Link]
why-69ee91e91000
[Link]

CrewAI Enterprise:
[Link]
CrewAI now lets you build fleets of enterprise AI agents:
[Link]

Cognizant Neuro AI Platform press release:

[Link]
AI-Platform
Cognizant Neuro AI Platform:
[Link]

Microsoft Magentic One:

[Link]
ng-complex-tasks

Amazon Web Services (AWS) Multi-Agent Orchestrator:

[Link]
[Link]
87de927

Agentic Mesh:
[Link]
e09f

© 2024 M. Kiumarse Zamanian Page 12