Agent AI:
Surveying the Horizons of Multimodal Interaction APREPRINT
1 Introduction
1.1 Motivation
Historically, AI systems were defined at the 1956 Dartmouth Conference as artificial life forms that could collect
information from the environment and interact with it in useful ways. Motivated by this definition, Minsky’s MIT
group built in 1970 a robotics system, called the “Copy Demo,” that observed “blocks world” scenes and successfully
reconstructed the observed polyhedral block structures. The system, which comprised observation, planning, and
manipulation modules, revealed that each of these subproblems is highly challenging and further research was necessary.
The AI field fragmented into specialized subfields that have largely independently made great progress in tackling these
and other problems, but over-reductionism has blurred the overarching goals of AI research.
To advance beyond the status quo, it is necessary to return to AI fundamentals motivated by Aristotelian Holism.
Fortunately, the recent revolution in Large Language Models (LLMs) and Visual Language Models (VLMs) has made it
possible to create novel AI agents consistent with the holistic ideal. Seizing upon this opportunity, this article explores
models that integrate language proficiency, visual cognition, context memory, intuitive reasoning, and adaptability.
It explores the potential completion of this holistic synthesis using LLMs and VLMs. In our exploration, we also
revisit system design based on Aristotle’s Final Cause, the teleological “why the system exists”, which may have been
overlooked in previous rounds of AI development.
With the advent of powerful pretrained LLMs and VLMs, a renaissance in natural language processing and computer
vision has been catalyzed. LLMs now demonstrate an impressive ability to decipher the nuances of real-world linguistic
data, often achieving abilities that parallel or even surpass human expertise (OpenAI, 2023). Recently, researchers have
shown that LLMs may be extended to act as agents within various environments, performing intricate actions and tasks
when paired with domain-specific knowledge and modules (Xi et al., 2023). These scenarios, characterized by complex
reasoning, understanding of the agent’s role and its environment, along with multi-step planning, test the agent’s ability
to make highly nuanced and intricate decisions within its environmental constraints (Wu et al., 2023; Meta Fundamental
AI Research (FAIR) Diplomacy Team et al., 2022).
Building upon these initial efforts, the AI community is on the cusp of a significant paradigm shift, transitioning from
creating AI models for passive, structured tasks to models capable of assuming dynamic, agentic roles in diverse and
complex environments. In this context, this article investigates the immense potential of using LLMs and VLMs as
agents, emphasizing models that have a blend of linguistic proficiency, visual cognition, contextual memory, intuitive
reasoning, and adaptability. Leveraging LLMs and VLMs as agents, especially within domains like gaming, robotics,
and healthcare, promises not just a rigorous evaluation platform for state-of-the-art AI systems, but also foreshadows
the transformative impacts that Agent-centric AI will have across society and industries. When fully harnessed, agentic
models can redefine human experiences and elevate operational standards. The potential for sweeping automation
ushered in by these models portends monumental shifts in industries and socio-economic dynamics. Such advancements
will be intertwined with multifaceted leader-board, not only technical but also ethical, as we will elaborate upon in
Section 11. We delve into the overlapping areas of these sub-fields of Agent AI and illustrate their interconnectedness
in Fig.1.
1.2 Background
We will now introduce relevant research papers that support the concepts, theoretical background, and modern
implementations of Agent AI.
Large Foundation Models: LLMs and VLMs have been driving the effort to develop general intelligent machines
(Bubeck et al., 2023; Mirchandani et al., 2023). Although they are trained using large text corpora, their superior
problem-solving capacity is not limited to canonical language processing domains. LLMs can potentially tackle complex
tasks that were previously presumed to be exclusive to human experts or domain-specific algorithms, ranging from
mathematical reasoning (Imani et al., 2023; Wei et al., 2022; Zhu et al., 2022) to answering questions of professional
law (Blair-Stanek et al., 2023; Choi et al., 2023; Nay, 2022). Recent research has shown the possibility of using LLMs
to generate complex plans for robots and game AI (Liang et al., 2022; Wang et al., 2023a,b; Yao et al., 2023a; Huang
et al., 2023a), marking an important milestone for LLMs as general-purpose intelligent agents.
5
犀牛老爹2025-01-22很好，是原版的，感谢。
