Understanding how and why humans make decisions has been extensively studied across various disciplines. Researchers at the Princeton Neuroscience Institute have explored decision-making in machine learning and proposed an approach that improves upon traditional single-agent processes. Their study demonstrates enhanced AI learning capabilities and adaptability by utilizing modular multi-AI agent systems.
The researchers conducted a study comparing reinforcement learning approaches in single AI agent systems and modular multi-AI agent systems. The agents were trained in a survival game on a two-dimensional grid, where they sought hidden resources and aimed to maintain sufficient supply levels.
Single-Agent vs. Modular-Agent approach
In the single-agent approach, a unified brain self-evaluated each objective step-by-step, learning through trial and error to determine the best solutions. On the other hand, the modular agent incorporated input from sub-agents, each with narrowly defined goals and unique experiences. The collective input from the sub-agents was evaluated in a single brain, enabling the agent to make informed choices.
Principles of conflicting needs and objectives
The researchers compared their approach to the longstanding debate on how individuals manage conflicting needs and objectives. This debate is prevalent across various scientific disciplines, including neuroscience, psychology, economics, sociology, and artificial intelligence. The modular agent design mirrors the competition among multiple modular agents, similar to how conflicting needs are managed in human decision-making.
The study results showed that the modular agent outperformed the single-agent approach. The modular agent learned significantly faster, achieving significant progress after only 5,000 learning steps compared to the single agent’s 30,000 steps. Modular agents demonstrated superior maintenance of internal variables in both static and changing environments, maintaining homeostasis more effectively. The sub-agents limited objectives allowed them to adapt more quickly to environmental challenges.
Exploration and adaptation in AI Learning
The actions determined by one sub-agent served as a source of exploration for others within the modular agent. This facilitated the discovery of valuable actions that may not have been chosen in a given state otherwise. In contrast, the monolithic approach struggled with the curse of dimensionality, which refers to the exponential growth of options as the environment complexity increases. The modular agents, acting as specialists with limited objectives, focused on smaller individual tasks and rapidly adapted to environmental shifts.
The study’s findings suggest that designing agents in a modular fashion, with separate sub-agents dedicated to specific needs, significantly enhances the agent’s overall capacity to satisfy its objectives. The modular approach not only improves AI learning and decision-making but also provides insights into the psychological conflicts inherent in the human psyche.
The use of modular agents in AI systems offers a more effective and adaptable approach to decision-making and learning. By leveraging the principles of conflicting needs and objectives, modular agents demonstrate enhanced adaptability to changing environments. The findings contribute to a deeper understanding of human decision-making processes and pave the way for more intelligent and flexible AI systems in the future.
Read more here: Zack Dulberg et al, Having multiple selves helps learning agents explore and adapt in complex changing worlds, Proceedings of the National Academy of Sciences (2023). DOI: 10.1073/pnas.2221180120
