ai llm agents reading/researchpaper
Core idea
An AI system should not spend expensive, deliberate reasoning on every problem. It can try a fast, experience-based response first, then use a separate metacognitive controller to decide whether the problem is important or difficult enough to justify slower reasoning.
Paper
Title: Thinking Fast and Slow in AI: the Role of Metacognition
Authors: M. Bergamaschi Ganapini, M. Campbell, F. Fabiano, L. Horesh, J. Lenchner, A. Loreggia, N. Mattei, F. Rossi, B. Srivastava, and K. B. Venable
Version: arXiv:2110.01834v1, 5 October 2021
Paper: arXiv abstract · PDF
System 1 vs System 2
The paper draws on Daniel Kahneman’s Thinking, Fast and Slow. System 1 is fast, intuitive, approximate, and relies heavily on experience. System 2 is slower, more deliberate, resource-intensive, and capable of more sophisticated reasoning.
For humans, recognizing a road sign can happen almost automatically, while solving a difficult arithmetic problem takes deliberate attention. The authors apply this distinction to AI systems.
The architecture needs more than a fast solver and a slow solver. It also needs a way to decide when the fast answer is not good enough:
How does the AI know when it should stop trusting its quick answer and start thinking harder?
SOFAI Architecture
A task first goes to a System 1 solver, which proposes an answer and a confidence value. A metacognitive module then decides whether to accept that answer or activate a more expensive System 2 solver.
System 1 (S1) is the default. The paper assumes it uses past experience and operates in constant time, rather than scaling its computation with problem size. It outputs a proposed solution and a confidence estimate.
System 2 (S2) performs more explicit reasoning and is generally more costly. Crucially, it does not activate itself; the metacognitive module must invoke it.
Metacognitive module
The paper proposes that the AI should reason about whether further reasoning is worth the cost.
The metacognitive component considers questions like:
- How confident is the cheap solver?
- How important is it to get this problem right?
- How much time or memory is available?
The paper divides this process into two stages, MC1 and MC2:
- MC1 is a fast screening step. It checks whether the system has enough resources to think further and whether the S1 answer is confident enough for the task’s importance. If so, it accepts the S1 answer. Otherwise, it passes the decision to MC2.
- MC2 makes a more careful cost–benefit decision. It asks whether System 2 is likely to improve the result enough to justify the extra computational cost. If the expected improvement outweighs the cost, it activates S2. Otherwise, it keeps the S1 answer.
In short:
MC1: “Is the fast answer good enough?”
MC2: “If not, is deeper reasoning worth the extra cost?”
What the MC agent knows
| Source | Information used in arbitration |
|---|---|
| Model of self | Solvers’ past decisions, rewards, skills, resource use, and available resources |
| Model of the world | Tasks, goals, environment, and available actions |
| Model of others | Knowledge and beliefs about other agents in the same environment |
| S1 solver | Proposed solution and confidence estimate |
A background model updater maintains these models as the system gathers information and evaluates decisions. It can also retrain S1 solvers.
Connections to existing work
Multi-agent systems use a similar pattern for task orchestration. One model decides which model or agent should handle a task.
Speculative decoding is related: a draft model generates an answer and a larger model verifies it. But speculative decoding does not consider the task itself. It is an internal mechanism for making autoregressive generation faster.
Jev is described as a System 1 model, which seems to align with the S1 model proposed in this paper.
The open question is what the metacognitive controller should be:
Is it another LLM, or does it need a specialized mechanism?