Open Problems: Robustness, Alignment, Long-Horizon Memory

Despite huge progress, fundamental agent problems remain unsolved. Awareness of these helps you scope what is realistic to build today.

Agents that work 99% of the time still fail catastrophically on edge cases:

• Prompt injection attacks succeed too often
• Tools fail in novel ways
• Long-horizon tasks compound small errors
• Distribution shift breaks production agents weekly

• Constitutional AI (Anthropic)
• Self-consistency and verification chains
• Adversarial training
• Spec-driven development

Agents act on behalf of users — but the user, the agent, and the AI company may have different goals. Whose values win when they conflict?

• Should an agent refuse a legal but unethical task?
• Should it act on long-term welfare against short-term wishes?
• Should it correct user misinformation?

• RLHF — train on human preferences
• Constitutional AI — model self-critiques against principles
• Debate — two agents argue, human judges
• Scalable oversight — AI helps humans supervise AI

Specific to agents: even an aligned LLM can become misaligned when given tools. The model "wants" to help, but its tool-use shortcuts user safety. Active research area at all major labs.

Agents that work for hours, days, months still struggle to remember what mattered:

• Vector stores recall TEXT, not understanding
• Summaries lose details
• Procedural memory doesn't generalize
• No good way to "merge" facts across contradictions

• MemGPT / Letta — explicit memory tiers
• Sparse mixture-of-experts for storage
• Self-editing weights (test-time training)
• Episodic-to-semantic distillation

Agents that plan 5 steps ahead are common; 50-step plans regularly derail. Hierarchical decomposition helps but does not solve.

• Plans drift over time
• Failures compound
• Replanning is expensive

State-of-the-art agents cost $1-$10 per complex task. Pricing must drop 10-100x for ambient consumer use. Trends:

• Distillation to smaller models
• Hardware (Cerebras, Groq, Etched)
• Better routing and caching

Multi-agent systems exhibit emergent miscoordination — agents may compete, lie, or collude when their incentives conflict. Mechanism design and game theory are entering the agent stack.

Today's benchmarks barely cover real production work. Coverage of long-tail, multi-modal, and multi-day tasks is poor. Building better benchmarks is an open research effort.

• Agents accumulate personal data — how is it protected?
• Multi-tenant agents leak between users
• Models may memorise sensitive training data
• Liability when agents make harmful decisions remains unclear

The path forward likely includes:

• Better reasoning at inference time
• Tighter integration of symbolic + neural
• Robust multi-agent coordination protocols
• Verifiable, auditable agents in regulated domains
• Better human-in-the-loop primitives

Today's agents are immensely useful for many tasks but should NEVER be trusted for high-stakes autonomous decisions without human review. Design for the boundary between "agent helps" and "human approves".

Agents are the most consequential paradigm in software since the web. Build them carefully. Evaluate honestly. Defend users. Stay current.

Which of these is still an UNSOLVED problem in agents?