There are two fundamentally different failure modes in modern AI systems.

They are often confused. They should not be.

Resource Link
Related Sleepy Coder: Routing Prevents Forgetting | RLM

The Two Failures

Split diagram showing catastrophic forgetting (weight interference) vs context rot (attention dilution)
Two different failure modes require two different solutions.

Catastrophic Forgetting (Weight-Space Failure)

When you fine-tune a model on new tasks, performance on older tasks may degrade.

This happens because gradient descent updates overlap in parameter space. The model does not “know” which weights correspond to which task. It optimizes globally.

Example: Fine-tune a model on medical text. Its ability to write code degrades. The new learning overwrote old capabilities.

Why It Happens

Neural networks store knowledge distributed across many weights. When you update those weights for Task D, you modify the same parameters that encoded Task A. The old knowledge gets overwritten.

This is the stability vs plasticity tradeoff:

  • Plasticity: Learn new things quickly
  • Stability: Retain old things reliably

You cannot maximize both simultaneously.

Solutions

Method How It Helps
Replay Train on old + new data
Subspace regularization Constrain weight updates to avoid interference
Shared Low-Rank Adaptation (LoRA) spaces Modular updates that don’t overwrite base weights
Freezing base weights Keep foundation stable, train adapters only

Context Rot (Inference-Time Failure)

Context rot is not weight damage.

It happens when:

  • Prompts grow too large
  • Earlier instructions get diluted
  • Attention spreads thin
  • The model begins averaging patterns instead of reasoning

Example: A 50,000 token conversation. The original system prompt is still there, but the model stops following it. Earlier context gets “forgotten” even though it’s technically present.

Why It Happens

Transformer attention is finite. With limited attention heads and capacity, the model cannot attend equally to everything. As context grows, earlier tokens receive less attention weight.

This creates:

  • Instruction drift: Original instructions lose influence
  • Pattern averaging: The model reverts to generic responses
  • Lost coherence: Multi-step reasoning fails

Solutions

Method How It Helps
Retrieval-based context Pull relevant passages, not everything
Recursive Language Models (RLM) Rebuild context each step
Summarization Compress old context
Memory indexing Constant-time lookup instead of linear attention
Structured tool calls Offload state to external systems

The Critical Distinction

Aspect Catastrophic Forgetting Context Rot
Where Weights Prompt window
When During training During inference
Persists? Permanently Session only
Analogy Brain damage Working memory overload

Why This Matters

If you confuse these failure modes, you apply the wrong fix.

  • Forgetting problem? Don’t add more context. Fix your training.
  • Context rot problem? Don’t retrain. Fix your context management.

Many “AI agents that forget” discussions conflate both. Modern systems need solutions for both simultaneously.

References

Concept Paper
Catastrophic Forgetting Overcoming Catastrophic Forgetting in Neural Networks (Kirkpatrick et al. 2017)
Continual Learning Survey A Comprehensive Survey of Continual Learning (Wang et al. 2023)
ELLA ELLA: Subspace Learning for Lifelong Machine Learning (2024)
Share Share: Shared LoRA Subspaces for Continual Learning (2025)
RLM Recursive Language Models (Zhou et al. 2024)

Coming Next

In Part 3, we’ll examine weight-based learning in detail: pretraining, fine-tuning, LoRA, alignment methods, and distillation.

Different failures need different fixes.