When Transformers Keep the Shape of Thought: A Topological Look at Context Memory

Persistent homology reveals the hidden geometry of how large language models remember, and forget.

---

---

1. Introduction: The Shape of Meaning

For years, transformer interpretability has revolved around attention heatmaps and token-level attribution.
These give local snapshots, but what about the global shape of the meaning space as it flows through the model?

Topology: the mathematics of shape and connectivity, offers a radical new view:
Instead of looking at where attention goes, we study how the whole space bends, loops, and merges over layers.

---

Visual: Concept Flow of Topological Information

---

2. Problem

Let L be the number of transformer layers, T tokens, and d hidden dimensions.
We have hidden states H_l ∈ R^T×d at layer l.

Current limitations:

• Attention maps → token-to-token focus, but no global memory metric.

• PCA/t-SNE/UMAP → distort distances, hide structural persistence.

• No scale-invariant measure → can’t compare models fairly.

Research Question: How do we measure global, scale-invariant memory stability in LLMs?

---

3. Proposed Solution: Topological Information Flow (TIF)