Presentation overview
This presentation studies whether recurrent computation helps Transformers maintain and update latent state beyond the sequence lengths seen during training. The experiments use a synthetic ball-swap task in which a model must track five entities through an ordered sequence of state changes and predict every final assignment exactly.
Research questions
- Does Fan-aligned recurrence generalize beyond compute-matched fixed-depth Transformers?
- Which token representations and positional encodings support out-of-distribution extrapolation?
- How do model width, recurrent depth, and inference-time loops shift the generalization boundary?
Key findings
Fan-aligned recurrent models outperform matched fixed-depth baselines beyond the training range. Length-invariant atomic tokens and the removal of absolute positional encodings are important for extrapolation. Additional width and depth extend the usable range, but performance still collapses on the longest 40-80-swap sequences, and excessive recurrence can introduce state drift.