We train transformers on mazes and use linear probing to show that they form internal representations of the entire maze, and find evidence for Adjacency Heads which attend to valid "next moves"

The paper accompanying our maze-dataset library, which was used in our first two papers and is publically available.