Nick Turner and Jack Strand

Attention head connectomes for large language models

In light of recent developments in the capabilities of large language models, making sense of the internal representations within these models is growing in importance. We seek to extend some of this research by looking for larger scale patterns, and have built a codebase to extract a connectome from transformer models by using the composition metric from Elhage et al., 2021. This gives a complete picture of the direct “edges” between the communication graph of larger models than previously analyzed in this way. This graph may show interesting patterns that elucidate function in specific networks, or show general principles in how transformers learn. We’ve seen some initial results from this research, and are looking for feedback on whether this lens seems promising or useful to others.

Keywords

• Interpretability
• LLMs
• Prosaic