In our paper, we show that large language models represent categorical concepts as simplices and hierarchical relations as orthogonality.

We confirm our theory with Gemma-2B representations and this repo provides the code for the experiments.

animals.json and plants.json are sets of words generated by ChatGPT-4.

***_gemma.json and ***_graph.adjlist files in data are the collections of words and hierarchy graph of WordNet for noun and verb, which are obtained by get_wordnet_hypernym.ipynb.

You need to install Python packages transformers, networkx, scikit-learn, nltk, inflect, torch, numpy, seaborn, matplotlib, json, and tqdm to run the codes. Also, some GPUs would be helpful to implement the codes efficiently.

Run store_matrices.py first to store the unembedding vectors, before you run other jupyter notebooks.

• 1_Animal.ipynb: We display 2D plots in Figure 2 and 3D plots in Figure 3.
• 2_Noun_Test.ipynb: We validate the existence of the vector representations for each feature in WordNet noun hierarchy in Figure 4.
• 3_Noun_Heatmap.ipynb: We confirm that the hierarchical relation in WordNet noun hierarchy is encoded as orthogonality in Figure 5. Also, we zoom in the heatmaps in Figure 8 for the subtree in Figure 7.
• 4_Verb_Test.ipynb: We validate the existence of the vector representations for each feature in WordNet verb hierarchy in Figure 9.
• 5_Verb_Heatmap.ipynb: We confirm that the hierarchical relation in WordNet verb hierarchy is encoded as orthogonality in Figure 10.