Thanks for open-sourcing this great work!

I was reading the source code in model.py file and got really confused by the implementation of RoPE. RoPE frames the multi-dimension rotation matrix as a bunch of 2x2 rotation matrix. So for a d-dimensional array [x1, x2, ..., xd], x1 and x2 goes through a rotation matrix while xd-1 and xdanother.

The problem is

freqs = einsum("i , j -> i j", seq, inv_freq)
positions = torch.cat((freqs, freqs), dim=-1)

If we concat the freqs in the last dimension, the resulting matrix will be 0, 2, 2i, ..., d, 0, 2, 2i, ..., d. In this case, if we later apply the rotary embedding, x1 and x2 accept different rotation angles.

Thus, I would suggest the correct way is to perform torch.repeat_interleave in the last dimension as is specified in huggingface timm library.

Correct me if I was wrong. Appreciated for any replies.