...because it supports setups with 6 GPUs (me) and optimally aligns with NVIDIA tiles (128) and AMD tiles (192).

I'm using six AMD 7900 XTX GPUs. I see this message in the log:
Padding vocab_size from 32760 to 32832 for efficiency

So I wondered if 32,832 should be the default... No, it breaks some NVIDIA alignment. Then what next? 33,024 seems to play nice with 4,6, and 8 GPUs, and meets alignment on both NVIDIA and AMD ...in theory.

32,768
Factors include: 2, 4, 8, 16, 128

33,024
Factors include: 2, 4, 6, 8, 12, 16, 128, 192

Thoughts?