Fabric server 1.19.4 golf Version: 0.63

When I ( as a server admin) go from creative to survival i am unable to break any more blocks.

When I rejoin, I am able to break blocks again.

However, I also found that without switching game modes, it randomly doesn't let me break any more blocks. while not switching game modes.

I have removed the mod for now.

Vanilla 1.19.4 client