I tried reloading the fees on the go, network was bad, so I hit it again, as I didn't know if it triggered correctly. This lead to the job executing twice and booking the fees twice as well. To revert it I had to change the fee to a different value (added 1ct), fee refresh, revert it and fee refresh again.
So on the one hand the fee refresh sometimes times out, but continues well with a refresh afterwards. But concurrent refreshes (potentially also by different users) lead to multiple bookings of the same fee.