Hello,
Thank you for your excellent work! I’m currently exploring this approach and encountered a small issue during the pretraining phase of the GNN encoder. Specifically, I trained the tokenizer using the default settings from the paper, and that worked well. However, in the following pretraining stage, when I used the default temperature of 0.1 for the contrastive loss, I noticed the loss dropped to negative values within the first epoch. Additionally, the node prediction accuracy stayed below 0.4.
I found that increasing the temperature to values like 0.5 or 1 helps stabilize the training, with the accuracy converging to around 0.9. However, I’m concerned that this adjustment might affect the final performance of the pretrained model weights. Could you provide any insights on whether this behavior is expected or if there are any potential downsides to increasing the temperature?
Thank you!
Hello,
Thank you for your excellent work! I’m currently exploring this approach and encountered a small issue during the pretraining phase of the GNN encoder. Specifically, I trained the tokenizer using the default settings from the paper, and that worked well. However, in the following pretraining stage, when I used the default temperature of 0.1 for the contrastive loss, I noticed the loss dropped to negative values within the first epoch. Additionally, the node prediction accuracy stayed below 0.4.
I found that increasing the temperature to values like 0.5 or 1 helps stabilize the training, with the accuracy converging to around 0.9. However, I’m concerned that this adjustment might affect the final performance of the pretrained model weights. Could you provide any insights on whether this behavior is expected or if there are any potential downsides to increasing the temperature?
Thank you!