Thanks for your great work! I notice that you use "per_token_loss = torch.exp(logp - logp.detach()) * advantages.unsqueeze(1)" to calculate the loss. It seems to be a trick since it is equivalent to "per_token_loss = logp * advantages.unsqueeze(1)". Could you explain the motivation behind this trick?
Thanks for your great work! I notice that you use "per_token_loss = torch.exp(logp - logp.detach()) * advantages.unsqueeze(1)" to calculate the loss. It seems to be a trick since it is equivalent to "per_token_loss = logp * advantages.unsqueeze(1)". Could you explain the motivation behind this trick?