Repository navigation
Performance on GenImage lower than reported #24
Description
Activity
I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.
Isn't this pretty much the same?
I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.
forgot to use pretrained resnet. Maybe the results will be the same if pretrained ones are used.
I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.
CNNDetection: How much accuracy did you achieve after training for 20 epochs? The mean accuracy? I got a very low accuracy rate. I don't know why.
I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.
CNNDetection: How much accuracy did you achieve after training for 20 epochs? The mean accuracy? I got a very low accuracy rate. I don't know why.
trained 20 epochs on CNNDetection (resnets forgot to use pretrained weight),epoch12:
CNNDetection 测试集
测试集 Accuracy Avg Precision biggan 0.72025 0.8000844691830253 crn 0.5365872767157631 0.6421978263673008 cyclegan 0.9072672218016654 0.9829853367473628 deepfake 0.5398704902867715 0.6892271403795274 gaugan 0.6069 0.7286883767469993 imle 0.5373707301786274 0.7834663542279399 progan 0.996375 0.9998983927237407 san 0.6917808219178082 0.8000423166181219 stargan 0.9942471235617809 0.999944842431401 stylegan 0.9831413787347688 0.998229921425742 stylegan2 0.9767776664997496 0.997156066106713 whichfaceisreal 0.889 0.96896608318801 GenImage 测试集
测试集 Accuracy Avg Precision adm 0.8910833333333333 0.9688910822097034 biggan 0.8599166666666667 0.9415706366912934 glide 0.9134166666666667 0.9832866998232219 midjourney 0.7631666666666667 0.8437958577052245 sd14 0.9064166666666666 0.9795961927299013 sd15 0.896556034752172 0.9721477597462465 vqdm 0.8668333333333333 0.9440023234856975 wukong 0.9003333333333333 0.9703778911169451 {"train_lr": 5.07786304123261e-05, "train_loss": 0.23767979066172626, "train_class_acc": 0.9891468965043055, "train_min_lr": 5.07786304123261e-05, "test_loss": 0.5558900009078456, "test_acc1": 75.90385891815451, "test_acc5": 100.0, "epoch": 0, "n_parameters": 54432466}
{"train_lr": 5.076291203383308e-05, "train_loss": 0.21532221244741248, "train_class_acc": 0.9963892483607938, "train_min_lr": 5.076291203383308e-05, "test_loss": 0.3683272960285346, "test_acc1": 86.2042039547151, "test_acc5": 100.0, "epoch": 1, "n_parameters": 54432466}
{"train_lr": 5.073148378575147e-05, "train_loss": 0.21160123621401622, "train_class_acc": 0.9970669595100622, "train_min_lr": 5.073148378575147e-05, "test_loss": 0.35336238008435117, "test_acc1": 86.86634627273635, "test_acc5": 100.0, "epoch": 2, "n_parameters": 54432466}
{"train_lr": 4.709174377446268e-05, "train_loss": 0.20996936735510288, "train_class_acc": 0.997344709966432, "train_min_lr": 4.709174377446268e-05, "test_loss": 0.38198807171973165, "test_acc1": 85.49190874479763, "test_acc5": 100.0, "epoch": 3, "n_parameters": 54432466}
{"train_lr": 4.479838119291914e-05, "train_loss": 0.20844324520938898, "train_class_acc": 0.9976557902985046, "train_min_lr": 4.479838119291914e-05, "test_loss": 0.34143052139802044, "test_acc1": 87.17995910371359, "test_acc5": 100.0, "epoch": 4, "n_parameters": 54432466}
{"train_lr": 4.203944690266106e-05, "train_loss": 0.2073439945434702, "train_class_acc": 0.9978696579514783, "train_min_lr": 4.203944690266106e-05, "test_loss": 0.3502461508979941, "test_acc1": 87.3558110505868, "test_acc5": 100.0, "epoch": 5, "n_parameters": 54432466}。我正确配置了权重，貌似也无法到达作者的精度请问单卡4090训练20epoch要多久时间啊
I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.
I would like to ask approximately how long it takes to train for 20 epochs on a single RTX 5090 in your setup.Thank you for your time and assistance.
I used CNNDetection as the training set and trained for 20 epochs on one 5090, using the shell script parameter and found the best result of the 20 ckpts different from the paper. the results are averagely high but not as high as the paper said.
I would like to ask approximately how long it takes to train for 20 epochs on a single RTX 5090 in your setup.Thank you for your time and assistance.
I remember it took about a day or more
Hi, thank you for sharing this great work!
I was trying to reproduce the results on the GenImage dataset following the training and evaluation instructions in the repository, but I found that the reproduced performance is significantly lower than what was reported in the paper.
Following settings are chosen by myself, while others remain the same with the original paper.
Settings:
Results:
As you can see, the Performance is lower than reported.
I set batchsize as 32 and choose the best ckpt in the 5 epochs, but I'did use 8 * A100 as reported in the paper. Since Learning rate is not changed, I am not sure if I need to scale lr according to my hardware setting. I wonder if you have seen such behavior in your experiment. Thank you.