The BEVFusion model trained on SimBEV reports a validation mAP of only ~48+, much lower than typical nuScenes results, yet visualization shows near-perfect 3D boxes within the 51.2 m range, indicating a clear evaluation flaw: the data collection uses bbox_collection_radius: 200.0, while the model’s point_cloud_range is only [-51.2, -51.2, -5.0, 51.2, 51.2, 3.0]; the custom SimBEVDetectionEval applies no distance-based clipping and computes metrics on all incoming GT boxes directly, so distant objects that the model cannot physically detect are all counted as false negatives (FN), severely suppressing recall and artificially causing a drastic drop in the reported validation mAP.
The BEVFusion model trained on SimBEV reports a validation mAP of only ~48+, much lower than typical nuScenes results, yet visualization shows near-perfect 3D boxes within the 51.2 m range, indicating a clear evaluation flaw: the data collection uses bbox_collection_radius: 200.0, while the model’s point_cloud_range is only [-51.2, -51.2, -5.0, 51.2, 51.2, 3.0]; the custom SimBEVDetectionEval applies no distance-based clipping and computes metrics on all incoming GT boxes directly, so distant objects that the model cannot physically detect are all counted as false negatives (FN), severely suppressing recall and artificially causing a drastic drop in the reported validation mAP.