Canadian Adverse Driving Conditions Dataset

Matthew Pitropov1 , Danson Garcia2 , Jason Rebello3 , Michael Smart4 ,

Carlos Wang4 , Krzysztof Czarnecki1 and Steven Waslander3

Abstract— The Canadian Adverse Driving Conditions In this work, we present an autonomous driving dataset
(CADC) dataset was collected with the Autonomoose au- filled with representative examples of winter driving con-
tonomous vehicle platform, based on a modified Lincoln MKZ. ditions in the Waterloo region, in an effort to compel
The dataset, collected during winter within the Region of Water-
loo, Canada, is the first autonomous vehicle dataset that focuses the self-driving vehicle research community to address the
shortcomings of current methods in adverse weather. Cover-
arXiv:2001.10117v3 [cs.CV] 27 Feb 2020

on adverse driving conditions specifically. It contains 7,000

frames collected through a variety of winter weather conditions ing 75 driving sequences and over 20 km of driving, with
of annotated data from 8 cameras (Ximea MQ013CG-E2), varying levels of both traffic and snowfall, the Canadian
Lidar (VLP-32C) and a GNSS+INS system (Novatel OEM638). Adverse Driving Conditions (CADC1 ) dataset includes a
The sensors are time synchronized and calibrated with the
intrinsic and extrinsic calibrations included in the dataset. wide range of perception challenges collected over 3 days in
Lidar frame annotations that represent ground truth for 3D the Waterloo region in Ontario, Canada, during March 2018
object detection and tracking have been provided by Scale AI. and February 2019. We describe the technical details of the
data collected and object detection labels provided with the
CADC dataset below.
II. R ELATED DATASETS
As of December 2019, there has not been a release of
a multi-modal dataset for autonomous vehicles containing
annotated lidar data and RGB images with the focus on
adverse driving conditions and specifically snowy weather.
The KITTI dataset [1] released 2012 was the first multi-
modal dataset for autonomous vehicles which contained data
for 4 cameras, 1 lidar and a GPS/IMU system. All cameras
were forward facing and all annotated objects are within this
forward facing direction. The dataset also only contains clear
weather conditions.
In 2016, the Oxford RoboCar Dataset [2] was released. It
contained 1 trinocular stereo camera, 3 cameras, 2 2D lidars,
1 3D lidar and 1 GPS/IMU system. Notably, the dataset was
collected in several different weather conditions; however,
Fig. 1. Autonomoose, our autonomous vehicle testing platform.
only one drive was taken during snow. It was released with
a focus on localisation and mapping and as such, no 2D or
3D object annotations are included.
I. I NTRODUCTION In 2018, two new datasets were released, the ApolloScape
Open Dataset [3] and the KAIST dataset [4]. ApolloScape
The safe operation of self-driving cars across a variety
contains 2 forward facing cameras, 2 lidars and 1 GPS/IMU.
of adverse weather conditions remains a challenging open
It contains 70,000 3D fitted car annotations. It also contains
problem in robotics. Perception tasks become more difficult
a variety of weather conditions, including bright sunlight,
as precipitation, and in particular snowfall, degrades sensor
cloud cover and rain, and the authors suggest that they
returns and obfuscates the surroundings. Existing methods
will be adding more weather conditions such as snow in
for localization and mapping, object detection, semantic
the future. The Kaist Dataset contains 2 RGB cameras, 1
segmentation, tracking and prediction, all suffer as snowfall
thermal camera, 1 lidar and 1 GPS/IMU. It includes 3D
thickens and accumulates on the ground and sensor optics,
object annotations for objects visible in the forward-facing
thereby changing the appearance of the driving environment.
cameras, but does not include diverse weather conditions or
1 Department of Electrical and Computer Engineering, University of snowfall.
Waterloo, Canada More recently, the H3D Dataset [5], nuScenes Dataset [6],
2 Department of Electrical and Computer Engineering, University of Argoverse Dataset [7], A*3D Dataset [8] and the Waymo
Toronto, Canada
3 Institute for Aerospace Studies, University of Toronto, Canada 1 Pronounced cad-see. The dataset is available at http://cadcd.
4 University of Waterloo, Canada uwaterloo.ca
 TABLE I
Open Dataset [9] have been released. H3D contains 3 cam-
S ENSOR S UITE
eras, 1 lidar and GPS/IMU. All cameras point in the forward-
facing direction and the weather is not diverse. nuScenes Sensor Parameter Description
contains 40,000 annotated frames, which contain 1 lidar, 1x Velodyne Data Rate 10 Hz
6 cameras and 5 radar. Argoverse contains data from 7 VLP-32C Lidar Beam Count 32
Range 200 m
cameras, 2 stereo cameras, 2 lidars as well as detailed map Field of View (FOV)
data. The A*3D dataset contains 39,179 labeled point cloud Horizontal 360°
Vertical 40° (−25° to 15°)
frames with two front facing cameras for stereo vision. Angular Resolution
It contains data collected in the sun, cloud and rain. The Horizontal 0.2°
Minimum Vertical 0.3° (non-linear dist.)
Waymo Open Dataset is one of the largest publicly available Distance Accuracy 3 cm
autonomous driving datasets at 200,000 frames, and contains Rate ∼600,000 points/s
data for 1 mid-range lidar, 4 short-range lidars and 5 cameras. 8x Ximea Data Rate 10 Hz
MQ013CG-E2 Resolution 1280x1024 (1.3 MP)
Both the nuScenes, Argoverse and Waymo dataset provide Camera 1x Horizontal FOV 102.8°
360° FOV coverages with their camera configurations. They 8x Horizontal FOV 360°
Sensor Size 1/1.8”
also list diverse weather conditions as part of their contri- Type Global Shutter
butions, but currently none of them contains sequences with Exposure Range (texps ) 800 µs to 1000 µs
Acquisition Time 29 µs + texps
snow. There are therefore no comparable datasets available 8x Edmund Optics Focal Length 3.5 mm
that capture both surround vision and lidar data as well C Series Fixed Working Distance 0 to ∞ mm
Focal Length Lens Aperture f/8
as ground truth motion and 3D object labels in snow-filled 1x Dataspeed Data Rate
driving conditions. Advanced Driver Brake 50 Hz
Assistance Gear 20 Hz
Systems Steer 50 Hz
(ADAS) Kit Surround 5 Hz
III. V EHICLE AND S ENSORS Throttle 50 Hz
Wheel Speed 100 Hz
Miscellaneous 20 Hz
The Autonomoose (Figure 1) is an autonomous vehicle 1x Novatel Data Rate
platform created as a joint effort between Toronto Robotics OEM638 INSPVAX 20 Hz
Triple-Frequency BESTPOS 20 Hz
and AI Lab (TRAIL)2 at the University of Toronto and GNSS Receiver INSCOV 1 Hz
Waterloo Intelligent Systems Engineering Lab (WISE Lab) at RTK 100 Hz
PPP 3 Accuracy 4 cm
the University of Waterloo. This platform has been developed RTK 4 Accuracy 1 cm + 1 ppm
to demonstrate research progress on perception, planning, Time Accuracy 20 ns RMS
control and safety assurance methods for autonomous driv- Velocity Accuracy 0.03 m/s RMS
1x Sensonor Data Rate 100 Hz
ing. In addition, a complete autonomous driving software STIM300 MEMS Bias Stability
stack has been developed and tested on over 100 km of public IMU Accelerometer 0.05 mg
Gyroscope 0.5°/h
road driving. Attitude Accuracy
Roll/Pitch 0.015°
Table I provides a brief summary of the sensor outputs that Heading 0.08°
are included in the CADC dataset. It contains information for 2x Xsens Data Rate 200 Hz
1x MTi-300-AHRS Bias Stability
our GPS/IMU, cameras, lidar, Xsens IMUs and Dataspeed 1x MTi-30-AHRS Accelerometer 15 µg
Advanced Driver Assistance Systems (ADAS) Kit. The IMUs Gyroscope 10°/h (MTi-300), 18°/h (MTi-30)
Attitude Accuracy
ADAS Kit provides feedback (as well as actuation) through Average Roll/Pitch 0.3° (MTi-300), 0.5° (MTi-30)
drive-by-wire, including 4x wheel speeds, steering, throttle, Average Heading 1.0°
brake, gear, turn signals, and other vehicle information. More
detail is provided in Section Raw Data.
1) GPS Timestamps:: From Figure 3, the Sensonor IMU
A. Time Synchronization is synchronized to GPS reference clock from the Novatel
Each sensor data folder has a timestamps.txt file OEM638 receiver, which is also used as the primary ref-
containing timestamps corresponding to the data files. On the erence clock on the Autonomoose computer. Two VEXXIS
Autonomoose, sensors either directly sync to the GPS time or GNSS-502 Antennae provides the GPS radio signals to the
the computer’s system time. Figure 3 shows how sensor data Novatel OEM638 GNSS receiver. The receiver outputs GPS
and time synchronization are distributed across the platform. NMEA messages which contain a UTC timestamp, and a
The Autonomoose Time Synchronizer is a custom-designed Pulse Per Second (PPS) signal set to rising edge output
signal distribution board, using two ADG3123 CMOS to high at the start of every second. These signals are sent to the
voltage level translators. computer, lidar and Xsens IMUs. The GPS also outputs a
variable frequency (VARF) signal which is set to the rising
2 Formerly the Waterloo Autonomous Vehicles Laboratory (WAVELab) at
edge output at 10 Hz (synchronized to the PPS signal). The
the University of Waterloo
VARF signal is used to hardware trigger the 8x cameras to
3 Precise Point Positioning start their next image acquisition cycle. Both the VARF and
4 Real-time Kinematic PPS signals are sent across the system via the I/O COM.
Fig. 2. Top: Cropped picture of Autonomoose. Bottom: Side profile and overhead view of the Autonomoose CAD file with each sensor frame axis
overlaid.

Dataspeed USB RTK

ADAS Kit Autonomoose Antenna
Computer
8x USB

RS232 COM + PPS RF Coax

8x Ximea Ethernet
Cameras 8x VARF 2x USB X
Xbee RF
Transceiver Y
Ethernet
RS232 COM + PPS Autonomoose
Velodyne Time Synchronizer
VLP-32C RS232 COM

2x USB COM
2x PPS I/O COM
2x RS232 COM
1x RS232 COM + PPS
2x Xsens
IMUs Novatel 2x RF Coax
GNSS Receiver
Camera
IMU COM 2x VEXXIS Lidar
Sensonor
MEMS IMU GNSS Antennae Phase Lock Angle
180° Cut Angle
Pointcloud Corrected Behind
Data
Pointcloud Corrected Forward
Time Sync
Both
Fig. 4. Lidar-camera motion correction diagram (top-down view).
Fig. 3. Sensor communication and synchronization diagram.

of the Network Time Protocol (NTP) is then used to syn-

chronize the GPS reference clock with the system clock of
2) Computer Timestamps:: With the GPS as the source, the computer and correct for clock drift from accumulating
the Autonomoose computer receives the GPS PPS signals over time. With the KPPS support the computer is stated
and NMEA messages from the COM port through a mod- to have a typical accuracy of ±1 µs within the GPSD Linux
ified DB9 serial cable. Using GPSD to process the NMEA documentation[10].
messages, as well as pps-tools to interface the Kernel PPS 3) Lidar Timestamps:: The lidar is time synchronized
(KPPS) from the Linux OS, the GPS reference clock is made with the GPS PPS signal and NMEA messages. Each lidar
available in the computer system. Chrony, an implementation point cloud contains a full 360° sweep of the lidar beams,
 TABLE II
starting from the 180° cut angle (directly behind the car) and
T HE STANDARD DEVIATION OF THE PERIOD FOR EACH DATASPEED
rotating clockwise, as shown on Figure 4. The message is
MESSAGE OVER DRIVE 0027 ON 2019 02 27.
timestamped to the phase lock angle, which is at the point
when the sensor array is aligned with the positive x-axis of Message Type Standard deviation (ms)
the lidar frame (front side of Autonomoose). The sensor array brake info report 1.849
passes the phase lock angle at every rising edge from the brake report 1.036
gear report 0.743
VARF signal and motion corrected messages are corrected misc 1 report 1.162
to this timestamp. During motion correction, the lidar points steering report 1.054
surround report 0.671
on the left hemisphere are transformed forward in time, while throttle report 1.042
the right hemisphere are transformed backward in time based wheel speed report 2.431
on the vehicle motion to match the time at the phase lock
angle. Through this process, the point cloud looks stationary
and is synchronized to the VARF signal. z upwards. The origin of the GPS/IMU frame is within the
4) Cameras Timestamps:: As shown on Table I and trunk inside the IMU. It is rotated 90° clockwise along the
Figure 3, all 8 cameras are hardware triggered at 10 Hz with z axis compared to the base link frame. The lidar frame
the VARF signal from the GPS. Camera images are taken at has the same orientation as the base link and is located at
the same instant when the lidar passes the phase lock angle. the lidar’s optical center. Each camera frame is located at
Unfortunately, the Ximea MQ013CG-E2 does not have an its respective optical center, with z axis pointing towards
internal time and therefore can not directly timestamp its own the image plane, x axis to the right and y axis downwards.
data. Therefore it is timestamped when the image is acquired The three forward-facing cameras are aligned in a trinocular
in Robot Operating System (ROS), and is also truncated to configuration, allowing for convenient rectification if needed.
within 0.1 s to match the VARF output rate. This truncation is Each Xsens IMU is placed on top of the camera with the
based on the validity that under normal operation the image same orientation of the base link but with its x axis being
acquisition period is well below the 0.1 s VARF period (see aligned with the camera’s z axis. Small misalignments exist
Table I acquisition delay). By doing this the image timestamp in each sensor axis position and exact transforms are included
should more accurately represent the time which the image with the extrinsic calibration file.
is captured by the camera. V. C ALIBRATION
5) Vehicle Control Timestamps:: The primary vehicle
control messages required for an automated driving system Figure 5 details the contents of a calibration zip file which
(ADS) are the vehicle’s throttle, brake, steering and gear is discussed further in this section.
shift. These control messages, received from the vehicle’s date calib.zip
0X.yaml X={0,1,2,3,4,5,6,7}
controller area network (CAN) bus, are decoded by the extrinsics.yaml
Dataspeed ADAS kit and are sent to the computer via a
Fig. 5. Contents of a calibration zip file, which contains intrinsics for each
Universal Serial Bus (USB) port. Since the Dataspeed ADAS 8 cameras, and extrinsics between each pair of sensors.
kit cannot be directly connected to a GPS PPS signal or
an NMEA message, each decoded message is timestamped
with the latest computer timestamp once the ROS CAN A. Camera intrinsics
driver receives it. All CAN Bus messages are sent to the Each of the eight camera yaml files labeled 00.yaml
CAN-bus-to-USB converter, which is part of the Dataspeed to 07.yaml store intrinsic calibration information: camera
ADAS Kit that relays them to the Autonomoose computer name, height and width (pixels), camera matrix, distortion
via USB, as soon as possible. There are two exceptions. One model and the distortion coefficients.
is the surround report which is event periodic and depends Equation 1 contains the camera matrix K. The focal length
on potential alerts. Another is the miscellaneous message is fx and fy in pixels, the axis skew s causes shear distortion
which is aggregated over the cycle time. Table II contains the and the coordinates for the optical center are cx and cy in
standard deviation of the period for each Dataspeed decoded pixels.
message on one drive showcasing low standard deviations  
for each message. fx s cx
6) Xsens Timestamps:: Time synchronization for the K =  0 fy cy  (1)
Xsens IMUs is a two step process that involves receiving 0 0 1
PPS from the GPS and timestamps from the Computer. As We provide the distortion coefficients for the Brown-
these devices cannot receive NMEA messages directly, any Conrady or “Plumb Bob” distortion model [11]. The ordering
error in the computer time will propagate to these IMUs. of coefficients in equation 2 is also used in OpenCV, Matlab
and ROS. The values for k1 , k2 , k3 are the 2nd, 4th and
IV. S ENSOR F RAMES 6th order radial coefficients, respectively. The two tangential
Figure 2 contains the full sensor diagram for Au- distortion coefficients are p1 and p2 .
tonomoose. The origin of the base link frame is at the center  
of the rear axle with x pointing forwards, y to the left and kc = k1 k2 p1 p2 k3 (2)
B. Sensor extrinsics There is a subsection for each sensor: camera images, lidar,
The extrinsics.yaml file contains an array of 4x4 novatel GPS/IMU, vehicle control and the Xsens IMUs. The
homogenous transformation matrices. There are transforms data rate for each message is included within Table I.
between these pairs of frames: base link to lidar, GPS to raw.zip
lidar, camera to right neighbouring camera and camera to image 0X X={0,1,2,3,4,5,6,7}
lidar points
lidar. Each transform has been attained through a calibration lidar points corrected
novatel
procedure with two exceptions. The lidar to base link trans- novatel bestpos
novatel imu
form was measured using a 3D scan of Autonomoose and novatel inscov
novatel rtk
direct camera to lidar transforms for each camera is provided brake info report
brake report
for convenience. Equation 3 contains an example transform gear report
misc 1 report
converting data within the GPS/IMU frame (FG ) to the lidar steering report
frame (FL ). The rotation matrix from FG → FL is defined surround report
throttle report
as RLG ∈ R3×3 . The translation matrix from FG → FL is turn signal cmd
wheel speed report
defined as tLG ∈ R3×1 . xsens 300 imu
xsens 300 velocity
  xsens 30 imu
RLG tLG xsens 30 velocity
TLG = (3)
0 1 Fig. 8. Contents of a raw data zip file.
VI. DATASET FORMAT
The CADC dataset has been stored in a format similar to A. Image data
the KITTI Raw dataset. Sensor data are provided for each There are 8 camera image folders, one for each of the 8
recorded drive, along with the calibration data for each given cameras denoted image 0X for X from 0 to 7. Driving se-
calendar day of recording. Each drive has raw output from quence 0066 from 2019 02 27 contains contains 15 missing
all sensors at their full sample rate; labeled data, which is a cam 00 images, this segment was not chosen to be part of
sampled subset of the raw dataset at ∼3 Hz; and lastly the the labeled data.
corresponding 3D annotations for the labeled data. Figure 6 1) images:: Each original image has been stored as a
contains the file structure used for all data available in this 1280x1024 PNG image with a size of ~2 MB.
dataset.
date/
B. Lidar data
drive/
labeled.zip 1) lidar points:: Each lidar point cloud file has a times-
raw.zip tamp which corresponds to when the lidar beams pass the
3d ann.json
calib.zip phase lock angle.
Fig. 6. Folder structure for downloading CADC. 2) lidar points corrected:: Each lidar point cloud file
has been corrected for motion distortion relative to vehicle
There are three types of sensor folders, which are all motion using post-processed GPS/IMU data to the timestamp
displayed in Figure 7. Camera image folders contain PNG which corresponds to when the lidar beams pass the phase
images. Lidar point clouds are stored as binary files, which lock angle.
can be read using the development kit. All other sensor C. GPS/IMU data
data have text files with space-separated values. Detailed
1) novatel:: A Novatel Inertial Position Velocity Attitude
information for each value can be found within the sensor’s
- Extended (INSPVAX) message, which contains the most
dataformat.txt file.
recent position, velocity and orientation as well as standard
labeled.zip deviations.
image 0X X={0,1,2,3,4,5,6,7}
timestamps.txt 2) novatel bestpos:: A Novatel Best Position (BESTPOS)
data
frame number.png message, which contains the best available current position.
lidar points
timestamps.txt 3) novatel imu:: IMU message, which contains the
data
frame number.bin change in orientation and acceleration about each axis cor-
novatel
dataformat.txt rected for gravity, the earth’s rotation and sensor errors. Pitch,
timestamps.txt roll and yaw are defined as right handed with pitch about
data
frame number.txt the x axis, roll about the y axis and yaw about the z axis.
Fig. 7. Contents of a labeled drive zip file, which contains data from 8 To retrieve instantaneous acceleration or rotation, the data
cameras, 1 lidar and 1 GPS/IMU. should be multiplied by the sample rate of 100 Hz.
4) novatel inscov:: A Novatel Inertial Covariance (IN-
SCOV) message, which contains the uncertainty of the
VII. R AW DATA current pose solution. These are three 3x3 matrices, each with
A raw data zip file contains content from all available the variance along the diagonal corresponding to position,
sensors on Autonomoose. Figure 8 contains the complete list. attitude and velocity.
 5) novatel rtk:: A Novatel RTK message, created by post- [
{
processing the GPS data with base station data. It contains a cuboids :
subset of the INSPVAX message: the position and orientation [
{
as well as standard deviations. ” u u i d ” : ”0241 bd75−b41f −4c67−8f c d −9388 f d 7 e 2 c 8 b ” ,
” label ” : ” Pedestrian ” ,
D. Vehicle control data ” position ” : {
” x ” : −48.85306519173028 ,
Each data type was converted from their respective ” y ” : −10.954928897518318 ,
” z ” : −0.728937152344576
Dataspeed ROS message, with detailed information provided },
within their dataformat.txt file as well as online at ” dimensions ” : {
”x” : 0.828 ,
ROS dbw mkz msgs. The messages contain information on ”y” : 0.766 ,
braking, gear, steering, surround, throttle and wheel speed, ”z” : 1.688
},
sorted by folder. The miscellaneous report messages contain ” yaw ” : −0.03539451751530657 ,
data from various sensors, for example the state of seat belts, ” stationary ” : false ,
” camera used ” : 7 ,
doors and the outside air temperature. ” attributes ”: {
” age ” : ” Adult ”
E. Xsens IMU data },
” p o i n t s c o u n t ” : 11
1) xsens 300 imu and xsens 30 imu:: Contains the ori- },
...
entation (with respect to the ENU frame), angular velocity ]
(rad s−1 ) and linear acceleration (m s−2 ) as well as the corre- },
...
sponding covariance matrices. ]
2) xsens 300 velocity and xsens 30 velocity:: Contains
the linear velocity (m s−1 ) and angular velocity (rad s−1 ). Fig. 9. The JSON structure of a 3D annotation file.

VIII. L ABELED DATA TABLE III

Object labels are provided for a subset of the raw data. The U NIQUE ATTRIBUTES .
subset is obtained by down sampling the raw data to select
one frame every 300 ms for a total of 50-100 frames per drive Relevant Label(s) Attribute Values

(15-30 seconds in length). As shown in Figure 7, there are 10 Truck truck type Snowplow Truck
Semi Truck
folders, one for each of the eight cameras denoted image 0X Construction Truck
for X from 0 to 7, a lidar points folder for the Velodyne lidar Garbage Truck
Pickup Truck
data and a novatel folder containing GPS/IMU data. All data Emergency Truck
is synced such that data for all sensors was matched to the Bus bus type Coach Bus
Transit Bus
closest data timestamp of the GPS/IMU. Standard School Bus
Van School Bus
A. Data (∼3 Hz) Bicycle rider state With Rider
Without Rider
1) images:: This data is equivalent to the image data in Pedestrian age Adult
Pedestrian With Object Child
the Raw Data section with the only difference being that Traffic Guidance Objects traffic guidance type Permanent
these images have been undistorted. Moveable
2) lidar points:: These lidar points are equivalent to the
motion corrected lidar points in the Raw Data Section.
3) novatel (GPS/IMU):: This data is equivalent to the post 4) dimensions:: The dimensions of the cuboid with x
processed Novatel RTK data within the Raw Data section. being the width from left to right, y being the length from
front to back and z being the height from top to bottom.
IX. 3D A NNOTATIONS 5) yaw:: The orientation of the object in the lidar frame.
Each 3D annotation file contains a list of frames with A yaw of zero will occur when when the cuboid is aligned
the detected cuboids in each frame. Figure 9 contains the with the positive direction of the lidar frame’s x-axis. A yaw
structure of this JSON file with an example cuboid. value of π/2 occurs when the cuboid is aligned with the
positive direction of the lidar frame’s y-axis.
A. cuboid 6) stationary:: A boolean value that describes if the
1) uuid:: A string used to identify this cuboid across all object is stationary across all frames in a drive.
frames within a drive. 7) points count:: An integer value of how many lidar
2) camera used:: The index of the camera used to label points are contained within the cuboid at this frame.
this cuboid. 8) label and attributes:: Each object has a label as well
3) position:: The center of the object in the lidar frame. as any attributes that are defined for the specified label. The
The object has its z axis pointing upwards, x axis pointing following labels are used: Car, Truck, Bus, Bicycle, Horse
to the forward facing direction of the object and the y axis and Buggy, Pedestrian, Pedestrian with Object, Animal,
pointing to the left direction of the object. Garbage Container on Wheels, and Traffic Guidance Object.
 104

Count (log)
102

100
s ts e t
Ca
r
ria
n ck eel jec Bu
s
ycl jec ggy als
est Tru Wh Ob Bic Ob Bu im
Ped on ce ith and An
rs a n W e
ine id ria
n rs
nta Gu est Ho
Co ffic Ped
age Tra
a r b
G

Label

Count (log)
104

102
s ts e t
Ca
r
stri
an ck Bu
s
eel jec ycl jec ggy als
e Tru Wh Ob Bic Ob Bu im
Ped on ith and An
rs da nce W se
ine i n r
Fig. 10. A map of data collected for CADC. nta Gu est
ria Ho
Co ffic Ped
age Tra
Garb
TABLE IV
TABLE CONTAINING DRIVING ASPECTS OF CADC DATASET. Label

Dataset type # of point clouds # of images Distance (Km) Fig. 12. Top: Number of unique labeled objects across all frames.
Bottom: Number of object instances across all frames.
Raw 32887 263637 20.33
Labeled 7000 56000 12.94

TABLE V on the given object class. The Truck label’s major attributes
TABLE CONTAINING DATA ASPECTS OF CADC DATASET. are pickup trucks and semi-trucks. The Bus label has four
different attributes: two transit types for coach and public
Dataset type Compressed size (GB) Uncompressed size (GB) transit and two school types for standard and van-length
Raw 472.7 514.73 school buses. The Bicycle label has an attribute for whether
Labeled 92.76 97.79
it is, or is not, being ridden. The Pedestrian label has an age
attribute set to Adult or Child. A Pedestrian with Object also
150 exists and is used, for example, when a person is pushing a
Frame count

stroller. The Animal label is used for wild animals as well as

100 pets. The Garbage Container on Wheels label is used for tall
50 garbage containers. Lastly, Traffic Guidance Objects could
be pylons with the Moveable attribute or permanent vertical
0 delineators for bike lanes with the Permanent attribute.
The Car label contains cars, SUVs and vans, and has no
0 50 100 150 200 250
unique attributes. One drawback of this design is that the
Annotations per frame single label covers several vehicle body sizes. We plan to
subsequently release 2D annotations that introduce attributes
Fig. 11. Each frame has been binned by the number of cuboid annotated.
to subdivide this class.

X. DATASET S TATISTICS
The set of vehicle classes includes [Car, Truck, Bus,
Bicycle, Horse and Buggy] and have attributes [Parked, We provide several figures and tables showcasing the wide
Stopped, Moving] in every frame. A Horse and Buggy range of data that has been collected. Figure 10 shows a
label is included, although there is only one in the training map with GPS points placed from the data. The red points
data. Pedestrian, Pedestrian with Object, Animal, Garbage represent the full dataset, whereas the overlayed green points
Container on Wheels and Traffic Guidance Object are not in are the subset of the data that is labeled.
vehicle class. Table IV contains information on the number of point
Table III lists potential attributes that are unique based clouds, images and distance traveled. Table V contains the
 ·104 TABLE VI
3 TOTAL INSTANCES FOR EACH VEHICLE LABEL SORTED BY STATE
Cars
ATTRIBUTE .
Trucks
2
Count

Pedestrians
Label (and attribute) Total Parked Stopped Moving
1
Car 281941 193246 18002 70693
Truck 20411 9959 2060 8392
0 Snowplow Truck 1497 628 221 648
Semi Truck 4653 1547 516 2590
0 50 100 150 200 Construction Truck 715 433 45 237
Radial distance (m) Garbage Truck 27 1 0 26
Pickup Truck 13045 7178 1138 4729
Percentage of annotations/class

Emergency Truck 474 172 140 162

Bus 4867 476 1513 2878
Coach Bus 751 193 135 423
Cars
Transit Bus 2899 196 847 1856
15 Trucks Standard School Bus 983 0 531 452
Pedestrians Van School Bus 234 87 0 147
Bicycle 785 520 0 265
10 With Rider 265 0 0 265
Without Rider 520 520 0 0
Horse and Buggy 75 0 75 0
5
TABLE VII
0 TOTAL INSTANCES FOR EACH PEDESTRIAN LABEL SORTED BY AGE
−2 0 2 ATTRIBUTE .

Orientation (rad)
Label Total Adult Child

Fig. 13. Top: Radial distance from the center of a cuboid to the origin of Pedestrian 62851 61664 1187
the lidar frame. Bottom: Orientation of cuboids within the lidar frame. Pedestrian With Object 638 638 0

30
20 LIDAR point cloud data. The number of points removed
10 is taken to be the approximate number of snow reflectance
0 points in a LIDAR scan. Bins for snowfall intensity, depicted
Light Snowfall Medium Snowfall Heavy Snowfall Extreme Snowfall
in Figure 14 are defined as: Light (25-249), Medium (250-
499), Heavy (500-749) and Extreme (750-1500). Lastly
Fig. 14. Number of drives for each level of snowfall.
Figure 15 contains an example LIDAR scan and image for
each snowfall level. The snow covering of the road is also
included. There are 18 driving sequences with bare road and
compressed and uncompressed sizes of the raw and labeled 57 with snow covering the road.
data. They are similar due to the images being the largest
sensor data size and already losslessly compressed in the raw
data. XI. D EVELOPMENT KIT
Figure 11 shows the number of frames for each number A basic development kit implemented in python is avail-
of annotations per frame. This is restricted to objects with at able at cadc devkit with the ability to view the vehicle
least one lidar point within its cuboid for a specific frame. path, the lidar projected onto the images and lastly the 3D
Figure 12 contains two graphs. The top graph shows the annotations projected onto the images and lidar.
number of unique objects binned by their label across all
frames based on uuid. The bottom graph displays the number
of object instances by how many frames they appear in. A. run demo vehicle path.py
Figure 13 also contains two graphs for the three objects This script loads all GPS messages in a drive, converts
with the most instances. The first displays the radial distance them to an ENU frame with the origin at the first message
of objects, and the second displays the orientation of each and plots each message as an axis frame. Figure 16 is an
object. Table VI contains the number of instances for each image displaying the output of this script.
vehicle label and type. Table VII contains the number of
instances for each pedestrian with age attribute.
B. run demo lidar.py
Each drive has been given a snowfall level from: light,
medium, heavy and extreme. This was done by first cropping This script loads a camera image and the corresponding
the point cloud to a cube that spans -4 to 4 in the x and y lidar file in a drive, loads the calibration data, then projects
direction and -3 to 10 in the z direction within the lidar frame, each lidar point onto the camera image. Point color is scaled
and then applying the Dynamic Radius Outlier Removal by depth. Figure 17 is an image displaying the output of this
(DROR)[12], which is designed to remove snowfall from script.
Fig. 15. Top down lidar view of each snowfall levels with the corresponding front camera image. Top: left image couple is the light snow and the right
side is medium snow. Bottom: left image couple is heavy snow and the right is extreme snow.

D. run demo lidar bev.py

This script loads lidar data and the corresponding 3D
annotation file in a drive, then creates a birds eye view of
the lidar point cloud with the cuboid boxes overlaid. Figure
19 is an image displaying the output of this script.
XII. C ONCLUSIONS
We present the CADC dataset, an annotated object dataset
containing lidar and images collected within the Region
of Waterloo during winter driving conditions. This dataset
Fig. 16. Output from the run demo vehicle path.py script on the drive will enable researchers to test their object detection and
0027 from 2019 02 27. It shows the path of Autonomoose as it makes a localization and mapping techniques on challenging winter
left hand turn
weather. In the future, we plan to release 2D annotations for
each individual image containing truncation and occlusion
values. We also will be creating a benchmark for 3D object
detection using this dataset.
ACKNOWLEDGEMENT
CADC Dataset would not have been created without
the help of many members of the Autonomoose project
over the past several years. Juichung Kuo, Ji-won Park
and Archie Lee contributed to the initial development and
setup of the sensor platform and triggering, as well as the
time synchronization required for data capture. Kevin Lee
supported this research through driving and collecting data
with the Autonomoose. Our SLAM team, consisting of Nav
Ganti, Leo Koppel and Ben Skikos created the ROS packages
we used to localize Autonomoose and motion correct the
Fig. 17. Output from the run demo lidar.py script on drive 0027 from lidar scans. Asha Asvathaman created several scripts for the
2019 02 27. The lidar points are projected onto the image of the front development kit used to visualize and process the dataset.
camera.
Lastly Jessen Liang designed and produced a calibration rig
to make calibration more efficient. We would also like to
thank Scale AI for annotating our dataset as well as the
C. run demo tracklets.py members of TRAIL and WISELab for assisting with auditing
This script loads a camera image and the corresponding the annotations.
3D annotation file in a drive, loads the calibration data, then R EFERENCES
creates and projects each cuboid within the frame onto the
[1] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics:
camera image. Figure 18 is an image displaying the output The KITTI dataset,” International Journal of Robotics Research
of this script. (IJRR), 2013.
Fig. 18. Output from the run demo tracklets.py script on drive 0033 from 2019 02 27. Each cuboid from the 3D annotation file is projected onto the
image of the front camera.

[2] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 Year, Vision and Pattern Recognition (CVPR), 2019.
1000km: The Oxford RobotCar Dataset,” The International Journal [8] Q.-H. Pham, P. Sevestre, R. S. Pahwa, H. Zhan, C. H. Pang, Y. Chen,
of Robotics Research (IJRR), vol. 36, no. 1, pp. 3–15, 2017. [Online]. A. Mustafa, V. Chandrasekhar, and J. Lin, “A*3D dataset: Towards
Available: http://dx.doi.org/10.1177/0278364916679498 autonomous driving in challenging environments,” 2019.
[3] P. Wang, X. Huang, X. Cheng, D. Zhou, Q. Geng, and [9] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui,
R. Yang, “The ApolloScape open dataset for autonomous driving J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam,
and its application,” IEEE Transactions on Pattern Analysis H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi,
and Machine Intelligence, p. 11, 2019. [Online]. Available: Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov, “Scalability in
http://dx.doi.org/10.1109/TPAMI.2019.2926463 perception for autonomous driving: Waymo open dataset,” 2019.
[4] Y. Choi, N. Kim, S. Hwang, K. Park, J. S. Yoon, K. An, and I. S. [10] G. E. Miller and E. S. Raymond, “Gpsd time service
Kweon, “KAIST multi-spectral day/night data set for autonomous howto,” Jun 2018. [Online]. Available: https://gpsd.gitlab.io/gpsd/
and assisted driving,” IEEE Transactions on Intelligent Transportation gpsd-time-service-howto.html
Systems, vol. 19, no. 3, pp. 934–948, March 2018. [11] D. C. Brown, “Decentering distortion of lenses,” Photometric Engi-
[5] A. Patil, S. Malla, H. Gang, and Y.-T. Chen, “The H3D dataset for neering, vol. 32, no. 3, pp. 444–462, 1966.
full-surround 3D multi-object detection and tracking in crowded urban [12] N. Charron, S. Phillips, and S. L. Waslander, “De-noising of lidar point
scenes,” 2019. clouds corrupted by snowfall,” in 2018 15th Conference on Computer
[6] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, and Robot Vision (CRV), May 2018, pp. 254–261.
A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuScenes: A
multimodal dataset for autonomous driving,” 2019.
[7] M.-F. Chang, J. W. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett,
D. Wang, P. Carr, S. Lucey, D. Ramanan, and J. Hays, “Argoverse: 3D
tracking and forecasting with rich maps,” in Conference on Computer
Fig. 19. Output from the run demo lidar bev.py script on drive 0033 from
2019 02 27. Each cuboid from the 3D annotation file is displayed within
the lidar frame from a top down perspective.