This repository houses the SNVA application and additional code used to develop the computer vision models at the core of SNVA. Model development code is based on TensorFlow-Slim.
v0.1 of the project is described in detail in our paper: arXiv preprint arXiv:1811.04250, 2018. If you were directed here by our paper, the v0.1 code may be found here.
SNVA is intended to expand the Roadway Information Database (RID)’s ability to help transportation safety researchers develop and answer research questions. The RID is the primary source of data collected as part of the FHWA’s SHRP2 Naturalistic Driving Study, including vehicle telemetry, geolocation, and roadway characteristics data. Missing from the RID are the locations of work zones driven through by NDS volunteer drivers. The app’s first release will focus on enabling/enhancing research questions related to work zone safety by using machine learning-based computer vision techniques to exhaustively and automatically detect the presence of work zone features across the entire ~1 million-hour forward-facing video data set, and then conflating that information with the RID’s time-series records. Previously, researchers depended on sparse and low fidelity 511 data provided by states that hosted the routes driven by volunteers. A successful deployment of the SNVA app will make it possible to query the RID for the exact start/stop locations, lengths, and frequencies of work zones in trip videos; a long-standing, highly desired ability within the SHRP2 community.
SNVA v0.2 is intended to run in a networked environment, and is comprised of three main components:
Manages the assignment of tasks to other working nodes. For more details view here.
The anaylzer node is a tf-serving 2.1 instance built from the official docker image. For more details, view here.
The processor node is assigned videos by the Control Node. It then handles making inference requests to the analyzer node, as well as pre/post processing and writing the results.
To deploy the SNVA application, follow the below steps:
All control and inference traffic requires verified mutual TLS. There is no ws://, plaintext gRPC, or certificate-verification bypass. Upgrade the control node and processors together: old clients cannot use the new assignment/reconnect protocol.
-
Provision certificates through your organization's PKI. Each server needs a PEM certificate chain and private key with
serverAuthusage and a subject alternative name matching the hostname clients use. Each processor and GUI user needs a trustedclientAuthcertificate and private key. Use a dedicated SNVA client CA: any certificate it signs grants access to the processor and status endpoints. Use distinct certificates for users/processors, restrict network access, and rotate/revoke credentials through your PKI. A reconnect token is also tied to the original client certificate, so a certificate change requires a new registration. -
Place
ca.pem,cert.pem, andkey.pemin separate control and processor credential directories outside the repository/build context.ca.pemmust trust the relevant server and client issuers. Protect directories and keys with owner-only POSIX permissions, or equivalent Windows ACLs. Never commit keys, GUI PKCS#12 files, or the generated serving config. On Compose, the control and analyzer server certificates must include DNS SANscontrolandanalyzer, respectively; add real external names if remote clients use them. Do not connect to0.0.0.0, a listening address rather than a server identity. -
Generate the TensorFlow Serving 2.1.4 SSLConfig, which embeds the analyzer key, certificate, and client CA and requires client certificates. The helper needs only Python's standard library:
python3 -m utils.transport --tls-ca /secure/analyzer/ca.pem \
--tls-cert /secure/analyzer/cert.pem --tls-key /secure/analyzer/key.pem \
--serving-config /secure/analyzer/serving-tls.pbtxtKeep the generated file private (the helper creates mode 0600 on POSIX). On Windows, explicitly apply restrictive ACLs. Start serving with TLS and disable its cleartext REST API, not just port publication:
docker run --runtime=nvidia -p 8500:8500 \
--mount type=bind,src=/path/to/model,dst=/models/mobilenet_v2,readonly \
--mount type=bind,src=/secure/analyzer/serving-tls.pbtxt,dst=/run/secrets/serving-tls.pbtxt,readonly \
--entrypoint tensorflow_model_server docker.io/tensorflow/serving:2.1.4-gpu \
--port=8500 --rest_api_port=0 --model_name=mobilenet_v2 \
--model_base_path=/models/mobilenet_v2 --enable_batching \
--ssl_config_file=/run/secrets/serving-tls.pbtxt- Start the control node with its server cert/key and client CA. Supply video paths relative to the processor input directory; nested directories are supported. Start processors with their client cert/key and server CA using the commands below. Open the GUI at
https://<control-host>:8081/snvaGui.htmlafter importing the trusted GUI client certificate/key into the browser/OS certificate store.
The provided docker-compose.yml mounts only each service's required credentials read-only. Set these absolute paths in your shell (all are required): SNVA_MODEL_PATH, SNVA_MODEL_METADATA, SNVA_VIDEO_LIST, SNVA_VIDEOS, SNVA_CONTROL_LOGS, SNVA_CONTROL_OUTPUT, SNVA_PROCESSOR_LOGS, SNVA_PROCESSOR_OUTPUT, SNVA_CONTROL_TLS_DIR, SNVA_PROCESSOR_TLS_DIR, and SNVA_SERVING_TLS_CONFIG. Also set SNVA_CONTROL_ORIGIN to the exact canonical HTTPS origin used by GUI browsers, such as https://control.example.org:8081 (no trailing slash, no explicit default :443). Foreign-Origin browser requests are rejected even when the browser has a trusted client certificate; Python clients do not send Origin. Then run docker compose up --build on a GPU-enabled Linux host. SNVA_MODEL_METADATA contains class_names.txt and the model subdirectory with input_size.txt; SNVA_MODEL_PATH is the versioned SavedModel directory. No port 8501 or processor logging port is enabled. Custom deployments must preserve these TLS boundaries.
Only validated assignments may be acknowledged or completed. PROCESS includes an assignmentId; processors echo that ID and the exact relative video path in REQUEST_RECEIVED and COMPLETE. Reconnect uses the registered ID plus reconnectToken received over WSS. Treat tokens as credentials, not loggable URLs. The final outputList.txt is now JSON Lines, one {"video":"relative/path.mp4","output":"..."} object per line, written at successful control-node shutdown. Update any external parser previously expecting video: output text; newlines and delimiters in fields are escaped rather than creating forged entries.
Processors retain completions until an exact COMPLETE_ACCEPTED receipt arrives, and replay unconfirmed results after reconnect without double-counting. Completed controller sessions can recover terminal status, but expired sessions cannot reconnect. Processor logging uses spawned workers, trusted child-process queues, and one main-process rotating writer, with child cleanup preceding writer shutdown. The former TCP pickle receiver on localhost:9020 has been removed; do not recreate it in custom deployments.
Upon connecting to the Control Node, the Processor Node will request the name of a video to process and begin work. When all videos are complete, the Control Node will request each Processor shut down before stopping.
The rest of this document outlines the Processor Node
SNVA has been tested using the following software stack:
- Ubuntu = 16.04
- Python >= 3.5
- TensorFlow = 2.1 (and its published dependencies)
- TensorBoard = 2.1
- FFmpeg >= 2.8
- websockets
- numpy
- scikit-image
export SNVA_HOME=/path/to/parent/folder/of/snva.py
export FFMPEG_HOME=/path/to/parent/folder/of/ffmpeg/binary
export FFPROBE_HOME=/path/to/parent/folder/of/ffprobe/binary
cd /path/to/parent/folder/of/SNVA/repo
mkdir SNVA
git clone https://github.com/VolpeUSDOT/SNVA.git SNVApython3 snva.py \
-et --modelname desired_model_name \
-cnh control.example.org:8081 \
--tls-ca /secure/processor/ca.pem \
--tls-cert /secure/processor/cert.pem --tls-key /secure/processor/key.pem \
-l /path/to/your/desired/log/directory \
--modelsdirpath /path/to/your/model/directory \
-msh analyzer.example.org:8500 \
-ip /path/to/directory/containing/your/video/files \
--writeinferencereports True sudo docker run \
--runtime=nvidia \
--mount type=bind,\
src=/path/to/your/model/directory,dst=/usr/model \
--mount type=bind,\
src=/path/to/your/desired/output/directory,dst=/usr/output \
--mount type=bind,\
src=/path/to/directory/containing/your/video/files,dst=/usr/videos,readonly \
--mount type=bind,\
src=/path/to/your/desired/log/directory,dst=/usr/logs \
--mount type=bind,src=/secure/processor,dst=/run/secrets/tls,readonly \
snva-processor -et -cnh control.example.org:8081 -msh analyzer.example.org:8500 \
--tls-ca /run/secrets/tls/ca.pem --tls-cert /run/secrets/tls/cert.pem \
--tls-key /run/secrets/tls/key.pem -wir true \
-mn desired_model_name -pm workzoneThe model directory should contain subdirectories for each available model. The specific model to use is specified by the '--modelname' argument, which should match the name of one of these directories.
This directory should also contain two nonstandard files. The first, class_names.txt, should be saved in the model directory itself. It will contain a list mapping numeric class id values to the appropriate string class name, in the format 'id:class_name'. Items should be separated by newlines. The processor will use this to parse model output.
The second file is the input_size.txt file, which should be saved in the subdirectory for each modelname. It will contain a single numeric value between 224 and 299, and indicates the square input dimension of the neural net.
| Flag | Short Flag | Properties | Description |
|---|---|---|---|
| --batchsize | -bs | type=int, default=32 | Number of concurrent neural net inputs |
| --binarizeprobs | -b | action=store_true | Round probs to zero or one. For distributions with two 0.5 values, both will be rounded up to 1.0 |
| --classnamesfilepath | -cnfp | Path to the class ids/names text file | |
| --numprocesses | -np | type=int, default=3 | Number of videos to process at one time |
| --crop | -c | action=store_true | Crop video frames to [offsetheight, offsetwidth, targetheight, targetwidth] |
| --cropheight | -ch | type=int, default=320 | y-component of bottom-right corner of crop |
| --cropwidth | -cw | type=int, default=474 | x-component of bottom-right corner of crop |
| --cropx | -cx | type=int, default=2 | x-component of top-left corner of crop |
| --cropy | -cy | type=int, default=0 | y-component of top-left corner of crop |
| --deinterlace | -d | action=store_true | Apply de-interlacing to video frames during extraction |
| --extracttimestamps | -et | action=store_true | Crop timestamps out of video frames and map them to strings for inclusion in the output CSV |
| --gpumemoryfraction | -gmf | type=float, default=0.9 | % of GPU memory available to this process |
| --inputpath | -ip | required=True | Path to a directory containing the video files to be processed |
| --ionodenamesfilepath | -ifp | Path to the io tensor names text file | |
| --loglevel | -ll | default=info | Defaults to 'info'. Pass 'debug' or 'error' for verbose or minimal logging, respectively |
| --logmode | -lm | default=verbose | If verbose, log to file and console. If silent, log to file only |
| --logpath | -l | default=logs | Path to the directory where log files are stored |
| --logmaxbytes | -lmb | type=int | default=2**23 |
| --modelsdirpath | -mdp | default=models/work_zone_scene_detection | Path to the parent directory of model directories |
| --modelname | -mn | required=True | The subdirectory of modelsdirpath to use |
| --numchannels | -nc | type=int, default=3 | The fourth dimension of image batches |
| --numprocessesperdevice | -nppd | type=int, default=1 | The number of instances of inference to perform on each device |
| --protobuffilename | -pbfn | default=model.pb | Name of the model protobuf file |
| --outputpath | -op | default=reports | Path to the directory where reports are stored |
| --smoothprobs | -sp | action=store_true | Apply class-wise smoothing across video frame class probability distributions |
| --smoothingfactor | -sf | type=int, default=16 | The class-wise probability smoothing factor |
| --timestampheight | -th | type=int, default=16 | The length of the y-dimension of the timestamp overlay |
| --timestampmaxwidth | -tw | type=int, default=160 | The length of the x-dimension of the timestamp overlay |
| --timestampx | -tx | type=int, default=25 | x-component of top-left corner of timestamp (before cropping) |
| --timestampy | -ty | type=int, default=340 | y-component of top-left corner of timestamp (before cropping) |
| --writeeventreports | -wer | type=bool, default=True | Output a CVS file for each video containing one or more feature events |
| --writeinferencereports | -wir | type=bool, default=False | For every video, output a CSV file containing a probability distribution over class labels, a timestamp, and a frame number for each frame |
| --controlnodehost | -cnh | default=localhost:8081 | Control hostname:port or wss://hostname:port; verified TLS required |
| --modelserverhost | -msh | default=localhost:8500 | TensorFlow Serving hostname:port; verified mutual TLS required |
| --tls-ca | required=True | PEM CA bundle trusted for control and inference servers | |
| --tls-cert | required=True | PEM processor client certificate chain | |
| --tls-key | required=True | PEM private key matching the client certificate | |
| --processormode | -pm | default=workzone | Indicates what model pipeline to use: 'workzone', 'signalstate', or 'weather' |
| --writebbox | -bb | action=store_true | Create JSON files with raw bounding box coordinates when run in 'signalstate' mode |
If a timestamp cannot be interpreted, a -1 will be written in its place in the output CSV.
While inference speed has been observed to monotonically increase with batch size, it is important to not exceed the GPU's memory capacity. The SNVA app does not automatically determine the optimal batch size for maximum inference speed. It is best to discover the optimal batch size by testing the app on a small sample of videos (say ~15) starting at a relatively low batch size, then iteratively incrementing the batch size while monitoring GPU memory utilization (e.g. using the NVIDIA X Server Settings GUI app or nvidia-smi CLI app: nvidia-smi --query-compute-apps=process_name,pid,used_gpu_memory --format=csv) and also observing the cumulative analysis duration printed at the end of each run. GPU memory is set to be dynamically allocated, so one should monitor its usage over time to increase the chance of observing peak utilization.
When terminating the app using ctrl-c, there may be a delay while the app terminates gracefully.
When terminating the dockerized app, use ctrl-c to let the app terminate gracefully before invoking the nvidia-docker stop command (which actually shouldn't be needed).
Windows is not officially supported but may be used with minor code tweaks.
Security regression tests run without models or GPUs: python3 -B -m unittest discover -s tests -v (install grpcio and provide OpenSSL for live TLS tests), and npm ci && npm test in ControlNode. Tests generate temporary certificates and exercise mutual TLS, rejected identities, reconnect cleanup, assignment ownership/replay, escaped manifests, and queue-only logging. Legacy Python/TensorFlow/container versions remain a separate maintenance concern; these security fixes do not modernize the inference stack.
When using Docker, some extraneous C++ output is passed to the host machine's console that is not actually logged to file and is not intended to be seen. Consider this a bug and ignore it.
@article{SNVA2018,
title={Detecting Work Zones in SHRP 2 NDS Videos Using Deep Learning Based Computer Vision},
author={Abodo, Rittmuller, Sumner, Berthaume},
journal={arXiv preprint arXiv:1811.04250},
year={2018}
}