This article has been accepted for inclusion in a future issue of this journal.

Content is final as presented, with the exception of pagination.

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 1

ReViewNet: A Fast and Resource Optimized

Network for Enabling Safe Autonomous
Driving in Hazy Weather Conditions
Aryan Mehra, Murari Mandal , Member, IEEE, Pratik Narang, and Vinay Chamola , Senior Member, IEEE

Abstract— Adverse weather conditions such as fog, haze, snow, mist, and haze drastically degrade the visual quality of images.
mist and glare create visibility problems for applications of Such visibility degradation has negative impacts on the per-
autonomous vehicles. To ensure safe and smooth operations formance of other vision-based systems as mentioned above.
in frequent bad weather scenarios, image dehazing is crucial
to any vehicular motion and navigation task on road or air. More importantly, effective dehazing is one of the primary
Moreover, the commonly deployed mobile systems are resource tasks to avoid accidents in driver-less vehicular applications [9]
constrained in nature. Therefore, it is important to ensure on land [10], water or air [11]. The dehazing methods also
memory, compute and run-time efficiency of dehazing algorithms. need to be fast and close to real-time since they have to
In this manuscript we propose ReViewNet, a fast, lightweight and extended to real-time video applications, which necessitates
robust dehazing system suitable for autonomous vehicles. The
network uses components like spatial feature pooling, quadruple the need for resource efficiency. While deep learning has been
color-cue, multi-look architecture and multi-weighted loss to used for image processing in general [12], [13], the onset of
effectively dehaze images captured by cameras of autonomous deep learning in vehicular technologies is ever increasing and
vehicles. The effectiveness of the proposed model is analyzed by efficient as well [14].
exhaustive quantitative evaluation on five benchmark datasets The haze in the atmosphere creates whitening effect which
demonstrating its supremacy over other existing state-of-the-art
methods. Further, a component-wise ablation and loss weight occludes and deforms both the foreground and background.
ratio analysis demonstrates the contribution of each and every Distant haze further reduces the visibility by the accumulated
component of the network. We also show the qualitative analysis veiling effect. In addition to the deterioration in color, contrast
with special use cases and visual responses on distinctive vehicu- and texture features, the degradation in hazy images also
lar vision instances, establishing the effectiveness of the proposed increases non-linearly with change in the distance between the
method in numerous hazy weather conditions for autonomous
vehicular applications. camera lens and the scene, making accurate dehazing a very
challenging task. The effect of fog/haze is significant in street
Index Terms— Vehicular vision, dehazing, adverse weather, scenes resulting in degradation of high-level perception func-
deep learning, resource-efficient, lightweight.
tions of autonomous vehicles and surveillance systems [15].
In most of the dehazing algorithms, the physical scattering
I. I NTRODUCTION model is frequently used to represent image formation. In this
model, the image is formulated based on the properties of
V ISIBILITY for autonomous vehicular systems powered
with perception based sensors for navigation and sur-
veillance is severely hindered by adverse weather condition.
light transmission through the air. The earlier learning-based
dehazing methods in the literature have used the physical
For autonomous vehicles, the low-level visual perception func- scattering model for dehazing. The network usually learns
tions such as object detection [1]–[3], segmentation [4]–[6], one or more of the components of the scattering model.
object tracking [7], [8] require clear image representation However, the accuracy of the estimated atmospheric light
of the street scenes. Similarly, for accurate analysis of the and transmission map greatly influence the quality of the
surveillance videos, good quality image/frames are desired. dehazed image. The disjoint optimization of transmission
As some of the most common bad-weather conditions, fog, map or atmospheric light may hamper the overall dehazing
performance.
Manuscript received March 15, 2020; revised May 21, 2020 and In this paper, we formulate the image dehazing prob-
June 29, 2020; accepted July 20, 2020. This work was supported by the
BITS Additional Competitive Research Grant through the Project titled lem as an end-to-end image-to-image mapping task, free
“Disaster Monitoring from Aerial Imagery using Deep Learning” under from the intermediate computation of transmission map with-
Grant PLN/AD/2018-19/5. The Associate Editor for this article was A. Jolfaei. out relying on the physical scattering model. We propose
(Corresponding author: Vinay Chamola.)
Aryan Mehra and Pratik Narang are with the Department of Computer a fast, lightweight network, ReViewNet, for dehazing in
Science and Information Systems, BITS Pilani, Pilani 333031, India (e-mail: autonomous vehicles. To the best of our knowledge, only few
f20170077@pilani.bits-pilani.ac.in; pratik.narang@pilani.bits-pilani.ac.in). papers [16], [17] have adopted such intermediate-computation
Murari Mandal is with the Department of Computer Science and Engineer-
ing, Malaviya National Institute of Technology (MNIT), Jaipur 302017, India free approach using the Pix-to-Pix and Cycle-GAN archi-
(e-mail: murarimandal.cv@gmail.com). tectures respectively. However, re-purposing image-to-image
Vinay Chamola is with the Department of Electrical and Electronics translation GANs for dehazing can be very difficult to optimize
Engineering, BITS Pilani, Pilani 333031, India (e-mail: vinay.chamola@
pilani.bits-pilani.ac.in). and not necessarily produce the optimal results, as can be
Digital Object Identifier 10.1109/TITS.2020.3013099 verified in the experimental results comparison in Section IV.
1524-9050 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 09:09:44 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

2 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Moreover, GANs tend to be heavier in computation at train and Liu et al. [24] learned haze-relevant priors with an iteration
test time, having indirectly encompassed two networks within algorithm using deep CNNs. The estimation of transmission
them. The proposed ReViewNet generates more realistic haze- maps or atmospheric light (or both) for image dehazing has
free images in terms of color and details in comparison been performed by several CNN-based architectures proposed
to existing state-of-the-art approaches, with lesser trainable in literature [25]–[29].
parameters and faster runtime. The main contributions of this The success of GANs in image-to-image translation tasks
work can be summarized as follows: has also attracted its use in image dehazing. Zhu et al. [30]
1) We propose a fast and lightweight end-to-end model proposed DehazeGAN which utilizes differential program-
ReViewNet for image dehazing. The model is free ming to re-formulate the atmospheric scattering model. More
from any intermediate component computation for the recently, CD-Net [31] and RI-GAN by Dudhane et al. [32]
physical scattering model and, thus, learns the most re-purposed the Cycle-GAN architecture to learn the trans-
optimal mapping between the hazy and haze-free image mission map. Some researchers [16], [17], [33] have pro-
while being easily extendable to real time applications posed GAN based architectures which argue in favor of
that process multiple frames per second. direct image-to-image mapping over intermediate transmission
2) The ReViewNet is designed to have multiple looks at map estimation. The estimation of intermediate transmission
different stages in the network for dehazing. We use map or atmosphere light through CNN or GAN based methods
different loss weights ratio for the first and second look. increases the training cost and result generation time. Further-
We introduce a new bottleneck parallel spatial cleaning more, such architecture often fail on dense haze conditions.
module. Moreover, we feed the multi-cue color space
(RGB, HSV, YCrCb and LAB) to the network for robust III. P ROPOSED M ETHOD
haze removal. ReViewNet is unique in design and functioning, and has
3) ReViewNet is a fast and highly resource-efficient (5 MB four pivotal contributions which make it the lightest learning-
model size, 399,670 trainable parameters) network and based dehazing solution – the use of quadruple color space,
can perform image dehazing in a highly resource- the double look architecture, the different loss weights ratio for
constrained environment with high speed (CPU speed the first and second look, and the use of bottleneck parallel
– 0.28 seconds per frame, GPU speed – 0.025 seconds spatial cleaning. The use of multiple color spaces provides
per frame) for real-time applications. the network a wholesome input feature vector, which enables
4) ReViewNet significantly outperforms the existing state- a faster convergence. Fig. 1 shows the network architecture in
of-the-art methods in terms of PSNR and SSIM in Haz- greater detail.
eRD, D-Hazy and the more recent RESIDE-Standard The following sections explain each component of the net-
(SOTS), RESIDE-β (SOTS) and RESIDE-β (HSTS) work and their contribution towards the performance achieved.
datasets.
A. Quadruple-Color Space
II. R ELATED W ORK
Most of the current work in the field of dehazing uses a
A. Prior Based Approaches single color space, mostly restricted to RGB or HSV. While
The Dark Channel Prior (DCP) [18] approach by He et al. RGB is the most common, YCrCb is also an absolute color
estimates the transmission map and soft matts the response, space lying in the family of 3 dimensional vector color spaces.
while Berman et al. [19] employed Non local priors (NLD) to HSV aligns more closely with how the humans perceive
model the hazy image with lines in the RGB space. Further, the colors around them. The change in the amount of color
Ancuti et al. [20] used a multi-fusion algorithm to propose perceived in Lab color space is same as the numerical change
a night-time dehazing approach. Authors such as [19], [21] in the values, which is the inspiration behind it’s inception. The
have also estimated the atmospheric light to remove the haze. proposed method leverages information from above mentioned
These approaches are limited by their dependence on the 4 color spaces, namely RGB, HSV, YCrCb and Lab, thus
priors, and this causes the dehazing approach to fail in case mapping the 12 channeled input to an RGB output. The inter-
of complicated image structures. While the results produced conversion between these color spaces is simply mathematical,
by such approaches often have unrealistic color distortion and requiring no additional storage for learnt parameters while pre-
contrast, their performance is also limited by the accuracy of processing. Let an image matrix of height m and width n,
the assumptions they make. consisting of 3 channels be represented as I3 . Then the
quadruple-color space matrix I12 consisting of 12 channels
B. Learning Based Approaches be represented as in Eq. 1.
Learning based approaches for dehazing are generally I12 = [Rm,n,1 , G m,n,1 , Bm,n,1 , Hm,n,1 ,
trained to directly learn the atmospheric light, transmission Sm,n,1 , Vm,n,1 , Ym,n,1 , Cbm,n,1 ,
map, or both. Li et al. [22] proposed AOD-Net which learns
Crm,n,1 , L m,n,1 , am,n,1 , bm,n,1 ] (1)
a CNN-based mapping function for the reformulated physical
scattering model, while Cai et al. [23] proposed DehazeNet where R, G, B, H, S, V, Y, Cb, Cr, L, a, b, denote the red,
which estimates the intermediate transmission map which green, blue, hue, saturation, value, luma, blue-difference, red-
is used to generate the haze-free image. The approach of difference, lightness, chromaticity coordinates between red to

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 09:09:44 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

MEHRA et al.: ReViewNeT: A FAST AND RESOURCE OPTIMIZED NETWORK FOR ENABLING SAFE AUTONOMOUS DRIVING 3

Fig. 1. ReViewNet Network Architecture: Inter and intra-carry connections with 4 path spatial pooling bottleneck.

green axis, chromaticity coordinates between yellow to blue to understand that the the multi-look architecture used by
axis, respectively. The RGB space is the most universal image ReViewNet does not add on to the number of parameters
depiction color space and has special importance because the because the second look or pass is not added after a static
output of the network is also RGB. Research has conclusively first look. Instead, keeping the number of parameters and
shown that the HSV color space is beneficial for dehazing [34], size of the architecture fixed, the entire network is split into
[35]. The YCrCb color space contributes to better luminance two looks – making the entire structure more efficient for
and color contrast to the dehazed output, as analyzed by image enhancement and light enough to be incorporated in
Tufail et al. [36] and Bianco et al. [37]. The YCrCb channel autonomous vehicles.
contributes to a lesser mean squared error as compared to The proposed ReViewNet consist of two set of encoder-
the RGB channel, making it a suitable choice specifically decoder modules. We denote them as the first look (RV Net f )
for image dehazing and enhancement tasks. It is intuitive and second look (RV Nets ) modules, respectively. For an input
that the L channel of the Lab color space is pivotal to the tensor Ix , the RV Net f is computed using Eq.2.
dehazing task because haze primarily affects the lightness of
RV Net f = DeM f (En M f (Ix )) (2)
the pixels, thus affecting clarity. Wang et al. [38] and [39]
use the CIELAB color space modelling and remove the where the first look encoder (En M f ) and decoder (DeM f )
haze further by processing the image using simple linear modules are computed as shown in Fig. 1. For the tensor Ix
iterative clustering to generate super-pixels of the image. All of size P × P with the number of channels denoted as ch,
these advantages are clear from the performance boost that the response of the convolutional layer (conv) with a kernel
we witness by incorporating quadruple-color-space into the function f (·) and size h × h is computed by Eq. 3.
model, as explained in the ablation section and Table XI. 
ch
conv(Ix ) = f k (h) × I nj |dk=1 (3)
j =1
B. Multi-Look Architecture
where n ∈ [1, P] and d is the filter depth. Eq. 4 shows the
Most prior work on image dehazing incorporate a single
constituent operations in the En M f (Ix ) module.
encoder-decoder based networks or GANs that are heavy in
terms of the number of parameters and require deepening of En M f (Ix ) = C f 3 (C f2 (C f 1 (Ix ))) (4)
the networks for an increase in accuracy. Moreover, a single
Each C f i for i ∈ [1, 2, 3] is composed of two convolution
look does not give an opportunity to create customised losses
(conv) operations and one maxpool (mp) operation.
and quantitatively perceive the improvement taking place in
the image as it passes through the network. It is imperative C f i = mp(conv(conv(F))) (5)

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 09:09:44 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

4 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

where F denotes the input feature maps. Similarly, TABLE I

the DeM f (·) module is composed of the following set of D ESCRIPTION OF THE T RAINING AND T ESTING D ATASETS
(N UMBER OF I MAGES C ONTRIBUTED IN E ACH C ATEGORY )
operations.
DeM f (En M f ) = up(conv(D f 3 (D f2 (D f1 (En M f ))))) (6)
The D fi for i ∈ [1, 2, 3] is composed of a transpose
convolution (conv T ) and an upsample (up) operation.
D fi = up(conv T (F)) (7)
The convolutional layers in the encoder and decoder mod-
ules have internal skip connections to accelerate the learning
process and enable the low-level features to propagate without otherwise build up in deeper layers. The SPB is computed as
progressively degrading through the network and even circum- given in Eq. 13 and Eq. 14.
vent the bottleneck. It also helps in improving the colour and
edge feature extraction process in the network. S P B = [Q, Q1, Q2, Q3, Q4] (13)
While a primary glance is essential for learning the dehazing Q = [En Ms , Cs3 ]
task, the second pass acts as a revision with certain insights of Q1 = up(conv(mp(Q)))
its own. The second pass also has over six carry connections Q2 = up(conv(ap4(Q)))
from the first look. They act as an inter-network feed for-
ward system, ensuring that the second look definitely learns Q3 = up(conv(ap8(Q)))
something more than the first. The features thus circumvent Q4 = up(conv(gp(Q))) (14)
the bottleneck and take the context straight to the output [40].
where ap, gp, mp denote the average, global and max pooling
The second look module RV Nets is computed using Eq. 8.
respectively. The ap4 and ap8 downsample the input feature
RV Nets = DeMs (S P B(En Ms ([Ix , RV Net f ]))) (8) maps by the factor of 4 and 8, respectively. The Q1, Q2, Q3,
Q4 are encoded in parallel and concatenated with Q before
The second look encoder (En Ms ) is composed of convo- passing on to the next layer.
lutional blocks along with the carry responses from the first The spatial pooling aspect is added in the second and not
look encoder (En M f ) as shown in Eq. 9. the first look because the second look builds upon the output
En Ms = Cs3 (Cs2 (Cs1 ([Ix , RV Net f ]))) (9) of the first, and as demonstrated in the ablation section and
Table XI, giving spatial advantage to the second look yields
where the Csi is computed as better results.

mp(conv(conv([F, C f i−1 ]))), if i ∈ [2, 3]
Csi = (10) C. Different Loss Weights
mp(conv(conv([Ix , RV Net f ]))), if i = 1
The network is capable of producing two outputs, referred
The encoded features are cleansed with a spatial pooling as the auxiliary and the main output, produced at the end of the
block (SPB). More details about the SPB is discussed in first and second look respectively. The network is thus trained
the next subsection. The response of SPB block is decoded like a multi-output image-to-image mapping architecture and
through DeMs module as given in Eq. 11. the two MSE losses are combined by multiplying them with
suitable scaling factors. These scaling factors or loss weights
DeM f = conv t (conv(conv([Ds3 (Ds2 (Ds1 (S P B))), D f3 ]))
can be treated as the importance given to the two looks. Thus
(11) if L 1 is the MSE loss for the auxiliary output and L 2 is the
The Dsi is computed as in Eq. 12. The carry branches from MSE loss for the main output, the net loss or principle target
the first look decoder and second look encoder are also fed to function for the network is given as
these decoder blocks. Lossnet = W1 L 1 + W2 L 2 (15)

up(conv T ([F, D fi−1 , Cs4−i ])), if i ∈ [2, 3]
Dsi = (12) where W1 is the scaling factor of the first look and W2 is
up(conv T (S P B)), if i = 1 the scaling factor for the second look. In order to analyse the
principle behind introducing a second look and understand
Spatial Pooling Block: The second look has an additional
the importance of the two looks, the scaling factors are
parallel pooling layer that leverages the spatial dimensions
varied across five different combinations, as discussed further
of the feature vector after the encoding process as shown
in Section IV-D.
in Fig. 1. The feature vector is pooled via global pooling
layers to extract coarse features, average pooling to extract
medium level features and maxpooling to extract fine features. IV. E XPERIMENTS AND A NALYSIS
This bottleneck also uses different spatial strides in these The model is trained and tested on a total of 5 different
pooling layers to facilitate a similar intuition. Such spatial datasets. The details of the train-test split are highlighted
pooling helps shallow networks to extract features that would in Table I along with descriptions in the datasets section

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 09:09:44 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

MEHRA et al.: ReViewNeT: A FAST AND RESOURCE OPTIMIZED NETWORK FOR ENABLING SAFE AUTONOMOUS DRIVING 5

TABLE II TABLE III

C OMPARATIVE R ESULTS OVER R ESIDE -S TANDARD C OMPARATIVE R ESULTS OVER R ESIDE -β
S OTS [41] I NDOOR D ATASET S OTS [41] O UTDOOR D ATASET

TABLE IV
C OMPARATIVE R ESULTS OVER H AZE RD [52] D ATASET

below. For quantitative evaluation, structural similarity index

(SSIM) and peak signal to noise ratio (PSNR) metrics are
used, which are the most frequently adopted metrics for image
dehazing algorithms. These metrics enable comparison of TABLE V
the proposed method to a wide range of existing solutions. C OMPARATIVE R ESULTS OVER HSTS D ATASET [41]
Training takes place separately for indoor and outdoor models,
primarily because autonomous vehicles for the outdoors differ
significantly from indoor applications of the same.

A. Datasets
We use 5 datasets in this work – RESIDE-standard indoor
(13,990 image pairs), RESIDE-β outdoor (72,135 image
pairs), RESIDE HSTS (10 image pairs) [41], HazeRD
(75 image pairs) [52] and D-Hazy (1,499 image pairs) [53].
We choose RESIDE dataset for the training because, apart
from being one of the largest publicly available datasets
for dehazing, it benchmarks nine representative state of the
art dehazing methods by providing full reference evaluation TABLE VI
metrics like PSNR and SSIM for the synthetic objective testing C OMPARATIVE R ESULTS OVER D-HAZY [53] D ATASET
set (SOTS). For robustness, the indoor models are also trained
on the D-Hazy [53] dataset. Further, the train-test split is
highlighted in the Table I.

B. Quantitative Analysis
We report the average PSNR and SSIM of all stated net-
works and the proposed method. Since the proposed method
outperforms all the existing state-of-the-art on all mentioned
datasets, we also report the percentage increase it brings on
every existing method. Table II to Table VI clearly demonstrate
the supremacy of the proposed method on PSNR and SSIM
as compared to other benchmarks that exist on these datasets.
As an example, Table III demonstrates how ReViewNet outper- C. Qualitative Analysis for Safe Autonomous Driving
forms the existing methods with improvements ranging from We present a detailed qualitative analysis which visu-
8.08 to 33.56 percent in PSNR on MADN and DCP methods ally depicts the efficacy of ReViewNet for enabling safe
respectively. There is a 2.70 percent improvement over Deep autonomous driving in hazy weather conditions. A visual com-
DCP method on the HSTS dataset in Table V on SSIM and a parison with existing approaches also establishes the superior
remarkable 29.80 percent over BCCR method. performance of ReViewNet. We delineate several use-cases

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 09:09:44 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

6 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Fig. 2. Qualitative comparison: Multiple objects in a scene. Magnify for minor details.

Fig. 3. Qualitative comparison: Roadside signs and distant T-Point dead-end enhancement. Magnify for minor details.

related to the autonomous vehicular context in hazy weather 2) Road Sign and Structural Navigation Integrity: Road
conditions. ReViewNet is able to perform effective dehazing in signs and intersections like T-points are extremely important in
a large variety of driving scenarios and consistently produces the context of autonomous vision based driving applications.
haze-free images with clear color and contrast details. These Fig. 3 shows how the parking sign is most clearly visible in
results further strengthen the application of dehazed output of the dehazed image produced by ReViewNet, the output being
ReViewNet for Vision-based vehicular applications like object closest to the available ground truth as well. Similarly the
detection or semantic segmentation. distant T-point is most clearly dehazed by the proposed method
1) Multiple Objects and Contextual Clarity: Real life sit- as compared to other methods demonstrated in Fig. 3. For net-
uations will require the autonomous vehicles to occasion- works with extremely low runtime for real time applications,
ally enter busy streets full of two-wheelers, pedestrians and it is rare to witness such structural integrity and nuance.
dynamic road surfaces as depicted in Fig. 2. The highlighted 3) Short Distance Obstacles and Urban Dynamic Traffic
ares of the image depict how ReViewNet elegantly dehazes the Situations: It is important for the image to maintain a color
image with clarity in context of the pedestrian, vehicles and contrast similar to the actual truth and have clearly visible road
distant objects. While the other deep learning based solutions surfaces for navigation. ReViewNet effectively uses its spatial
like AODNet and DCPDN fail to provide complete dehazing, feature extraction and skip connections to obtain the dehazed
mathematical models like DCP, NLD and FVR introduce an output as close to the ground truth as possible, as depicted
unusual color contrast, hindering object detection or road in Fig. 4. Similarly Fig. 5 shows the same effect on a real
segmentation tasks performed by an autonomous vehicle. dynamic traffic image taken from a traffic camera.

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 09:09:44 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

MEHRA et al.: ReViewNeT: A FAST AND RESOURCE OPTIMIZED NETWORK FOR ENABLING SAFE AUTONOMOUS DRIVING 7

Fig. 4. Qualitative comparison: Short distance and color contrast quality in dehazing outputs. Magnify for minor details.

Fig. 5. Qualitative comparison: Urban traffic and dynamic scenarios. Magnify for minor details.

Fig. 6. Qualitative comparison: Comparison of dehazing in atmospheric dispersion. Magnify for minor details.

TABLE VII
L OSS W EIGHT A NALYSIS FOR D IFFERENT W EIGHT R ATIOS FOR THE F IRST AND S ECOND L OOK

4) Atmospheric Light and Dispersion: Fig. 6 depicts how instead form a circumvented image about the source of light
non-learning based methods like DCP, BCCR, NLD and in haze. While the learning based methods like AODNet
FVR cannot sometimes deal with atmospheric dispersion and and DCPDN do overcome that issue, they fail to dehaze the

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 09:09:44 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

8 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

Fig. 7. Vehicular vision specific qualitative use cases. Magnify for minor details.

image completely. ReViewNet, on the other hand, distinctively driving road navigation, road navigation and pedestrian prone
dehazes the aerial view with minimal distortion in structure crossings, railroad and waterway use cases and aerial (drone-
and color contrast. based) building and rooftop applications.
5) Specific Use Cases for Autonomous Transportation The proposed system is a useful preprocessor for several
Safety and Surveillance: In this section, we discuss several vision based traffic analysis tasks such as object detection,
use cases for the utility of the proposed ReViewNet. Fig. 7 lane detection, and segmentation. The first five rows in Fig. 7
highlights over 32 examples of hazy and our dehazed outputs, highlight the effectiveness of dehazing in the prominence of
divided into 5 categories – traffic camera view, autonomous zebra crossings and turnings of the road that are essential

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 09:09:44 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

MEHRA et al.: ReViewNeT: A FAST AND RESOURCE OPTIMIZED NETWORK FOR ENABLING SAFE AUTONOMOUS DRIVING 9

Fig. 8. Graphs depicting loss weight ratio to decide W 1 and W 2. (a) PSNR values for all the datasets at different W 1/ W 2 ratios, (b) SSIM values for all
the datasets at different W 1/ W 2 ratios.

for automated decisions required in autonomous vehicles, TABLE VIII

such road lane segmentation and lane changes. The photos C OMPARISON B ETWEEN E QUALLY W EIGHTED F IRST
(AUXILIARY ) AND S ECOND (M AIN ) O UTPUTS
of pedestrian-prone areas clearly show how, in some cases,
object which were not clearly visible in the hazy image are
made visible in the dehazed output, which in turn improves
object detection. The dehazed objects do not merge with the
haze anymore, and they become more prominent and natural,
despite the lightweight and fast architecture of the model.
These objects range from parking lot signs to number plates.
The sixth row in Fig. 7 extend to applications beyond roads
to railroad and water bodies. Sample images in the last two look as depicted by the Table VII. The table depicts the final
rows in Fig. 7 demonstrate that aerial imagery, that is sim- output (second look) of the five trials. The Fig. 8 shows the
ulated from the perspective of delivery drones, is effectively graphical plots of these values for SSIM and PSNR. We can
dehazed despite the adverse solar glare present in the images. clearly see that though the second look is more important
Detecting rooftops better can act as a big advantage to aerial than the first, the sharp distinctive peak at a weight ratio
vehicular technology for adverse weather conditions. During of 0.4 for the first versus 0.6 for the second look shows
our experimentation, the only limitation that was observed an optimal configuration. After this, the graphs takes a dip
was on images with artificial dense haze combined with again. The values at 0.3/0.7 weight are still better than the
mushy or fuzzy green backgrounds. ReViewNet could not fully 0.6/0.4 and 0.7/0.3, demonstrating that the second look is
dehaze them. As a future prospect, it is planned to overcome learning much more than the first. Thus, context specific
this difficulty. novelty of the experimentation is observed. We successfully
establish the technique of dividing the network into two parts,
D. Loss Weight Ratio Analysis and Explainable Revision boosting performance without increase in parameters with
Improvement unique dehazing-specific yet computationally simple compo-
nents for image enhancement.
We now answer the mathematical basis for varying the
loss ratio of the first and second look. As stated before,
the main impact of the multi-look architecture is seen in the E. Running Time Analysis
reduction of running time and lightweight architecture of the For all vehicular vision applications, dehazing will pre-
model. We now experimentally demonstrate that given the dominantly be used as a pre-processing step and hence it
same weight (50-50 ratio) for both the looks, the second look necessitates that it be fast as well as computationally efficient.
does in fact learn more than the first through Table VIII. We present a comparison of the state of the art learning based
It depicts how the main output is consistently better than the and mathematical methods in terms of their running time on
first on all four datasets in the table on both PSNR and SSIM, CPU and GPU environments. We see from Table IX and X
thus reiterating the fact and assumption that the second look that the proposed ReViewNet is the fastest strategy among
improves upon the first like a “revision” or “review”. all methods. AODNet [22] is famous for being one of the
To decide the weights of the first and second look, we create fastest networks, and ReViewNet is almost three times faster
five trials that have ratio of losses for the first and second in computation, without any compromise in PSNR or SSIM.

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 09:09:44 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

10 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS

TABLE IX
AVERAGE PER I MAGE CPU RUNNING T IME ( IN SECONDS ) C OMPARISON OF VARIOUS M ETHODS OVER RESIDE SOTS I MAGES

TABLE X
AVERAGE PER I MAGE GPU RUNNING T IME ( IN SECONDS ) C OMPARISON OF VARIOUS M ETHODS

TABLE XI V. C ONCLUSION
A BLATION A NALYSIS . T 1 I S W ITHOUT THE M ULTI -C OLOR S PACE , T 2 H AS
S PATIAL P OOLING IN THE F IRST L OOK AND T 3 I S A S INGLE L OOK
This work presents a fast and resource-efficient network,
A RCHITECTURE . T HE R EST OF THE PARAMETERS A RE C ONSTANT ReViewNet, for image dehazing which is suitable for real
time applications in autonomous driving. The ReViewNet
architecture looks twice over the hazy image and the network
is optimized with a hybrid weighted loss. Ablation analysis of
components and experimentation on ratios of the two different
losses are conducted to determine the optimal weight ratio.
We demonstrate sample qualitative results over 32 different
scenarios for specific use-cases of vehicular vision. This work
The running time shown here includes the multi color-cue adds to the state-of-the-art by demonstrating conclusive proof
spacial modelling and inter-conversion as well. This will give that using context specific components and features of a deep
a tremendous boost to the top level vision tasks on aerial learning network can result in faster (0.025 seconds per frame
drones and autonomous cars and boats, making the model most on a GPU) and more accurate image enhancement modules.
suitable for typical autonomous vehicle video frame rates up- This further leads better preprocessing and higher perfor-
to the range of 40 frames per second. mance in Vision-based tasks for vehicular technologies such
as object detection and road segmentation, thus enabling safer
F. Ablation and Novelty Analysis autonomous driving. An exhaustive experimental analysis on
five benchmark haze datasets demonstrates that the proposed
The ablation analysis in Table XI shows the rationale
ReViewNet network significantly outperforms all the existing
behind several design choices by comparing performance on
state-of-the-art CNN and GAN based methods in quantitative,
two of the largest datasets in the experiment. Comparison
qualitative, and computation speed evaluations.
of ReViewNet with t1 shows the quantitative contribution
of multi-color space, while comparison with t2 shows the R EFERENCES
increase in accuracy when spatial pooling is used in the second
[1] L. Zhou, W. Min, D. Lin, Q. Han, and R. Liu, “Detecting motion
look rather than the first. An explanation for this is also blurred vehicle logo in IoV using filter-DeblurGAN and VL-YOLO,”
hinted in the fact that since the second look uses the output IEEE Trans. Veh. Technol., vol. 69, no. 4, pp. 3604–3614, Apr. 2020.
of the first along with the original image as well, giving it [2] M. Mandal, M. Shah, P. Meena, and S. K. Vipparthi, “SSSDET:
Simple short and shallow network for resource efficient vehicle detection
the spatial component gives the network an upper hand. The in aerial scenes,” in Proc. IEEE Int. Conf. Image Process. (ICIP),
performance increase by dividing the network into two parts Sep. 2019, pp. 3098–3102.
is seen by comparison with t3, showing how ReViewNet is [3] M. Mandal, M. Shah, P. Meena, S. Devi, and S. K. Vipparthi, “AVDNet:
A small-sized vehicle detection network for aerial visual data,” IEEE
better than a simple encoder decoder with spatial pooling. Geosci. Remote Sens. Lett., vol. 17, no. 3, pp. 494–498, Mar. 2020.
Hence, the novelty of the network is observed in its use [4] M. Mandal, M. Chaudhary, S. K. Vipparthi, S. Murala, A. B. Gonde, and
of multi-color spaces, multi-look architecture, loss weighting, S. K. Nagar, “ANTIC: Antithetic isomeric cluster patterns for medical
image retrieval and change detection,” IET Comput. Vis., vol. 13, no. 1,
spatial pooling, skip connections and computation efficient pp. 31–43, Feb. 2019.
dehazing without dependency on physical phenomenon. It is [5] M. Mandal, P. Saxena, S. K. Vipparthi, and S. Murala, “CANDID:
the harmonious synchronisation of all these aspects that makes Robust change dynamics and deterministic update policy for dynamic
background subtraction,” in Proc. 24th Int. Conf. Pattern Recognit.
the network unique and, more importantly, context-specific for (ICPR), Aug. 2018, pp. 2468–2473.
dehazing in real-time autonomous driving applications. This [6] M. Mandal, V. Dhar, A. Mishra, and S. K. Vipparthi, “3DFR: A swift 3D
harmony is also demonstrated in the fact that the network feature reductionist framework for scene independent change detection,”
IEEE Signal Process. Lett., vol. 26, no. 12, pp. 1882–1886, Dec. 2019.
outperforms all state of the art in the field, in both PSNR [7] P.-H. Chiu, P.-H. Tseng, and K.-T. Feng, “Interactive mobile augmented
ans SSIM metrics. Moreover, we experimentally establish a reality system for image and hand motion tracking,” IEEE Trans. Veh.
precedent that image enhancement tasks will see an increase Technol., vol. 67, no. 10, pp. 9995–10009, Oct. 2018.
[8] M. Mandal, L. K. Kumar, M. S. Saran, and S. K. Vipparthi, “MotionRec:
in performance by dividing the network into two parts, keeping A unified deep framework for moving object recognition,” in Proc. IEEE
the training parameters same. Winter Conf. Appl. Comput. Vis. (WACV), Mar. 2020, pp. 2734–2743.

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 09:09:44 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

MEHRA et al.: ReViewNeT: A FAST AND RESOURCE OPTIMIZED NETWORK FOR ENABLING SAFE AUTONOMOUS DRIVING 11

[9] A. Jolfaei and K. Kant, “Privacy and security of connected vehicles [31] A. Dudhane and S. Murala, “CDNet: Single image de-hazing using
in intelligent transportation system,” in Proc. 49th Annu. IEEE/IFIP Int. unpaired adversarial training,” in Proc. IEEE Winter Conf. Appl. Comput.
Conf. Dependable Syst. Netw.-Supplemental Volume (DSN-S), Jun. 2019, Vis. (WACV), Jan. 2019, pp. 1147–1155.
pp. 9–10. [32] A. Dudhane, H. S. Aulakh, and S. Murala, “RI-GAN: An end-to-
[10] M. Shafiq, Z. Tian, A. K. Bashir, A. Jolfaei, and X. Yu, “Data mining and end network for single image haze removal,” in Proc. IEEE/CVF
machine learning methods for sustainable smart cities traffic classifica- Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2019,
tion: A survey,” Sustain. Cities Soc., vol. 60, Sep. 2020, Art. no. 102177. pp. 1–10.
[11] S. Kouroshnezhad, A. Peiravi, M. S. Haghighi, and A. Jolfaei, [33] A. Mehta, H. Sinha, P. Narang, and M. Mandal, “HIDeGan:
“An energy-aware drone trajectory planning scheme for terrestrial A hyperspectral-guided image dehazing GAN,” in Proc. IEEE/CVF
sensors localization,” Comput. Commun., vol. 154, pp. 542–550, Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2020,
Mar. 2020. pp. 212–213.
[12] G. Bansal, V. Chamola, P. Narang, S. Kumar, and S. Raman, [34] Y. Wan and Q. Chen, “Joint image dehazing and contrast enhancement
“Deep3DSCan: Deep residual network and morphological descrip- using the HSV color space,” in Proc. Vis. Commun. Image Process.
tor based framework for lung cancer classification and 3D seg- (VCIP), Dec. 2015, pp. 1–4.
mentation,” IET Image Process., vol. 14, no. 7, pp. 1240–1247, [35] T. Zhang, H.-M. Hu, and B. Li, “A naturalness preserved fast
May 2020. dehazing algorithm using HSV color space,” IEEE Access, vol. 6,
[13] A. Mehra, N. Jain, and H. S. Srivastava, “A novel approach to pp. 10644–10649, 2018.
use semantic segmentation based deep learning networks to clas- [36] Z. Tufail, K. Khurshid, A. Salman, I. F. Nizami, K. Khurshid, and
sify multi-temporal SAR data,” Geocarto Int., pp. 1–16, 2020, doi: B. Jeon, “Improved dark channel prior for image defogging using
10.1080/10106049.2019.1704072. RGB and YCbCr color space,” IEEE Access, vol. 6, pp. 32576–32587,
[14] V. Hassija, V. Gupta, S. Garg, and V. Chamola, “Traffic jam prob- 2018.
ability estimation based on blockchain and deep neural networks,” [37] S. Bianco, L. Celona, F. Piccoli, and R. Schettini, “High-resolution
IEEE Trans. Intell. Transp. Syst., early access, Jun. 3, 2020, doi: single image dehazing using encoder-decoder architecture,” in Proc.
10.1109/TITS.2020.2988040. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW),
[15] S. Huang, B. Chen, and Y. Cheng, “An efficient visibility enhancement Jun. 2019.
algorithm for road scenes captured by intelligent transportation systems,” [38] P. Wang, Q. Fan, Y. Zhang, F. Bao, and C. Zhang, “A novel dehaz-
IEEE Trans. Intell. Transp. Syst., vol. 15, no. 5, pp. 2321–2332, ing method for color fidelity and contrast enhancement on mobile
Oct. 2014, doi: 10.1109/TITS.2014.2314696. devices,” IEEE Trans. Consum. Electron., vol. 65, no. 1, pp. 47–56,
[16] Y. Qu, Y. Chen, J. Huang, and Y. Xie, “Enhanced Pix2pix dehazing Feb. 2019.
network,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. [39] M. Yang, J. Liu, and Z. Li, “Superpixel-based single nighttime image
(CVPR), Jun. 2019, pp. 8160–8168. haze removal,” IEEE Trans. Multimedia, vol. 20, no. 11, pp. 3008–3018,
[17] D. Engin, A. Genc, and H. K. Ekenel, “Cycle-dehaze: Enhanced Nov. 2018.
CycleGAN for single image dehazing,” in Proc. IEEE/CVF Conf. [40] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image trans-
Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2018, lation with conditional adversarial networks,” 2016, arXiv:1611.07004.
pp. 825–833. [Online]. Available: https://arxiv.org/abs/1611.07004
[18] K. He, J. Sun, and X. Tang, “Single image haze removal using dark [41] B. Li et al., “Benchmarking single-image dehazing and beyond,” IEEE
channel prior,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 12, Trans. Image Process., vol. 28, no. 1, pp. 492–505, Jan. 2019.
pp. 2341–2353, Dec. 2011. [42] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image
translation using cycle-consistent adversarial networks,” in Proc. IEEE
[19] D. Berman, T. Treibitz, and S. Avidan, “Non-local image dehazing,” in
Int. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 2223–2232.
Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016,
[43] J.-P. Tarel and N. Hautiere, “Fast visibility restoration from a single
pp. 1674–1682.
color or gray level image,” in Proc. IEEE 12th Int. Conf. Comput. Vis.,
[20] C. Ancuti, C. O. Ancuti, C. De Vleeschouwer, and A. C. Bovik, “Night-
Sep. 2009, pp. 2201–2208.
time dehazing by fusion,” in Proc. IEEE Int. Conf. Image Process.
[44] G. Meng, Y. Wang, J. Duan, S. Xiang, and C. Pan, “Efficient image
(ICIP), Sep. 2016, pp. 2256–2260.
dehazing with boundary constraint and contextual regularization,” in
[21] M. Sulami, I. Glatzer, R. Fattal, and M. Werman, “Automatic recovery of Proc. IEEE Int. Conf. Comput. Vis., Dec. 2013, pp. 617–624.
the atmospheric light in hazy images,” in Proc. IEEE Int. Conf. Comput. [45] A. Dudhane and S. Murala, “C2 MSNet: A novel approach for single
Photogr. (ICCP), May 2014, pp. 1–11. image haze removal,” in Proc. IEEE Winter Conf. Appl. Comput. Vis.
[22] B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng, “AOD-Net: All-in- (WACV), Mar. 2018, pp. 1397–1404.
one dehazing network,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), [46] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation
Oct. 2017, pp. 4770–4778. with conditional adversarial networks,” in Proc. IEEE Conf. Comput. Vis.
[23] B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao, “DehazeNet: An end-to-end Pattern Recognit. (CVPR), Jul. 2017, pp. 1125–1134.
system for single image haze removal,” IEEE Trans. Image Process., [47] X. Yang, Z. Xu, and J. Luo, “Towards perceptual image dehazing by
vol. 25, no. 11, pp. 5187–5198, Nov. 2016. physics-based disentanglement and adversarial training,” in Proc. 32nd
[24] Y. Liu, J. Pan, J. Ren, and Z. Su, “Learning deep priors for image dehaz- AAAI Conf. Artif. Intell., 2018, pp. 1–8.
ing,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2019, [48] Q. Zhu, J. Mai, and L. Shao, “A fast single image haze removal algorithm
pp. 2492–2500. using color attenuation prior,” IEEE Trans. Image Process., vol. 24,
[25] Y.-T. Peng, Z. Lu, F.-C. Cheng, Y. Zheng, and S.-C. Huang, “Image no. 11, pp. 3522–3533, Nov. 2015.
haze removal using airlight white correction, local light filter, and aerial [49] C. Chen, M. N. Do, and J. Wang, “Robust image and video dehazing
perspective prior,” IEEE Trans. Circuits Syst. Video Technol., vol. 30, with visual artifact suppression via gradient residual minimization,”
no. 5, pp. 1385–1395, May 2020. in Proc. Eur. Conf. Comput. Vis. Cham, Switzerland: Springer, 2016,
[26] S. Chen, Y. Chen, Y. Qu, J. Huang, and M. Hong, “Multi-scale adaptive pp. 576–591.
dehazing network,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern [50] A. Dudhane and S. Murala, “RYF-Net: Deep fusion network for single
Recognit. Workshops (CVPRW), Jun. 2019, pp. 1–9. image haze removal,” IEEE Trans. Image Process., vol. 29, pp. 628–640,
[27] W. Ren, S. Liu, H. Zhang, J. Pan, X. Cao, and M.-H. Yang, “Sin- 2020.
gle image dehazing via multi-scale convolutional neural networks,” [51] H. Zhang and V. M. Patel, “Densely connected pyramid dehazing
in Proc. Eur. Conf. Comput. Vis. Cham, Switzerland: Springer, 2016, network,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.,
pp. 154–169. Jun. 2018, pp. 3194–3203.
[28] W. Ren et al., “Gated fusion network for single image dehazing,” [52] Y. Zhang, L. Ding, and G. Sharma, “HazeRD: An outdoor scene dataset
in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, and benchmark for single image dehazing,” in Proc. IEEE Int. Conf.
pp. 3253–3261. Image Process. (ICIP), Sep. 2017, pp. 3205–3209.
[29] C. O. Ancuti et al., “NTIRE 2020 challenge on NonHomogeneous [53] C. Ancuti, C. O. Ancuti, and C. De Vleeschouwer, “D-HAZY: A dataset
dehazing,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. to evaluate quantitatively dehazing algorithms,” in Proc. IEEE Int. Conf.
Workshops (CVPRW), Jun. 2020, pp. 490–491. Image Process. (ICIP), Sep. 2016, pp. 2226–2230.
[30] H. Zhu, X. Peng, V. Chandrasekhar, L. Li, and J.-H. Lim, “DehazeGAN: [54] A. Golts, D. Freedman, and M. Elad, “Unsupervised single image dehaz-
When image dehazing meets differential programming,” in Proc. IJCAI, ing using dark channel prior loss,” 2018, arXiv:1812.07051. [Online].
Jul. 2018, pp. 1234–1240. Available: http://arxiv.org/abs/1812.07051

Authorized licensed use limited to: Cornell University Library. Downloaded on September 02,2020 at 09:09:44 UTC from IEEE Xplore. Restrictions apply.