Fast and Effective AI Approaches for Video Quality Improvement
Marco Bertini, Leonardo Galteri, Lorenzo Seidenari, Tiberio Uricchio, Alberto Del Bimbo∗
[name.surname]@unifi.com
[name]@small-pixels.com
Università di Firenze - MICC
Small Pixels
Firenze, Italy
ABSTRACT
In this work we present solutions based on AI techniques to the
problem of real-time video quality improvement, addressing both
video super resolution and compression artefact removal. These
solutions can be used to revamp video archive materials allowing
their reuse in modern video production and to improve the end
user experience playing streaming videos in higher quality while Figure 1: System overview: left) GAN-based training; right)
requiring less bandwidth for their transmission. The proposed ap- use of the network to improve video quality of a generic
proaches can be used on a variety of devices as a post-processing video (top), use of a network specialized on a specific video
step, without requiring any change in existing video encoding and (bottom)
transmission pipelines. Experiments on standard video datasets
have shown that the proposed approaches improve video quality
metrics considering either fixed bandwidth budgets or fixed quality artefacts like blocking, mosquito noise, posterization, etc. that ham-
goals. per user experience. In this work, we present a set of techniques
based on AI that can be used to revamp video archive materials [5, 6]
CCS CONCEPTS or increase the visual quality of streaming videos [1, 7]. The devel-
• Computing methodologies → Learning from critiques; Image oped neural networks, trained using the Generative Adversarial
compression; • Computer systems organization → Neural net- Networks (GANs) framework [3] (Fig. 1 left), can be optimized to
works. run in real-time [2, 4, 7] or faster than real-time even on mid-level
GPUs, allowing their deployment for video restoration, and can be
KEYWORDS further optimized to run in real-time on mobile devices, exploiting
Video quality enhancement,GANs,video players,real-time video CoreML and Neural Engine hardware on iOS devices [4], and ex-
enhancement ploiting WebGL and mobile GPU acceleration on Android and web
browsers. The main scientific contributions of our work are:
ACM Reference Format:
Marco Bertini, Leonardo Galteri, Lorenzo Seidenari, Tiberio Uricchio, Al- (1) development of losses that combine perceptual and signal
berto Del Bimbo. 2022. Fast and Effective AI Approaches for Video Quality based metrics that help to reconstruct perceptually pleasant
Improvement. In Mile-High Video Conference (MHV ’22), March 1–3, 2022, frames;
Denver, CO, USA. ACM, New York, NY, USA, 2 pages. https://doi.org/10. (2) development of neural network designs that allow to reduce
1145/3510450.3517270 their computational costs;
(3) development of GANs training regimes that generate realis-
1 INTRODUCTION AND PROPOSED tic details.
METHODS Furthermore, we present a set of products, based on these contribu-
Lossy video compression algorithms such as H.264, H.265, AV1, tions, that can be deployed on a variety of end user devices. These
etc. are the foundation of video streaming but, in order to optimize products can be embedded in video players, to process the frames
available bandwidth and transmission costs, they introduce visual immediately before showing them to the user (Fig. 1 right-top).
They can upscale video frames, thus effectively reducing the band-
∗ These authors contributed equally to the paper. width required to stream videos, and at the same time eliminate
compression artefacts and add image details that were lost due to
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed lossy compression. Our networks can be customized to specific
for profit or commercial advantage and that copies bear this notice and the full citation video types (e.g. soccer, cartoons, documentaries) or on a per-title
on the first page. Copyrights for components of this work owned by others than ACM basis (Fig. 1 right-bottom), allowing to obtain a required video qual-
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a ity with a lower bitrate, even if this latter approach requires to send
fee. Request permissions from permissions@acm.org. the weights of the network for each title; this is possible thanks
MHV ’22, March 1–3, 2022, Denver, CO, USA to the compactness of the designed networks that require an ex-
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9222-8/22/03. . . $15.00 tremely limited space. The proposed method can be adapted and
https://doi.org/10.1145/3510450.3517270 effectively used also for video conferencing applications.
77
MHV ’22, March 1–3, 2022, Denver, CO, USA Bertini et al.
REFERENCES the 27th ACM International Conference on Multimedia. ACM, ACM, 1065–1067.
[1] Leonardo Galteri, Marco Bertini, Lorenzo Seidenari, Tiberio Uricchio, and Alberto [5] Filippo Mameli, Marco Bertini, Leonardo Galteri, and Alberto Del Bimbo. 2020.
Del Bimbo. 2020. Increasing video perceptual quality with gans and semantic Image and video restoration and compression artefact removal using a NoGAN
coding. In Proceedings of the 28th ACM International Conference on Multimedia. approach. In Proceedings of the 28th ACM International Conference on Multimedia.
ACM, ACM, 862–870. ACM, ACM, 4539–4541.
[2] Leonardo Galteri, Lorenzo Seidenari, Marco Bertini, and Alberto Del Bimbo. 2019. [6] Filippo Mameli, Marco Bertini, Leonardo Galteri, and Alberto Del Bimbo. 2021.
Towards real-time image enhancement GANs. In International Conference on A NoGAN approach for image and video restoration and compression artifact
Computer Analysis of Images and Patterns. Springer, Springer, 183–195. removal. In 2020 25th International Conference on Pattern Recognition (ICPR). IEEE,
[3] Leonardo Galteri, Lorenzo Seidenari, Marco Bertini, and Alberto Del Bimbo. 2019. IEEE, 9326–9332.
Deep universal generative adversarial compression artifact removal. IEEE Trans- [7] Federico Vaccaro, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. 2021.
actions on Multimedia 21, 8 (2019), 2131–2145. Fast Video Visual Quality and Resolution Improvement using SR-UNet. In Pro-
[4] Leonardo Galteri, Lorenzo Seidenari, Marco Bertini, Tiberio Uricchio, and Alberto ceedings of the 29th ACM International Conference on Multimedia. ACM, ACM,
Del Bimbo. 2019. Fast video quality enhancement using GANs. In Proceedings of 1221–1229.
78
