Lecture 3: Image Restoration
B14 Image Analysis Michaelmas 2014 A. Zisserman
• Image degradations
• motion blur, focus blur, resolution
• The inverse filter
• The Wiener filter
• MAP formulation
In contrast to image enhancement, in image restoration the
degradation is modelled. This enables the effects of the
degradation to be (largely) removed
Degradations
• original
• optical blur
• motion blur
• spatial quantization (discrete pixels)
• additive intensity noise
Overview – Deconvolution
The objective is to restore a degraded image to its original
form.
An observed image can often be modelled as:
where the integral is a convolution, h is the point spread
function of the imaging system, and n is additive noise.
The objective of image restoration in this case is to
estimate the original image f from the observed degraded
image g.
Degradation model
Model degradation as a convolution with a linear, shift
invariant, filter h(x,y)
• Example: for out of focus blurring, model h(x,y) as a Gaussian
original blurred
f(x,y) h(x,y) g(x,y)
i.e. : g(x,y) = h(x,y) * f(x,y)
h(x,y) is the impulse response or point spread function of the imaging system
The challenge: loss of information and noise
Gaussian
scale=3 pixels
*
FT FT Inverse FT
Blurring acts as a low pass filter and attenuates higher spatial frequencies
Definitions
• f(x,y) – image before degradation, ‘true image’
• g(x,y) – image after degradation, ‘observed image’
• h(x,y) – degradation filter
• f(x,y) – estimate of f(x,y) computed from g(x,y)
• n(x,y) – additive noise
f(x,y) g(x,y) f(x,y)
degradation restoration
h(x,y) n(x,y)
g(x,y) = h(x,y) * f(x,y) + n(x,y) G(u,v) = H(u,v) F(u,v) + N(u,v)
The inverse filter
Start from the generative model
g(x,y) = h(x,y)* f(x,y) + n(x,y) G(u,v) = H(u,v) F(u,v) + N(u,v)
and for the moment ignore n(x,y), then an estimate of f(x,y) is
obtained from
F(u,v) = G(u,v) / H(u,v)
Restoration with an inverse filter
g(x,y) G(u,v) F(u,v) f(x,y)
F.T. Inverse filter I.F.T.
1D vector explanation
Fourier trick
Example : Deblurring (deconvolution)
Image blurred with Gaussian point spread function
h(x,y) = n(x,y) = Normal distribution, mean zero
blur = 1.0 pixels
noise = 0.3 grey levels
f(x,y) g(x,y)
Restoration with an inverse filter
F(u,v) = G(u,v) / H(u,v) where H(u,v), is the FT of the Gaussian
Deblurring with an inverse filter
noise = 0.3 grey levels F(u,v) = G(u,v) / H(u,v)
blur = 0.5 pixels blur = 1.0pixels blur = 1.5pixels
g(x,y)
f(x,y)
The problem of noise amplification
G(u,v) = H(u,v) F(u,v) + N(u,v)
F(u,v) = G(u,v) / H(u,v) = F(u,v) + N(u,v) / H(u,v)
Schematically …
1 N(u,v)
F(u,v) H(u,v)
F(u,v)
x +
H(u,v)F(u,v)
0 u,v 0 u,v 0 u,v 0 u,v
1/H(u,v) G(u,v)
F(u,v)
x
1
0 u,v 0 u,v 0 u,v
high spatial frequency sinusoids
f(x,y)
blur = 1.0 pixels
The Wiener filter
The Wiener filter
F(u,v) = W(u,v) G(u,v)
Frequency behaviour
F(u,v) = W(u,v) G(u,v)
• If K = 0 then W(u,v) = 1 / H(u,v), i.e. an inverse filter
• If K >> |H(u,v)| for large u,v, then high frequencies are attenuated
• |F(u,v)| and |N(u,v)| are often known approximately, or
• K is set to a constant scalar which is determined empirically
• A Wiener filter minimizes the least square error
F(u,v) = W(u,v) G(u,v)
Schematically …
G(u,v) = H(u,v) F(u,v) + N(u,v)
1 N(u,v)
F(u,v) H(u,v)
F(u,v)
x +
H(u,v)F(u,v)
0 u,v 0 u,v
1/H(u,v) G(u,v)
F(u,v) x
1
W(u,v)
Restoration with a Wiener filter
G(u,v) = H(u,v) F(u,v) + N(u,v)
F(u,v) = W(u,v) G(u,v)
g(x,y) G(u,v) F(u,v) f(x,y)
F.T. Wiener filter I.F.T.
Example 1: Focus deblurring with a Wiener filter
blur = 1.5 pixels
noise = 0.3 grey levels F(u,v) = W(u,v) G(u,v)
g(x,y) f(x,y)
K = 1.0 e -5 K = 1.0 e -3 K = 1.0 e -1
blur = 3.0 pixels
noise = 0.3 grey levels
f(x,y) g(x,y) f(x,y)
K = 5.0 e -4
Wiener filter – sketch derivation
Parseval’s Theorem
since f(x,y) and (x,y) uncorrelated
• Note, integrand is sum of two squares
Minimize integral if integrand minimum for all (u,v)
NB
Note: filter is defined in the Fourier domain
Example 2: Motion deblurring
Suppose there is blur only in the horizontal direction
e.g. camera pans as image is acquired
Degradation model
Require H(u,v) for Wiener filter
interchange order of spatial and temporal integration
where
pixels
FT of …
Note, H(u,v) has zeros – a problem for an inverse filter
Motion deblurring with a Wiener filter
blur = 20 pixels
1. Compute the FT of the blurred image
2. Multiply the FT by the Wiener filter F(u,v) = W(u,v) G(u,v)
3. Compute the inverse FT
Application: Reading number plates
Algorithm
1. Rotate image so that blur is horizontal
2. Estimate length of blur
3. Construct a bar modelling the convolution
4. Compute and apply a Wiener filter
5. Optimize over values of K
f(x,y) h(x,y) f(x,y)
blur = 30 pixels
Maximum a posteriori (MAP)
Estimation
Generative model (forward process)
• original f(x,y)
• motion blur
• additive intensity noise
For an image with n pixels, write this process as
ĝ = Af + n
where ĝ and f are n-vectors, and A is an n × n matrix.
Inverse problem
• Estimate f(x,y) by optimizing a cost function:
observed generated
image image
f̂ 2
= arg min (g − Af ) + λp (f )
f
Likelihood/ prior/
loss function regularization
Example
p (f ) = (∇f ) 2
to suppress high frequency noise
Example 3: Super resolution
Suppose there are multiple images of the same scene
each displaced spatially …
After registration the samples are not coincident
and this may be used to defeat the Nyquist limit.
Intuitive model
Treat images as point samples
low more images
• increase resolution
high • reduce noise
Generative Model
High-resolution
image, f.
Registrations,
lighting and
M1 M2 M3 M4 blur.
g1 g2 g3 g4
Low-resolution images
Sketch solution Non-examinable
• Estimate the super resolution image which minimizes
the error between predicted and observed images.
Write the generative model for one image i as
gi = Mif + η i
where Mi combines registration, lighting and down-sampling.
likelihood prior
Super resolution example I: Mars
25 JPEG images courtesy of the Mars lander
images are from different sweeps of a rotating camera
Super resolution result
Original frame Average image Super-resolution
2x zoom from 25 JPEG images.
Super resolution example II: car sequence
rotating DV camera
Mosaic
Super-resolution result for ROI
85 JPEG images
original ROI four times resolution
35 x 20 pixels
Super resolution example III: Run Lola Run
Input – low resolution
Super-resolution output
Blind deblurring Non-examinable
So far we have assumed that we know the generative model, e.g.
g = A(h) f
G = HF
= *
i.e. that h(x,y) is known, so that given the observed
image g(x,y), then the original image f(x,y) can be
estimated (restored)
Consider if only the observed image g(x,y) is known.
This is the problem of blind estimation.
Blind deblurring continued
• Estimate f(x,y) and h(x,y) by optimizing a cost function:
observed generated
image image
2
min (g − A(h) f ) + λpf (f ) + μph (h)
f ,h
Likelihood/ image blur
loss function prior prior
Example I: Blind deblurring
estimated
blurred image restored image
blur filter
More examples of blind deblurring
Blurry input Deblurred output
