Restoration Model

Restoration in presence of noise only

Restoration in presence of blur and noise

Image Restoration
EE 604

Govind Sharma

Department of Electrical Engineering

Indian Institute of Technology, Kanpur

11/4/2019

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Outline

Restoration Model

Restoration in presence of noise only

Restoration in presence of blur and noise

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Image Restoration

Image restoration may be viewed as an estimation process in which

operations are performed on an observed or measured image to
estimate the ideal image that would be observed if no degradation
were present in an imaging system.

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Restoration Model

Image Perfect Perfect

light imaging image fI [k, l]
distribution system digitizer

Physical Physical Image Image

imaging image restoration display
system digitizer system system

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Potential degradations

(a) Diffraction in optical system

(b) Sensor nonlinearities
(c) Optical system aberrations
(d) Atmospheric turbulence effects
(e) Image motion blur
(f) Camera motion blur
(g) Geometric distortions
(h) Noise due to electronics, film granularities etc.

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Restoration Goal
Usually, it will not be possible to restore perfectly the observed
image. We try to minimize an error criterion. Minimum mean
squared error criterion
h i
E = E |fI (x , y ) − fˆ(x , y )|2

where fI (x , y ) is output of perfect imaging system and fˆ(x , y ) is

display image of a practical system.
Since restoration process is to be performed digitally, we use the
criterion on the sampled images.
h i
E = E |f [k, l] − fˆ[k, l]|2

where f [k, l] is output of the perfect digitizer and fˆ[k, l] is the

output of the image restoration system.
G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Simplified model of degradation

There are no general solutions for the restoration problem. We

take a simplified model

g[k, l] = H(f [k, l]) + n[k, l]

where H() is a general transformation, f [k, l] is the output of the

perfect digitizer, g[k, l] is degraded image and n[k, l] is noise.
First we assume that H() is an Identity operator. That is
degradation is due to noise only.

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Some noise models

We assume that noise is independent of spatial coordinates and

that it is uncorrelated to the image pixel values. Some important
noise probability densities are
(a) Gaussian (circuit and sensor noise)
(b) Rayleigh (in range imaging)
(c) Gamma (laser imaging)
(d) Exponential (laser imaging)
(e) Uniform (quantization)
(f) Impulse or salt-pepper (due to faulty switching)

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Test Pattern

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Effect of noise

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Effect of noise

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Spatial filtering

When only additive noise is present we usually use spatial filtering.

Some of the filters are
(a) Arithmetic mean filter:

1 X
fˆ[k, l] = g[i, j]
MN [i,j]∈S
kl

where Skl is a rectangular neighborhood of size M × N

centered at point [k, l]. This operation can be implemented by
a spatial filter whose all coefficients are 1/MN. Image is
blurred by this filter.

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Spatial filtering

(b) Geometric mean filter:

  1
MN

fˆ[k, l] = 
Y
g[i, j]
[i,j]∈Skl

It performs slightly better than arithmetic mean filter.

(c) Harmonic mean filter:

MN
fˆ[k, l] = P
1/g[i, j]
[i,j]∈Skl

It works well for salt noise, but fails for pepper noise.

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Image restoration example

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Spatial filtering

(d) Contraharmonic mean filter:

g[i, j]Q+1
P
[i,j]∈Skl
fˆ[k, l] =
g[i, j]Q
P
[i,j]∈Skl

where Q is called the order of the filter. It works well for salt
noise if Q < 0, and for pepper noise if Q > 0. It reduces to
arithmetic mean if Q = 0 and harmonic mean if Q = −1.

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Restoration example

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Spatial filtering

(e) Median filter: Median of the neighborhood

(f) Midpoint filter: average of minimum and maximum values in
the neighborhood
(g) Alpha trimmed mean: d/2 smallest and largest values in the
neighborhood are dropped and arithmetic mean of the
remaining values is taken.
(h) Adaptive filtering

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Restoration example

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Adaptive mean filter

σ 2
fˆ[k, l] = g[k, l] − n2 (g[k, l] − mL )
σL
where ML = local mean of the pixel values in Skl , σL2 = local
variance of the pixel values in Skl and σn2 = noise variance. If noise
variance is much smaller than local variance this filter returns the
estimated value close to observed value.

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Restoration example

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Adaptive median filter

We use the following definitions

zmax = maximum intensity value in Skl

zmin = minimum intensity value in Skl
zmed = median intensity value in Skl
zkl = observed intensity value g[k, l]
Smax = maximum size of neighborhood Skl
A1 = zmed − zmin
A2 = zmed − zmax
B1 = zkl − zmin
B2 = zkl − zmax

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Adaptive median filter

Stage A: If A1 > 0 and A2 < 0, go to Stage B

else increase the window size
If window size ≤ Smax repeat Stage A
else output zmed
Stage B: If B1 > 0 and B2 < 0, output zkl
else output zmed

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Restoration example

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Linear shift invariant degradation

Simplified degradation model we are using is

g[k, l] = H(f [k, l]) + n[k, l]

We further make assumption that H is a linear shift invariant

operator. So

g[k, l] = h[k, l] ∗ f [k, l] + n[k, l]

G(u, v ) = H(u, v )F (u, v ) + N(u, v )

h[k, l] is also known as point spread function (PSF). Restoration

method in this case is also known as deconvolution.

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Estimation of degradation function

(a) Estimation by image observation

(b) Estimation by experimentation
(c) Estimation by modeling

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

PSF estimation by experimentation

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Example estimation by modeling

Suppose image f (x , y ) undergoes planar motion x0 (t), y0 (t) are
components of motion. If duration of exposure is T , then observed
image is

ZT
g(x , y ) = f (x − x0 (t), y − y0 (t))dt
0
Z∞ Z∞
G(u, v ) = g(x , y )e −j2π(ux +vy ) dxdy
−∞ −∞
Z∞ Z∞ ZT
= f (x − x0 (t), y − y0 (t))dte −j2π(ux +vy ) dxdy
−∞ −∞ 0

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Example estimation by modeling

ZT Z∞ Z∞
G(u, v ) = f (x − x0 (t), y − y0 (t))e −j2π(ux +vy ) dxdydt
0 −∞ −∞
ZT
= F (u, v )e −j2π(ux0 (t)+vy0 (t)) dt
0
ZT
= F (u, v ) e −j2π(ux0 (t)+vy0 (t)) dt
0
ZT
H(u, v ) = e −j2π(ux0 (t)+vy0 (t)) dt
0

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Example estimation by modeling

If image has uniform motion x0 (t) = at/T , y0 (t) = bt/T then

ZT
H(u, v ) = e −j2π(uat/T +vbt/T ) dt
0
T
= sin(π(ua + vb))e −jπ(ua+vb)
π(ua + vb)

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Motion Blur example

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Restoration methods

(a) Inverse filtering

(b) Wiener filtering
(c) Least squares and Constrained least squares

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Inverse filtering

1
Q(u, v ) = H(u,v )
F̂ (u, v ) = Q(u, v )G(u, v )
N(u,v )
= F (u, v ) + H(u,v )

If H(u, v ) has small value inverse filter may blow up. So it is

modified as
(
1
Q(u, v ) = H(u,v ) , if H(u, v ) > threshold
0, otherwise

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Wiener Filter

Let us first consider 1-D case. Let y [k] be observed sequence and
s[k] is a sequence we want to estimate. We assume that these two
processes are zero mean jointly wide sense stationery sequences
with known auto and cross correlation functions. Estimation
method is linear minimum mean squared error criterion. Let w [k]
represent the impulse response and W (u) its Fourier transform
then we want to minimize

E = E [|s[k] − ŝ[k]|2 ]

This can be done using orthogonality principle.

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Wiener filter
∞
X
E [(s[k] − w [l]y [k − l])y [m]] = 0, −∞ < m < ∞
l=−∞
Taking expectation inside summation sign we get
∞
X
E [s[k]y [m]] = w [l]E [y [k − l]y [m]]
l=−∞
X∞
Rsy [k − m] = w [l]Ryy [k − m − l]
l=−∞
X∞
Rsy [n] = w [l]Ryy [n − l] (k-m=n)
l=−∞
Rsy [n] = w [n] ∗ Ryy [n]
Ssy (u) = W (u)Syy (u)

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Wiener filter

Let
g[k] = h[k] ∗ f [k] + n[k]
Since noise n[k] is uncorrelated to f [k], we get Rgg [m] as sum of
two terms and so

Sgg (u) = |H(u)|2 Sff (u) + Snn (u)

where we use the relationship between power spectral densities

when a random process is passed through a linear system. Similarly

Sfg (u) = H ∗ (u)Sff (u)

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Wiener filter

Wiener filter is
H ∗ (u)Sff (u)
W (u) =
|H(u)|2 Sff (u) + Snn (u)

In 2-D case we will get

H ∗ (u, v )Sff (u, v )

W (u, v ) =
|H(u, v )|2 Sff (u, v ) + Snn (u, v )
H ∗ (u, v )
=
|H(u, v )|2 + SSnn (u,v )
ff (u,v )

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Motion blur restoration example

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Wiener filter noise and blur

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Least squares
Recall that linear shift invariant model
g[k, l] = h[k, l] ∗ f [k, l] + n[k, l]
Suppose image has size M × N and filter has size J × K then
blurred image will have size (M + J − 1) × (N + K − 1). We can
write this relation ship in vector space form
g = Bf + n
We want to find a matrix C so that fˆ = C g and
k g − B fˆk2
is minimized. The minimum norm least square solution is given by
fˆ = B − g
where pseudo inverse B − is obtained via singular value
decomposition.
G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Least square restoration example

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Least square restoration example

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Constrained least squares

Least squares method is very sensitive to noise. To reduce noise

sensitivity a measure of smoothness like second derivative is
minimized. Thus criterion is
M−1
X N−1
X 2
ξ= ∇2 f [k, l]
k=0 l=0

subject to constraint

k g − B fˆk2 = knk2

Second derivative is calculated using Laplacian operator

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Constrained least squares

Laplacian operator is given by ∇2 f [k, l] = p[k, l] ∗ f [k, l] where

 
0 1 0
p[k, l] = 1 −4 1
 
0 1 0

Problem is solved by converting it into frequency domain and

Langrangian multiplier method. The solution is given by

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Constrained least squares

H ∗ (u, v )
F̂ (u, v ) = G(u, v )
|H(u, v )|2 + λ|P(u, v )|2
where λ is chosen to satisfy the constraint. If noise has variance σ 2
then knk2 = (M + J − 1) × (N + K − 1)σ 2 . For λ = 0 then we get
inverse filter. Problem is solved iteratively by trying various values
of λ.

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

Constrained least square restoration example

G. Sharma Restoration
 Restoration Model
Restoration in presence of noise only
Restoration in presence of blur and noise

References

1. Introduction to Digital Image Processing, William K. Pratt,

CRC Press.
2. Digital Image Processing, Rafael C. Gonzalez and Richard E.
Woods, Prentice Hall.
3. Fundamentals of Digital Image Processing, Anil K. Jain,
Prentice Hall.

G. Sharma Restoration