Computer Vision

Lec. 6: Image filtering

Spring 2025
Zezhou Cheng
University of Virginia
Welcome to Computer Vision!
Last: Photometric image formation

Surface

1. Biological eyes: human and animal vision

2. Color: RGB and other color space HW01 is out!
3. Light: BRDF, etc.
Today: Image filtering

1. Image filtering
2. Convolution and its properties
3. Filters and their applications
Today: Image filtering

1. Image filtering
2. Convolution and its properties
3. Filters and their applications
How can we reduce noise in a photograph?

Why do noise appear? Camera sense limitations, low lighting, high ISO, etc.
Image Credits: Subhransu Maji
How can we reduce noise in a photograph?
• Replace each pixel with a weighted average of its neighborhood
• Weights are called the filter kernel
• What are the weights for the average of a 3x3 neighborhood?

Apply this operation on every pixel.

Image Credits: Subhransu Maji

1D Case

Signal 10 12 9 11 10 11 12

What’s the average

Filter 1/3 1/3 1/3
of 9, 10, 12?

(a) 9 (b) 11.5

Output 10.33
(c) 10.33 (d) 11.66

Image Credits: David Fouhey

1D Case

Signal 10 12 9 11 10 11 12

Filter 1/3 1/3 1/3

Output 10.33

Done! Next?
Image Credits: David Fouhey
1D Case

Signal 10 12 9 11 10 11 12

Filter 1/3 1/3 1/3

Output 10.33 10.66

Image Credits: David Fouhey

1D Case

Signal 10 12 9 11 10 11 12

Filter 1/3 1/3 1/3

Output 10.33 10.66 10

Image Credits: David Fouhey

1D Case

Signal 10 12 9 11 10 11 12

Filter 1/3 1/3 1/3

Output 10.33 10.66 10 10.66

Image Credits: David Fouhey

1D Case

Signal 10 12 9 11 10 11 12

Filter 1/3 1/3 1/3

Output 10.33 10.66 10 10.66 11

Image Credits: David Fouhey

1D Case
10 12 9 11 10 11 12

⁎
1/3 1/3 1/3

=
10.33 10.66 10 10.66 11

You lose pixels (maybe…)

Filter “sees” only a few pixels at a time
What does an image look like for a computer?

2D Projection of 3D world

an array of numbers!

Image Credits: Andreas Geiger & Bill Freeman and Phillip Isola
Image filtering
Input Filter Output
I11 I12 I13 I14 I15 I16

I21 I22 I23 I24 I25 I26 F11 F12 F13 O11 O12 O13 O14

I31 I32 I33 I34 I35 I36 F21 F22 F23 O21 O22 O23 O24

I41 I42 I43 I44 I45 I46 F31 F32 F33 O31 O32 O33 O34

I51 I52 I53 I54 I55 I56

Image Credits: David Fouhey

Image filtering
Input & Filter Output
F11
I11 F12
I12 F13
I13 I14 I15 I16

F21
I21 F22
I22 F23
I23 I24 I25 I26 O11

F31
I31 F32
I32 F33
I33 I34 I35 I36

I41 I42 I43 I44 I45 I46

I51 I52 I53 I54 I55 I56

𝑂11 = 𝐼11 ∗ 𝐹11 + 𝐼12 ∗ 𝐹12 + ⋯ + 𝐼33 ∗ 𝐹33

Image Credits: David Fouhey
Image filtering
Input & Filter Output
I11 F11
I12 F12
I13 F13
I14 I15 I16

I21 F21
I22 F22
I23 F23
I24 I25 I26 O11 O12

I31 F31
I32 F32
I33 F33
I34 I35 I36

I41 I42 I43 I44 I45 I46

I51 I52 I53 I54 I55 I56

𝑂12 = 𝐼12 ∗ 𝐹11 + 𝐼13 ∗ 𝐹12 + ⋯ + 𝐼34 ∗ 𝐹33

Image Credits: David Fouhey
Image filtering
Input Filter Output
I11 I12 I13 I14 I15 I16

I21 I22 I23 I24 I25 I26 F11 F12 F13

I31 I32 I33 I34 I35 I36 F21 F22 F23

I41 I42 I43 I44 I45 I46 F31 F32 F33

I51 I52 I53 I54 I55 I56

How many times can we apply a 3x3 filter to a 5x6 image?

Image Credits: David Fouhey

Image filtering
Input Filter Output
I11 I12 I13 I14 I15 I16

I21 I22 I23 I24 I25 I26 F11 F12 F13 O11 O12 O13 O14

I31 I32 I33 I34 I35 I36 F21 F22 F23 O21 O22 O23 O24

I41 I42 I43 I44 I45 I46 F31 F32 F33 O31 O32 O33 O34

I51 I52 I53 I54 I55 I56

𝑂𝑖𝑗 = 𝐼𝑖𝑗 ∗ 𝐹11 + 𝐼𝑖,𝑗+1 ∗ 𝐹12 + ⋯ + 𝐼𝑖+2,𝑗+2 ∗ 𝐹33

Image Credits: David Fouhey

Practice with Linear Filters

0 0 0
0 1 0
?
0 0 0

Original

Slide Credit: D. Lowe

Practice with Linear Filters

0 0 0
0 1 0
0 0 0

Original The Same!

Slide Credit: D. Lowe

Practice with Linear Filters

0 0 0
0 0 1 ?
0 0 0

Original

Slide Credit: D. Lowe

Practice with Linear Filters

0 0 0
0 0 1
0 0 0

Original Shifted
LEFT
1 pixel

Slide Credit: D. Lowe

Practice with Linear Filters

0 1 0
0 0 0 ?
0 0 0

Original

Slide Credit: D. Lowe

Practice with Linear Filters

0 1 0
0 0 0
0 0 0

Original Shifted
DOWN
1 pixel

Slide Credit: D. Lowe

Practice with Linear Filters

1/9 1/9 1/9

1/9 1/9 1/9 ?
1/9 1/9 1/9

Original

Slide Credit: D. Lowe

Practice with Linear Filters

1/9 1/9 1/9

1/9 1/9 1/9
1/9 1/9 1/9

Original Blur
(Box Filter)

Slide Credit: D. Lowe

Practice with Linear Filters
0 0 0
0 2 0
0 0 0

-
?
1/9 1/9 1/9
Original
1/9 1/9 1/9
1/9 1/9 1/9

Slide Credit: D. Lowe

Practice with Linear Filters
0 0 0
0 2 0
0 0 0

-
1/9 1/9 1/9
Original Sharpened
1/9 1/9 1/9
1/9 1/9 1/9

Slide Credit: D. Lowe

Image filtering
Input Filter Output
I11 I12 I13 I14 I15 I16

I21 I22 I23 I24 I25 I26 F11 F12 F13 O11 O12 O13 O14

I31 I32 I33 I34 I35 I36 F21 F22 F23 O21 O22 O23 O24

I41 I42 I43 I44 I45 I46 F31 F32 F33 O31 O32 O33 O34

I51 I52 I53 I54 I55 I56

We lose some pixels! How to fix it?

Image Credits: David Fouhey

Painful Details – Edge Cases
• Filtering doesn’t keep the whole image.
• Suppose f is the image and g the filter.

g g g g
g g

f f f

g g
g g g g
Full Same Valid
Output size: f + g Output size: f Output size: f - g

Image Credits: Subhransu Maji

Painful Details – Edge Cases
What to about the “?” region?
???? Symmetry: fold sides over
g g

Circular/Wrap: wrap around

f

pad/fill: add value, often 0

g g

f/g Diagram Credit: D. Lowe

Painful Details – Edge Cases
Zero padding

11x11 avg

Image Credits: Bill Freeman and Phillip Isola

Painful Details – Edge Cases

Input
Output
Error
Zero-padding: simply, most used
Today: Image filtering

1. Image filtering
2. Convolution and its properties
3. Filters and their applications
Image filtering
• Let 𝑓 be the image and 𝑔 be the kernel. The output of filtering 𝑓
with 𝑔 denoted f ∗ 𝑔 is given by:

𝑓 ∗ 𝑔 𝑚, 𝑛 = ෍ 𝑓 𝑚 + 𝑘, 𝑛 + 𝑙 𝑔[𝑘, 𝑙]
𝑘,𝑙

• Filtering computes cross-correlation between the 𝑔 and 𝑓 at each location

• Convolution is filtering with a flipped 𝑔 (by notation)
Image Credits: Subhransu Maji
Convolution vs Cross-correlation
Cross-Correlation Convolution
(Original Orientation) (Flip filter in x,y first)

𝑓 ∗ 𝑔 𝑚, 𝑛 = ෍ 𝑓 𝑚 + 𝑘, 𝑛 + 𝑙 𝑔[𝑘, 𝑙] 𝑓 ∘ 𝑔 𝑚, 𝑛 = ෍ 𝑓 𝑚 − 𝑘, 𝑛 − 𝑙 𝑔[𝑘, 𝑙]
𝑘,𝑙 𝑘,𝑙

• Painful source of confusion

• Does not matter if the kernel is symmetric
• Does not matter in convolutional networks – kernels are learned
Image Credits: Subhransu Maji
Convolution Properties – Linear
Assume: 𝐼 image 𝑓1 , 𝑓2 filters
Linear: apply(𝐼, 𝑓1 + 𝑓2 ) = apply(𝐼, 𝑓1 ) + apply(𝐼, 𝑓2 )
𝐼 is a white box on black, and 𝑓1 , 𝑓2 are rectangles

A( , + ) =A( , )=

A( , )+A( , )= + =

Note: I am showing filters un-normalized and blown up. They’re a smaller box
filter (i.e., each entry is 1/(size^2))
Image Credits: David Fouhey
Convolution Properties – Shift-Invariant
Assume: 𝐼 image, 𝑓 filter
Shift-invariant: shift(apply(𝐼, 𝑓)) = apply(shift(𝐼, 𝑓))
Intuitively: only depends on filter neighborhood

A( , )=

A( , )=
Image Credits: David Fouhey
Convolution neural networks

• Convolution kernels: 𝑓 ∘ 𝑔 𝑚, 𝑛 = σ𝑘,𝑙 𝑓 𝑚 − 𝑘, 𝑛 − 𝑙 𝑔[𝑘, 𝑙]

• Why do we get the kernel?
• Manually designed → machine learning

LeCun et al. 1989

We need translation invariance
Image Credits: Bill Freeman and Phillip Isola
Image Credits: Bill Freeman and Phillip Isola
Classifier “Bird”

Image Credits: Bill Freeman and Phillip Isola

Classifier “Bird”
Bird

Image Credits: Bill Freeman and Phillip Isola

 Bird

Classifier “Sky”
Sky

Image Credits: Bill Freeman and Phillip Isola

Sky Sky Sky Sky Sky Sky Sky Bird

Sky Sky Sky Sky Sky Sky Sky Sky

Sky Sky Sky Sky Sky Sky Sky Sky

Bird Bird Bird Sky Bird Sky Sky Sky

Sky Sky Sky Bird Sky Sky Sky Sky

Image Credits: Bill Freeman and Phillip Isola

Image Credits: Bill Freeman and Phillip Isola
Convolution Properties – Some others
• Any shift-invariant, linear operation is a convolution (⁎)
• Commutative: 𝑓 ⁎ 𝑔 = 𝑔 ⁎ 𝑓
• Associative: (𝑓 ⁎ 𝑔) ⁎ ℎ = 𝑓 ⁎ (𝑔 ⁎ ℎ)
• Distributes over addition: 𝑓 ⁎ (𝑔 + ℎ) = 𝑓 ⁎ 𝑔 + 𝑓 ⁎ ℎ
• Scalars factor out: 𝑘𝑓 ⁎ 𝑔 = 𝑓 ⁎ 𝑘𝑔 = 𝑘 (𝑓 ⁎ 𝑔)
• Identity (a single one with all zeros):
• unit impulse 𝑒 = [… , 0, 0, 1, 0, 0, … ], 𝑎 ∗ 𝑒 = 𝑎

Image Credits: Subhransu Maji

Today: Image filtering

1. Image filtering
2. Convolution and its properties
3. Filters and their applications
Box filtering: some artifacts
• If filter touches it, it gets a contribution
• but images have local similarity
1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Input Filter Output

Solution – Weighted Combination
Weight according to closeness to center.
Define 0,0 to be center pixel, then:

𝐹𝑖𝑙𝑡𝑒𝑟𝑖𝑗 ∝ 1
What’s this?

𝑥2 + 𝑦2
𝐹𝑖𝑙𝑡𝑒𝑟𝑖𝑗 ∝ exp −
2𝜎 2
Recognize the Filter?
It’s a Gaussian!
1 𝑥2 + 𝑦2
𝐹𝑖𝑙𝑡𝑒𝑟𝑖𝑗 ∝ 2
exp −
2𝜋𝜎 2𝜎 2

0.003 0.013 0.022 0.013 0.003

0.013 0.060 0.098 0.060 0.013
0.022 0.098 0.162 0.098 0.022
0.013 0.060 0.098 0.060 0.013
0.003 0.013 0.022 0.013 0.003
Smoothing With A Box & Gauss
Still have some speckles, but it’s not a big box
Input Box Filter Gauss. Filter
Gaussian Filters

σ=1 σ=2 σ=4 σ=8

filter = 21x21 filter = 21x21 filter = 21x21 filter = 21x21

Note: filter visualizations are independently normalized throughout

the slides so you can see them better
Applying Gaussian Filters
Applying Gaussian Filters
Input Image
(no filter)
Applying Gaussian Filters
σ=1
Applying Gaussian Filters
σ=2
Applying Gaussian Filters
σ=4
Applying Gaussian Filters
σ=8
Picking a Filter Size
Too small filter → bad approximation
Want size ≈ 6σ (99.7% of energy)
Left far too small; right slightly too small!
σ = 8, size = 21 σ = 8, size = 43
Runtime Complexity
Image size = NxN = 6x6
Filter size = MxM = 3x3
I11 I12 I13 I14 I15 I16 for ImageY in range(N):
I21 F11
I22 F12
I23 F13
I24 I25 I26 for ImageX in range(N):
I31 I32
F21 I33
F22 I34
F23 I35 I36 for FilterY in range(M):
I41 F31
I42 F32
I43 F33
I44 I45 I46
for FilterX in
range(M):
I51 I52 I53 I54 I55 I56
…
I61 I62 I63 I64 I65 I66

Time: O(N2M2)
Separability

Conv(vector, transposed vector) → outer product

Fx1 * Fx2 * Fx3 *

Fy1
Fy1 Fy1 Fy1

Fy2 ⁎ Fx1 Fx2 Fx3 = Fx1 *

Fy2
Fx2 *
Fy2
Fx3 *
Fy2

Fx1 * Fx2 * Fx3 *

Fy3
Fy3 Fy3 Fy3
Separability

1 𝑥2 + 𝑦2
𝐹𝑖𝑙𝑡𝑒𝑟𝑖𝑗 ∝ 2
exp −
2𝜋𝜎 2𝜎 2

→
1 𝑥2 1 𝑦2
𝐹𝑖𝑙𝑡𝑒𝑟𝑖𝑗 ∝ exp − 2 exp − 2
2𝜋𝜎 2𝜎 2𝜋𝜎 2𝜎
Separability
1D Gaussian ⁎ 1D Gaussian = 2D Gaussian
Image ⁎ 2D Gauss = Image ⁎ (1D Gauss ⁎ 1D Gauss )
= (Image ⁎ 1D Gauss) ⁎ 1D Gauss

⁎ =
Runtime Complexity
Image size = NxN = 6x6
Filter size = Mx1 = 3x1
I11 I12 I13 I14 I15 I16 for ImageY in range(N):
I21 F1
I22 I23 I24 I25 I26 for ImageX in range(N):
I31 I32
F2 I33 I34 I35 I36 for FilterY in range(M):
I41 F3
I42 I43 I44 I45 I46
…
for ImageY in range(N):
I51 I52 I53 I54 I55 I56
for ImageX in range(N):
I61 I62 I63 I64 I65 I66
for FilterX in range(M):
…
Time: O(N2M)
Where Gaussian Fails
Applying Gaussian Filters
σ=1
Why Does This Fail?
Means can be arbitrarily distorted by outliers

Signal 10 12 9 8 1000 11 10 12

Filter 0.1 0.8 0.1

Output 11.5 9.2 107.3 801.9 109.8 10.3

What else is an “average” other than a mean?

Non-linear Filters (2D)
[040, 081, 013, 125, 830, 076, 144, 092, 108]
Sort
40 81 13 22
[013, 040, 076, 081, 092, 108, 125, 144, 830]
125 830 76 80
144 92 108 95 92
132 102 106 87
[830, 076, 080, 092, 108, 095, 102, 106, 087]

Sort
[076, 080, 087, 092, 095, 102, 106, 108, 830]

95
Applying Median Filter
Median
Filter
(size=3)
Applying Median Filter
Median
Filter
(size = 7)
Is Median Filtering Linear?

1 1 1 0 0 0 1 1 1
1 1 2 + 0 1 0 = 1 2 2
2 2 2 0 0 0 2 2 2
Median Filter

1 + 0 = 2

Example from (I believe): Kristen Grauman

Some Other Examples of Filtering
Filtering – Sharpening
Image Smoothed

-
Details

=
Filtering – Sharpening
Image Details

+α
“Sharpened” α=1

=
Filtering – Sharpening
Image Details

+α
“Sharpened” α=0

=
Filtering – Sharpening
Image Details

+α
“Sharpened” α=2

=
Filtering – Sharpening
Image Details

+α
“Sharpened” α=0

=
Filtering – Extreme Sharpening
Image Details

+α
“Sharpened” α=10

=
Filtering
What’s this Filter?
T
-1 0 1 -1 0 1

Dx Dy
Filtering – Derivatives

(Dx2 + Dy2 )1/2

Filtering – Counting
How many “on” pixels have
10+ neighbors within 10 pixels?

Pixels Disk ???

⁎ r=10
=
Filtering – Counting
How many “on” pixels have
10+ neighbors within 10 pixels?

Pixels Density Answer

x =
 Questions?

1. Image filtering
2. Convolution and its properties
3. Filters and their applications