4.

Image Processing and Computer Vision

CCST9049
The Metaverse: From Human Vision to Extended Reality

Dirk Schnieders
CCST 9049 2

What is Image Processing (IP)?

● Improvement of pictorial information for human interpretation
○ e.g. image enhancement, image restoration
● Processing of image data for storage, transmission, and
representation for autonomous machine perception
○ e.g. image compression, image segmentation
● Digital IP refers to processing of digital images by means of a
computer
CCST 9049 3

What is Computer Vision (CV)?

● A branch of artificial intelligence
● Analyse and understand images
● Comparable to the human vision system, including
○ Learning
○ Being able to make inferences
○ Taking actions based on visual inputs
■ e.g. navigation, manipulation, recognition
CCST 9049 4

IP vs CV
● There is a continuum from IP at one end to CV at the other
○ No clear-cut boundaries in this continuum
● The area Image Analysis is in between IP and CV
● To study CV we need to learn the fundamentals of IP first

Image Image Computer

Processing Analysis Vision
CCST 9049 5

The Origins of Digital Images

● First used in the newspaper industry
○ Sent via submarine cable between London and New York
● Bartlane cable picture transmission system
○ Introduced in the early 1920s
○ Reduced time required to transport a picture across the Atlantic
■ From one week down to less than three hours
● Specialized printing equipment coded pictures for cable transmission and
then reconstructed them at the receiving end
● The early Bartlane systems were capable of coding images in five distinct
levels of gray
○ This capability was increased to 15 levels in 1929
CCST 9049 6

The Origins of Digital Images

● The digital picture produced in 1921 from a coded tape by a
telegraph printer with special type faces
CCST 9049 7

Digital Image Processing

● The example shown involves a digital image, but a computer was
not involved in its creation
● Digital images require large storage, computational power, display
and transmission technologies
● Meaningful tasks with digital images appeared first in the early
1960s
● The birth of digital image processing can be traced to the onset of
the US space program
CCST 9049 8

The First Image Processing Task

● In early 1960s, the potential of digital image processing was first
explored
● Ranger 7, a space probe of the moon, captured close images of the
lunar surface on July 31, 1964
● A computer corrected various
types of image distortions inherent
in the on-board camera
● Since the 1960s, the field of image
processing has grown vigorously
and it is now used in a broad range
of application
CCST 9049 9

Image Sensing and Acquisition

● Images are generated by the combination of an illumination source and the
reflection or absorption of energy from that source by the elements of the scene
being imaged
● The reflected or transmitted energy from the scene is focused onto a photo-
converter, which transforms the received energy into a electrical voltage
● The sensor material of the photo-converter is responsive to the particular type of
energy being detected
CCST 9049 10
CCST 9049 11

Digital Images as 2D Functions

● Consider a monochromatic image
○ i.e., grayscale image
● Denote images by 2D functions of the form f(x, y)
● The value or amplitudes of f at spatial coordinates (x, y) is a
positive scalar quantity called the intensity or gray level, whose
physical meaning is determined by the source of the image

When x, y and the amplitude values of f

are all finite and discrete quantities, the
image is called a digital image
CCST 9049 12

Image Sampling & Quantization

● An image may be continuous with respect to the x and y spatial
coordinates and also in amplitude
● To convert an image into digital form, the function has be be
sampled in both coordinates and amplitudes
● Digitizing the coordinate values is called sampling
● Digitizing the amplitude values is called quantization
● The quality of a digital image is determined to a large degree by the
no. of samples (M in x direction and N in y direction) and discrete
gray levels (L) used in sampling and quantization
CCST 9049 13
CCST 9049 14
CCST 9049 15

Representing Digital Images

● The result of sampling and quantization is a M×N matrix of real
numbers
● By using integer values for the discrete coordinates, the resulting
digital image can be written as:

● Each element of the above matrix is called a pixel

CCST 9049 16
CCST 9049 17
CCST 9049 18

Resolutions of a Digital Image

● Spatial resolution depends on the no. of samples used in sampling
● Gray-level resolution depends on the no. of discrete gray levels
used in quantization
● A L-level digital image of size M × N is said to have a spatial
resolution of M × N pixels and a gray-level resolution of L levels
( i.e., [ 0, L−1 ] )
● Due to hardware considerations, no. of gray levels used is typically
an integer power of 2, i.e., L = 2k, and the image is commonly
referred to as a “k-bit image”
● Storage requirement
○ b = M × N × k bits and if M = N, b = N2 × k bits
CCST 9049 19

Quiz 1
● Consider the Canon 5D Mark IV
digital camera
● The camera produces a picture with a
spatial resolution of 6,720 x 4,480
and a gray-level resolution of 16,384
levels
● Calculate the storage requirement in
bits
○ b=?
Image Sensing and Acquisition

What about color images?

CCST 9049 21

Color Sensing: Bayer Grid

Estimate RGB
values from
neighboring
values
CCST 9049 22

B
CCST 9049 23
CCST 9049 24

Enter the following code on this site

im.setGreen(x, y, 0);
im.setBlue(x, y, 0);
red = im.getRed(x, y);
im.setRed(x, y, red*10);
CCST 9049 25

Quiz 2
● You would like to store a photo taken
with your Canon 5D Mark IV on
your computer
● Your camera produces a picture with
a spatial resolution of 6,720 x 4,480
and a color resolution of 16,384
levels per color channel
○ You may assume that 3 color
channels (Red, Green, Blue) are
used
● Calculate the storage requirement in
bits
○ b=?
CCST 9049 26

Quiz 3
● You would like to store a photo taken
with your Canon 5D Mark IV on
your computer
● Your camera produces a picture with
a spatial resolution of 6,720 x 4,480
and a color resolution of 16,384
levels per color channel
○ You may assume that 3 color
channels are used
● 60 frames (images) per second
○ 60 fps
● Calculate the storage requirement for
a 1 minute video in bits
○ b=?
CCST 9049 27

Saturation and Noise

CCST 9049 28

Sampling Checkerboards
● Insufficient no. samples

1250 PPI 300 PPI 150 PPI 72 PPI

CCST 9049 29

False Contouring
● Insufficient no. of gray levels

k=8 k=7 k=6 k=5

k=4 k=3 k=2 k=1

CCST 9049 30

Spatial Domain Process

● f(x, y) is the input image(s)

● g(x, y) is the output image
● T is an operation on f, defined over
some neighborhood of (x, y)
● T utilizes only the pixels in the area
of the image spanned by the
neighborhood of (x, y) to produce g(x, y)
● The neighborhood shapes are usually square or rectangular due to
the ease of implementation
CCST 9049 31

Gray Level Transformation

● Simplest form: T with a neighborhood of size 1 x 1
s and r denote intensity
of g and f at any point (x,y)
● Referred to as point processing since the enhancement at any point in an image
depends only on that point
● For digital images, the mapping from r to s can be implemented as a lookup-table
using a 1D array
CCST 9049 32

Image Negative

● Produces the equivalent of a photographic negative

● For enhancing white or gray details embedded in dark regions

input output
CCST 9049 33

Piecewise-Linear Transformation
● Advantage:
○ The form of the piecewise functions can be arbitrarily complex
● Disadvantage:
○ Specification requires considerably more user input
● Examples:
○ Contrast stretching
○ Gray-level slicing
CCST 9049 34

Contrast Stretching
● Increases the dynamic
range of the gray levels

input
CCST 9049 35

Gray-Level Slicing
● Highlights a specific range of
gray levels

input
CCST 9049 36

Histogram
● The histogram of a digital image with grey levels in the range [0, L−1] is a
discrete function

where rk is the k-th grey level and nk is the number of pixels in the image having
grey level rk
● A normalized histogram is given by

● p(rk) gives an estimate of the probability of occurrence of grey level r k, and

Code
Code
CCST 9049 38

Task
● Consider the two 3-bit images shown below
○ There is a black border around the image that is not part of the image
● Let the images be of the same spatial dimensions
● Suppose that the white pixels in both images are of intensity 7 and that the black
pixels are of intensity 0
● Draw the histogram of the two images
CCST 9049 40

Cumulative Histogram
● A cumulative histogram counts the number of intensities up to the
specified intensity

Code
CCST 9049 43

Histogram Equalization
● A processed (output) image is obtained by mapping each pixel in
the input image with intensity rk into a corresponding pixel with
level sk in the output image using
Code
Code
CCST 9049 46

Histogram Equalization
CCST 9049 47

Fundamentals of Spatial Filtering

● A spatial filter consists of
○ a neighborhood, and
○ a predefined operation that is
performed on the image pixels
encompassed by the neighborhood
● Filtering creates a new pixel with
coordinates equal to the coordinates of the
center of the neighborhood
● A processed (filtered) image is generated
as the center of the filter visits each pixel in
the input image
CCST 9049 48

Example: Box Filter

1 1 1

1 1 1

1 1 1
CCST 9049 49

1 1 1

1 1 1

1 1 1

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0
0 0
0 0
0 90
90 0
0 90
90 90
90 90
90 0
0 0
0

0
0 0
0 0
0 90
90 90
90 90
90 90
90 90
90 0
0 0
0

0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0

0
0 0
0 90
90 0
0 0
0 0
0 0
0 0
0 0
0 0
0

0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0
CCST 9049 50

1 1 1

1 1 1

1 1 1

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0
CCST 9049 51

1 1 1

1 1 1

1 1 1

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0
CCST 9049 52

1 1 1

1 1 1

1 1 1

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0
CCST 9049 53

1 1 1

1 1 1

1 1 1

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30 30

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0
CCST 9049 54

1 1 1

1 1 1

1 1 1

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30 30

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0
?
0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0
CCST 9049 55

1 1 1

1 1 1

1 1 1

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30 30

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0
?
0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0 50

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0
CCST 9049 56

1 1 1

1 1 1

1 1 1

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10

0 0 0 90 90 90 90 90 0 0 0 20 40 60 60 60 40 20

0 0 0 90 90 90 90 90 0 0 0 30 60 90 90 90 60 30

0 0 0 90 90 90 90 90 0 0 0 30 50 80 80 90 60 30

0 0 0 90 0 90 90 90 0 0 0 30 50 80 80 90 60 30

0 0 0 90 90 90 90 90 0 0 0 20 30 50 50 60 40 20

0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 30 20 10

0 0 90 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0
CCST 9049 57

Example: Box Filter

What does it do?

1 1 1
• Replaces each pixel with an
average of its neighborhood 1 1 1

1 1 1
• Achieve smoothing effect
(remove sharp features)

• Why does it sum to one?

CCST 9049 58

Example: Box Filter

CCST 9049 59
1 1 1 1 1
Quiz 4 1 1 1 1 1
1 1 1 1 1
1 1 1 1 1

0 0 0 0 0 0 0 0 0 0 1 1 1 1 1
0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0
0 0
0 0
0 90
90 0
0 90
90 90
90 90
90 0
0 0
0

0
0 0
0 0
0 90
90 90
90 90
90 90
90 90
90 0
0 0
0

0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0

0
0 0
0 90
90 0
0 0
0 0
0 0
0 0
0 0
0 0
0

0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0
CCST 9049 60

Task
0 0 0
1. 0 1 0
0 0 0
0 0 0
2. 0 0 1
0 0 0
1 0 -1
3. 2 0 -2
1 0 -1
0 0 0 1 1 1
4. 0 2 0
0 0 0 - 1 1 1
1 1 1
CCST 9049 61

?
0 0 0
0 1 0
0 0 0

Original
CCST 9049 62

0 0 0
0 1 0
0 0 0

Original Filtered
(no change)
CCST 9049 63

?
0 0 0
0 0 1
0 0 0

Original
CCST 9049 64

0 0 0
0 0 1
0 0 0

Original Shifted left

By 1 pixel
CCST 9049 65

1 0 -1
2 0 -2
1 0 -1
Sobel

Vertical Edge
(absolute value)
CCST 9049 66

1 2 1
0 0 0
-1 -2 -1
Sobel

Horizontal Edge
(absolute value)
CCST 9049 67

-
0 0 0 1 1 1
0
0
2
0
0
0
1
1
1
1
1
1
?
(Note that filter sums to 1)
Original
CCST 9049 68

Unsharp masking

-
0 0 0 1 1 1
0 2 0 1 1 1
0 0 0 1 1 1

Original Sharpening filter

- Accentuates differences with local
average
CCST 9049 69

Smoothing Filters
● Used for blurring
○ Removal of small details from an image prior to object
extraction
○ Bridging of small gaps in lines of curves
○ Create beautiful graphical user interfaces
● Used for noise reduction
○ Removal of random noise in an image
○ Accomplished by blurring with a linear filter and also by
nonlinear filtering
CCST 9049 70

Smoothing Filters

3x3
box filter

5x5
9x9

weighted
average

15x15 35x35
box filter
CCST 9049 71

References
● Digital Image Processing, Gonzalez and Woods, 3rd Edition
● CSCI 1430, James Tompkin, Brown University