photo by unsplash user @maickmacielfotos

BBM444
FUNDAMENTALS OF
COMPUTATIONAL
PHOTOGRAPHY
Lecture #04 – Exposure and High-Dynamic-Range Imaging
Erkut Erdem // Hacettepe University // Spring 2022
Today’s Lecture
• Controlling exposure
• High-dynamic-range imaging
• Tonemapping

Disclaimer: The material and slides for this lecture were borrowed from
—Ioannis Gkioulekas’ 15-463/15-663/15-862 “Computational Photography” class
—Wojciech Jarosz’s CS 89.15/189.5 “Computational Aspects of Digital Photography” class
—Derek Hoiem’s CS 498 “Computational Photography” class
2
Light, exposure and dynamic range
• Exposure: how bright is the scene overall?
• Dynamic range: contrast in the scene
• ratio of brightest to darkest intensity

3
Today’s Lecture
• Controlling exposure
• High-dynamic-range imaging
• Tonemapping

4
Exposure control

5
What is exposure?
Roughly speaking, the “brightness” of a captured image given a fixed scene.

Exposure = Gain x Flux x Time

• Flux is controlled by the aperture.

• Time is controlled by the shutter speed.
• Gain is controlled by the ISO.

6
Exposure controls brightness of image

Aperture

Exposure

Shutter ISO
7
Exposure controls brightness of image

Aperture

Exposure

Shutter ISO
8
Shutter speed
Controls the length of time that shutter remains open.

incoming
light

shutter

sensor
closed shutter

9
Shutter speed
Controls the length of time that shutter remains open.

incoming
light

shutter

sensor
open shutter

10
Shutter speed

11
Shutter speed
Controls the period of time that shutter remains open.

incoming
light

shutter

sensor
open shutter
What happens to the image as we increase shutter speed?
12
Side-effects of shutter speed
Moving scene elements appear blurry.

How can we “simulate” decreasing the shutter speed?

13
Motion deblurring

Shah et al. High-quality Motion Deblurring from a Single Image, SIGGRAPH 2008 14
Exposure controls brightness of image

Aperture

Exposure

Shutter ISO
15
Aperture size
Controls area of lens that lets light pass through.

Also determines
circle of confusion.

in-focus object distance S focal length f

actual object distance O sensor distance S’

16
Aperture size
Controls area of lens that lets light pass through.

Also determines
circle of confusion.

in-focus object distance S focal length f

actual object distance O sensor distance S’

17
Aperture size
Most lenses have apertures of variable size.
• The size of the aperture is expressed as the “f-number”: The bigger this
number, the smaller the aperture.

f / 1.4 f / 2.8 f/4 f/8 f / 16

You can see the aperture by removing the lens and looking inside it.

18
Side-effects of aperture size
Depth of field decreases as aperture size increases.
• Having a very sharp depth of field is known as “bokeh”.

19
How can we simulate bokeh?

20
How can we simulate bokeh?
Infer per-pixel depth, then blur with depth-dependent kernel.
• Example: Google camera “lens blur” feature

Barron et al., “Fast Bilateral-Space Stereo for Synthetic Defocus,” CVPR 2015 21
 Back plane focus, close-up Back planeTraining with triplets
focus, close-up Training w
How
(3C)can
Effectwe simulate
of photo triplets (3C) bokeh?
in training. Wephoto
Effect of compare the in
triplets disparity
training.map
We co
ames and shallow DOF rendering and using networks
shallow DOFtrained w and
rendering usingw networks
triplet tra
Employ a learning-based strategy, i.e. an image-to-image translation model
consistency. The results withconsistency.
triplet consistency are geometrically
The results more
with triplet consistency
accurate in the background region of high
accurate disparity.
in the background region of high dispar

close-up

Zhang et al., “Synthetic Defocus and Look-Ahead Autofocus for Casual Videography,” SIGGRAPH 2019 22
Exposure controls brightness of image

Aperture

Exposure

Shutter ISO
23
The (in-camera) image processing pipeline
The sequence of image processing operations applied by the camera’s image signal
processor (ISP) to convert a RAW image into a “conventional” image.

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
24
Analog front-end

analog analog discrete discrete

voltage voltage signal signal

analog amplifier (gain): look-up table (LUT):

analog-to-digital
• gets voltage in range • corrects non-linearities in
converter (ADC):
needed by A/D converter. sensor’s response
• depending on sensor,
• accommodates ISO function (within proper
output has 10-16 bits.
settings. exposure).
• most often (?) 12 bits.
• accounts for vignetting. • corrects defective pixels.
25
Side-effects of increasing ISO
Image becomes very grainy because noise is amplified.

26
Note about the name ISO

ISO is not an acronym.

• It refers to the International Organization for Standardization.
• ISO comes from the Greek word ίσος, which means equal.
• It is pronounced (roughly) eye-zo, and should not be spelled out.

27
 Camera modes
Aperture priority (“A”): you set aperture, camera sets everything else.
• Pros: Direct depth of field control.
• Cons: Can require impossible shutter speed (e.g. with f/1.4 for a bright scene).

Shutter speed priority (“S”): you set shutter speed, camera sets everything else.
• Pros: Direct motion blur control.
• Cons: Can require impossible aperture (e.g. when requesting a 1/1000 speed for a dark
scene)

Automatic (“AUTO”): camera sets everything.

• Pros: Very fast, requires no experience.
• Cons: No control.

Manual (“M”): you set everything.

• Pros: Full control.
• Cons: Very slow, requires a lot of experience.
generic camera mode dial
28
 Camera modes
Aperture priority (“A”): you set aperture, camera sets everything else.
• Pros: Direct depth of field control.
• Cons: Can require impossible shutter speed (e.g. with f/1.4 for a bright scene).

Shutter speed priority (“S”): you set shutter speed, camera sets everything else.
• Pros: Direct motion blur control.
• Cons: Can require impossible aperture (e.g. when requesting a 1/1000 speed for a dark
scene)

Automatic (“AUTO”): camera sets everything.

• Pros: Very fast, requires no experience.
• Cons: No control.

Manual (“M”): you set everything.

• Pros: Full control.
• Cons: Very slow, requires a lot of experience.
generic camera mode dial
29
Light metering

30
 Light metering in modern cameras

• SLR cameras use a separate low-

resolution sensor that is placed at the
focusing screen.

• Mirrorless cameras use measurements

directly from the main sensor.
31
 Light metering in modern cameras
• Measurements are averaged to produce a single intensity estimate, which is assumed to
correspond to a scene of 18% reflectance (the “key”).

• Exposure is set so that this average is exposed at the middle of the sensor’s dynamic
range.

• Averaging can be done in many ways:

1. Center-weighed.

2. Spot.

3. Scene-specific preset
(portrait, landscape, horizon).

4. “Intelligently” using proprietary algorithm.

32
Metering challenges: low resolution
Low-resolution can make it difficult to correctly meter the scene and set exposure.
• In which of these scenes is it OK to let the brightest pixels be overexposed?

33
Metering challenges: low resolution
Low-resolution can make it difficult to correctly meter the scene and set exposure.
• In which of these scenes is it OK to let the brightest pixels be overexposed?

34
Today’s Lecture
• Controlling exposure
• High-dynamic-range imaging
• Tonemapping

35
Light, exposure and dynamic range
• Exposure: how bright is the scene overall?
• Dynamic range: contrast in the scene
• ratio of brightest to darkest intensity

36
Our devices do not match the world

37
 The world has a high dynamic range

1500

25,000 2,000,000,000

400,000
Relative brightness of different scenes, ranging from 1 inside
a dark room lit by a monitor to 2,000,000 looking at the Sun. 38
The world has a high dynamic range

10-6 106
common real-world scenes

adaptation range of our eyes

39
(Digital) sensors also have a low dynamic range

10-6 106
sensor

10-6 106
common real-world scenes

adaptation range of our eyes

40
(Digital) images have an even lower dynamic range
high exposure

10-6 106
image

10-6 106
common real-world scenes

adaptation range of our eyes

41
(Digital) images have an even lower dynamic range
low exposure

10-6 106
image

10-6 106
common real-world scenes

adaptation range of our eyes

42
 Outside is too bright
The dynamic range challenge
Sun overexposed Inside is istoo dark
Examples
Sun overexposed
Inside
Outside is is
Outside too
too dark
toobright
bright
Foreground too dark
Foreground too dark
Sun overexposed
Foreground too dark
After a slide by Frédo Durand

43
Low Dynamic
Low Dynamic Range
Range (LDR)
(LDR)
Longer exposure Shorter exposure

[Durand and Dorsey 02] [Durand and Dorsey 02]

✓ detail
detail ininshadows
shadows ✓ detail
detail ininhighlights
highlights
✗ clipped highlights
clipped highlights ✗ noisy/clipped shadows
noisy/clipped shadows
5 44
(Digital) images have an even lower dynamic range
high exposure

10-6 106
image

10-6 106
common real-world scenes

adaptation range of our eyes

45
(Digital) images have an even lower dynamic range
low exposure

10-6 106
image

10-6 106
common real-world scenes

adaptation range of our eyes

46
(Digital) images have an even lower dynamic range
Any guesses about the dynamic range of a standard 0-255 image?

pure black pure white

47
(Digital) images have an even lower dynamic range
Any guesses about the dynamic range of a standard 0-255 image?

about 50x
brighter

pure black pure white

48
 Our devices do not match the real world
• 10:1 photographic print (higher for glossy paper)
• 20:1 artist's paints
• 200:1 slide film
• 500:1 negative film
• 1000:1 LCD display
• 2000:1 digital SLR (at 12 bits)
• 100000:1 real world
Two challenges:
1. HDR imaging – which parts of the world do we measure in the 8-14 bits available to our sensor?
2. Tonemapping – which parts of the world do we show in the 4-10 bits available to our display?

49
 Our devices do not match the real world
• 10:1 photographic print (higher for glossy paper)
• 20:1 artist's paints
• 200:1 slide film
• 500:1 negative film
• 1000:1 LCD display
• 2000:1 digital SLR (at 12 bits)
HDR imaging and tonemapping are distinct
• 100000:1 real world techniques with different goals
Two challenges: HDR imaging compensates for sensor limitations
1. HDR imaging – which parts of the world do we measure in the 8-14 bits available to our sensor?
2. Tonemapping – which parts of the world do we show in the 4-10 bits available to our display?

Tonemapping compensates for display limitations 50

High dynamic range imaging

51
52
53
54
55
Slide credits

56
57
 Key idea
1. Exposure bracketing: Capture multiple LDR images at different exposures

2. Merging: Combine them into a single HDR image

58
 Application: Photography

59
“Sunset from Rigi Kaltbad” [Wojciech Jarosz 2014]
 60
“Sunset from Rigi Kaltbad” [Wojciech Jarosz 2014]
 61
“Camogli Lighthouse” [Wojciech Jarosz 2012]
 62
“Camogli Lighthouse” [Wojciech Jarosz 2012]
 63
“Florence” [Wojciech Jarosz 2011]
 64
“Florence” [Wojciech Jarosz 2011]
 65
“Matterhorn and Riffelsee” [Wojciech Jarosz 2010]
 66
“Matterhorn and Riffelsee” [Wojciech Jarosz 2010]
 Key idea
1. Exposure bracketing: Capture multiple LDR images at different exposures

2. Merging: Combine them into a single HDR image

67
Multiple exposure photography

low contrast
10-6 106

image

10-6 106
real
high dynamic range world
68
Multiple exposure photography

low contrast
10-6 106

image

10-6 106
real
high dynamic range world
69
Multiple exposure photography

low contrast
10-6 106

image

10-6 106
real
high dynamic range world
70
Multiple exposure photography

low contrast
10-6 106

image

10-6 106
real
high dynamic range world
71
Multiple exposure photography

low contrast
10-6 106

image

10-6 106
real
high dynamic range world
72
Multiple exposure photography

low contrast
10-6 106

image

10-6 106
real
high dynamic range world
73
Ways to vary exposure
1. Shutter speed

2. F-stop (aperture, iris)

3. ISO

4. Neutral density (ND) filters

Pros and cons of each for HDR?

74
Ways to vary exposure
1. Shutter speed
– Range: about 30 sec to 1/4000 sec (6 orders of magnitude)
– Pros: repeatable, linear
– Cons: noise and motion blur for long exposure
2. F-stop (aperture, iris)
– Range: about f/0.98 to f/22 (3 orders of magnitude)
– Pros: fully optical, no noise
– Cons: changes depth of field
3. ISO
– Range: about 100 to 1600 (1.5 orders of magnitude)
– Pros: no movement at all
– Cons: noise
4. Neutral density (ND) filters
– Range: up to 6 densities (6 orders of magnitude)
– Pros: works with strobe/flash
– Cons: not perfectly neutral (color shift), extra glass (interreflections, aberrations),
need to touch camera (shake)
75
Exposure bracketing with shutter speed
Note: shutter times usually obey a power series – each “stop” is a factor of 2
1/4, 1/8, 1/15, 1/30, 1/60, 1/125, 1/250, 1/500, 1/1000 sec
usually really is
1/4, 1/8, 1/16, 1/32, 1/64, 1/128, 1/256, 1/512, 1/1024 sec

Questions:
1. How many exposures?
2. What exposures?

76
Exposure bracketing with shutter speed
Note: shutter times usually obey a power series – each “stop” is a factor of 2
1/4, 1/8, 1/15, 1/30, 1/60, 1/125, 1/250, 1/500, 1/1000 sec
usually really is
1/4, 1/8, 1/16, 1/32, 1/64, 1/128, 1/256, 1/512, 1/1024 sec

Questions:
1. How many exposures?
2. What exposures?

Answer: Depends on the scene, but a good default is 5 exposures,

the metered exposure and +/- 2 stops around that.

77
 Key idea
1. Exposure bracketing: Capture multiple LDR images at different exposures

2. Merging: Combine them into a single HDR image

78
The image processing pipeline
The sequence of image processing operations applied by the camera’s image signal
processor (ISP) to convert a RAW image into a “conventional” image.

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
79
The image processing pipeline
The sequence of image processing operations applied by the camera’s image signal
processor (ISP) to convert a RAW image into a “conventional” image.

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
80
RAW images have a linear response curve
Colorchecker: Great tool for radiometric
and color calibration.

when not over/under exposed

Patches at bottom row have log-

reflectance that increases linearly.
81
 82

Over/under exposure
in highlights we are
limited by clipping

in shadows we are
limited by noise
RAW (linear) image formation model
Real scene flux for image pixel (x,y): Φ(x, y)
Exposure time:
t5 t4 t3 t2 t1

What is an expression for the image Ilinear(x,y) as a function of Φ(x,y)?

83
RAW (linear) image formation model
Real scene flux for image pixel (x,y): Φ(x, y)
Exposure time:
t5Scene radiance t4 Φ(x,y) reachest3 t2 at a pixel x, ty
the sensor 1

For each image I,

- radiance gets multiplied by exposure factor ti
(depends on shutter speed, aperture, ISO)
- noise gets added
- values above 1 get clipped
(depends on photosite well capacity)

What is an expression for the image Ilinear(x,y) as a function of Φ(x,y)?

84
RAW (linear) image formation model
Real scene flux for image pixel (x,y): Φ(x, y)
Exposure time:
t5 t4 t3 t2 t1

What is an expression for the image Ilinear(x,y) as a function of Φ(x,y)?

Ilinear(x,y) = clip[ ti ⋅ Φ(x,y) + noise ]

85
RAW (linear) image formation model
Real scene flux for image pixel (x,y): Φ(x, y)
Exposure time:
t5 t4 t3 t2 t1

What is an expression for the image Ilinear(x,y) as a function of Φ(x,y)?

Ilinear(x,y) = clip[ ti ⋅ Φ(x,y) + noise ]
How would you merge these images into an HDR one? 86
2-image example
2-image example
Simple in principle:
Simple in principle:
- -imageA = 1/30th
imageA = 1/30th secondsecond
(“brighter”(“brighter”
image) image)
- -imageB = 1/120th
imageB = 1/120th secondsecond (“darker” image)
(“darker” image)

- -imageHDR = average(4·imageB,
imageHDR = average(4·imageB, remove-clipped(imageA))
remove-clipped(imageA))
- assumes images have been linearized
- assumes images have been linearized

imageA imageB imageHDR 87

Merging RAW (linear) exposure stacks
For each pixel:

1. Find “valid” images How would you

implement steps 1-2?
2. Weight valid pixel values appropriately

3. Form a new pixel value as the weighted average of valid pixel values
t5 t4 t3 t2 t1

88
Merging RAW (linear) exposure stacks
For each pixel:

1. Find “valid” images (noise) 0.05 < pixel < 0.95 (clipping)

2. Weight valid pixel values appropriately noise

valid
3. Form a new pixel value as the weighted average of valid pixel values clipped
t5 t4 t3 t2 t1

89
Merging RAW (linear) exposure stacks
For each pixel:

1. Find “valid” images (noise) 0.05 < pixel < 0.95 (clipping)

2. Weight valid pixel values appropriately (pixel value) / ti

3. Form a new pixel value as the weighted average of valid pixel values
t5 t4 t3 t2 t1

90
Merging result (after tonemapping)

91
What if I cannot use raw?

92
Radiometric calibration

93
The image processing pipeline
• Can you foresee any problem when we switch from RAW to rendered images?

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
94
The image processing pipeline
• Can you foresee any problem when we switch from RAW to rendered images?
• How do we deal with the nonlinearities?

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
95
Radiometric calibration
The process of measuring the camera’s response curve. Can be done in three ways:

• Take images of scenes with different flux while keeping exposure the same.

• Take images under different exposures while keeping flux the same.

• Take images of scenes with different flux and under different exposures.

96
Same camera exposure, varying scene flux
Colorchecker: Great tool for radiometric
and color calibration.

Patches at bottom row have log- Different values correspond to patches

reflectance that increases linearly. of increasing reflected flux.
97
Same scene flux, varying camera exposure
White balance card: Great tool for white
balancing and radiometric calibration.

known exposure
Different values correspond to images
All points on (the white part of) the
taken under increasing camera
target have the same reflectance.
exposure.
98
Varying both scene flux and camera exposure
You can do this using the LDR exposure stack itself.

Different scene flux, same camera exposure

Same scene flux, different camera exposure

99
 Non-linear image formation model
Real scene flux for image pixel (x,y): Φ(x, y)
Exposure time: ti

Ilinear(x,y) = clip[ ti ⋅ Φ(x,y) + noise ]

Inon-linear(x,y) = f[ Ilinear(x,y) ]

How would you merge the non-linear images into an HDR one?
100
 Non-linear image formation model
Real scene flux for image pixel (x,y): Φ(x, y)
Exposure time: ti

Ilinear(x,y) = clip[ ti ⋅ Φ(x,y) + noise ]

Inon-linear(x,y) = f[ Ilinear(x,y) ] Iest(x,y) = f-1[ Inon-linear(x,y) ]

Use inverse transform to estimate linear image, then proceed as before

101
 Linearization

Inon-linear(x,y) = f[ Ilinear(x,y) ]

Iest(x,y) = f-1[ Inon-linear(x,y) ]

102
Merging non-linear exposure stacks
1. Calibrate response curve

2. Linearize images

For each pixel:

3. Find “valid” images (noise) 0.05 < pixel < 0.95 (clipping)

4. Weight valid pixel values appropriately (pixel value) / ti

5. Form a new pixel value as the weighted average of valid pixel values

Same steps as in the RAW case.

103
What if I cannot measure the response curve?

104
You may find information in the image itself

If you cannot do calibration, take a look at the image’s

EXIF data (if available).

Often contains information about tone reproduction

curve and color space.

105
Tone reproduction curves
The exact tone reproduction curve depends on the camera.
• Often well approximated as Lγ, for different values of the power γ (“gamma”).
• A good default is γ = 1 / 2.2.

before gamma after gamma

If nothing else, take the square of your image to approximately remove effect of tone
reproduction curve. 106
What if I cannot measure the response curve?
• Predict an approximated camera response function from the observed images.

107
The Approach
• Get pixel values Zij for image with shutter time Δtj (ith pixel location, jth image)
• Exposure is radiance integrated over time:
Eij = Ri × Dt j Þ ln Eij = ln Ri + ln Dt j
• To recover radiance Ri, we must map pixel values to log exposure: ln(Eij)= g(Zij)

• Solve for R, g by minimizing:

[ ]
N P Z max

åå w( Zij ) ln Ri + ln Dt j - g ( Zij ) + l
i =1 j =1
2
å w( z )
z = Z min
g ¢¢( z ) 2

108
The objective
Solve for radiance R and mapping g for each of 256 pixel values to
minimize:

åå w( Z )[ln R + ln Dt - g ( Z )] + l å w( z ) g ¢¢( z )
N P Z max
2 2
ij i j ij
i =1 j =1 z = Z min

give pixels near 0 known shutter exposure should smoothly

or 255 less weight time for image j increase as pixel intensity
increases

radiance at particular exposure, as a function of

pixel site is the same pixel value
for each image

109
The Math
• Let g(z) be the discrete inverse response function
• For each pixel site i in each image j, want:

ln Radiancei +ln Dt j = g(Zij )

• Solve the overdetermined linear system:

åå [ ]
N P Z max
ln Radiancei + ln Dt j - g ( Zij ) + l å g ( z )
¢¢
2 2

i =1 j =1 z = Z min

fitting term smoothness term

110
 [4] FAUGERAS , O. Three-Dimensional Computer Vision. MIT
Press, 1993.

Matlab Code
%
[5] F ERWERDA , J. A., PATTANAIK , S. N., S HIRLEY, P., AND %
%
gsolve.m − Solve for imaging system response function

G REENBERG , D. P. A model of visual adaptation for realistic %

%
Given a set of pixel values observed for several pixels in several
images with different exposure times, this function returns the
image synthesis. In SIGGRAPH ’96 (1996), pp. 249–258. %
%
imaging system’s response function g as well as the log film irradiance
values for the observed pixels.
%
[6] G ORTLER , S. J., G RZESZCZUK , R., S ZELISKI , R., AND C O - % Assumes:
%
HEN , M. F. The Lumigraph. In SIGGRAPH ’96 (1996), % Zmin = 0
% Zmax = 255

• 21 lines of code!
pp. 43–54.
[7] H ORN , B. K. P. Robot Vision. MIT Press, Cambridge, Mass.,
%
%
%
%
Arguments:

Z(i,j) is the pixel values of pixel location number i in image j

% B(j) is the log delta t, or log shutter speed, for image j
1986, ch. 10, pp. 206–208. % l is lamdba, the constant that determines the amount of smoothness
% w(z) is the weighting function value for pixel value z
[8] J AMES , T., Ed. The Theory of the Photographic Process. %
% Returns:
Macmillan, New York, 1977. %
% g(z) is the log exposure corresponding to pixel value z
% lE(i) is the log film irradiance at pixel location i
[9] K AUFMAN , J. E., Ed. IES Lighting Handbook; the standard %

lighting guide, 7th ed. Illuminating Engineering Society, New function [g,lE]=gsolve(Z,B,l,w)

York, 1987, p. 24. n = 256;

A = zeros(size(Z,1)*size(Z,2)+n+1,n+size(Z,1));
[10] K OLB , C., M ITCHELL , D., AND H ANRAHAN , P. A realis- b = zeros(size(A,1),1);

tic camera model for computer graphics. In SIGGRAPH ’95 %% Include the data−fitting equations

(1995). k = 1;
for i=1:size(Z,1)
for j=1:size(Z,2)
[11] L AVEAU , S., AND FAUGERAS , O. 3-D scene representation wij = w(Z(i,j)+1);
A(k,Z(i,j)+1) = wij; A(k,n+i) = −wij; b(k,1) = wij * B(i,j);
as a collection of images. In Proceedings of 12th International k=k+1;
end
Conference on Pattern Recognition (1994), vol. 1, pp. 689– end
691. %% Fix the curve by setting its middle value to 0

[12] L EVOY, M., AND H ANRAHAN , P. Light field rendering. In A(k,129) = 1;

k=k+1;
SIGGRAPH ’96 (1996), pp. 31–42. %% Include the smoothness equations

[13] M ADDEN , B. C. Extended intensity range imaging. Tech. for i=1:n−2

A(k,i)=l*w(i+1); A(k,i+1)=−2*l*w(i+1); A(k,i+2)=l*w(i+1);
rep., GRASP Laboratory, University of Pennsylvania, 1993. k=k+1;
end
[14] M ANN , S., AND P ICARD , R. W. Being ’undigital’ with dig- %% Solve the system using SVD

ital cameras: Extending dynamic range by combining differ- x = A\b;

ently exposed pictures. In Proceedings of IS&T 46th annual g = x(1:n);

lE = x(n+1:size(x,1));
conference (May 1995), pp. 422–428.
[15] M C M ILLAN , L., AND B ISHOP, G. Plenoptic Modeling: An 111
Illustration
Exposure stack

•1 •1 •1 •1 •1

•2 •2 •2 •2 •2

•3 •3 •3 •3 •3

∆t = ∆t = ∆t = ∆t = ∆t =
1/64 sec 1/16 sec 1/4 sec 1 sec 4 sec

Pixel Value Z = f(Exposure)

Exposure = Radiance * Dt
log Exposure = log Radiance + log Dt
112
 Response Curve
Computer Vision: Algorithms and Applications, 2nd ed. (final draft, S
Assuming unit radiance After adjusting radiances to obtain
for each pixel a smooth response curve

3
value

value
Pixel value
Pixel value

Pixel
Pixel

log
log Exposure
Exposure log
log Exposure
Exposure
113
scaling terms can be adjusted by photographing a cali-
naire of known color.
Response Curve
ng virtual photographs
ed response functions can also be used to map radiance Figure 3: (a) Eleven grayscale photographs of an indoor scene ac
to pixel values for a given exposure using Equa- quired with a Kodak DCS460 digital camera, with shutter speed
process can be thought Kodak DCS460
of as taking a virtual photograph progressing in 1-stop increments from of a second to 30 seconds
Recovered response curve
ce map, in that the resulting
1/30image
to 30willsecexhibit the re-
ties of the modeled imaging system. Note that the re-
hree un- 250
ions used need not be the same response functions used
ute radi-
the original radiance map, which allows photographs
of these
ce
h map.
one imaging process to be rendered as if they were 200
ch that a 9
hixelanother.
with

value
ance val-

pixel value Z
c. If the 150
ults
qually to
rocedure

Pixel
how the results of using our algorithm to determine the
matically 100
ent light.
ve ofbea DCS460 digital camera. Eleven grayscale pho-
hould
ered down to
o a radi- resolution (Fig. 3) were taken at
the color 50
osure times ranging from
film re-
of a second to 30 seconds,
mage receiving twice the exposure of the previous one.
g a cali-
ve recovered by our algorithm from 45 pixel locations 0
−10 −5 0 5
oss the image sequence is shown in Fig. 4. Note that al- log exposure X
image arrays
radiance Figurenaturally
3: (a) Elevenproduce linear output,
grayscale photographs from
of an indoor theac-
scene log Exposure
quired with a Kodak DCS460 digital camera, with shutter speeds Figure 4: The response function of the DCS460 recovered by our al
ident
ng Equa-that the camera nonlinearly remaps the data, pre- 114
 However, films are usually calibrated to respond achromatically
to a particular color of light , such as sunlight or fluorescent light.
623
In this case, the radiance values of the three channels should be

Radiance Image
scaled so that the pixel value maps to a radi-
ance with the same color ratios as . To properly model the color
response of the entire imaging process rather than just the film re-
sponse, the scaling terms can be adjusted by photographing a cali-
bration luminaire of known color.

2.7 Taking virtual photographs

The recovered response functions can also be used to map radiance Figure 3: (a) Eleven grayscale photographs of an indoor scene
values back to pixel values for a given exposure using Equa- quired with a Kodak DCS460 digital camera, with shutter spee
tion 1. This process can be thought of as taking a virtual photograph progressing in 1-stop increments from of a second to 30 secon
of the radiance map, in that the resulting image will exhibit the re- Recovered response curve
sponse qualities of the modeled imaging system. Note that the re-
250
sponse functions used need not be the same response functions used
to construct the original radiance map, which allows photographs
acquired with one imaging process to be rendered as if they were 200
acquired with another.9

Pixel value
pixel value Z
150
3 Results
Figures 3-5 show the results of using our algorithm to determine the 100
response curve of a DCS460 digital camera. Eleven grayscale pho-
tographs filtered down to resolution (Fig. 3) were taken at
50
f/8 with exposure times ranging from of a second to 30 seconds,
with each image receiving twice the exposure of the previous one.
The film curve recovered by our algorithm from 45 pixel locations 0
−10 −5 0 5
observed across the image sequence is shown in Fig. 4. Note that al- log exposure X
though CCD image arrays naturally produce linear output, from the log
Figure 4: The response Exposure
function of the DCS460 recovered by our
curve it is evident that the camera nonlinearly remaps the data, pre- gorithm, with the underlying data shown as light c
sumably to mimic the response curves found in film. The underlying cles. The logarithm is base .
registered data are shown as light circles underneath
the curve; some outliers are due to sensor artifacts (light horizontal
bands across some of the darker images.)
Fig. 5 shows the reconstructed high dynamic range radiance map. 115
624 Computer Vision: Algorithms and Applications, 2nd ed. (final draft, Sept. 2021)
Radiance Image

Kodak Gold ASA 100, PhotoCD

Figure 10.15 Bracketed set of exposures captured with a film camera and the resulting
116
 Figure 6: Sixteen photographs of a church taken at 1-stop increments from 30 sec to sec. The sun is directly behind the rightmost stained

Recovered Response Curves

Figure 6: Sixteen photographs of a church taken at 1-stop increments from 30 sec to sec. The sun is directly behind the rightmost stained
glass window, making it especially bright. The blue borders seen in some of the image margins are induced by the image registration process.
glass window, making it especially bright. The blue borders seen in some of the image margins are induced by the image registration process.
Red Green
Red Green

250 250

Red Green
250 250

200 200
200 200

value

value
Z Z

Z Z
value

value
value 150 150

value
150 150
pixel

pixel
Pixel

Pixel
pixel

pixel
100 100
100 100

50 50
50 50

0 0
−10
0 −5 0 5 −10
0 −5 0 5
−10 −5 log exposure X 0 5 −10 −5 log exposure X 0 5
log exposure X log exposure X
(a) (b)
log Exposure
(a)
Blue
log Exposure (b)
Red (dashed), Green (solid), and Blue (dash−dotted) curves
Blue Red (dashed), Green (solid), and Blue (dash−dotted) curves

250 250

Blue RGB
250 250

200 200
200 200
value

value
Z Z

Z Z
value

value
150 150
value

value
150 150
Pixel

Pixel
pixel

pixel
pixel

pixel
100 100
100 100

50 50
50 50

0 0
−10
0 −5 0 5 −5
0 −4 −3 −2 −1 0 1 2
−10 −5 log exposure X 0 5 −5 −4 −3 −2 −1X
log exposure 0 1 2

log Exposure
log exposure X
(c)
(c) log Exposure
log exposure X
(d)
(d) 117
Other aspects of HDR imaging

118
Relative vs absolute flux
Final fused HDR image gives flux only up to a global scale
• If we know exact flux at one point, we can convert relative HDR image
to absolute flux map

HDR image spotmeter (absolute absolute

(relative flux) flux at one point) flux map 119
Basic HDR approach
1. Capture multiple LDR images at different exposures

2. Merge them into a single HDR image

Any problems with this approach?

120
Basic HDR approach
1. Capture multiple LDR images at different exposures

2. Merge them into a single HDR image

Problem: Very sensitive to movement

• Scene must be completely static

• Camera must not move

Most modern automatic HDR solutions include an alignment step before merging
exposures

121
 HDR Deghosting
• Family algorithms suggested for eliminating the artefacts
occur due to moving objects/camera and/or dynamic
backgrounds during HDR reconstruction.
• Mostly the motion is compensated by selecting or removing
moving
684 objects and
O. T. Tursun, A. O.finding
Akyüz, A. alignments
Erdem & E. Erdembetween
/ The State ofimages.
the Art in HDR Deghosting: A Survey and Evaluation

m / The State of the Art in HDR Deghosting: A Survey and Evaluation

(a) A moving light source with (b) Non-deformable body motion (c) Deformable body motion (d) Deformable body motion with
high noise with large displacements occlusions

Figure 1: Different types of ghost artifacts.

Table 1: Notation used in this survey Once f is recovered, the HDR value H(p) can be com-
122
HDR Deghosting
Okan Tarhan Tursun1 , Ahmet Oğuz Akyüz1 , Aykut Erdem2 and Erkut Erdem2
• Family algorithms suggested for eliminating the artefacts
1 Dept. of Computer Engineering, Middle East Technical University, Turkey
occur due to moving objects/camera and/or dynamic
2 Dept. of Computer Engineering, Hacettepe University, Turkey
backgrounds during HDR reconstruction.
• Mostly the motion is compensated by selecting or removing
moving objects and finding alignments between images.

123
How do we store HDR images?
• Most standard image formats store integer 8-bit images
• Some image formats store integer 12-bit or 16-bit images
• HDR images are floating point 32-bit or 64-bit images

124
How do we store HDR images?
Use specialized image formats for HDR images
32 bits

portable float map (.pfm)

• very simple to implement

sign exponent mantissa

Radiance format (.hdr)

• supported by Matlab
red green blue exponent

OpenEXR format (.exr)

• multiple extra features

sign exponent mantissa

125
Another type of HDR images
Light probes: place a chrome sphere in the scene and capture an HDR image
• Used to measure real-world illumination environments (“environment maps”)

Application: image-
based relighting

126
Another way to create HDR images

Physics-based renderers simulate flux maps

(relative or absolute)

• Their outputs are very often HDR images

127
 Our devices do not match the real world
• 10:1 photographic print (higher for glossy paper)
• 20:1 artist's paints
• 200:1 slide film
• 500:1 negative film
• 1000:1 LCD display
• 2000:1 digital SLR (at 12 bits)
HDR imaging and tonemapping are distinct
• 100000:1 real world techniques with different goals
Two challenges: HDR imaging compensates for sensor limitations
1. HDR imaging – which parts of the world do we measure in the 8-14 bits available to our sensor?
2. Tonemapping – which parts of the world do we show in the 4-10 bits available to our display?

Tonemapping compensates for display limitations 128

Today’s Lecture
• Controlling exposure
• High-dynamic-range imaging
• Tonemapping

129
Tonemapping

130
How do we display our HDR images?
10-6 106
display

10-6 106
image HDR image

10-6 106
common real-world scenes

adaptation range of our eyes

131
Tonemapping
• Called tone mapping operators
• Two general categories:
• Global (spatially invariant)
• Local (spatially varying)

132
 Linear scaling
Scale image so that maximum value equals 1.

HDR image looks

underexposed because of
the display’s limited
dynamic range, but is not
actually underexposed.
133
 Linear scaling
Scale image so that 10% value equals 1.

Can you think

of something
better?

HDR image looks

saturated because of
the display’s limited
dynamic range, but is not
actually saturated.
134
 Photographic tonemapping
Apply the same non-linear scaling to all pixels in the image so that:
• Bring everything within range → asymptote to 1
• Leave dark areas alone → slope = 1 near 0

I HDR
I display =
1 + I HDR
(exact formula more complicated)

• Photographic because designed to approximate film zone system.

• Perceptually motivated, as it approximates our eye’s response curve.
135
 What is the zone system?
• Technique formulated by Ansel Adams for film development.
• Still used with digital photography.

136
Examples

photographic tonemapping linear scaling (map 10% to 1) linear scaling (map 100% to 1) 137
Compare with LDR images

photographic tonemapping high exposure low exposure 138

Dealing with color
If we tonemap all channels the same, colors are washed out

Can you think of a way to deal with this?

139
 Intensity-only tonemapping

tonemap
intensity
(e.g., luminance
Y in xyY)

leave color the

same (e.g., xy
in xyY)

How would you implement this?

140
Comparison
Color now OK, but some details are washed out due to loss of contrast

Can you think of a way to deal with this?

141
The importance of local contrast
The importance of local contrast

142
Purposes of tone mapping
Technical:
- fitting a wide range of values into a small space while preserving
differences between values as much as possible
Artistic
- reproduce what the photographer/artist feels she saw
- stylize the look of a photo

143
 Low-frequency intensity-only tonemapping
tonemap low-frequency
intensity component

leave high-frequency
intensity component
the same

leave color the same

How would you implement this?

144
Comparison
We got nice color and contrast, but now we’ve run into the halo plague

Can you think of a way to deal with this?

145
Edge-aware filtering and tonemapping
Separate base and detail using
edge-preserving filtering
(e.g., bilateral filtering).

More in later lecture.

146
Comparison
We fixed the halos without losing contrast

147
148
 149

Gradient-domain processing and tonemapping

Compute gradients, scale and merge them, then integrate (solve Poisson problem).
• More in later lecture.
Comparison (which one do you like better?)

photographic bilateral filtering gradient-domain

150
Comparison (which one do you like better?)

photographic bilateral filtering gradient-domain

151
Comparison (which one do you like better?)

There is no ground-truth: which one looks better is entirely subjective

photographic bilateral filtering gradient-domain

152
 Tonemapping for a single image
Modern DSLR sensors capture about 3 stops of dynamic range.
• Tonemap single RAW file instead of using camera’s default rendering.

result from image

tonemapping using
processing pipeline
bilateral filtering
(basic tone
(I think)
reproduction)

153
Tonemapping for a single image
Modern DSLR sensors capture about 3 stops of dynamic range.
• Tonemap single RAW file instead of using camera’s default rendering.

Careful not to “tonemap” noise.

154
Some notes about HDR imaging and tonemapping

155
 Our devices do not match the real world
• 10:1 photographic print (higher for glossy paper)
• 20:1 artist's paints
• 200:1 slide film
• 500:1 negative film
• 1000:1 LCD display
• 2000:1 digital SLR (at 12 bits)
HDR imaging and tonemapping are distinct
• 100000:1 real world techniques with different goals
Two challenges: HDR imaging compensates for sensor limitations
1. HDR imaging – which parts of the world do we measure in the 8-14 bits available to our sensor?
2. Tonemapping – which parts of the world do we show in the 4-10 bits available to our display?

Tonemapping compensates for display limitations 156

A note about terminology
“High-dynamic-range imaging” is used to refer to a lot of different things:
1. Using single RAW images.
2. Performing radiometric calibration.
3. Merging an exposure stack.
4. Tonemapping an image (linear or non-linear, HDR or LDR).
5. Some or all of the above.
Technically, HDR imaging and tonemapping are distinct processes:
• HDR imaging is the process of creating a radiometrically linear image, free of
overexposure and underexposure artifacts. This is achieved using some combination of
1-3, depending on the imaging scenario.
• Tonemapping (step 4) process of mapping the intensity values in an image (linear or non-
linear, HDR or LDR) to the range of tones available in a display.
But:
• In consumer photography, “HDR photography” is often used to refer to both HDR
imaging (steps 1-3) and tonemapping (step 4).
157
Another note about terminology
Tonemapping is just another form of tone reproduction.
• Many ISPs implement the tonemapping algorithms we discussed for tone
reproduction.

analog front-
end
RAW image
(mosaiced,
linear, 12-bit)
CFA white
denoising
demosaicing balance

final RGB
color tone
compression image (non-
transforms reproduction
linear, 8-bit)
158
A note of caution

• HDR photography can produce very visually compelling results.

159
160
161
162
A note of caution

• HDR photography can produce very visually compelling results.

• It is also a very routinely abused technique, resulting in awful results.

163
164
165
166
167
A note of caution

• HDR photography can produce very visually compelling results.

• It is also a very routinely abused technique, resulting in awful results.

• The problem typically is tonemapping, not HDR imaging itself.

168
A note about HDR today

• Most cameras (even phone cameras)

have automatic HDR modes/apps.

• Popular-enough feature that phone

manufacturers are actively competing
about which one has the best HDR.

• The technology behind some of those apps

(e.g., Google’s HDR+) is published in
SIGGRAPH and SIGGRAPH Asia
conferences.

169
Optimal weights for HDR merging

170
Merging non-linear exposure stacks
1. Calibrate response curve

2. Linearize images

For each pixel:

3. Find “valid” images (noise) 0.05 < pixel < 0.95 (clipping)

4. Weight valid pixel values appropriately (pixel value) / ti

5. Form a new pixel value as the weighted average of valid pixel values

Same steps as in the RAW case.

Note: many possible weighting schemes

171
Many possible weighting schemes
“Confidence” that pixel is noisy/clipped
• What are the optimal weights for
merging an exposure stack?

172
RAW (linear) image formation model
(Weighted) radiant flux for image pixel (x,y): α ⋅ Φ(x, y)
Exposure time:
t5 t4 t3 t2 t1

What weights should we use to merge these Different images in the

images, so that the resulting HDR image is an exposure stack will have
optimal estimator of the weighted radiant flux? different noise characteristics
173
 Simple estimation example
We have two independent unbiased estimators x and y of the same quantity I (e.g., pixel
intensity) with variance σ[x]2 and σ[y]2.
What does unbiased mean?

174
 Simple estimation example
We have two independent unbiased estimators x and y of the same quantity I (e.g., pixel
intensity) with variance σ[x]2 and σ[y]2.
What does unbiased mean?
E[x] = E[y] = I
Assume we form a new estimator from the convex combination of the other two:
z = a ⋅ x + (1 – a) ⋅ y
Is the new estimator z unbiased?

175
 Simple estimation example
We have two independent unbiased estimators x and y of the same quantity I (e.g., pixel
intensity) with variance σ[x]2 and σ[y]2.
What does unbiased mean?
E[x] = E[y] = I
Assume we form a new estimator from the convex combination of the other two:
z = a ⋅ x + (1 – a) ⋅ y
Is the new estimator z unbiased? → Yes, convex combination preserves unbiasedness.
E[z] = E[a ⋅ x + (1 – a) ⋅ y] = a ⋅ E[x] + (1 – a) ⋅ E[y] = I
How should we select a?

176
 Simple estimation example
We have two independent unbiased estimators x and y of the same quantity I (e.g., pixel
intensity) with variance σ[x]2 and σ[y]2.
What does unbiased mean?
E[x] = E[y] = I
Assume we form a new estimator from the convex combination of the other two:
z = a ⋅ x + (1 – a) ⋅ y
Is the new estimator z unbiased? → Yes, convex combination preserves unbiasedness.
E[z] = E[a ⋅ x + (1 – a) ⋅ y] = a ⋅ E[x] + (1 – a) ⋅ E[y] = I
How should we select a? → Minimize variance (= expected squared error for unbiased
estimators).
E[(z – I) ] = E[z ] – 2 ⋅ E[z] ⋅ I + I = E[z ] – E[z] = σ[z]
2 2 2 2 2 2

What is the variance of z as a function of a?

177
 Simple estimation example
We have two independent unbiased estimators x and y of the same quantity I (e.g., pixel
intensity) with variance σ[x]2 and σ[y]2.
What does unbiased mean?
E[x] = E[y] = I
Assume we form a new estimator from the convex combination of the other two:
z = a ⋅ x + (1 – a) ⋅ y
Is the new estimator z unbiased? → Yes, convex combination preserves unbiasedness.
E[z] = E[a ⋅ x + (1 – a) ⋅ y] = a ⋅ E[x] + (1 – a) ⋅ E[y] = I
How should we select a? → Minimize variance (= expected squared error for unbiased
estimators).
E[(z – I) ] = E[z ] – 2 ⋅ E[z] ⋅ I + I = E[z ] – E[z] = σ[z]
2 2 2 2 2 2

What is the variance of z as a function of a?

σ[z]2 = a2 ⋅ σ[x]2 + (1 – a)2 ⋅ σ[y]2
What value of a minimizes σ[z]2? 178
 Simple estimation example
Simple optimization problem:

𝜕σ[z]2
=0
𝜕𝑎

𝜕(𝑎2 ⋅ σ[x]2 + (1 –𝑎)2 ⋅ σ[y]2)

⇒ =0
𝜕𝑎

⇒ 2 ⋅ 𝑎 ⋅ σ[x]2 − 2 ⋅ (1 –𝑎) ⋅ σ[y]2= 0

σ[y]2 σ[x]2
⇒ 𝑎= and 1−𝑎 =
σ[x]2 +σ[y]2 σ[x]2 +σ[y]2

179
 Simple estimation example
Putting it all together, the optimal linear combination of the two estimators is

σ[x]2σ[y]2 1 1
𝑧= ⋅ 𝑥+ 𝑦
σ[x] + σ[y]2 σ[x]2
2 σ[y]2

normalization weights inversely

factor proportional to variance

180
 Simple estimation example
Putting it all together, the optimal linear combination of the two estimators is

σ[x]2σ[y]2 1 1
𝑧= ⋅ 𝑥+ 𝑦
σ[x] + σ[y]2 σ[x]2
2 σ[y]2

normalization weights inversely

factor proportional to variance
More generally, for more than two estimators,
$
1 1
𝑧= ⋅= 𝑥!
$ 1 σ[𝑥! ]2
∑!"# !"#
σ[𝑥! ]2
This is weighting scheme is called Fisher weighting and is a BLUE estimator.
181
 Back to HDR
Given unclipped and dark-frame-corrected intensity measurements Ii[x, y] at pixel [x, y] and
exposures ti, we can merge them optimally into a single HDR intensity I[x, y] as
$ $
1 1 1 1 1
𝐼 𝑥, 𝑦 = $ ⋅ = 𝑤𝑖 𝑥, 𝑦 𝐼𝑖 𝑥, 𝑦 = ⋅= 𝐼𝑖 𝑥, 𝑦
∑!"# 𝑤𝑖 𝑥, 𝑦 𝑡i 1 1 2 𝑡i
!"# ∑$ !"# σ[ 𝐼 [𝑥, 𝑦]]
!"# 1 𝑡 i
𝑖
σ[ 𝑡 𝐼𝑖 [𝑥, 𝑦]]2
i

182
 Back to HDR
Given unclipped and dark-frame-corrected intensity measurements Ii[x, y] at pixel [x, y] and
exposures ti, we can merge them optimally into a single HDR intensity I[x, y] as
$ $
1 1 1 1 1
𝐼 𝑥, 𝑦 = $ ⋅ = 𝑤𝑖 𝑥, 𝑦 𝐼𝑖 𝑥, 𝑦 = ⋅= 𝐼𝑖 𝑥, 𝑦
∑!"# 𝑤𝑖 𝑥, 𝑦 𝑡i 1 1 2 𝑡i
!"# ∑$ !"# σ[ 𝐼 [𝑥, 𝑦]]
!"# 1 𝑡 i
𝑖
σ[ 𝑡 𝐼𝑖 [𝑥, 𝑦]]2
i
The per-pixel weights wi x, y should be selected to be inversely proportional to the variance
$
σ[ Ii[x, y]]2 at each image in the exposure stack.
%!
• How do we compute this variance?

183
 Pixel noise and variance
• Recall: noise is characterized by its variance
• i.e. each pixel value comes from a true value plus some noise added

• We can calibrate this noise by taking multiple exposures, or we can

derive variance equations using pen and paper

184
 Sources of noise
• Photon noise
• variance proportional to signal
• dominates for dark pixels

• Read noise
• constant variance
• dominates for dark pixels

• Affine noise model: 𝐼 = 𝐿 ⋅ 𝑔 + 𝑛"## where 𝑛"## = 𝑛$%"# ⋅ 𝑔 + 𝑛&'(

! !
• For a pixel value 𝐼 : 𝜎 𝐼 = 𝑡 ⋅ 𝑎 ⋅ Φ + 𝐷 ⋅ 𝑔! + 𝜎"##
' ' ' '
• where 𝜎%&& = 𝜎()%& ⋅ 𝑔' + 𝜎*+, , 𝑎 and 𝜎-./0 depend on the camera and ISO
185
 Back to HDR
Given unclipped and dark-frame-corrected intensity measurements Ii[x, y] at pixel [x, y] and
exposures ti, we can merge them optimally into a single HDR intensity I[x, y] as
$ $
1 1 1 1 1
𝐼 𝑥, 𝑦 = $ ⋅ = 𝑤𝑖 𝑥, 𝑦 𝐼𝑖 𝑥, 𝑦 = ⋅= 𝐼𝑖 𝑥, 𝑦
∑!"# 𝑤𝑖 𝑥, 𝑦 𝑡i 1 1 2 𝑡i
!"# ∑$ !"# σ[ 𝐼 [𝑥, 𝑦]]
!"# 1 𝑡 i
𝑖
σ[ 𝑡 𝐼𝑖 [𝑥, 𝑦]]2
i
The per-pixel weights wi x, y should be selected to be inversely proportional to the variance
!
σ[ Ii[x, y]]2 at each image in the exposure stack.
"!
• How do we compute this variance?

186
 Back to HDR
Given unclipped and dark-frame-corrected intensity measurements Ii[x, y] at pixel [x, y] and
exposures ti, we can merge them optimally into a single HDR intensity I[x, y] as
$ $
1 1 1 1 1
𝐼 𝑥, 𝑦 = $ ⋅ = 𝑤𝑖 𝑥, 𝑦 𝐼𝑖 𝑥, 𝑦 = ⋅= 𝐼𝑖 𝑥, 𝑦
∑!"# 𝑤𝑖 𝑥, 𝑦 𝑡i 1 1 2 𝑡i
!"# ∑$ !"# σ[ 𝐼 [𝑥, 𝑦]]
!"# 1 𝑡 i
𝑖
σ[ 𝑡 𝐼𝑖 [𝑥, 𝑦]]2
i
The per-pixel weights wi x, y should be selected to be inversely proportional to the variance
!
σ[ Ii[x, y]]2 at each image in the exposure stack.
"!
• How do we compute this variance? → Use affine noise model.

#
σ[1 𝐼𝑖 [𝑥, 𝑦]]2 = ?
!

187
 Back to HDR
Given unclipped and dark-frame-corrected intensity measurements Ii[x, y] at pixel [x, y] and
exposures ti, we can merge them optimally into a single HDR intensity I[x, y] as
$ $
1 1 1 1 1
𝐼 𝑥, 𝑦 = $ ⋅ = 𝑤𝑖 𝑥, 𝑦 𝐼𝑖 𝑥, 𝑦 = ⋅= 𝐼𝑖 𝑥, 𝑦
∑!"# 𝑤𝑖 𝑥, 𝑦 𝑡i 1 1 2 𝑡i
!"# ∑$ !"# σ[ 𝐼 [𝑥, 𝑦]]
!"# 1 𝑡 i
𝑖
σ[ 𝑡 𝐼𝑖 [𝑥, 𝑦]]2
i
The per-pixel weights wi x, y should be selected to be inversely proportional to the variance
!
σ[ Ii[x, y]]2 at each image in the exposure stack.
"!
• How do we compute this variance? → Use affine noise model.

# #
σ[1 𝐼𝑖 [𝑥, 𝑦]] = 1 !
2 σ[𝐼𝑖 [𝑥, 𝑦]]2
! !

#
⇒ σ[1 𝐼𝑖 [𝑥, 𝑦]]2 = ?
!
188
 Back to HDR
Given unclipped and dark-frame-corrected intensity measurements Ii[x, y] at pixel [x, y] and
exposures ti, we can merge them optimally into a single HDR intensity I[x, y] as
$ $
1 1 1 1 1
𝐼 𝑥, 𝑦 = $ ⋅ = 𝑤𝑖 𝑥, 𝑦 𝐼𝑖 𝑥, 𝑦 = ⋅= 𝐼𝑖 𝑥, 𝑦
∑!"# 𝑤𝑖 𝑥, 𝑦 𝑡i 1 1 2 𝑡i
!"# ∑$ !"# σ[ 𝐼 [𝑥, 𝑦]]
!"# 1 𝑡 i
𝑖
σ[ 𝑡 𝐼𝑖 [𝑥, 𝑦]]2
i
The per-pixel weights wi x, y should be selected to be inversely proportional to the variance
!
σ[ Ii[x, y]]2 at each image in the exposure stack.
"!
• How do we compute this variance? → Use affine noise model.
Computing the optimal weights
# # requires:
σ[1 𝐼𝑖 [𝑥, 𝑦]] = 1 ! σ[𝐼𝑖 [𝑥, 𝑦]]2
2
! ! 1. calibrated noise characteristics.
# # ' 2. knowing the radiant flux α ⋅ Φ[x, y].
⇒ σ[1 𝐼𝑖 [𝑥, 𝑦]] = 1 ! 𝑡i ⋅ 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔 + σadd
2 2
! !
189
 Back to HDR
Given unclipped and dark-frame-corrected intensity measurements Ii[x, y] at pixel [x, y] and
exposures ti, we can merge them optimally into a single HDR intensity I[x, y] as
$ $
1 1 1 1 1
𝐼 𝑥, 𝑦 = $ ⋅ = 𝑤𝑖 𝑥, 𝑦 𝐼𝑖 𝑥, 𝑦 = ⋅= 𝐼𝑖 𝑥, 𝑦
∑!"# 𝑤𝑖 𝑥, 𝑦 𝑡i 1 1 2 𝑡i
!"# ∑$ !"# σ[ 𝐼 [𝑥, 𝑦]]
!"# 1 𝑡 i
𝑖
σ[ 𝑡 𝐼𝑖 [𝑥, 𝑦]]2
i
The per-pixel weights wi x, y should be selected to be inversely proportional to the variance
!
σ[ Ii[x, y]]2 at each image in the exposure stack.
"!
• How do we compute this variance? → Use affine noise model.
Computing the optimal weights
# # requires:
σ[1 𝐼𝑖 [𝑥, 𝑦]] = 1 ! σ[𝐼𝑖 [𝑥, 𝑦]]2
2
! ! 1. calibrated noise characteristics.
# # ' 2. knowing the radiant flux α ⋅ Φ[x, y].
⇒ σ[1 𝐼𝑖 [𝑥, 𝑦]] = 1 ! 𝑡i ⋅ 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔 + σadd
2 2
! !
This is what we wanted to estimate!
190
 Simplification: only photon noise
If we assume that our measurements are dominated by photon noise, the variance
becomes:

# #
σ[1 𝐼𝑖 [𝑥, 𝑦]] = 1 !
2 𝑡i ⋅ 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔'+ σadd2 ≃ ?
! !

191
 Simplification: only photon noise
If we assume that our measurements are dominated by photon noise, the variance
becomes:

# # #
σ[1 𝐼𝑖 [𝑥, 𝑦]] = 1 !
2 𝑡i ⋅ 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔'+ σ add
2 ≃ 1!
𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔 '
! !

By replacing in the merging formula and assuming only valid pixels, the HDR estimate
becomes:

$ $
1 1 1 1
𝐼 𝑥, 𝑦 = ⋅= 𝐼𝑖 𝑥, 𝑦 = $ ⋅ = 𝐼𝑖 [𝑥, 𝑦]
1 1 ' 𝑡i ∑!"# 𝑡𝑖
∑$ !"# 𝑡 𝛼 ⋅ Φ 𝑥, 𝑦 ⋅ 𝑔 !"#
!"# 1 ' i
𝑡i 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔

192
 Simplification: only photon noise
If we assume that our measurements are dominated by photon noise, the variance
becomes:

# # #
σ[1 𝐼𝑖 [𝑥, 𝑦]] = 1 !
2 𝑡i ⋅ 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔'+ σ add
2 ≃ 1!
𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔 '
! !

By replacing in the merging formula and assuming only valid pixels, the HDR estimate
becomes:

$ $
1 1 1 1
𝐼 𝑥, 𝑦 = ⋅= 𝐼𝑖 𝑥, 𝑦 = $ ⋅ = 𝐼𝑖 [𝑥, 𝑦]
1 1 ' 𝑡i ∑!"# 𝑡𝑖
∑$ !"# 𝑡 𝛼 ⋅ Φ 𝑥, 𝑦 ⋅ 𝑔 !"#
!"# 1 ' i
𝑡i 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔

193
 Simplification: only photon noise
If we assume that our measurements are dominated by photon noise, the variance
becomes:

# # #
σ[1 𝐼𝑖 [𝑥, 𝑦]] = 1 !
2 𝑡i ⋅ 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔'+ σ add
2 ≃ 1!
𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔 '
! !

By replacing in the merging formula and assuming only valid pixels, the HDR estimate
becomes:

$ $
1 1 1 1
𝐼 𝑥, 𝑦 = ⋅= 𝐼𝑖 𝑥, 𝑦 = $ ⋅ = 𝐼𝑖 [𝑥, 𝑦]
1 1 ' 𝑡i ∑!"# 𝑡𝑖
∑$ !"# 𝑡 𝛼 ⋅ Φ 𝑥, 𝑦 ⋅ 𝑔 !"#
!"# 1 ' i
𝑡i 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔

Notice that we no longer weight each image in the exposure stack by its exposure time!
194
Some comparisons

original weights optimal weights assuming

only photon noise
195
 More general case
If we cannot assume that our measurements are dominated by photon noise, we can
approximate the variance as:

# # #
σ[1 𝐼𝑖 [𝑥, 𝑦]] = 1 !
2 𝑡i ⋅ 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔'+ σ add
2 ≃ 1!!
𝐼𝑖 [𝑥, 𝑦] ⋅ 𝑔 + σadd2
! !

Where does this approximation come from?

196
 More general case
If we cannot assume that our measurements are dominated by photon noise, we can
approximate the variance as:

# # #
σ[1 𝐼𝑖 [𝑥, 𝑦]] = 1 !
2 𝑡i ⋅ 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ 𝑔'+ σ add
2 ≃ 1!!
𝐼𝑖 [𝑥, 𝑦] ⋅ 𝑔 + σadd2
! !

Where does this approximation come from?

• We use the fact that each pixel intensity (after dark frame subtraction) is an unbiased
estimate of the radiant flux, weighted by exposure and gain:

E[𝐼𝑖 [𝑥, 𝑦]] = 𝑡i ⋅ 𝛼 ⋅ Φ[𝑥, 𝑦] ⋅ g

197
 standard weights optimal weights ground-truth
Some
comparisons

tone-mapped merged HDR

2.8 dB 14.6 dB

198
What about ISO?

• We need to separately account for

read and ADC noise, as read noise
is gain-dependent.
• We can optimize our exposure
bracket by varying both shutter
speed and ISO

199
 “ground truth” SNR-optimal standard exposure
sequence bracketing
1/125 s 1/1600 s 1/6400 s 1/200 s 1/400 s 1/800 s
ISO 6400 ISO 6400 ISO 800 ISO 100 ISO 100 ISO 100

……

+15 more
input

200
Real capture results
tone-mapped merged HDR
bright region
dark region

16.2 dB
5.9 dB

200
Recap
• High dynamic range (HDR) imaging is useful, and a new aesthetic
• but is not necessary in all photographic situations
• Low dynamic range (LDR) tone mapping methods can also be applied
to HDR scenes
• but reducing very HDR scenes to 8 bits for JPEG using only global
methods is hard
• Local methods reduce large-scale luminance changes (across the
image) while preserving local contrast (across edges)
• use edge-preserving filters to avoid halos

201
 Next Lecture:
Edge-aware filtering,
Gradient-domain image
processing

202