Radiometry
CMPSCI 670: Computer Vision
Grant Van Horn
February 6, 2024
Administrivia
Homework 1 will be released today
Python tutorial — Friday 2/9, 2 - 3 pm, CS 150/151
TAs will cover python basics (e.g. numpy, matplotlib, scipy, skimage)
Project Poster Presentation May 10th, 2-4pm, LGRC A112.
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 2
Your Perception
Figures from:
https://www.scientificamerican.com/article/seeing-is-
believing-aug-08/
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 3
Your Perception
Figures from:
https://www.scientificamerican.com/article/seeing-is-
believing-aug-08/
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 4
Your Perception
Figures from:
https://www.scientificamerican.com/article/seeing-is-
believing-aug-08/
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 5
Your Perception
Figures from:
https://www.scientificamerican.com/article/seeing-is-
believing-aug-08/
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 6
Your Perception
https://macaulaylibrary.org/asset/557939521
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 7
Your Perception
CMPSCI 670 Grant Van Horn (UMass, Fall 24) https://macaulaylibrary.org/asset/424082941 8
Radiometry
Questions:
• How “bright” will surfaces be?
• What is “brightness”?
‣ measuring light
‣ interactions between light and
surfaces source: https://www.umass.edu/
Core idea — think about light arriving
at a surface around any point is a
hemisphere of directions # d#
Simplest problems can be dealt with
by reasoning about this hemisphere
Computer Vision - A Modern Approach
Set: Radiometry
Slides by D.A. Forsyth
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 9
Lambert’s wall
What is the distribution
of brightness on the ground?
Computer Vision - A Modern Approach
Set: Radiometry
Slides by D.A. Forsyth
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 10
More complex wall
Computer Vision - A Modern Approach
Set: Radiometry
Slides by D.A. Forsyth
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 11
More complex wall
Computer Vision - A Modern Approach
Set: Radiometry
Slides by D.A. Forsyth
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 12
Light at surfaces
What happens when a light ray hits a point on an object?
Some of the light gets absorbed
• converted to other forms of energy (e.g., heat)
Some gets transmitted through the object
• possibly bent, through refraction
• or scattered inside the object (subsurface scattering)
Some gets reflected
• possibly in multiple directions at once
Really complicated things can happen
• uorescence
CMPSCI 670 Grant Van Horn (UMass, Fall 24) Source: Steve Seitz 13
fl
Fluorescence
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 14
Modeling surface re ectance
Bidirectional re ectance distribution function (BRDF)
• How bright a surface appears when viewed from one direction when light falls on it
from another
• De nition: ratio of the radiance in the emitted direction to irradiance in the
incident direction
L(p, ✓e , e )
⇢(p, ✓i , i , ✓e , e ) =
L(p, ✓i , i ) cos ✓i d!
<latexit sha1_base64="Yh+qIxnLnyY3XiGHjox1w5RlM1M=">AAACZXicbZFLSwMxFIUz46vWV33gxoXBIihImVFBN4LoxoULBatCp5RMeqcTmpkMyR2hjP2T7ty68W+YtoPUx4WQw3dy8zgJMykMet67487Mzs0vVBarS8srq2u19Y1Ho3LNocmVVPo5ZAakSKGJAiU8ZxpYEkp4CvvXI//pBbQRKn3AQQbthPVSEQnO0KJO7TXQsTrIjmiAMSDrCKuyWIznCYGSwCG9oEGkGS9u6VTHlD8sbv/b6pAGXJlvTLs0UAn02LBTq3sNb1z0r/BLUSdl3XVqb0FX8TyBFLlkxrR8L8N2wTQKLmFYDXIDGeN91oOWlSlLwLSLcUpDum9Jl0ZK25EiHdPpjoIlxgyS0K5MGMbmtzeC/3mtHKPzdiHSLEdI+eSgKJcUFR1FTrtCA0c5sIJxLexdKY+ZDRLtx1RtCP7vJ/8Vj8cN/6RxfH9av7wq46iQHbJHDohPzsgluSF3pEk4+XAqzrqz4Xy6K+6Wuz1Z6jplzyb5Ue7uF5HmtKM=</latexit>
Simplifying assumptions
locality, no uorescence,
does not generate light
CMPSCI 670 Grant Van Horn (UMass, Fall 24) Source: Steve Seitz 15
fi
fl
fl
fl
Goniore ectometer
The University of Virginia spherical gantry, an example of a
modern image-based goniore ectometer
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 16
fl
fl
BRDFs can be incredibly complicated…
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 17
Suppressing the angles in the BRDF
BRDF is a very general notion
• some surfaces need it (underside of a CD; tiger eye; etc)
• very hard to measure — illuminate from one direction, view from another,
repeat
• very unstable — minor surface damage can change the BRDF
• e.g. ridges of oil left by contact with the skin can act as lenses
• However, for many surfaces, light leaving the surface is largely independent
of exit angle
• surface roughness is one source of this property
Computer Vision - A Modern Approach
Set: Radiometry
Slides by D.A. Forsyth
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 18
Special cases: Diffuse re ection
Light is re ected equally in all directions
• Dull, matte surfaces like chalk or cotton cloth
• Microfacets scatter incoming light randomly
• Effect is that light is re ected (approximately) equally in all directions
Brightness of the surface depends on the incidence of illumination
brighter darker
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 19
fl
fl
fl
Diffuse re ection: Lambert’s law
N
S
B = ρ (N ⋅ S)
θ
= ρ S cos θ
B: radiosity (total power leaving the
surface per unit area)
ρ: albedo (fraction of incident irradiance
reflected by the surface)
N: unit normal
S: source vector (magnitude proportional
to intensity of the source)
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 20
fl
Specular re ection
Radiation arriving along a source direction
leaves along the specular direction (source
direction re ected about normal)
Some fraction is absorbed, some re ected
On real surfaces, energy usually goes into a
lobe of directions
n
Phong model: re ected energy falls of with cos (δθ )
Lambertian + specular model: sum of diffuse
and specular term
• a reasonable approximation to lot of
surfaces we see
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 21
fl
fl
fl
fl
Specular re ection
Moving the light source
n
cos (δθ )
Changing the exponent
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 22
fl
Photometric stereo
Can we reconstruct the shape of an object based on shading cues?
Luca della Robbia,
Cantoria, 1438
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 23
Ambiguities in shape and shading
The workshop metaphor from Adelson and Pentland, 1996
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 24
Photometric stereo
S1 S2 S3 Sn
S2
S1 ???
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 25
Photometric stereo
Assume:
‣ A Lambertian object
‣ A local shading model, i.e., each point on a surface receives light only from sources
visible at that point
‣ A set of known light source directions
‣ A set of pictures of an object, obtained in exactly the same camera and object
con guration but using different sources
‣ Orthographic projection
Goal: reconstruct object shape and albedo
S2
S1 ???
Sn
F&P 2nd ed., sec. 2.2.4
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 26
fi
Surface model: Monge patch
z = f(x,y)
F&P 2nd ed., sec. 2.2.4
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 27
Image model
Known: source vectors Sj and pixel values Ij(x,y)
Unknown: surface normal N(x,y) and albedo ρ(x,y)
Assume that the response function of the camera is a linear scaling by a factor of k
Lambert’s law:
I j ( x, y ) = k ρ (x, y )(N(x, y )⋅ S j )
= (ρ (x, y )N(x, y ))⋅ (k S j )
= g ( x, y ) ⋅ V j
F&P 2nd ed., sec. 2.2.4
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 28
Least squares problem
For each pixel, set up a linear system:
T
& I1 ( x, y ) # & V # 1
$ I ( x, y ) ! $ ! T
$ 2 !=$ V ! g ( x, y )
2
$ ! ! $ ! !
$ ! $ T!
I
% n ( x , y ) " $%Vn !"
(n × 1) (n × 3) (3 × 1)
known known unknown
Obtain least-squares solution for g(x,y) de ned as N(x,y) ρ(x,y)
Since N(x,y) is the unit normal, ρ(x,y) is given by the magnitude of g(x,y)
Finally, N(x,y) = g(x,y) / ρ(x,y)
F&P 2nd ed., sec. 2.2.4
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 29
fi
Example
Recovered albedo Recovered normal field
F&P 2nd ed., sec. 2.2.4
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 30
Recovering a surface from normals
Recall the surface is written as If we write the estimated
vector g as
( x, y, f ( x, y ))
& g1 ( x, y ) #
$ !
This means the normal has the form: g ( x, y ) = $ g 2 ( x, y ) !
$ g ( x, y ) !
& fx # % 3 "
1 $ !
N ( x, y ) = $ fy !
2 2
fx + f y +1 $ ! Then we obtain values for
1
% " the partial derivatives of the
surface:
f x ( x, y ) = g1 ( x, y ) / g 3 ( x, y )
f y ( x, y ) = g 2 ( x, y ) / g 3 ( x, y )
CMPSCI 670 Grant Van Horn (UMass, Fall 24) F&P 2nd ed., sec. 2.2.4 31
Recovering a surface from normals
Integrability: for the surface f to exist, the We can now recover the surface height at
mixed second partial derivatives must be any point by integration along some path
equal:
2 2
@ f @ f
= i.e.,
@x@y @y@x
<latexit sha1_base64="dOePCpZubdRkWi+RG/b7wG7tqvg=">AAACQnicfVBLS8NAGNzUV62vqEcvi0XwVJIq6EUoevFYwT6kiWWz3bRLNw92N9IQ8tu8+Au8+QO8eFDEqwc3bbDaigMLszPz7WOckFEhDeNJKywsLi2vFFdLa+sbm1v69k5TBBHHpIEDFvC2gwRh1CcNSSUj7ZAT5DmMtJzhRea37ggXNPCvZRwS20N9n7oUI6mkrn5juRzhxAoRlxQxeFuFbjrdjuA3jVN4Bv9Px9P0KO3qZaNijAHniZmTMshR7+qPVi/AkUd8iRkSomMaobST7EDMSFqyIkFChIeoTzqK+sgjwk7GFaTwQCk96AZcLV/CsfpzIkGeELHnqKSH5EDMepn4l9eJpHtqJ9QPI0l8PLnIjRiUAcz6hD3KCZYsVgRhTtVbIR4g1ZJUrZdUCebsl+dJs1oxjyrVq+Ny7Tyvowj2wD44BCY4ATVwCeqgATC4B8/gFbxpD9qL9q59TKIFLZ/ZBb+gfX4BU1yxOg==</latexit>
∂ For example:
( g1 ( x, y ) / g 3 ( x, y )) =
∂y
∂
( g 2 ( x, y ) / g 3 ( x, y ))
∂x
In practice, they should be similar For robustness, take integrals over many
different paths and average the results
F&P 2nd ed., sec. 2.2.4
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 32
Surface recovered by integration
F&P 2nd ed., sec. 2.2.4
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 33
Works for more complicated surfaces
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 34
Limitations
Orthographic camera model
Simplistic re ectance and lighting model
No shadows
No inter-re ections
No missing data
Integration is tricky
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 35
fl
fl
GELSIGHT
https://www.youtube.com/watch?v=S7gXih4XS7A
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 36
GELSIGHT
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 37
Finding the direction of the light source
P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 38
Finding the direction of the light source
I(x,y) = N(x,y) ·S(x,y) + A
N For points on the occluding contour:
S
& N x ( x1 , y1 ) N y ( x1 , y1 ) 1# & I ( x1 , y1 ) #
$ !& S x # $ !
$ N x ( x2 , y 2 ) N y ( x2 , y 2 ) 1!$ ! $ I ( x2 , y2 ) !
$ !$ Sy ! = $ !
! ! ! $ ! !
$ !% A " $ !
$ N (x , y ) N (x , y ) 1!" $ I (x , y )!
% x n n y n n % n n "
P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 39
Detecting composite photos
Fake photo Real photo
M. K. Johnson and H. Farid, Exposing Digital Forgeries by Detecting Inconsistencies in
Lighting, ACM Multimedia and Security Workshop, 2005.
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 40
Neural radiance elds (NeRF)
CMPSCI 670 Grant Van Horn (UMass, Fall 24) Slide credit: Matthew Tancik 41
fi
Neural radiance elds (NeRF)
https://www.matthewtancik.com/nerf
CMPSCI 670 Grant Van Horn (UMass, Fall 24) 42
fi
