Photometric stereo
Radiance
• Pixels measure radiance
This pixel
Measures radiance
along this ray
Where do the rays come from?
• Rays from the
light source
“reflect” off a
surface and
reach camera
• Reflection:
Surface absorbs
light energy and
radiates it back
Light rays interacting with a
surface
• Light of radiance !" comes from light source at an
incoming direction #"
• It sends out a ray of radiance !$ in the outgoing
direction #$
• How does !$ relate to !" ?
• N is surface normal
• L is direction of light, making #"
with normal
#" # • V is viewing direction, making #$
$ with normal
Light rays interacting with a
surface
• N is surface normal
• L is direction of light, making !"
with normal
!" ! • V is viewing direction, making !#
# with normal
Incoming
Output radiance
along V $# = & !" , !# $" cos !" irradiance along
L
Bi-directional reflectance function (BRDF)
Light rays interacting with a
surface
"# "
(% = ! "# , "% (# cos "#
%
• Special case 1: Perfect mirror
• ! "# , "% = 0 unless "# = "%
• Special case 2: Matte surface
• ! "# , "% = !' (constant)
Special case 1: Perfect mirror
• ! "# , "% = 0 unless "# = "%
• Also called “Specular surfaces”
• Reflects light in a single, particular direction
Special case 2: Matte surface
• ! "# , "% = !&
• Also called “Lambertian surfaces”
• Reflected light is independent of viewing direction
Lambertian surfaces
• For a lambertian surface:
Lr = ⇢Li cos ✓i
) Lr = ⇢Li L · N
"# "
$
• ! is called albedo
• Think of this as paint
• High albedo: white colored surface
• Low albedo: black surface
• Varies from point to point
Lambertian surfaces
• Assume the light is directional: all
rays from light source are parallel
• Equivalent to a light source
infinitely far away
!" !
• All pixels get light from the same #
direction L and of the same
intensity Li
Lambertian surfaces
Intrinsic Image
Decomposition
I(x, y) = ⇢(x, y)Li L · N(x, y)
Shading
Reflectance
image
image
Reconstructing Lambertian
surfaces
I(x, y) = ⇢(x, y)Li L · N(x, y)
• Equation is a constraint on albedo and normals
• Can we solve for albedo and normals?
Solution 1: Recovery from a single
image
• Step 1: Intrinsic image
decomposition
• Reflectance image ! ", $
• Shading image Li L · N(x, y)
• Decomposition relies on priors on
reflectance image
• What kind of priors?
• Reflectance image captures the
“paint” on an object surface
• Surfaces tend to be of uniform color
with sharp edges when color
changes
Images from Barron et al, TPAMI 13
Solution 1: Recovery from a single
image
• Step 2: Decompose shading image into illumination
and normals
Li L · N(x, y)
• Called Shape-From-Shading
• Relies on priors on shape: shapes are smooth
Far
Near
Solution 2: Recovery from
multiple images
I(x, y) = ⇢(x, y)Li L · N(x, y)
• Represents an equation in the albedo and normals
• Multiple images give constraints on albedo and normals
• Called Photometric Stereo
Image credit: Wikipedia
Multiple Images: Photometric Stereo
N
L3 L2
L1
V
Photometric stereo - the math
I(x, y) = ⇢(x, y)Li L · N(x, y)
• Consider single pixel
• Assume !" = 1
I = ⇢L · N
T
I = ⇢N L
• Write G = ⇢N
• G is a 3-vector
• Norm of G = '
• Direction of G = N
Photometric stereo - the math
• Consider single pixel
• Assume !" = 1
I = ⇢NT L
• Write G = ⇢N
• G is a 3-vector
• Norm of G = '
• Direction of G = N
I = GT L = L T G
Photometric stereo - the math
I = LT G
• Multiple images with different light sources but
same viewing direction?
I1 = LT1 G
I2 = LT2 G
..
.
Ik = LTk G
Photometric stereo - the math
I1 = LT1 G
I2 = LT2 G
..
.
Ik = LTk G
• Assume lighting directions are known
• Each is a linear equation in G
• Stack everything up into a massive linear system of
equations!
Photometric stereo - the math
k x 3 matrix
of lighting
I1 = LT1 G directions
I2 = LT2 G T
.. I=L G
.
k x 1 vector 3x1 vector of
Ik = LTk G of intensities unknowns
Photometric stereo - the math
T
I=L G
kx1 kx3 3x1
T
G=L I
• What is the minimum value of k to allow recovery
of G?
• How do we recover G if the problem is
overconstrained?
Photometric stereo - the math
• How do we recover G if the problem is
overconstrained?
• More than 3 lights: more than 3 images
• Least squares
T 2
min kI L Gk
G
• Solved using normal equations
T 1
G = (LL ) LI
Normal equations
T 2 T T T T
kI L Gk = I I + G LL G 2G LI
• Take derivative with respect to G and set to 0
T
2LL G 2LI = 0
T 1
) G = (LL ) LI
Estimating normals and albedo
from G
• Recall that G = ⇢N
kGk = ⇢
G
=N
kGk
Multiple pixels
• We’ve looked at a single pixel till now
• How do we handle multiple pixels?
• Essentially independent equations!
Multiple pixels: matrix form
• Note that all pixels share the same set of lights
(1) T (1)
I =L G
(2) T (2)
I =L G
..
.
I(n) = LT G(n)
Multiple pixels: matrix form
• Can stack these into columns of a matrix
I(1) = LT G(1)
I(2) = LT G(2)
..
.
I(n) = LT G(n)
⇥ (1) (2) (n)
⇤ T
⇥ (1) (2) (n)
⇤
I I ··· I =L G G ··· G
T
I=L G
Multiple pixels: matrix form
T
I=L G
#pixels 3 #pixels
3 G
=
#lights
#lights
I LT
Estimating depth from normals
• So we got surface normals, can we get depth?
• Yes, given boundary conditions
• Normals provide information about the derivative
Brief detour: Orthographic
projection
• Perspective projection
# '
• != $
,& = $
• If all points have similar depth Perspective
• ( ≈ (*
# '
• !≈ $+
,& ≈ $+
• ! ≈ ,-, & ≈ ,.
• A scaled version of orthographic
projection Scaled
orthographic
• ! = -, & = .
Depth Map from Normal Map
• We now have a surface normal, but how do we get
depth? Assume a smooth surface
(cx, c(y + 1), Zx,y+1 )
V2
(cx, cy, Zx,y ) V1 V1 = (c(x + 1), cy, Zx+1,y ) (cx, cy, Zx,y )
N
= (c, 0, Zx+1,y Zx,y )
(c(x + 1), cy, Zx+1,y ) 0 = N · V1
= (nx , ny , nz ) · (c, 0, Zx+1,y Zx,y )
= cnx + nz (Zx+1,y Zx,y )
Get a similar equation for V2
• Each normal gives us two linear constraints on z
• compute z values by solving a matrix equation
32
Determining Light Directions
• Trick: Place a mirror ball in the scene.
• The location of the highlight is determined by the
light source direction.
33
Determining Light Directions
• For a perfect mirror, the light is reflected across N:
34
Determining Light Directions
N
= "⋅$ "
L || || R
=$ − "⋅$ "
=$ − "⋅$ "
= -2
=$ −2 $ −"⋅$ "
So the light source direction = 2 " ⋅ $ " − $
is given by:
35
Determining Light Directions
• Assume orthographic projection
• Viewing direction R = [0,0,-1]
• Normal?
(# and (' are unknown, but:
"# − "' * + %# − %' *
("# , %# , (# ) + (# − (' * = - *
("# , %# ) (# − (' can be computed
("' , %' , (' ) "# − "' , %# − %' , (# − (' is
("' , %' ) the normal
Z=1
Photometric Stereo
What results can you get?
Input Normals (RGB Normals (vectors) Shaded 3D Textured 3D
(1 of 12) colormap) rendering rendering
Results
from Athos Georghiades
38
Results
Input Normals (RGB Normals (vectors) Shaded 3D Textured 3D
(1 of 12) colormap) rendering rendering
Questions?
