Last Lecture
• Single View Modeling
Vermeer’s Music Lesson
Reconstructions by Criminisi et al.
Today
• Photometric Stereo
• Separate Global and Direct Illumination
Photometric Stereo
Photometric Stereo
Merle Norman Cosmetics, Los Angeles
Readings
• R. Woodham, Photometric Method for Determining Surface Orientation from
Multiple Images. Optical Engineering 19(1)139-144 (1980). (PDF)
Diffuse reflection
image intensity of P
Simplifying assumptions
• I = Re: camera response function f is the identity function:
– can always achieve this in practice by solving for f and
applying f -1 to each pixel in the image
• Ri = 1: light source intensity is 1
– can achieve this by dividing each pixel in the image by Ri
Shape from shading
Suppose
You can directly measure angle between normal and light source
• Not quite enough information to compute surface shape
• But can be if you add some additional info, for example
– assume a few of the normals are known (e.g., along silhouette)
– constraints on neighboring normals—―integrability‖
– smoothness
• Hard to get it to work well in practice
– plus, how many real objects have constant albedo?
Photometric stereo
N
L3 L2
L1
V
Can write this as a matrix equation:
Solving the equations
More than three lights
Get better results by using more lights
Least squares solution:
Solve for N, kd as before
What’s the size of LTL?
Computing light source directions
Trick: place a chrome sphere in the scene
• the location of the highlight tells you where the light source is
Recall the rule for specular reflection
For a perfect mirror, light is reflected about N
Ri if V R
Re
0 otherwise
We see a highlight when V = R
• then L is given as follows:
Depth from normals
V2
V1
N
Get a similar equation for V2
• Each normal gives us two linear constraints on z
• compute z values by solving a matrix equation
Example
What if we don’t have mirror ball?
• Hayakawa, Journal of the Optical Society of America,
1994, Photometric stereo under a light source with
arbitrary motion.
Limitations
Big problems
• doesn’t work for shiny things, semi-translucent things
• shadows, inter-reflections
Smaller problems
• camera and lights have to be distant
• calibration requirements
– measure light source directions, intensities
– camera response function
Newer work addresses some of these issues
Some pointers for further reading:
• Zickler, Belhumeur, and Kriegman, "Helmholtz Stereopsis: Exploiting
Reciprocity for Surface Reconstruction." IJCV, Vol. 49 No. 2/3, pp 215-227.
• Hertzmann & Seitz, ―Example-Based Photometric Stereo: Shape
Reconstruction with General, Varying BRDFs.‖ IEEE Trans. PAMI 2005
Example-based
Photometric Stereo
Aaron Hertzmann Steven M. Seitz
University of Toronto University of Washington
Shiny things
“Orientation consistency”
same surface normal
Virtual views
Velvet
Virtual Views
Brushed Fur
Brushed Fur
Virtual Views
Linear combinations of materials
= + +
= + +
0.9 + 0.6 + 0.2 + 2.0 + 2.1 + 2.1
=
0.9
0.6
0.2
2.0
2.1
2.1
Virtual views
photometric
laser scanstereo laser scanstereo
photometric
Problem definition
Estimate 3D shape by varying
illumination, fixed camera
Operating conditions
• any opaque material
• distant camera, lighting
• reference object available
• no shadows, interreflections,
transparency
Fast Separation of Direct and Global Images
Using High Frequency Illumination
Shree K. Nayar
Gurunandan G. Krishnan
Columbia University
Michael D. Grossberg
City College of New York
Ramesh Raskar
MERL
SIGGRAPH Conference
Boston, July 2006
Support: ONR, NSF, MERL
Direct and Global Illumination
participating
medium
surface
source
D B
A
C
P A : Direct
E B : Interrelection
C : Subsurface
camera translucent D : Volumetric
surface
E : Diffusion
Fast Separation of Direct and Global Images
• Create Novel Images of the Scene
• Enhance Brightness Based Vision Methods
• New Insights into Material Properties
Direct and Global Components: Interreflections
surface
source j
camera
L[c, i] Ld [c, i] Lg [c, i] Lg [c, i] A[i, j ] L[i, j ]
P
radiance direct global BRDF and geometry
High Frequency Illumination Pattern
surface
source
camera
+
L [c, i] Ld [c, i] Lg [c, i]
fraction of activated source elements
High Frequency Illumination Pattern
surface
source
camera
+
L [c, i] Ld [c, i] Lg [c, i]
-
L [c, i] (1 ) Lg [c, i]
fraction of activated source elements
Separation from Two Images
1
: Ld Lmax Lmin , Lg 2Lmin
2
direct global
Other Global Effects: Subsurface Scattering
translucent
surface
source j
camera
Other Global Effects: Volumetric Scattering
participating medium
surface
source
j
i
camera
Diffuse
Interreflections
Specular
Interreflections
Diffusion
Volumetric
Subsurface
Scattering
Scattering
Scene
Scene
Direct Global
Real World Examples:
Can You Guess the Images?
Eggs: Diffuse Interreflections
Direct Global
Wooden Blocks: Specular Interreflections
Direct Global
Novel Images
Mirror Ball: Failure Case
Direct Global
Photometric Stereo using Direct Images
Source 1 Source 2 Source 3
Bowl
Shape
Global
Direct
Nayar et al., 1991
Variants of Separation Method
• Coded Structured Light
• Shifted Sinusoids
• Shadow of Line Occluder
• Shadow of Mesh Occluders
Building Corner
3D from Shadows:
Bouguet and Perona 99
Stick
Shadow
Ld Lmax Lmin , Lg Lmin
direct global
Building Corner
Direct Global
Shower Curtain: Diffuser
Mesh
Shadow
Ld Lmax Lmin , Lg Lmin
direct global
Shower Curtain: Diffuser
Direct Global
Kitchen Sink: Volumetric Scattering
Volumetric Scattering:
Chandrasekar 50, Ishimaru 78
Direct Global
Peppers: Subsurface Scattering
Direct Global
Hand
Skin: Hanrahan and Krueger 93,
Uchida 96, Haro 01, Jensen et al. 01,
Cula and Dana 02, Igarashi et al.
05, Weyrich et al. 05
Direct Global
Hands
Afric. Amer. Chinese Spanish
Female Male Male
Afric. Amer. Chinese Spanish Afric. Amer. Chinese Spanish
Female Male Male Female Male Male
Direct Global
Tea Rose Leaf
Leaf Anatomy: Purves et al. 03
Direct Global
Translucent Rubber Balls
Direct Global
Face: Without and With Makeup
Without Makeup Direct Global
With Makeup Direct Global
Blonde Hair
Hair Scattering: Stamm et al. 77,
Bustard and Smith 91, Lu et al. 00
Marschner et al. 03
Direct Global
Pebbles: 3D Texture
Direct Global
Pink Carnation
Spectral Bleeding: Funt et al. 91
Direct Global
Summary
• Fast and Simple Separation Method
• No Prior Knowledge of Material Properties
• Wide Variety of Global Effects
• Implications:
• Generation of Novel Images
• Enhance Computer Vision Methods
• Insights into Properties of Materials
http://www1.cs.columbia.edu/CAVE/projects/separation/separation.php
