Modulated dark-field phasing

detection for automatic optical

inspection

Heejoo Choi
John Mineo Kam
Joel David Berkson
Logan Rodriguez Graves
Dae Wook Kim

Heejoo Choi, John Mineo Kam, Joel David Berkson, Logan Rodriguez Graves, Dae Wook Kim, “Modulated
dark-field phasing detection for automatic optical inspection,” Opt. Eng. 58(9), 092603 (2019),
doi: 10.1117/1.OE.58.9.092603.

Downloaded From: https://www.spiedigitallibrary.org/journals/Optical-Engineering on 21 Apr 2019

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Optical Engineering 58(9), 092603 (September 2019)

Modulated dark-field phasing detection for automatic

optical inspection
Heejoo Choi,a John Mineo Kam,a Joel David Berkson,a Logan Rodriguez Graves,a and Dae Wook Kima,b,*
a
University of Arizona, James C. Wyant College of Optical Sciences, Tucson, Arizona, United States
b
University of Arizona, Department of Astronomy and Steward Observatory, Tucson, Arizona, United States

Abstract. Dark-field illumination is a simple yet elegant imaging technique that can be used to detect the pres-
ence of particles on a specular surface. However, the sensitivity of dark-field illumination to initial conditions
affects its repeatability. This is problematic in cases where automation is desired. We present an improvement
to the current method of using a modulation field that relies on phase calculations rather than intensity.
As a result, we obtain a computational method that is insensitive to noise and provides clearly defined particle
information, allowing a global threshold to be set for autonomous measurement purposes. After introducing
the theory behind our method, we present experimental results for various scenarios and compare them to
those obtained using the dark-field approach. © 2019 Society of Photo-Optical Instrumentation Engineers (SPIE) [DOI: 10.1117/1
.OE.58.9.092603]
Keywords: automatic optical inspection; particle detection; dark field; phase measurement.
Paper 190052SS received Jan. 13, 2019; accepted for publication Mar. 4, 2019; published online Mar. 26, 2019.

1 Introduction is sometimes transparent and the scattered light from the

Fabrication of semiconductors and display panels is a cutting- object produces a nice high-contrast image.6,9,10
edge and competitive industry in which products must be In measurements where high repeatability and automation
manufactured reliably, with minimal defects, and in high vol- is needed, relying on the inherently noisy intensity is not
umes to maximize profit margins. Manufacturers implement ideal. The effectiveness of intensity data depends on the
automatic optical inspection (AOI) processes1–3 to interpret roughness and orientation of the particle scattering the light,
the results of continuous repetitive inspections with minimal as well as the signal-to-noise ratio (SNR). If the AOI thresh-
human interaction. As this is an automated process, reliable old for judging particles is fixed, the results will vary from
trial to trial because of changes in measurement conditions or
thresholds must be set to distinguish bad products from
even Gaussian noise in the sensor itself.
good ones.
This paper describes a new method of inspection that
The difficulty in AOI lies in detecting small defects over a
instead relies on phase differences to detect small defects on
large area in a dynamic environment. This kind of scenario is
a specular surface. In Sec. 2, we introduce the proposed modu-
often found in lithography processes applied to semiconduc-
lated field (MF) method. Section 3 describes a case study of
tors or display panels, where the wire grid is incorrectly various initial phase differences, before Sec. 4 presents the
patterned on the wafer because of defects on the surface of result of an experimental comparison between the proposed
the masks or substrates.4,5 The substrates (wafers) are typi- method and the conventional dark-field approach. Finally,
cally greater than 100 mm in diameter (sometimes reaching the conclusions to this study are summarized in Sec. 5.
1 to 2 m depending on the facility), yet particles on the order
of microns can result in errors. Inspection using human
knowledge might provide more trustworthy results but will 2 Modulated Dark-Field Detection
require much longer inspection times. To meet the needs 2.1 Modulated Dark-Field System Configuration
of industry, a robust AOI that produces reliable results and
minimizes runtime is essential. Our method, known as MF illumination, uses a similar setup
Dark-field illumination6–10 is one of the simplest and most to that used in the dark-field case but considers the phase
powerful solutions among current methods of detecting par- rather than the intensity to distinguish the target from the
ticles on substrates and masks. This method produces a high- background. This is achieved through the addition of a sec-
contrast image in which objects of interest appear bright atop ond light source (LCD monitor) located almost normal to the
a dark background. The images are achieved by illuminating surface that illuminates the entire surface, as shown in Fig. 1.
the specular surface at a grazing angle, with the detector Each light source is driven to produce a time-varying
placed almost normal to the surface of the unit under test sinusoid (brightness changes with time). The key lies in
phase shifting the signals such that the grazing LED produ-
(UUT). In this configuration, only light that has been scat-
ces a signal that is out of phase with that produced by the
tered by particles or defects on or in the surface can reach
monitor normal to the UUT’s surface. After image process-
the detector.7,8 This method is often associated with the
ing, the pixel values correspond to the newly defined phase-
microscopy of biological samples, where the sample itself
index, in which the particles and background have different,
specific values. This phase information provides stable
*Address all correspondence to Dae Wook Kim, E-mail: letter2dwk@hotmail
.com 0091-3286/2019/$25.00 © 2019 SPIE

Optical Engineering 092603-1 September 2019 • Vol. 58(9)

Downloaded From: https://www.spiedigitallibrary.org/journals/Optical-Engineering on 21 Apr 2019

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Choi et al.: Modulated dark-field phasing detection for automatic optical inspection

information of the background and particles. A pixel might

have a fully occupied signal from either light source, or
contain some fraction of both. Such a blended case with
mixed signals is simulated using


πn
I particle ðx; y; nÞ ¼ A1 ðx; yÞ sin þ Φ1 ; (1)
N
EQ-TARGET;temp:intralink-;e001;326;708


πn
I background ðx; y; nÞ ¼ A2 ðx; yÞ sin þ Φ2 ; (2)
N
EQ-TARGET;temp:intralink-;e002;326;653

Fig. 1 (a) Camera and monitor. (b) UUT, LED, and LED fan for cool-  PN 2πn 
ing. (c) Schematic diagram of experimental setup. All rays from the n¼1 Iðx; y; nÞ sin N
LED reflect off the specular surface of the UUT, except those scat- Φindex ðx; yÞ ¼ a tan Q PN 2πn ; (3)
n¼1 Iðx; y; nÞ cos N
EQ-TARGET;temp:intralink-;e003;326;618

tered by defects on the surface. The rays from the monitor fill the
surface of the UUT with uniform phase information, except in areas
where defects occur. where I particle is the intensity from light scattered by a par-
ticle and I background is the intensity reflected by the specular
back surface. A1 and A2 are the fill factors for each signal
results, even if the particle intensity values are low, as
(A1 þ A2 ¼ 1), which take account of the case of a mixed
the calculated phases are the same for every particle [see
signal (e.g., boundary of particle or particle smaller than the
Fig. 2(a)]. This allows for simple criteria to be used in deter-
resolution of our camera). N is the total number of phase
mining a threshold that yields reliable and repeatable results
steps and n is the index of the step number (e.g., n ¼
without any human interaction. In addition, if there is an
1 − 4. Φ1 and Φ2 denote the initial phase of each source.
intensity-sensitive mark (e.g., mask pattern) on the UUT,
We can insert any pair of Φ1 and Φ2 so that the resulting
this mark will show up in the intensity difference map but
map has a high contrast. Note that the goal of this calculation
not in the phase map, as shown in Fig. 2(b).
and treatment is to obtain a high-contrast map in which par-
ticle information can be easily distinguished from nonpar-
2.2 Detection Criteria Using Phase-Index Value ticle information. Ið¼ I particle þ I background Þ in Eq. (3) is the
When assigning different phases to the LED and the screen recorded value (mixed intensity) used to compute the ensem-
to achieve a phase contrast, our options are only limited by ble phase-index Φindex . The mixed intensity I is recorded for
the precision of the brightness used to drive each element. every single pixel (x; y) and all the data processing calcula-
However, as our goal is to obtain a high phase contrast tions are performed pixel by pixel.
between the background and particle, we opt to produce The phase-index Φindex ðx; yÞ is calculated from the
the largest possible phase difference. four-quadrant inverse tangent (a tan Q) from Eq. (3) using
When we take the image, some pixels may record the the intensity variation during the phase stepping (n ¼
physical boundary of the particle. Thus, the issue of the 1;2; 3; · · · ; N). It is important to use the four-quadrant
fill factor must be considered. The particle (30 to 50 μm) inverse tangent, because this can distinguish the sign of the
could occupy several pixels on the sensor (4 to 10 pixels), numerator and denominator in the parentheses in Eq. (3),
and these pixels on the sensor will record blended resulting in a phase range of [−π; π]. The normal arctangent

Fig. 2 The benefit of MF calculations. (a) Schematic diagram of the processed data image. MF method
shows the same phase-index difference for different intensity variation cases (particles #1 and #2).
(b) Measured data using MF (top) and dark-field (bottom) methods on a sample with mask patterns
(gray arrows). MF method applies the same phase contrast to both background and mask patterns
with the particle signals. The intensity-based dark-field measurement suffers from the reflectance
given by the mask patterns.

Optical Engineering 092603-2 September 2019 • Vol. 58(9)

Downloaded From: https://www.spiedigitallibrary.org/journals/Optical-Engineering on 21 Apr 2019

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Choi et al.: Modulated dark-field phasing detection for automatic optical inspection

two light sources, the phase-index may have a middle value

between the extreme cases (pure-particle and pure-back-
ground). To ensure the clear detection of particles, it is com-
putationally preferable for the middle phase values to have
a clear bimodal distribution, which directly depends on the
two initial phase values Φ1 and Φ2 in Eqs. (1) and (2).
Table 1 lists four exemplary Φ1 and Φ2 combinations that
can be used to demonstrate the dependency. In Fig. 4, the
variation of the final phase-index values is presented as a
function of the contribution of the scattered particle intensity
into a single pixel signal. In other words, 100% denotes the
pure particle signal and 0% denotes the background signal
only. Though all cases could be used for the phase-index
Fig. 3 Maximized angular range space of the phase-index using calculation, to achieve a highly bimodal distribution with
the four-quadrant inverse tangent. We can calculate the phase-
index value based on the measured intensity variation at every single
a clear distinction between the particle and the background
pixel during the phase-stepping process. cases, case 1 (Φ1 ¼ −π∕4 and Φ2 ¼ π∕4) is used to provide
the performance evaluation results presented in the remain-
calculation does not account for the sign of the numerator der of this paper.
and denominator, and so only produces values in the range If we examine the phase variation trend with respect to the
[−π∕2; π∕2]. For instance, a tanð−1∕4Þ and a tanð1∕ − 4Þ fill factor, unlike the monotonically changing phase-index
give the same result. In contrast, using a tan Q enables us value for case 2, which has a continuous distribution, case
to calculate the phase-index over the maximum range of 1 provides the highest contrast (i.e., discontinuity with 2π
angle information, as shown in Fig. 3. jump) at the mixed phase region around the 50% fill factor.
This is even larger than the phase-index difference in the two
3 Case-Study Simulation for Different Initial Phase extreme cases (4π∕3 between the 0% and 100% fill factor
Comparison case) in Fig. 4. This clear distinction between the particle-
The value given by Eq. (3) will be mapped to the angular dominant and background-dominant cases offers a simple
phase space of Fig. 3 as a single phase-index angle. and robust threshold value for an AOI implementation.
Because some pixels record the mixed intensity from the Cases 3 and 4 are worth discussing, as they have only two
step values (case 3: π and 0, case 4: π∕2 and −π∕2) rather
Table 1 Four exemplary Φ1 and Φ2 combinations. than varying phase-index values as a function of the fill fac-
tor. Although this appears to offer a convenient contrast for
Case 1 Case 2 Case 3 Case 4
an AOI application, it is easily affected by noise around the
50% fill factor. For instance, in case 3, two signals mixed in
Φ1 −π∕4 0 0 π∕2 a single pixel are effectively A1 sinðxÞ þ A2 sinðx þ πÞ, and
the similar A1 and A2 values from the 50% fill factor situation
Φ2 π∕4 π∕2 π −π∕2
yield a sinusoidal signal with almost zero amplitude, which
jΦ1 − Φ2 j π∕2 π∕2 π π could induce a poor SNR. Thus, the final phase-index value
is not robust and reliable as an AOI threshold.

Fig. 4 (a) Phase-index calculation results as a function of the particle fill factor on a detector pixel signal.
(b) Phase variation on a polar plot, showing the bimodal jump when the fill factor is exactly 50%. The
arrow indicates the direction of the phase change as the contribution of the particle fill factor increases.
The radial value (amplitude) of each sampling point is scaled arbitrarily to visualize the change in the
calculated phase-index value.

Optical Engineering 092603-3 September 2019 • Vol. 58(9)

Downloaded From: https://www.spiedigitallibrary.org/journals/Optical-Engineering on 21 Apr 2019

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Choi et al.: Modulated dark-field phasing detection for automatic optical inspection

Another interesting aspect is when we have two very (Dell 1907FP) and camera (Pointgrey, FL3-U3-13Y3M-C),
small initial phase values with opposite signs, such as Φ1 ¼ as well as a three-point emitting LED (Cree LEDs, XHP 35)
−0.001π and Φ2 ¼ 0.001π. While this works perfectly well driven by an Arduino controller to produce the required
in ideal simulation cases, in practical applications with noise modulated signal. The camera was placed 55 cm from the
in the raw data, the total signal across all the fill factor values sample UUT, which was the Al-coated mirror shown in Fig. 5.
is simply too small and well below the noise level. In this Under these conditions, a single detector pixel occupied an
case, we are subject to similar limitations as the dark-field area of ∼80 × 80 μm on the sample. To create a constant
approach, where we are restricted by the overall SNR during reference sample, we sprayed a clean surface with particles
the data processing stage. of a known size. A polycrystalline particle spray (Struers,
DP-Spray P 35 μm) was used to deposit equal-sized particles
4 Experimental Performance Demonstration (∼35 μm) on the surface of the Al-coated UUT.
For the four MF cases, as predicted from the simulation
4.1 Performance Comparison study, case 1 produces the best contrast and clear particle
The four cases listed in Table 1 were experimentally tested boundaries in the calculated map. Case 2 fails to provide
and verified alongside the standard dark-field method using good-quality phase information because of the low contrast
the setup shown in Fig. 1. We used an off-the-shelf monitor between the particle and background signals. In contrast,
cases 3 and 4 give gradually changing phase-index values
near the particle boundary, because the boundary is less
clear (compared with the large 2π phase step in case 1) in
the phase-index space (see Fig. 4) in the presence of actual
measurement noise such as Gaussian white noise. In addi-
tion, these cases are similar to the dark-field result (bottom
of Fig. 6), which suffers from a fuzzy intensity change near
the particle boundary. This highlights the robustness of case
1 in the practical application of the MF approach. Therefore,
the optimal initial phase values of the MF approach defined
Fig. 5 Detector image of the Al-coated mirror surface of the UUT. This in Eq. (3) are Φ1 ¼ –π∕4 and Φ2 ¼ π∕4 (case 1), as they
is one of the raw images from the eight phase shifts (i.e., N ¼ 8). The provide the sharpest AOI distinction criteria with a high-
yellow box indicates the region of the data comparisons in Figs. 6–8. contrast particle map.

Fig. 6 Experimental results for cases 1–4 using the MF method and for the dark-field case. The x- and
y-axes represent the pixel number (location) on the detector.

Optical Engineering 092603-4 September 2019 • Vol. 58(9)

Downloaded From: https://www.spiedigitallibrary.org/journals/Optical-Engineering on 21 Apr 2019

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Choi et al.: Modulated dark-field phasing detection for automatic optical inspection

The sensitivity obtained from dark-field measurements during the data processing step. There is a stark contrast
can be improved by increasing the brightness of the LED, between particle and background in the phase-index map,
but this does not mean that a uniform signal will be received whereas the dark-field measurement produces an ambiguous
from all particles. The mask pattern, which has high reflec- particle boundary. For an AOI to determine the presence
tance, will also affect the results in this case. At the same (and/or size) of a particle, it is critical to generate the nor-
time, increasing the signal from the particles always im- malized map using a certain threshold value, but this is
proves the signals received in the phase measurements. often based on subjective human intuition. The MF method
Hence, for a given hardware configuration, the MF approach offers a much wider range of safe threshold values when cre-
yields more objective inspection results than the dark field ating an AOI than in the dark-field case. Any fluctuations in
in a realistic environment. noise do not severely disturb the results in the MF. However,
because of the fluctuating results in the dark field, human
4.2 Automatic Optical Inspection Performance intervention will often be needed to set the threshold and
verify the results.
Next, we adjusted the phase deviation and LED brightness to
The AOI algorithm categorizes a particle based on the
obtain the optimal results from the MF and dark-field meth-
ods, respectively, for a reliability test. A total of 10 trials were threshold chosen from the histogram plot. When implement-
conducted to check the repeatability of the methods, with 10 ing the dark-field approach, even if a threshold value is care-
snapshots taken in each trial for averaging purposes. To fully chosen for one trial, fluctuations between trials may
ensure a fair evaluation, the total number of snapshots for change the intensity distribution. The green error bars in
each case was fixed (for each trial, MF: 10 snaps ×8 mod- Fig. 7(b) show the fluctuations in the intensity distribution
ulations, dark field: 40 snaps ×2 for LED on and off). over the 10 trials. The top histogram in Fig. 7(b) clearly dem-
The results from one of the trials are shown in Fig. 7(a) onstrates the additional stability that the MF approach offers
and the associated statistical distribution is shown in Fig. 7(b). over the intensity-based dark-field results (bottom), indicat-
To give a clear view of the particle, we used inverted images ing a high-fidelity method. In other words, the modulated

Fig. 7 Comparison of particle detection measurement results from the different methodologies. (a) Image
from a single trial. The MF method (left) clearly produces distinguishable particle information, unlike the
dark-field result (right), which gives an unclear boundary. (b) Histogram of the pixel value distribution,
showing the phase-index (top) and intensity (bottom) from all pixels. The possible range of threshold
settings is shown in the red zone.

Optical Engineering 092603-5 September 2019 • Vol. 58(9)

Downloaded From: https://www.spiedigitallibrary.org/journals/Optical-Engineering on 21 Apr 2019

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Choi et al.: Modulated dark-field phasing detection for automatic optical inspection

References
1. H. Golnabi and A. Asadpour, “Design and application of industrial
machine vision systems,” Rob. Comput. Integr. Manuf. 23, 630–637
(2007).
2. K. Harding, Handbook of Optical Dimensional Metrology, Chapter 2,
CRC Press, Boca Raton, Florida (2013).
3. E. N. Malamas et al., “A survey on industrial vision systems, applica-
tions and tools,” Image Vision Comput. 21(2), 171–188 (2003).
4. H. Rau and C. H. Wu, “Automatic optical inspection for detecting
defects on printed circuit board inner layers,” Int. J. Adv. Manuf.
Fig. 8 AOI defect detection percentage (%) results for (a) MF and Technol. 25(9–10), 940–946 (2005).
(b) dark-field measurements. In the case of a fixed threshold value, 5. D. M. Tsai and C. Y. Hung, “Automatic defect inspection of patterned
thin film transistor-liquid crystal display (TFT-LCD) panels using
the dark-field method failed to detect three particles, two completely one-dimensional Fourier reconstruction and wavelet decomposition,”
(red arrow) and one 60% of the time (blue arrow), whereas the MF AOI Int. J. Prod. Res. 43(21), 4589–4607 (2005)
was able to detect all one particle, which was still detected 90% of 6. S. H. Gage, “Modern dark-field microscopy and the history of its
the time. development,” Trans. Am. Microsc. Soc. 39(2), 95–141 (1920).
7. M. Feldman and O. W. Lynn, “Inspection system utilizing dark-field
illumination,” U.S. Patent 4,595,289 (1986).
8. G. Addiego, “Automated specimen inspection system for and method of
dark-field phasing detection enables an enhanced robustness distinguishing features or anomalies under either bright field or dark
of the AOI method against random noise in the actual meas- field illumination,” U.S. Patent 5917588A (1999).
urement signal. However, we also acknowledge that this 9. K. Maslov, G. Stoica, and L. V. Wang, “In vivo dark-field reflection-
mode photoacoustic microscopy,” Opt. Lett. 30(6), 625–627 (2005).
does not mean a 100% immunity to such noise. Just like 10. P. T. Goedhart et al., “Sidestream dark field (SDF) imaging: a novel
other systems with uncertainties, there is still a chance, albeit stroboscopic LED ring-based imaging modality for clinical assessment
small, for false-positives and missed particles depending on of the microcirculation,” Opt. Express 15(23), 15101–15114 (2007).
the actual magnitude of the noise and the intrinsic errors of
Heejoo Choi is a postdoctoral researcher at James C. Wyant College
the built system. of Optical Sciences at the University of Arizona. He has studied
Figure 8 represents the results of a fixed (constant) thresh- nonlinear optical harmonic generation and optical metrology. His
old AOI over the 10 trials. While most particles were repeat- current research mainly focuses on the development of optical
edly detected in the MF method (one particle missed in one metrology for science and industry.
of the ten trials), the dark-field method failed to detect three
John Mineo Kam is a master’s student in the James C. Wyant
particles (red and blue arrows) with high fidelity. A lower College of Optical Sciences at the University of Arizona. He has
threshold value could have been set to allow the dark-field a background in physics, building interferometry systems to test
AOI to detect the missing particles, but this may result in the alignment of surfaces. His work in optical metrology continues
false positives depending on the noise levels. in the Large Optics Fabrication and Testing group, where he works
on new deflectometry techniques to surpass traditional systems lim-
ited by the surface figure of the unit under test.
5 Conclusion
Joel David Berkson is a senior undergraduate student studying
Using modulated phase information in addition to the previ- optical sciences and engineering at the University of Arizona. He is
ously established dark-field method, we have developed beginning his PhD in the fall of 2019. His main research interests
a reliable particle detection method called MF detection. include deflectometry, optical system design, and image science.
Under noisy or dynamic testing environments (e.g., room
Logan Rodriguez Graves is a PhD candidate in the James C. Wyant
light and reflectance of the substrate), a single parameter College of Optical Sciences at the University of Arizona. His main
setting (e.g., brightness of light sources and threshold for research area covers precision optical metrology, focusing on deflec-
judgment of particles) might not produce optimal or univer- tometry hardware and software development for improved freeform
sal solutions. The MF approach provides more robust results measurement capabilities. He also has a background in biomedical
engineering, with research topics including early cancer detection
than the dark-field approach at the expense of an additional utilizing autofluorescence and nonlinear rod–cone interactions at the
LCD screen, as it does not require a tightly controlled testing retinal level.
environment or skilled human input.
Dae Wook Kim is an assistant professor of optical sciences and
astronomy in the James C. Wyant College of Optical Sciences at
Acknowledgments the University of Arizona. His research area covers precision optical
This research was made possible in part by the II-VI engineering including interferometry and deflectometry. He is the
chair of the Optical Manufacturing and Testing (SPIE) and Optical
Foundation Block-Gift Program and the Technology Research Fabrication and Testing (OSA) conferences. He is a senior member
Initiative Fund Optics/Imaging Program. The authors wish to of OSA and SPIE and has been serving as an associate editor for
thank Spectral Optics for the use of their equipment. Optics Express.

Optical Engineering 092603-6 September 2019 • Vol. 58(9)

Downloaded From: https://www.spiedigitallibrary.org/journals/Optical-Engineering on 21 Apr 2019

Terms of Use: https://www.spiedigitallibrary.org/terms-of-use