Face recognition System for Smartphone based on LBP

Jesus Olivares-Mercado, Karina Toscano-Medina, Gabriel Sanchez-Perez

Hector Perez-Meana and Mariko Nakano-Miyatake
Instituto Politecnico Nacional, ESIME Culhuacan,
Av. Santa Ana 1000 San Francisco Culhuacan, Coyoacan, Mexico City Email: jolivares@ipn.mx

Abstract—This paper presents a face recognition algorithm depending on the device and sensor with which account.
based on Local Binary Pattern (LBP) to be implemented in a This topic has received significant attention. Due to this,
Smartphone with Android operating system where the input different schemes have been proposed to reduce the variable
image is obtained using the camera of such Smartphone. The illumination problems [4]. Some processes take the input
LBP algorithm is used for Face characterization, due to its low image to reduce the illumination changes and improve the
complexity and its robustness light of this method is chosen to quality of the input face image; examples of these processes
be applied in a Smartphone, this is because the light sensor including illumination are histogram equalization [5], and
of smartphone could darken or lighten the captured image contrast-limited adaptive histogram equalization (CLAHE)
and affect a efficient recognition. To perform system testing [6].
on a Smartphone was used a standard database (AR Face Also another approach to address the illumination con-
database) to simulate the capture of images, the average of ditions is the development of face recognition algorithms
images was used for obtaining a template by person and using that are able to provide a good performance under such
Euclidean distance for classification, showing that the LBP conditions and this depend the accuracy of the feature
obtains good results using a simple classification algorithm extraction method. Thus, several methods have been pro-
with a Smartphone with limited processing power like a posed to solve the problem. One example is the eigen-
smartphone, further tests were performed with 1 to 9 training
phase approach, which uses the phase spectrum together
images, obtaining up to 90% of recognition.
with principal component analysis (PCA) and the support
vector machine (SVM) [7], [8]. Feature extraction methods
based on other frequency transforms, such as the discrete
1. Introduction cosine transform [9], [10], discrete Gabor transform [11],
[12], discrete wavelet transform [13], [14], and discrete
Face recognition is one of the most widely used bio- Haar transform [15], have been proposed. These approaches,
metric technologies because the data acquisition approach under controlled conditions, achieve recognition rates of
is non-intrusive. Face recognition is performed by taking a over 90%. Several other approaches have been proposed
picture and can be performed with or without the coopera- to solve the problems related to changes in illumination
tion of the person under analysis. Thus, face recognition is conditions using genetic algorithms [16], image processing
a biometric technology that has obtained high acceptance filters [17], and linear regression-based classification [18] in
among users [1], [2]. A face recognition system can be recent years.
used for either identity verification or person identification. The Local Binary Pattern (LBP) [19], [20] algorithm
In the identity verification task, the system is asked to is one of the best texture descriptor methods. The accep-
determine whether the person is who he/she claims to be, tance and popularity level of this feature extraction method
whereas during the person identification task, the system has propitiate its implementation even in facial expression
is asked to determine, among a set of persons whose facial recognition systems, achieving good results. Therefore, This
characteristics are stored in a database, the person who most texture descriptor method is chosen to be applied, because
closely resembles the image under analysis [3], in n this have a lot of benefits like a low complexity to develop on
work only part of facial identification is covered. . a Smartphone. The LBP algorithm has been implemented
Variable illumination, pose, facial expressions are impor- only in simulation environments, where it has responded
tant problems that must be considered in the development of satisfactorily, as is shown in [21], [22], but has not yet been
face recognition systems because these factors significantly implemented in a biometric system that works in an uncon-
decreasing the accuracy of face recognition performance trolled environment, to prove if it responds acceptably in
[4]. Among these factors, changes in lighting conditions situations that are not presented in a simulation environment.
are very important because they occur not only due to the There has been increasing interest in the development of
differences on illumination conditions between indoor and face recognition schemes that are suitable for implementa-
outdoor environments but also with the light sensor of each tion in mobile devices, such as smartphones, which gener-
Smartphone, which produces different amount of lighting ally have low computational power. Because these systems
must operate in environments with varying illumination, it is
advantageous to maintain minimal computational complex-
ity, i.e., one that does not require a preprocessing stage to
improve the image quality. Hence, this paper proposes the
implementation of the LBP algorithm for face recognition.
After take the photo from Smartphone camera, detect and
crop the face of the image, it is divided in sub-blocks of
3 × 3 pixels, which are characterized by the LBP coefficient
corresponding to the central pixel of each sub-block. Finally,
Euclidean distance is used to perform recognition. The
proposed methods are evaluated with several illumination, Figure 1. Worldwide Smartphones sales until 2015.
relying on the AR face databases and a database made
directly in the Smartphone using the camera.
LBP will be implemented as feature extractors as part
of a facial recognition system for the Android operating
system, in order to assess the effectiveness and efficiency
of this algorithm.
The remaining of this paper is organized as follows:
Section 2 presents the description of the proposed method.
Section 3 presents and analysis of the experimental results.
Finally, Section 4 provides the conclusions of this research.

2. Proposed Method
Biometric systems are a set of automated methods for
recognizing people using physiological or personal behavior
characteristics [1], [2]. A biometric system is, essentially,
Figure 2. Worldwide Smartphones OS Market until 2015.
a pattern recognition system and, generally, can be di-
vided into four main modules: a capture module, a feature
extraction module, a comparison and classification and a
database module. In capture module is taken the picture to where the central pixel is used as a threshold to compare
analyze; in the feature extraction module, biometric data are values of its 8 neighbors. The pixels whose values are less
processed, and a set of discriminatory features is extracted to than the threshold must be labeled with 0 and those that
represent the most important features of the person identity are greater or equal than the threshold are labeled with 1,
under analysis, in this work are used the LBP for feature as shown in Figure. 3(b). Then, the labels of pixels are
extraction. In the decision module is used the Euclidean multiplied by 2P , where0 ≤ P ≤ 7 represents the position
distance to determine the winner class, and finally in the of each pixel in the neighborhood, as shown in Figure.
database module it’s stored all models of each person to 3(c). Finally, the resulting values are added to obtain the
identify. label of the central pixel of that neighborhood, as shown in
Figure. 3(d). This method produces 256 possible values for
the label of the central pixel. This process is repeated for
2.1. Android OS
the entire image and produces a LBP matrix (LBP image).
The importance of mobile devices called ”smart phones” It’s important to mention that the original method use the
has been increasing in recent years, thanks to technological histogram of values obtained above, but in this proposal is
development in this area. Then the percentage of use of used directly the values of LBP, and shown that the results
smartphones by brand is show in the Figure 1, are good enought.
Android is an operating system based on Linux kernel,
and was developed by Android Inc., which was acquired
by Google in 2005. Android was introduced in 2007 and
is currently the most widely used mobile operating system
in the world, as is reported by IDC and is shown in Figure 2.

2.2. Local Binary Pattern (LBP)

Figure 3. LBP Algorithm: (a) Values of neighbors around the central pixel.
The original Local Binary Pattern (LBP) method uses (b) Comparison of each neighbor with the central pixel. (c) Substitution of
each value of the comparison by the corresponding 2P value. (d) Adding
windows of 3x3 pixels of an image representing a neigh- and replacing of the central pixel with the resultant value.
borhood around the central pixel, as shown in Figure. 3(a),
2.3. General Scheme System 2.3.1. Input (face). Using the Smartphone camera face
pictures are taken as shown in Figure. 7 and Figure. 8. The
In order to apply the LBP algorithm, it will be used in system will detect the face of the person and will eliminate
a face recognition system; this system will be implemented all leftover image parts. Finally the cropped face image will
on the Android operating system. In Figure. 4, the general be the one used in the following steps.
scheme of face recognition system is shown and blocks will
be explain below.

 
#
   




#


 


 

$  

%



$%
 
! 

 
 

Figure 7. Opening the Smartphone camera to take the sample.

Figure 4. Block diagram of proposed face recognition system.

In this work 2 different tests were performed:

• The first test is performed in a real uncontrolled en-
vironment taking photographs with the Smartphone
camera an example of these images are shown in
Figure 5 and is performed a database with 10 dif-
ferent people, for training process were used only 3
images per person, then a test is made with 3 other
images of each person.

Figure 8. Taking the test sample.

Figure 5. Example of image taken from smartphone camera. 2.3.2. Re-dimension. The images are redimensioned in or-
der to maintain a standard size in every shot no matter
• The second test is performed with the AR Face much the distance, this is because not always the photo was
database and for practical purposes the database is taken at exactly the same distance from the person. Also all
stored in Smartphone memory simulating that photos pictures already redimensioned are passed to grayscale to
have already been taken with Smartphone camera, in subsequently apply the LBP algorithm.
this test are obtained results using a different amount
of training images an example of these images are 2.3.3. Feature Extraction. For feature extraction the LBP
shown in Figure 6 and test with all images on the algorithm will be implemented like explained above, to
database. obtain the feature vector of the sample. Figure. 8 shows
that the movil application have a button with ”Get LBP”
which apply the LBP algorithm to capture image, the result
of this block in the application on the Smartphone is shown
in Figure. 9. After apply the LBP to the image, this is
converted to row vector concatenating each row one after
another. Finally the average of different number of images
Figure 6. Example of images in AR Face database.
of the same person is used like model to classification stage.
 Figure 10. The system identifies the person that has been entered. The ID
Figure 9. Sample after application of the LBP, the execution time of the and the execution time of the algorithm are shown.
algorithm is shown.

camera and an outcome with the ID assigned and the time

2.3.4. Classification. This block of the system is divided it take to perform the process as shown in Figure. 10.
in two parts:
3. Experimental Results
Training: It will take place when individuals are entered
to the database. The feature vectors of each person will be
marked with the ID of the person to which they belong, 3.1. Test 1: make a 10 people database
and then, will be stored in the database. For this, tests with
different number of training images were made, were used For the training phase a database of 10 people was
from 1 to 9 training images taking into account that while taken, using the Smartphone camera in the input module.
more images are used to train the system, the recognition The system was trained with three photographs of each
rate will be higher as we will see in the part of results. To person, which were classified using the average of this three
obtain the model of each person, the average between these image like model. It was stored into the database as the
is computed by: feature vector of the person concerned. For the recognition
phase three photographs of each person were taken. Table
1 shows the successes, errors, effectiveness rate and the
n
 average execution time obtained.
IT (x, y) = ( Ii (x, y))/n i = 0, 1, . . . , n;
i=0
TABLE 1. R ESULTS OBTAINED WITH THE LBP ALGORITHM IN THE
x = 1, 2, . . . , N ; y = 1, 2, . . . , M TEST 1, WITH A DATABASE OF 10 PEOPLE AND A TRAINING OF 3
IMAGES PER PERSON .
where n is the number of image to obtain the average, N
is the height of redimensioned image and M is the width Algorithm Successes Failures Effectiveness Execution time
LBP 28 2 93.33% 9.2 ms.
of redimensioned image resized.

Identification: The identification part is done when you

want to identify a person, this task is done by taking the 3.2. Test 2
photograph from the Smartphone camera, then the image is
cropped, is redimensioned and applies the LBP algorithm, For the training phase a standard AR Face database of
later the Euclidian distance (equation 2) is used to compare 120 people where 65 are males and 55 females and 14
with the model of each person and so make a decision and images per person were used, it’s important to say that
allocation of identity. in this work only taken the images without oclussion from
 database, and this were storing on memory of Smartphone,
dst = (xs − yt )(xs − yt )T (1) to avoid the take of each picture through the Smartphone
camera. The system was trained with diferent number of
where xs is the estimated feature vector of the image under photographs of each person, which were classified using
analysis and yt is the centre of the t-th class. the average sample. It was stored into the database as the
feature vector of the person concerned.
It should be noted that the whole process is carried For the recognition phase, a total of 5040 pictures were
out internally in the Smartphone, the sample images shown analyzed. For increasing the number of illumination con-
above are to illustrate how the process is done, but in ditions, the AR database was expanded only for testing,
real application only the image taken with the Smartphone including 2 additional images per each one contained in
the original AR database. This allows us to increase three TABLE 2. R ESULTS OBTAINED WITH THE LBP ALGORITHM WITH
times the number of face images in the AR database, DIFFERENT NUMBER OF TRAINING IMAGE .

obtaining a larger number of face images with different 1 image 2 image 3 image 4 image 5 image
illumination conditions, simulating the posible function of 38.7 45.2 68.1 72.3 77.0
the light sensor of the camera increasing or decreasing the 6 image 7 image 8 image 9 image
illumination on images. To this end, the 14 face images, of 78.0 82.9 89.6 90.8
each person were used to generate 28 additional images of
each person with different illumination conditions by means
of the intensity transformation given by: 4. Conclusions
I(x, y) = |255(Iorig (x, y)/255)γ(x) | (2) The application on a Smartphone of a method such
where γ(x) is defined according to desired effect on the as LBP for facial feature extraction shows it have a good
resulting image. However, to generate face images with performance, in addition to the computational cost is low,
spatially varying illumination γ(x) was selected as follows: when tests were done with a standard database with 120
people is obtained a recognition rate of up to 90% using
models with 9 training images per person. It notes that
2(C − 1)x
γ(x) = − + C, 0 ≤ x ≤ M/2 (3) the implementation of a method of facial recognition on
M a Smartphone has many applications, besides that, having a
to produce face images where the illumination increases database within the Smartphone does not limit its use only
from left to right, and with an Internet connection allowing the user to make an
identification in real time. One possible application may be
2(C − 1)x for police department in the pursuit of suspects. Another
γ(x) = − (C − 2), M/2 ≤ x ≤ M (4) advantage of this application is that the runtime for iden-
M
tification is very short which makes the application even
to produce images where the illumination decreases from more attractive. The use of Android operating system is
left to right. because it is the most currently used in the market. When
Table 2 shows the results obtained with up to 9 training test 1 was developed shows that when a small real database
images. It can be seen that the effectiveness of the LBP algo- (only 10 persons) is used, the identification rate is until
rithm implemented on Smartphone is acceptable considering 93%, obviously when the database increases the number of
that the database contains images with variations in lighting training individuals the identification rate it diminished and
and some facial expressions. Also whereas a basic classifier the confusion in the application increases because of the
is used as the Euclidean distance that has low computational possible similarity between individuals. Another advantage
cost and the model of each person is obtained from a non- of this application is that the procedure use the Smartphone
supervised method, as is the average between images, the microprocessor and you not need use internet to send the
results are considered satisfactory. The results obtained in image to a server to apply the algorithms, and you can
the work of Yang and Cheng [23] where the AR database do real time test and does not matter the sensibility of the
and chi-square distance used are similar to those obtained Smartphone light sensor or if you are in a place with a lot
in the proposed system, Yang and Cheng present a study on of light or in a place something dark, this is thanks to LBP
hLBPI, this is comparable method with used in this work, which has the property of being invariant to illumination.
which obtains recognition rates between 64 and 92 % which
allows us to say that the results obtained in the Smartphone
application of the proposed system and conducting tests in a Acknowledgments
real uncontrolled environment are good enough like in [23]
using a simulated and controlled environment for testing; We thanks the National Science and Technology Council
In addition, you can also compare the results with those of of Mexico and to the Instituto Politecnico Nacional for the
hLBPH, eLBPH, Eigenface, Laplacianface and Fisherface financial support during the realization of this work.
which according to [23] obtained in most cases lower results
than those presented in this article. References
Also is remarkable that the time of execution for all im-
ages is considerably short, considering that the application
[1] S. Y. Kung, M.-W. Mak, and S.-H. Lin, Biometric authentication: a
is on a Smartphone with a low computational power. machine learning approach. Prentice Hall Professional Technical
These results can be compared with other methods such Reference, 2005.
as those developed in [24], where a database of 10 people in
[2] H. M. El-Bakry and N. Mastorakis, “Personal identification through
total was used, and 20 images of each person were used for biometric technology,” in 9th WSEAS International Conference on
training and 40 for testing, showing that results using opti- Applied Informatics and Communications (AIC09), Moscow, Russia,
SRC method, proposed in [24] on average is 90%, where the 2009, pp. 325–340.
time of processing is 239 ms, which is considerably higher [3] R. Chellappa, P. Sinha, and P. J. Phillips, “Face recognition by
than reported in this work, which is 9.2 ms. computers and humans,” Computer, vol. 43, no. 2, pp. 46–55, 2010.
[4] J. Ruiz-del Solar and J. Quinteros, “Illumination compensation and [21] B. Yuan, H. Cao, and J. Chu, “Combining local binary pattern and
normalization in eigenspace-based face recognition: A comparative local phase quantization for face recognition,” in Biometrics and
study of different pre-processing approaches,” Pattern Recognition Security Technologies (ISBAST), 2012 International Symposium on.
Letters, vol. 29, no. 14, pp. 1966–1979, 2008. IEEE, 2012, pp. 51–53.
[5] K. Ramirez-Gutierrez, D. Cruz-Perez, J. Olivares-Mercado, [22] S. Zhang, X. Zhao, and B. Lei, “Robust facial expression recognition
M. Nakano-Miyatake, and H. Perez-Meana, “A face recognition via compressive sensing,” Sensors, vol. 12, no. 3, pp. 3747–3761,
algorithm using eigenphases and histogram equalization,” 2012.
International journal of Computers, vol. 5, no. 1, pp. 34–41,
2011. [23] B. Yang and S. Chen, “A comparative study on local binary
pattern (lbp) based face recognition: {LBP} histogram versus
[6] G. Benitez-Garcia, J. Olivares-Mercado, G. Aguilar-Torres, {LBP} image,” Neurocomputing, vol. 120, pp. 365 – 379, 2013,
G. Sanchez-Perez, and H. Perez-Meana, “Face identification image Feature Detection and Description. [Online]. Available:
based on contrast limited adaptive histogram equalization (clahe),” http://www.sciencedirect.com/science/article/pii/S0925231213003068
in International Conference on Image Processing, Computer Vision
and Pattern Recognition. http:// www. worldacademyofsc ience. org/ [24] Y. Shen, W. Hu, M. Yang, B. Wei, S. Lucey, and C. T. Chou,
worldcomp11/ ws/ conferences/ ipcv11, 2011. “Face recognition on smartphones via optimised sparse representation
classification,” in IPSN-14 Proceedings of the 13th International
[7] G. Benitez-Garcia, J. Olivares-Mercado, G. Sanchez-Perez, Symposium on Information Processing in Sensor Networks, April
M. Nakano-Miyatake, and H. Perez-Meana, “A sub-block-based 2014, pp. 237–248.
eigenphases algorithm with optimum sub-block size,” Knowledge-
Based Systems, vol. 37, pp. 415–426, 2013.
[8] E. Owusu, Y. Zhan, and Q. R. Mao, “An svm-adaboost facial ex-
pression recognition system,” Applied intelligence, vol. 40, no. 3, pp.
536–545, 2014.
[9] N. A. Krisshna, V. K. Deepak, K. Manikantan, and S. Ramachandran,
“Face recognition using transform domain feature extraction and pso-
based feature selection,” Applied Soft Computing, vol. 22, pp. 141–
161, 2014.
[10] S. Dabbaghchian, M. P. Ghaemmaghami, and A. Aghagolzadeh,
“Feature extraction using discrete cosine transform and discrimination
power analysis with a face recognition technology,” Pattern Recog-
nition, vol. 43, no. 4, pp. 1431–1440, 2010.
[11] G. Aguilar-Torres, K. Toscano-Medina, G. Sanchez-Perez,
M. Nakano-Miyatake, and H. Perez-Meana, “Eigenface-gabor
algorithm for feature extraction in face recognition,” International
Journal of Computers, vol. 3, no. 1, pp. 20–30, 2009.
[12] H. Qin, L. Qin, L. Xue, and C. Yu, “Gabor-based weighted region
covariance matrix for face recognition,” Electronics letters, vol. 48,
no. 16, pp. 992–993, 2012.
[13] A. Eleyan, H. Özkaramanli, and H. Demirel, “Complex wavelet
transform-based face recognition,” EURASIP Journal on Advances
in Signal Processing, vol. 2008, no. 1, pp. 1–13, 2009.
[14] H. Hu, “Variable lighting face recognition using discrete wavelet
transform,” Pattern Recognition Letters, vol. 32, no. 13, pp. 1526–
1534, 2011.
[15] K. Gautam, N. Quadri, A. Pareek, and S. S. Choudhary, “A face
recognition system based on back propagation neural network using
haar wavelet transform and morphology,” in Emerging Trends in
Computing and Communication. Springer, 2014, pp. 87–94.
[16] H. R. Kanan and K. Faez, “Ga-based optimal selection of pzmi
features for face recognition,” Applied Mathematics and Computation,
vol. 205, no. 2, pp. 706–715, 2008.
[17] O. Arandjelović and R. Cipolla, “A methodology for rapid
illumination-invariant face recognition using image processing fil-
ters,” Computer Vision and Image Understanding, vol. 113, no. 2,
pp. 159–171, 2009.
[18] J.-X. Mi, J.-X. Liu, and J. Wen, “New robust face recognition methods
based on linear regression,” PloS one, vol. 7, no. 8, p. e42461, 2012.
[19] T. Ojala, M. Pietikainen, and D. Harwood, “Performance evaluation of
texture measures with classification based on kullback discrimination
of distributions,” in Pattern Recognition, 1994. Vol. 1 - Conference
A: Computer Vision amp; Image Processing., Proceedings of the 12th
IAPR International Conference on, vol. 1, Oct 1994, pp. 582–585
vol.1.
[20] X. Zhao and S. Zhang, “Facial expression recognition based on local
binary patterns and kernel discriminant isomap,” Sensors, vol. 11,
no. 10, pp. 9573–9588, 2011.