Age and Gender Prediction From Face Images Using

Attentional Convolutional Network

Amirali Abdolrashidi1 , Mehdi Minaei2 , Elham Azimi3 , Shervin Minaee4

1 University of California, Riverside
2 Sama Technical College, Azad University
3 New York University
4 Snap Inc
arXiv:2010.03791v2 [cs.CV] 7 Dec 2020

Abstract—Automatic prediction of age and gender from face

images has drawn a lot of attention recently, due it is wide
applications in various facial analysis problems. However, due to
the large intra-class variation of face images (such as variation in
lighting, pose, scale, occlusion), the existing models are still behind
the desired accuracy level, which is necessary for the use of these
models in real-world applications. In this work, we propose a
deep learning framework, based on the ensemble of attentional
and residual convolutional networks, to predict gender and age
group of facial images with high accuracy rate. Using attention
mechanism enables our model to focus on the important and
informative parts of the face, which can help it to make a more
accurate prediction. We train our model in a multi-task learning
fashion, and augment the feature embedding of the age classifier,
with the predicted gender, and show that doing so can further
increase the accuracy of age prediction. Our model is trained on
a popular face age and gender dataset, and achieved promising
results. Through visualization of the attention maps of the train
Fig. 1. Three sample face images, and their corresponding attention map
model, we show that our model has learned to become sensitive outputs, from two different layers. Color scale is from blue (lowest) to red
to the right regions of the face. (highest).

I. I NTRODUCTION
as one of our backbone models, to better attend to the salient
Age and gender information are very important for various
and informative part of the face. Figure 1 provide three sample
real world applications, such as social understanding, biomet-
images, and the corresponding attention map outputs of two
rics, identity verification, video surveillance, human-computer
different layers of our model for these images. As we can see,
interaction, electronic customer, crowd behavior analysis, on-
the model outputs are mostly sensitive to the edge patterns
line advertisement, item recommendation, and many more.
around facial parts, as well as wrinkles, which are important
Despite their huge applications, being able to automatically
for age and gender prediction.
predicting age and gender from face images is a very hard
problem, mainly due to the various sources of intra-class As predicting age and gender from faces are very related,
variations on the facial images of people, which makes the we use a single model with multi-task learning approach to
use of these models in real world applications limited. jointly predict both gender and age bucket. Also, given that
knowing the gender of someone, we can better estimate her/his
There are numerous works proposed for age and gender
age, we augment the feature of the age-prediction branch with
prediction in the past several years. The earlier works were
the predicted gender output. Through experimental results, we
mainly based on hand-crafted features extracted facial images
show that adding the predicted gender information to the age-
followed by a classifier. But with the great success of deep
prediction branch, improves the model performance. To further
learning models in various computer vision problems in the
improve the prediction accuracy of our model, we combine the
past decade [1]–[5], the more recent works on age and gender
prediction of attentional network with the residual network,
predictions are mostly shifted toward deep neural networks
and use their ensemble model as the final predictor.
based models.
In this work, we propose a deep learning framework to Here are the contributions of this work:
jointly predict the age and gender from face images. Given • We propose a multi-task learning framework to jointly
the intuition that some local regions of the face have more predict the age and gender of individuals from their
clear signals about the age and gender of an individual (such face images.
as beard and mustache for male, and wrinkles around eyes and
mouth for age), we use an attentional convolutional network • We develop an ensemble of attentional and residual
 networks, which outperforms both individual models. classification in uncontrolled environments. Not surprisingly,
The attention layers of our model learn to focus on they were able to obtain promising results on the features
the most important and salient parts of the face. learned from these popular architectures.
• We further propose to feed the predicted gender label In [18], Lapuschkin et al. compared four popular neural
to the age prediction branch, and show that doing this network architectures, studied the effect of pre-training, evalu-
will improve the accuracy of age prediction branch. ated the robustness of the considered alignment preprocessings
• With the help of the attention mechanism, we can via cross-method test set swapping, and intuitively visualized
explain the predictions of the classifiers after they are the model’s prediction strategies in given pre-processing con-
trained, by locating the salient facial regions they are ditions using the Layer-wise Relevance Propagation (LRP)
focusing on each image. algorithm. They were able to obtain very interesting relevance
maps for some of the popular model architectures, as shown
The structure of the remaining parts of this paper is as in Figure 3. [15].
follows. Section II provides an overview of some of the previ-
ous works on age and gender prediction. Section III provides
the details of our proposed framework, and the architecture of
our multi-task learning model. Section IV, provides a quick
overview of the dataset used in our framework. Then, in Sec-
tion V, we provide the experimental studies, the quantitative
performance of our model, and also visual evaluation of model
outputs. Finally the paper is concluded in Section VI.

II. R ELATED W ORKS

Face is one of the most popular biometrics (along with
fingerprint, iris, and palmprint [6]–[11]), and face recognition
and facial attributes/characteristics prediction have attracted a
lot of attention in the past few decades [12]–[14]. Age and
gender prediction from face images, as a specific face analysis
problem have also drawn attention in the recent years, and
there have been several previous works for age and gender
prediction from face images. Here we provide an overview of Fig. 3. From top to bottom: Input image, followed by relevance maps for
some of the most promising works. the best performing CaffeNet, GoogleNet and the VGG16 model for gender
prediction. Courtesy of [18].
In [15], Levi and Hassner proposed a simple convolutional
neutral network architecture with 5 layers, to perform age and
gender prediction. Despite the simplicity of this model, they In [19], Rodriguez et al. proposed a novel feedforward at-
achieved promising results on Adience benchmark for age and tention mechanism that is able to discover the most informative
gender estimation. Figure 2 shows the architecture of the model and reliable parts of a given face for improving age and gender
proposed in [15]. classification. More specifically, given a downsampled facial
image, the proposed model is trained based on an end-to-end
learning framework to extract the most discriminative patches
from the original high-resolution image. They were able to
obtain promising results on several benchmarks, including
Adience, Images of Groups, and MORPH II. The block-
diagram of this work is shown in Figure 4.

Fig. 2. The architecture of the work by Levi and Hassner, courtesy of [15].

In [16], Duan et al. proposed a hybrid structure, which

combines Convolutional Neural Network (CNN) and Extreme
Learning Machine (ELM), to perform age and gender classi-
fication. CNN is used to extract the features from the input
images, while ELM classifies the intermediate results. They
were able to obtain reasonable accuracy on the MORPHII and
Adience Benchmarks. Fig. 4. The proposed attention model by Rodriguez et al. Courtesy of [19].

In [17], Ozbulak et al. analyzed the transferability of

existing deep convolutional neural network (CNN) models for Some of the other promising works for age and gender pre-
age and gender classification. The generic AlexNet-like archi- dictions includes: adversarial spatial frequency domain critic
tecture and domain specific VGG-Face CNN model are fine- learning [20], Region-SIFT and multi-layered SVM [21], and
tuned with the Adience dataset prepared for age and gender Landmark-Guided Local Deep Neural Networks [22].
Fig. 5. The architecture of residual attention network, courtesy of [23].

III. T HE P ROPOSED F RAMEWORK average predictions (output probabilities) are used as the final
prediction. We will give more details on the architecture of
In this section we provide the details of the proposed
each of these two models in the below parts.
age and gender prediction framework. We formulate this as
a multi-task learning problem, in which a single model is
used to predict both gender and age-buckets simultaneously. A. Residual Attentional Network
In another word, a single convolutional neural network with As mentioned above, an important piece of our framework
two heads (output branches) is used to jointly predict age and is the residual attentional network (RAN), which is a convo-
gender. Figure 6 shows the block-diagram of a simple multi- lutional neural network with attention mechanism, which can
task learning model for joint age and gender prediction. be incorporated with state-of-art feed forward network archi-
tecture in an end-to-end training fashion [23]. This network is
built by stacking attention modules, which generate attention-
aware features that adaptively changes as layers go deeper into
the network.
The composition of the Attention Module includes two
branches: the trunk branch and the mask branch. Trunk Branch
Fig. 6. The block-diagram of a multi-task learning network for joint age and performs feature processing with Residual Units. Mask Branch
gender prediction. uses bottom-up top-down structure softly weight output fea-
tures with the goal of improving trunk branch features. Bottom-
Up Step: collects global information of the whole image by
Given the intuition that knowing one’s gender, can enable
downsampling (i.e. max pooling) the image. Top-Down Step:
us to better predict her/his age, we augment the input feature
combines global information with original feature maps by
of the age prediction part of the model, with the the predicted-
upsampling (i.e. interpolation) to keep the output size the same
gender from the other head. This can help us the age model
as the input feature map. The full architecture of residual
to have access to a rough estimation of gender. Through
attention network is shown in Figure 5.
experimental study, we show that doing so improves the
performance of the age prediction. Figure 7, provides the block
diagram of the proposed model architecture. B. ResNet Model
Another model used in our framework is based on residual
convolutional network (ResNet) [24]. ResNet is known to
have a better gradient flow by providing the skip connection
in each residual block. Here we use a ResNet architecture
to perform gender classification on the input image. Then,
the predicted output of the gender branch is concatenated
with the last hidden layer of the age branch. The overall
block diagram of ResNet18 model is illustrated in Figure 8.
Fig. 7. The block-diagram of the framework for joint age and gender
prediction, in which the predicted gender is used along with the neural
ResNet50 architecture is pretty similar to ResNet18, the main
embedding as the input for the age prediction branch. difference being having more layers.

C. Ensemble of Attentional and Residual Networks

To further boost the accuracy of our model, we propose
to use an ensemble of attentional network, and a residual To boost the prediction accuracy of our model, we use the
convolutional network. Once these models are trianed, their ensemble of the above two architectures, and combine their
 7x7x64 kernel, /2
3x3 Max Pooling

7x7 Avg Pooling

3x3x128 kernel

3x3x128 kernel

3x3x128 kernel

3x3x128 kernel

3x3x256 kernel

3x3x256 kernel

3x3x256 kernel

3x3x512 kernel

3x3x512 kernel

3x3x512 kernel
3x3x256 kernel

3x3x512 kernel
3x3x64 kernel

3x3x64 kernel

3x3x64 kernel

3x3x64 kernel
Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

Convolution

512x1000
Softmax
FC layer
Input Classes

Fig. 8. The architecture of ResNet18 model. Courtesy of [24].

predictions. Through experimental results, we show that the

ensemble model achieves higher performance than both of the
individual models.

IV. UTK-FACE DATASET

Fig. 10. Sample images from UTK-Face database.
In this section, we provide a quick overview of the dataset
used in this work, UTK-Face dataset [25]. We used the cropped
and aligned version of this dataset, which consists of 9780 B. Model Quantitative Performance
images of size 200x200 (4372 male, 5408 female). The images
are taken from the faces of people from various ethnic groups In this part, we provide the model performance in terms
and ages (1 to 110 years). The age distribution of the used of various metrics, such as classification accuracy for both
dataset is shown in Figure 9. It can be seen that the largest branches (gender prediction, and age-bucket classification),
portion of the images belong to people under 2 years of age. as well as ”average age-bucket absolute difference (AABD)”.
AABD calculates the absolute difference between the ground-
1200
Age Distribution truth and the predicted age-bucket of each person, and average
1000 them out among all test samples. As an example, if someone’s
800
600
age is 25 (its age-bucket being 20-30, therefore 3), and the
400 predicted age by model is 4, then the AABD value will be 1.
200
0
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106
As mentioned earlier, each model predicts a probability
vector, which shows the likelihood of that image being in each
of the possible classes (for example, male vs female, for gender
Fig. 9. Age distribution in the used UTK-Face dataset.
model). In the ensemble model, the predicted probabilities
from Attentional-CNN and ResNet are averaged and used to
Also, Figure 10 denotes several images from UTK-Face infer the predicted classes of each task. In Table I, we presents
dataset. As we can see frm this figure, UKT-Face dataset has the comparison between the proposed ensemble model, with
a good diversity in terms of age and gender. It is worth to note each of the two individual models. As we can see the ensemble
that, in order to make the age prediction simpler, we group model outperforms both the individual models on both tasks
people whose age is in some range into the same age-bucket (gender and age prediction). In Table II, we compare the
(such as 0-10), and try to predict their age-bucket instead of performance of the proposed model with one of the previous
the exact age. Doing so, on one hand makes age prediction works, on gender classification task.
simpler, and on the other hand turns this model from regression
TABLE I. C OMPARISON OF ACCURACY OF DIFFERENT MODELS
to classification (which does not need any clipping of the
predicted values). Model Age Range Acc Gender Acc Age Bucket Diff
Attention CNN 0.742 0.552 0.33
ResNet 0.900 0.965 0.14
V. E XPERIMENTAL R ESULTS The Ensemble Model 0.913 0.965 0.11

Before presenting the quantitative and qualitative perfor-

mance of the proposed framework, let us first discuss about TABLE II. C OMPARISON WITH A PREVIOUS WORK .
the hyper-parameter values used in our training procedure. Model Gender Acc
These values are tuned based on the model performance on Automatic Gender Detection [27] 0.9485
Our model 0.965
the validation set.

A. Model Hyper-parameter Values C. Model Predicted Scores

The proposed model has been trained for over 100 epochs Figure 11 shows the histogram of the predicted probability
on an Nvidia Tesla GPU. The batch size is set to 16, and scores for gender by our model, for each class. Note that
number of workers to 8. ADAM optimizer with a learning here we are showing the histogram of the likelihood of an
rate of 0.005 is used to optimize the loss function. PyTorch image being female. It can be seen that in the majority of the
library is used for the implementation of experimental studies cases, the model’s prediction of gender is much higher in the
[26]. extremes, i.e. the model makes its decision with remarkable
certainty. On the other hand, more uncertain decisions, which
are shown in the middle of the chart, are much lower in
frequency.

Predicted Probability
0.5 Male Female

0.4

0.3

0.2

0.1

0
[0, 0.1] [0.1, 0.2] [0.2, 0.3] [0.3, 0.4] [0.4, 0.5] [0.5, 0.6] [0.6, 0.7] [0.7, 0.8] [0.8, 0.9] [0.9, 1]

Fig. 11. Predicted probability distribution for gender classification

D. Model Predictions for Some Sample Images

We also present the model prediction for some of the
sample images of our test set. In Figure 12, some of the Fig. 13. Confusion matrix for age range
model’s predictions are provided next to the true labels of
the face image. It can be seen that, for most of these sample
images, the true age label falls within the predicted age range
by our model.

Gender: Gender: Gender: Gender: Gender:

Predicted: male Predicted: female Predicted: male Predicted: male Predicted: female
Label: male Label: female Label: male Label: male Label: female
Age: Age: Age: Age: Age:
Predicted: 50-60 Predicted: 30-40 Predicted: 40-50 Predicted: 70+ Predicted: 20-30
Label: 58 Label: 35 Label: 49 Label: 70 Label: 25

Fig. 14. Confusion matrix for gender outputs

Fig. 12. Classification results for sample images

VI. C ONCLUSION

E. Model Confusion Matrix In this work we propose a multi-task learning framework,

to simultaneously predict age and gender from face images.
To provide a more detailed analysis of the proposed Our framework is based on an ensemble of ResNet-based
model’s performance, we also present the confusion matrix model and an attention-based model. We trained and tested
of this model for each of the two tasks. Figure 13 and the proposed model on the UTKFace dataset consisting a large
Figure 14 show the confusion matrix for age range and gender variety of faces from different ages, genders and ethnicities.
classification respectively. It can clearly be seen that a vast Through experimental studies, we show that the prediction
majority of the cases are predicted as the true label of their accuracy of the ensemble model (for both age and gender
class, seen on the main diagonal of the matrix. It is worth prediction tasks) surpasses in those of the separated models.
noting the largest false positive percentage in the age range We also showed that providing the prediction of the gender
classification model, corresponds to the images belonging to model as one of the input signal for the age-prediction branch,
the 30-40 age range which are mistaken for 20-30. can improve the accuracy of predicted age values. Through
visualization of the attention maps of the trained model, we
show that the model learned to focus on the most salient part
F. Model Attention Map Visualization of the face, useful for predicting age and gender.

In Figure 15, we show the output of two of the attention R EFERENCES

maps (from the three attention modules in our model) of the
trained model for a few sample images from the test set. As [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in neural
it can be seen from this figure, our model is learned to extract information processing systems, 2012, pp. 1097–1105.
features from different salient parts of the face, such as the [2] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time
outline, eyes, and wrinkles, which sounds reasonable when object detection with region proposal networks,” in Advances in neural
trying to make prediction about someone’s age and gender. information processing systems, 2015, pp. 91–99.
Fig. 15. The attention maps of our model, for eight sample images.

[3] S. Minaee, A. Abdolrashidi, H. Su, M. Bennamoun, and D. Zhang, Conference of the Biometrics Special Interest Group (BIOSIG). IEEE,
“Biometric recognition using deep learning: A survey,” arXiv preprint 2016, pp. 1–6.
arXiv:1912.00271, 2019. [18] S. Lapuschkin, A. Binder, K.-R. Muller, and W. Samek, “Understanding
[4] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, and comparing deep neural networks for age and gender classification,”
“Deeplab: Semantic image segmentation with deep convolutional nets, in Proceedings of the IEEE International Conference on Computer
atrous convolution, and fully connected crfs,” IEEE transactions on Vision Workshops, 2017, pp. 1629–1638.
pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834–848, [19] P. Rodrı́guez, G. Cucurull, J. M. Gonfaus, F. X. Roca, and J. Gonzàlez,
2017. “Age and gender recognition in the wild with deep attention,” Pattern
[5] S. Minaee, Y. Wang, A. Aygar, S. Chung, X. Wang, Y. W. Lui, Recognition, vol. 72, pp. 563–571, 2017.
E. Fieremans, S. Flanagan, and J. Rath, “Mtbi identification from [20] S. S. Lee, H. G. Kim, K. Kim, and Y. M. Ro, “Adversarial spatial
diffusion mr images using bag of adversarial visual features,” IEEE frequency domain critic learning for age and gender classification,” in
transactions on medical imaging, vol. 38, no. 11, pp. 2545–2555, 2019. 2018 25th IEEE International Conference on Image Processing (ICIP).
[6] Q. Zhao, L. Zhang, D. Zhang, and N. Luo, “Direct pore matching IEEE, 2018, pp. 2032–2036.
for fingerprint recognition,” in International Conference on Biometrics. [21] H. Kim, S.-H. Lee, M.-K. Sohn, and B. Hwang, “Age and gender esti-
Springer, 2009, pp. 597–606. mation using region-sift and multi-layered svm,” in Tenth International
Conference on Machine Vision (ICMV 2017), vol. 10696. International
[7] S. Minaee and Y. Wang, “Fingerprint recognition using translation
Society for Optics and Photonics, 2018, p. 106962J.
invariant scattering network,” in 2015 IEEE Signal Processing in
Medicine and Biology Symposium (SPMB). IEEE, 2015, pp. 1–6. [22] Y. Zhang and T. Xu, “Landmark-guided local deep neural networks for
age and gender classification,” Journal of Sensors, vol. 2018, 2018.
[8] M. De Marsico, A. Petrosino, and S. Ricciardi, “Iris recognition through
machine learning techniques: A survey,” Pattern Recognition Letters, [23] F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang,
vol. 82, pp. 106–115, 2016. and X. Tang, “Residual attention network for image classification,” in
Proceedings of the IEEE conference on computer vision and pattern
[9] S. Minaee, A. Abdolrashidiy, and Y. Wang, “An experimental study of recognition, 2017, pp. 3156–3164.
deep convolutional features for iris recognition,” in 2016 IEEE signal
[24] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
processing in medicine and biology symposium (SPMB). IEEE, 2016,
recognition,” in Proceedings of the IEEE conference on computer vision
pp. 1–6.
and pattern recognition, 2016, pp. 770–778.
[10] S. Minaee and A. Abdolrashidi, “Highly accurate palmprint recognition [25] Z. Zhang, Y. Song, and H. Qi, “Age progression/regression by condi-
using statistical and wavelet features,” in 2015 IEEE Signal Processing tional adversarial autoencoder,” in Proceedings of the IEEE conference
and Signal Processing Education Workshop (SP/SPE). IEEE, 2015, on computer vision and pattern recognition, 2017, pp. 5810–5818.
pp. 31–36.
[26] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin,
[11] S. A. Mistani, S. Minaee, and E. Fatemizadeh, “Multispectral palmprint A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in
recognition using a hybrid feature,” arXiv preprint arXiv:1112.5997, pytorch,” 2017.
2011.
[27] R. R. Nair, R. Madhavankutty, and S. Nema, “Automated detection of
[12] J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, “Robust face gender from face images,” 2019.
recognition via sparse representation,” IEEE transactions on pattern
analysis and machine intelligence, vol. 31, no. 2, pp. 210–227, 2008.
[13] O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,”
2015.
[14] S. Minaee, A. Abdolrashidi, and Y. Wang, “Face recognition using
scattering convolutional network,” in 2017 IEEE Signal Processing in
Medicine and Biology Symposium (SPMB). IEEE, 2017, pp. 1–6.
[15] G. Levi and T. Hassner, “Age and gender classification using convo-
lutional neural networks,” in Proceedings of the IEEE conference on
computer vision and pattern recognition workshops, 2015, pp. 34–42.
[16] M. Duan, K. Li, C. Yang, and K. Li, “A hybrid deep learning cnn–
elm for age and gender classification,” Neurocomputing, vol. 275, pp.
448–461, 2018.
[17] G. Ozbulak, Y. Aytar, and H. K. Ekenel, “How transferable are cnn-
based features for age and gender classification?” in 2016 International