Automated classification of tropical shrub species: a hybrid of leaf shape and machine learning approach

Sampling sites and data collection

This dataset is named as myDAUN, in which ‘my’ represents Malaysia and ‘DAUN’ means leaf in the Malay language. The images in the myDAUN dataset were sampled from the campus of University of Malaya (UM), Kuala Lumpur, Malaysia. UM is Malaysia’s oldest university and is situated in the southwest of Kuala Lumpur, the capital of Malaysia. There is a botanical garden, Rimba Ilmu situated in the UM campus with over 1,600 plant species that emphasises the flora of the Malaysian and Indonesian region (University of Malaya, 1991). The myDAUN dataset was initially focused only on shrub species that are commonly seen by the public. Four main locations in UM with more variety of tropical shrubs were chosen and the sampling took place in these locations. The four locations were Faculty of Science, Dewan Tunku Canselor (Tunku Canselor Hall), Varsity Lake and Main Library as shown in Fig. 1.

The species of tropical shrub were identified and selected with the help of botanists. Since shrubs have a variety of species and cultivar, thus the advice from the professional botanists and the staff from the botanical garden were crucial and valuable. In this study, 45 species of common tropical shrubs were selected and 30 leaf samples were collected for each species. Hence, there were 1350 images of tropical shrub leaf images in total. Table 1 shows the selected species in the myDAUN database.

Proposed framework

The implementation of the tropical shrub species classification system is shown in Fig. 2. There are four main steps; image acquisition, image pre-processing, features extraction and the output of the classification process. Matlab R2015b was used to develop and test the proposed method.

Image acquisition

The myDAUN image dataset was collected and compiled in the laboratory in UM. Firstly, leaf samples that could represent the existing population were identified and the leaf samples from different parts of the shrub and size were plucked and collected. The standard criteria is to select a leaf that has not ruptured, is abnormal or damaged. Secateurs was used for a clean cut of the stem. Thirty samples of leaf were collected for each species. Each of the samples was labelled and the tags were attached securely onto the samples. Next, the collected samples were brought back to the laboratory for image acquisition. The samples were compressed and flattened using newspapers, and the leaf stalks were removed. In order to obtain a quality leaf photo, the light boxes were designed and two sizes of light boxes were used which were 37 cm × 59 cm × 13.5 cm for small-sized leaves and 93 cm × 111 cm  × 13.5 cm for bigger-sized leaves. The setup of image acquisition step is shown in Fig. 3.

The captured images are taken in the same standard with uniform background. The image of the leaf samples is captured on the front side, from a distance of 55 cm from the camera lens using Nikon D750 DSLR camera with AF-S Nikkor 24–120 mm F4G ED VR lens, with a resolution of 6,014 × 4,016 pixels and stored as 32-bits RGB colour Tagged Image File Format (tiff). Adobe Photoshop CC was used to enhance the image quality and to eliminate the illumination and contrast problem, which would affect the process of object segmentation. Figure 4 shows the samples of all tropical shrub species in the myDAUN dataset.

Image pre-processing

The main objectives of image pre-processing are to identify the main object, which is the leaf shape, and get rid of all other unrelated information. Before morphological descriptors were extracted, the region of interest (ROI) of the leaf must be obtained. Firstly, the RGB image or original image is converted to grayscale image. Next, Canny edge detection method is applied to the leaf grey scale images. The image with detected edge is then converted to a binary image and the shape of object is obtained after filling the holes in the binary image. To obtain the desired shape of leaf in the image, the small particles in the surrounding are removed and the ROI was obtained through the segmentation process. The steps for image pre-processing is shown in Fig. 5.

Feature extraction

The ROIs obtained from the image pre-processing step are used as input in the feature extraction steps. Four types of shape representations are applied and tested, which were morphological shape descriptors (MSD), Histogram of Oriented Gradients (HOG), Hu invariant moments and Zernike moments.

Morphological shape descriptors (MSD)

Five basic shape descriptors commonly used for leaf analysis were used in this study, namely diameter, major axis length, minor axis length, area, and perimeter. Based on these basic shape descriptors, 15 morphological descriptors are computed as shown in Table 2.

Histogram of oriented gradients (HOG)

The Histogram of Oriented Gradients (HOG) are descriptors used in image processing for object detection and it is the local statistic of the orientations of the image gradients around key points (Dalal & Triggs, 2005; Xiao et al., 2010). HOG descriptors method determines occurrences of gradient orientation in localized portions of an image or ROI. This technique is alike to scale-invariant feature transformation descriptors, edge orientation histograms and shape context. Gradient computation, G and gradient orientation, θ are computed using Eqs. (1) and (2) respectively.

(1)${\displaystyle \left|G\right|=\sqrt{G_x^2+G_y^2};}$ (2)${\displaystyle \theta =arctan\frac{G_x}{G_y};}$

where G_x is gradient in X direction and G_y is gradient in Y direction.

Each pixel within a cell casts a weighted vote for an orientation based histogram based on the gradient magnitude, G, and gradient orientation, θ. One histogram was counted for each cell based on the number of bins orientation binning. After that, the image was split into a number of cells. A cell can be represented as a region like a square with a predefined size in pixels. Each block has 3 × 3 cells; for each cell, the histogram of the gradient is obtained by splitting votes into bins for each orientation. The normalization was executed among a group of cells called a block, and a normalization factor was calculated over the block. All the histograms within this block were normalized and link together in single feature vector. The normalized vector, V can be performed by (3)${\displaystyle V=\frac{V_K}{{\parallel V_K\parallel }_2^2+{\epsilon }^2};}$

where V_K is the vector for combined histogram and ε is a small constant.

The histogram of all the blocks accumulated into a whole HOG descriptor were processed. In this study, the number of bins, K was set to 9, whereas the block size was 3 × 3 cells. Thus, there were 81-dimensional vector for each of leaf image based on the computation of 3 × 3 × 9.

Hu invariant moments

The moment invariant was first introduced by Hu (1962). Hu defined seven invariant moments computed from central moments through order up to three and two-dimensional that are invariant under object translation, scaling and rotation. Hence, a set of seven invariant moments can be derived from the normalized central moments as stated in Eq. (4).

${\displaystyle Hu1={\eta }_{20}+{\eta }_{02};}$ ${\displaystyle Hu2={\left({\eta }_{20}-{\eta }_{02}\right)}^2+4{\eta }_{11}^2;}$ ${\displaystyle Hu3={\left({\eta }_{30}-3{\eta }_{12}\right)}^2+{\left({\eta }_{03}-3{\eta }_{21}\right)}^2;}$ (4)${\displaystyle Hu4={\left({\eta }_{30}+3{\eta }_{12}\right)}^2+{\left({\eta }_{03}+{\eta }_{21}\right)}^2;}$ ${\displaystyle Hu5=\left({\eta }_{30}-3{\eta }_{12}\right)\left({\eta }_{30}+{\eta }_{12}\right)\left[{\left({\eta }_{30}+{\eta }_{12}\right)}^2-3{\left({\eta }_{03}+{\eta }_{21}\right)}^2\right]+\left(3{\eta }_{21}-{\eta }_{03}\right)\left({\eta }_{21}+{\eta }_{03}\right)\left[3{\left({\eta }_{30}+{\eta }_{12}\right)}^2-{\left({\eta }_{03}+{\eta }_{21}\right)}^2\right];}$ ${\displaystyle Hu6=\left({\eta }_{20}-{\eta }_{02}\right)\left[{\left({\eta }_{30}+{\eta }_{12}\right)}^2-{\left({\eta }_{03}+{\eta }_{21}\right)}^2\right]+4{\eta }_{11}\left({\eta }_{30}+{\eta }_{12}\right)\left({\eta }_{03}+{\eta }_{21}\right);}$ ${\displaystyle Hu7=\left(3{\eta }_{21}-{\eta }_{03}\right)\left({\eta }_{30}+{\eta }_{12}\right)\left[{\left({\eta }_{30}+{\eta }_{12}\right)}^2-3{\left({\eta }_{03}+{\eta }_{21}\right)}^2\right]+\left(3{\eta }_{21}-{\eta }_{30}\right)\left({\eta }_{21}+{\eta }_{03}\right)\left[3{\left({\eta }_{30}+{\eta }_{12}\right)}^2-{\left({\eta }_{03}+{\eta }_{21}\right)}^2\right];}$

Zernike moments

Zernike moments was firstly introduced by Teague (1980). In order to compute Zernike moments, three steps are required, namely computation of radial polynomials, Zernike basis functions, and Zernike moments. The approach to obtain Zernike moments from an input image starts with the computation of Zernike radial polynomials (Hwang & Kim, 2006). Zernike moments are based on Zernike polynomials that are orthogonal to the circle x² + y² = 1. The form of these polynomials is formulated in Eq. (5) (5)${\displaystyle K_{ab}\left(x,y\right)=R_{ab}\left(r\right)exp\left(jb\theta \right);}$

where a is a non-negative integer, b is positive or negative integer satisfying constrains a − |b| = even and |b| ≤ a. r is the radius of (x, y) to the centroid where $r=\sqrt{x^2+y^2}$, θ is the angle between r and x-axis where $\theta =ta{n}^{-1}\frac{y}{x},j=\sqrt{-1}$. R_ab is the radial polynomial defined as (6)${\displaystyle R_{ab}\left(r\right)=\sum _{s=0}^{\left(a-\left|b\right|\right)∕2}{\left(-1\right)}^s\frac{\left(a-s\right)!}{s!\left[\frac{a+\left|b\right|}{2}-s\right]!\left[\frac{a-\left|b\right|}{2}-s\right]!}{r}^{a-2s};}$

The Zernike moments for order a and b repetition of continued function f(x,y), if f(x,y) is a digital image, is defined below: (7)${\displaystyle Z_{ab}=\frac{a+1}{\pi }\sum _x\sum _{y}f\left(x,y\right)K_{ab}^{\ast }\left(r,\theta \right);}$

In this case, K^∗ is the complex conjugate, while K_ab is the Zernike basis functions order a with b repetitions, where $K_{ab}\left(x,y\right)=K_{ab}\left(r,\theta \right)=R_{ab}\left(r\right)exp\left(jb\theta \right)$.

These descriptors need to be normalized before classification. The normalized Zernike moments can be calculated using (8)${\displaystyle Z_{ab}^{\prime }=\frac{Z_{ab}}{m_{00}};}$

where m₀₀ is spatial moment order (0,0) that indicates the area of a leaf.

The Zernike moments with order a counting from 0 to 8 as the descriptors were selected and 25 descriptors of Zernike moment were obtained.

Feature selection

Feature selection is a process of identifying and removing the irrelevant and redundant features to describe the target concept. Feature selection reduced the dimensionality of the data and allowed learning algorithms to operate faster and more effectively. In this study, three feature selection methods are proposed, which are Relief, Correlation-based feature selection (CFS), and Pearson’s correlation coefficient (PCC). Relief is a distance based filter model that distinguish classes based on how well a feature can separate classes. CFS is a simple filter algorithm that ranks feature subset according to a correlation based heuristic evolution function (Hall, 1999). The PCC is a statistical method to analyse the relationship between features and decide which features are selected to train classifier.

Classification

Six classifiers were tested and applied, which are artificial neural network (ANN), random forest (RF), support vector machine (SVM), k-nearest neighbour (k-NN), linear discriminant analysis (LDA) and directed acyclic graph multiclass least squares twin support vector machine (DAG MLSTSVM). All of the classification algorithms were tested using a random sampling approach. The myDAUN dataset was randomly divided into 80% for training and 20% for testing. This process was repeated 10 times and the average of 10 runs was taken as the final result.

Artificial neural network (ANN)

Artificial neural network (ANN) is used as the classification tool. ANN is a biologically inspired program designed to stimulate the system in which the human brain processes information. The general neural network consists of three layers, which are input layer, hidden layer, and output layer. The ANN is composed of a set of neurons that is interconnected with each other. The total of 133 descriptors that was obtained, which includes 20 descriptors from MSD descriptor, 81 descriptors from HOG descriptors, seven descriptors from Hu moments and 25 descriptors from Zernike moments. The number of neurons in the hidden layer was experimentally selected from the error set by comparing with the general training of the ANN. The number of output neurons was presented by the number of species classified, which in this case, are 45 classes. The networks were two-layer feed forward with 133 input nodes and 45 output nodes as shown in Fig. 6.

Random forest (RF)

Random forest (RF) is based on the classification tree approach (Breiman, 2001). Predictions of multiple classification trees are aggregated for a dataset and each tree in the forest is grown using bootstrap samples. At prediction time, trees in the forest use their votes for target class and classification results are taken from each tree. The forest selected the class that achieved the most votes among the separate trees. RF can give an estimate of important input variables in the classification and it runs efficiently on large dataset with high accuracy. However, RF has some constraints on computing time and memory.

Support vector machine (SVM)

Support vector machine (SVM) (Cortes & Vapnik, 1995) is a classification method that maps the input data to a high dimensional feature space through some nonlinear transformation that separated by optimal hyperplane, which maximizes the gap of positive samples and negative samples. SVM used the kernel function to transform the input data into a higher dimensional space and optimal hyperplane is constructed with maximum margin. The classification involved 45 classes in this study, therefore SVM classifier was trained using the one versus all approach. In this approach, every class was trained with test cases of that class as positive and all other as negative.

k-nearest neighbour (k-NN)

k-nearest neighbour (k-NN) is a non-parametric classification algorithm that classifies unknown samples based on their k-nearest neighbour (Singh, Haddon & Markou, 2001). The most frequent class among these k neighbours is selected as the class of this sample. A challenge of k-NN is to determine a fit value of k, which is based on error rates. In these experiments, k-NN classifier with k = 1, 2, 3, 4, 5 were simulated and the case of k = 1 showed the best classification result.

Linear discriminant analysis (LDA)

Linear discriminant analysis (LDA) is also known as the Fisher discriminant analysis (FDA) (Alpaydin, 2014). LDA proposed to find a linear combination of features, which characterizes or separates different classes. LDA maximizes the ratio of between class variance to within class variance, thus achieving maximum discrimination. LDA contains weights for each feature separately for every class that allows it to ignore features that have no significant meaning for some classes.

Directed acyclic graph multiclass least squares twin support vector machine (DAG MLSTSVM)

Directed acyclic graph multiclass least squares twin support vector machine (DAG MLSTSVM) is a learning framework using directed acyclic graph that choose and reconstruct classifiers in “one-versus-one” algorithm. DAG MLSTSVM can solve a linear equation problem instead of using quadratic programming in the multiclass classification, which can lead to higher computational costs (Tomar & Agarwal, 2016).

Validation using Flavia dataset and Swedish Leaf dataset

The purpose of validation is to test on the viability of using hybridisation of four descriptors on the classification of plant species in other datasets. To validate the proposed methods, the Flavia dataset and Swedish Leaf dataset were used. Flavia and Swedish Leaf dataset are currently the most popular benchmark datasets used by researchers to compare and evaluate methods across studies. The dataset of Flavia consists of 32 species with 50 samples for each species. Whereas, Swedish Leaf dataset consists of 15 species with 75 samples per species. The validation applied the same settings as in the myDAUN dataset with the ratio of training and testing data set to 80:20 and used ANN as the classifier. The results obtained from both datasets were comparable with the myDAUN dataset, where the combination of MSD, HOG, Hu moments and Zernike moments are descriptors obtained the best accuracy.

Table 7 shows the average accuracy for various sets of descriptors in Flavia dataset and Swedish Leaf dataset. For single descriptor method, MSD and HOG descriptor achieved more than 93% accuracy in the Flavia dataset and more than 98% in the Swedish Leaf dataset. Hu invariant moments obtained the lowest accuracy in the classification of plant species, which was only 80.46% in the Flavia dataset and 95.20% in the Swedish Leaf dataset. The highest accuracy for the hybrid of two, three and four descriptors increased slightly for both dataset. The combination of all descriptors improved the classification accuracy and produced the best result for classification of plant species in the Flavia and Swedish Leaf datasets. The results achieved in the Flavia and Swedish Leaf datasets were comparable to those achieved in the myDAUN dataset.