Department of Computer Science and Engineering University of Moratuwa
CS4202 - Research and Development Project
OPTICAL COLOR SORTING MACHINE FOR TEA PARTICLE SORTING
Group Members
120026E 120126K 120162P 120496R
J.P.B Anushka A.Y Dissanayake H.V.L.C Gamage K.A.D.B Priyadarshana
Supervisors Mr. Nalin Karunasinghe
Coordinated by Dr. Charith Chithranjan THIS REPORT IS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE AWARD OF THE DEGREE OF BACHELOR OF SCIENCE OF ENGINEERING AT UNIVERSITY OF MORATUWA, SRI LANKA. April 06, 2017
DECLARATION We, the project group Sortex hereby declare that except where specified reference is made to the work of others, the project Sortex - an optical tea sorter for Sri Lankan tea industry- is our own work and contains nothing which is the outcome of work done in collaboration with others, except as specified in the text and Acknowledgement.
Signatures of the candidates:
....................................................... J.P.B Anushka [120026E] ....................................................... A.Y Dissanayake [120126K] ....................................................... H.V.L.C Gamage [120162P] ....................................................... K.A.D.B Priyadarshana [120496R]
Supervisor:
....................................................... (Signature and Date) Mr. Nalin Karunasinghe Coordinator: ....................................................... (Signature and Date) Dr. Charith Chithranjan
i
ABSTRACT
Tea products are present in incredible varieties in terms of shape, size, color and as the market grows more demanding, the tea products are subdivided in various categories and are destined to different segments. The definition and characterization of different attributes of tea are very important for the business and for the consumer, making it necessary to establish norms of classification and standardization, thus making commercial trading more efficient and allowing for higher awareness on the part of consumers. A computer vision system (CVS) has proven very successful in the analysis on the basis of color, size, shape, texture etc in food engineering. Through this project we use these computer vision and image processing techniques for implementing the optical tea sorting machine. And apart from that through the research part, the advantages of using a color pixels classification algorithm in a Hybrid Color Space with derived color components were presented. This new color space depends on the application. This introduced a new method which is a lightweight solution for implementation and more accurate than feed forward neural network. To increase the accuracy of result evaluation the characteristic curve of histogram was used instead of histogram overlapping calculation method which reduces the dependency on training sample and the accuracy level.
ii
ACKNOWLEDGEMENT The final year research and development project module (CS4202) is great chance for the students to explore new fields of study in computer science and engineering discipline expand our knowledge. We would like express our gratitude towards Department of Computer Science & Engineering for giving us the opportunity to perform this kind of work by offering the module. We would like to thank Eng. Nalin for the support in both knowledge-wise and resources-wise to make this project a success. And also we would like to thank Eng. Yasura Vithana for sharing his expertise knowledge in the field and spending his valuable time to guide us through the project.
Last but not least, we would like to express our heartiest gratitude towards the evaluation panel members Dr. Chandana Gamage and Dr. Charith (project co-ordinator) for all the support & advices given and the effort made in conducting this module and for the resources we were given carry out the project.
Finally, we would like express our heartiest gratitude towards our friends and family for the immense support given throughout the project.
iii
TABLE OF CONTENTS Chapter 1: Introduction
9
1.1 Background
9
1.2 Problem statement
9
1.3 Motivation
10
1.4 Requirement Specification
10
Chapter 2 : Literature Review
11
2.1 Abstract
11
2.2 Introduction
11
2.3 Computer Vision System
13
2.4 Image Processing System
13
2.5 Color Feature extraction
14
2.6 Morphological feature extraction
16
2.7 Shape feature extraction
17
2.8 Texture feature extraction
18
2.9 Knowledge base Learning
23
2.10 Hardware Used for Sorting
27
Chapter 3: Experimental Procedure
30
3.1 Preprocessing
30
3.1.2 Background removing
58
3.1.3 White Balancing
58
3.1.3 Data extraction
59
3.1.4 Data cleaning
59
3.1.5 Feature Selections
60
3.1.6 Data Transformation
63
3.2 Testing With Different Machine Learning Model & Evaluation of the Results
64
Evaluation Criteria
64
3.2.1 Artificial Neural Network:
66
3.2.2 Perceptron
67
3.2.3 Naive Bayes
67
3.2.4 Support Vector Machine
68
ROC - SVM Linear kernel
69
ROC - SVM- rbf kernel
69
3.2.5 Random Forest
70
3.3 Result Optimization
70
iv
3.3.1 Feature Selection
70
3.3.2 Derive New Attributes.
71
3.3.3 Outlier Analysis
71
3.3.4
71
Optimized Results
3.3.5 Final Algorithm of the Research Component
76
Chapter 4 : System Design
77
4.1 Component Diagram
77
4.1.1 Image Acquisition Subsystem
77
4.1.2 Data Extracting module
77
4.1.3 Decision Making Module
78
4.1.4 Actuator Controller
78
4.1.5 Controlling Module
78
4.1.6 Human Interface Module
78
4.1.7 Machine learning System
78
4.2 High Level Hardware Architecture
79
4.3 Detailed Architecture
80
4.3.1 Real Time Stem/Leaf Classification Module
80
4.3.2 Controlling module
80
4.4 Data Flow Diagram
81
3.5 Decision Making Flowchart
83
Chapter 5 : Implementation
84
5.1 Real time Stem/Leaf Classification Module
84
5.1.1 Hardware used
84
5.1.2 Implementation Details
85
5.2 Machine Learning Module
87
5.2.1.Pre Processing Unit
87
5.2.3.Feature Extraction Unit
88
5.2.4.Feature Selection and analytical unit
88
5.3
Human Machine Interaction Module and Controlling Module
5.3.1 Hardware Used
91 91
Chapter 6: Discussion
95
Chapter 7: Conclusions
97
v
LIST OF FIGURES Figure 1: Tea Manufacturing Process Figure 2: Process of a Computer Vision Based System to Obtain the Tea Image Database Figure 3 : Example Visualization of Good Cluster Figure 4 : Example Visualization of Bad Cluster Figure 5 : Basic concept of a machine learning system Figure 6 : structure of the artificial neural network.
9 14 20 21 23 24
Figure 7 : Summary of ANN application for classification of food product done in [6]. Figure 8 : maximum margin hyper plane for a SVM with two class data set Figure 9 : SVM Classification Architecture of the literature [18] Figure 10 : Tea samples from different areas of the country Figure 11 : Y Component Histogram Figure 12 : Cb Component Histogram Figure 13 : Cr Component Histogram Figure 14 : R Component Histogram Figure 15 : G Component Histogram Figure 16 : B Component Histogram Figure 17 :L Component Histogram Figure 18 : *a Component Histogram Figure 19 : *b Component Histogram Figure 20 : H Component Histogram Figure 21 : S Component Histogram Figure 22 : V Component Histogram Figure 23 : Y Component Histogram Figure 24 : Y Component Histogram Figure 25 : Cr Component Histogram Figure 26 : R Component Histogram Figure 27 : G Component Histogram Figure 28 : B Component Histogram Figure 29 : H Component Histogram Figure 30 : S Component Histogram Figure 31 : V Component Histogram Figure 32 : L Component Histogram Figure 33 : a* Component Histogram Figure 34 : b* Component Histogram Figure 35 : Y Component Histogram Figure 36 : Cb Component Histogram Figure 37 : Cr Component Histogram Figure 38 : R Component Histogram Figure 39 ; G Component Histogram Figure 40 : B Component Histogram Figure 41 : Cr/Cb Component Histogram Figure 42 : 1.5*Cr-2*Cb Component Histogram Figure 43 : 9.3628*Y+8.9859*Cr+8.6208*Cb Component Histogram Figure 44 : -0.0015Y+ -0.0035*Cr+0.0003*Cb Component Histogram Figure 45 : -0.3339*Y -0.3069*Cr -0.1662*Cb Component Histogram Figure 46 : -0.0003*Y+ 0.0002*Cr+ 0.0027*Cb Component Histogram
25 26 27 30 30 31 31 32 32 32 33 33 33 34 34 34 42 42 42 43 43 43 44 44 44 45 45 45 46 46 47 47 47 48 48 49 49 50 50 50
vi
Figure 47 : Multi linear regression to Y Cr Cb Components Figure 48 : Multi linear regression to R G B Components Figure 49 : H/S Component Figure 50 : H-S Component Histogram Figure 51 : b*Cb Component Histogram Figure 52 : Cb-2*b3 Component Histogram Figure 53 : Cb-2.5*b3-0.8*Cr Component Histogram Figure 54 : -37.797*R-74.204*G+112*B Component Histogram
51 51 51 52 52 52 53 53
Figure 55 : 0.5687*R - 0.2813*G + 0.88*B Component Histogram Figure 56 : R2 + B2 Component Histogram Figure 57:Single Particle Identification Block Diagram Figure 58;Motion Detection Block Diagram Figure 59 : Object Detection Block Matching Diagram Figure 60 : RGB Color Space Figure 61 : YCbCr Color Space Figure 62 : HSV Color Space Figure 63 : L* a* b* Color Space Figure 64 : Shape Features Figure 65 : Textural Features Figure 66 : ROC Curve of ANN Figure 67: ROC Curve for perceptron
53 54 54 55 55 60 61 61 61 62 62 66 67
Figure 68 : ROC Curve of Naive Bayesian Figure 69 : ROC Curve of SVM Figure 70 : ROC Curve of Random Forest Figure 71 : Component Diagram of the System Figure 72 ; High Level Architecture Diagram Figure 73 : Logic Diagram of the Classification Module Figure 74 : Flow Diagram of Controlling Module Figure 75 : Logic Diagram of H/W implementation Figure 76 : Decision Making Flowchart Figure 77 : Logic Diagram of ML Module Figure 78 : UI of HMI Panel
67 69 70 77 79 80 80 81 83 87 93
vii
LIST OF ABBREVIATIONS
Abbreviation
Description
ML
Machine Learning
FPGA
Field Programmable Gate Array
PL
Programmable Logic
PS
Processing System
IPC
Industrial PC
SoC
System on Chip
viii
Chapter 1: INTRODUCTION 1.1 Background
Tea production is one of the main sources of foreign income in Sri Lanka. Sri Lankan tea recognized as a best tea category in the world because of the unique taste and the high quality. Production of high quality tea is not a simple task. After plucking tea leaves, they are transported to the tea factory and go through various steps before producing the consumable tea. Figure 1 shows the steps of tea producing process.
Figure 1: Tea Manufacturing Process
This project focuses on the last step of the process. Last step is removing stems from tea mixture. Removing stems is a very critical step in producing high quality tea. Because tea stems affect the taste and the quality of the tea. To produce high quality tea as much as possible amount of stems should be removed from the mixture. 1.2 Problem statement
Color sorting machine is the latest method used to remove stems from the tea mixture. This machine identifies stems using machine vision techniques and removes those stems using a ejecting mechanism. Color sorters use threshold values to differentiate stems from leaves. Initially those values need to configure manually by inspecting the machine output. Based on the misclassifications operator has to tune those threshold values. The problem is color of tea stems and leaves change due to various factors like whether conditions of the tea estate and variations of firing temperature. If color of the mixture changed due to any reason the color sorting machine need to reconfigure manually to match the new mixture. This is time consuming and accuracy of the stem removing also affected if the selected threshold values are not optimal. This is the main problem we are going to address in this project. The other problem we are going to solve is misclassifications due to partially burned tea stems and leaves. Current stem removing systems using only color data to identify stems. Hence, if a stem or leaf is not burned completely the machine can‘t identify it correctly
1
because the machine is configured to identify the stems and leaves based on the color of a completely burned stems and leaves. This project contains a research component on how other characteristics such as shape and size of stems and leaves can be used to differentiate tea stems from tea leaves. 1.3 Motivation
Stem removing is a very important process in tea industry. In our solution we are going to improve the accuracy of the stem removing process as well as to automate the machine configuration process which is currently done manually. This would be one of the greatest contributions done by a Sri Lankan group of students to the Sri Lankan tea industry.
1.4 Requirement Specification
The tea sorting machine should be able to
●
Learn and adjust the sorting parameters based on the tea produced in a particular factory.
●
Remove tea stems from tea with 95% confidence level
Chapter 2 : LITERATURE REVIEW 2.1 Abstract
Tea products are present in incredible varieties in terms of shape, size, color and as the market grows more demanding, the tea products are subdivided in various categories and are
2
destined to different segments. The definition and characterization of different attributes of tea are very important for the business and for the consumer, making it necessary to establish norms of classification and standardization, thus making commercial trading more efficient and allowing for higher awareness on the part of consumers. A computer vision system (CVS) has proven very successful in the analysis on the basis of color, size, shape, texture etc in food engineering. Computer vision is the science that develops the theoretical and algorithmic basis by which useful information about an object or scene can be automatically extracted and analyzed from an observed image, image set or image sequence. An image can be defined as a spatial representation of an object or scene. Image processing is base of computer vision system. Image acquisition, processing, segmentation are basic steps for extracting information of image. Computer vision is an emerging subject in food sector. CVS system is being used in tea industry for the detection of dust and for the categorization based on the quality. It is a fast, non-invasive cheap method for evaluating quality. The use of computers allows for increased efficiency with less manpower, reduces dependence on trained experts which is costly and time consuming. Also, we can do modeling of information obtained from image and can be used for future using machine learning techniques. 2.2 Introduction
The great concern with product quality due to new market restrictions have become important and the market competition become increasing and also new quality testing methods are developing with new technologies. Over the past decade, significant advances in techniques of tests have been observed, while extraordinary resources in electronics and informatics were identified as important factors in this development. Automation has motivated the development of testing equipment in production lines, and the evolution of sensor technology has led to the establishment of new techniques of measurement of the products, thus allowing permanent monitoring during the process, with the implementation of visual inspection systems. In the tea industry, some quality evaluation is still performed manually by trained inspectors, which is tedious, laborious, costly and inherently unreliable due to its subjective nature. Increased demands for objectivity, consistency and efficiency have necessitated the introduction of computer-based image processing techniques. Recently, computer vision employing image processing techniques has been developed rapidly, which can quantitatively characterize complex size, shape, color and texture
3
properties of foods. But less number of solutions for the tea grading and sorting systems is available. Computer vision based systems are basically a combination of a light source, a device for capturing the image and a computational system for the extraction of characteristics. Such systems are important for the product lines because it reduce the repetitive activities of the human, increase the efficiency of the manufacturing process and since the less human interaction , the accuracy and the quality of the output is increased. Different from the problems present in visual inspection performed by people, these kinds of systems offer accuracy and repeatability in measurements without contact, especially due to the elimination of aspects such as subjectivity, tiredness, slowness and costs associated with human inspection. By automating the grading and the sorting process in tea industry has become an interesting solution for the final analysis of the product quality and the assessed values or characteristics involve not only dimensional aspects, but also characteristics of color, texture and shape. Tea samples present a variety of shapes, sizes, colors and flavors, and as the market grows more demanding, tea products are subdivided in various categories. Over the past few years, the explosive growth in both computer hardware and software has led to many significant advances in computer vision technology. Computer vision technology provides a high level of flexibility and repeatability at relatively low cost. It also permits fairly high plant throughput without compromising accuracy. Currently, computer vision systems are being developed as an integral part of food processing plants for on-line, realtime quality evaluation and quality control [11] Therefore the computer vision based system can use for sorting and grading of agricultural products such as tea. In the case of detection of tea leaf, it is essential to correctly extract the object (leaf or stem) from the background. Image segmentation is an important and perhaps the most difficult image processing task. Segmentation refers to subdividing an image into regions exhibiting ―similar‖ characteristics. Subsequent image interpretation task s, such as feature extraction and object recognition, rely on the quality of the segmentation results. 2.3 Computer Vision System
Computer vision is the science that develops the theoretical and algorithmic basis by which useful information about an object or scene can be automatically extracted and analyzed from an observed image, image set or image sequence [11]. An image is the main component of a
4
computer vision based systems and it is defined as the spatial representation of an object. Computer vision based systems rapidly used in the food engineering and they use to inspect the quality, sorting and grading processes etc. Such systems are important because they reduce the human interaction in food engineering and increase the accuracy of the results. Research on tea industry was ranging from a simple evaluation of tea samples to a complicated robot guidance application. In past decade, computer vision based systems have been used in color measurement and sorting up to some accuracy level, including classification and grading by shape detection and extracting the texture of the tea particles [15]. Alternatively, computer vision system (CVS), a nondestructive method, offering objective measures for color and other physical factors. 2.4 Image Processing System
An image is a matrix which consists of square pixels. Each pixel indicates two factors, i.e. brightness value and the locations in the coordinates that are assigned to the image. The former is the color feature while features extracted from the latter are known as size or shape features. Image processing has become a main part of the tea sorting and grading process of the currently available systems. This technique plays an important role in the tea sorting and grading process while maintaining the accuracy and consistency. They offer flexibility in application and can be reasonable substitutes for the human visual decision-making process. In order to develop an automated system for tea sorting process, image processing techniques are often combined with mechanical and instrumental devices to replace human manipulative effort in the performance of a given process. In such a system, the image processing system is the center, which controls the operation of the machinery. Image acquisition Digital image analysis offers an objective and quantitative method for the estimation of morphological parameters. This process uses digital images to measure the size of individual grains and mathematically extract features and shape related information from the images.
The study ―Identification of Green, Oolong and Black Teas in China via Wavelet Packet Entropy and Fuzzy Support Vector Machine‖[18] had implemented the image acquiring experiment as shown in figure 1. They had included five parts i.e. a digital camera, an illumination platform, a capture board (digitizer or frame grabber), computer software, and computer hardware.
5
Figure 2: Process of a Computer Vision Based System to O btain the Tea Image Database
2.5 Color Feature extraction
According to the diverse information stored in pixels, image features obtained can be categorized into four types - color, size, shape, and texture. In this project we need to extract the quantitative measurement such as color, shape and texture to do the classification from tea leaves from the mixture. In literature, ―Classification system for rain fed wheat grain cultivars using artificial neural network‖[1], a variety of different methods have been developed to measure size, shape, color, and texture features. Color feature extraction An RGB image, sometimes referred as a true color image, is stored as an m-by-n-by-3 data array that defines red, green and blue color components for each individual pixel. Using MATLAB and the image processing toolbox software extract the individual color components like Y, Cb ,Cr, R, G, B etc in color spaces like RGB, YCbCR, HSV, L*a*b which are most common color schemes use for color classification in grains and I1I2I3 color
6
space is defined in [4] which was calculated from RGB color space, using the following equations
The study done for automated strawberry grading had used a technique called Dominant Color Method[3]. In this study they had considered about the L*a*b* color space for their experiment. Generally, the human sight is more interested in main color of the image means that color which appears frequently in the image. So this Dominant Color Method was used on a* channel to extract the color feature from the image. The methodology which was followed by them is as follows. Step 1: It first calculates the histogram of the image on a* channel. Step 2: Then it sets a window of width L, and then from the most left of the histogram the gray value of the window is calculated. Then window is moved one pixel towards right and again the gray value is calculated. This procedure continues until window is moved towards the most right. Step 3 : Then the major color window is found out i.e. the window having the largest frequency. And then by using the Eq. (1) the average color value of the dominant window is found out.
The L*a*b* model was also used in to detect the browning degree . The average L*a*b* values were obtained from the image using the Histogram Window of Adobe Photoshop CS5.
7
In the histogram window the parameters L, a, b were not standard colour values so they were converted as follows:
2.6 Morphological feature extraction
The following morphological features were extracted from labeled images of individual rain fed wheat grains cultivars. Geometry related features including area, perimeter and major and minor axis lengths were measured from the binary images[1]. Area (A): The area of a region is defined as the number of pixels contained within its boundary; perimeter (P): the perimeter is the length of its boundary. The length of the minor axis is the longest line that can be drawn through the object perpendicular to the major axis.
Equivalent diameter (Eq): It was the diameter of a circle with the same area as the wheat grain region.
Convex area (C): it is the number of pixels in the smallest convex polygon that can contain the wheat grains region; solidity (S): the
8
proportion of the pixels in the grains region that are also in the convex hull; extent (Ex): the proportion of the pixels in the bounding box which are also in the grains region; roundness (R): this is given by:
Compactness (CO): the compactness provides a measure of the object's roundness:
2.7 Shape feature extraction
Size reflects the number of pixels and shape describes the boundary of food products. Size features are usually obtained using measurements of area, perimeter, length, and width. Shape is characterized mostly in two ways, i.e., size dependent measurements such as compactness and elongation, and size independent measurements such as spatial moment and Fourier descriptor (boundary shape). In the study[2] , they had used the axis length and the area of the object to derive the shape factors as follows.
9
2.8 Texture feature extraction
Texture is normally the dependency between pixels and their neighboring pixels or the variation of intensity of pixel.
glcm is an m-by-n-by-p array of valid gray-level co-occurrence matrices. If glcm is an array of GLCMs, stats is an array of statistics for each glcm. graycoprops normalizes the gray-level co-occurrence matrix (GLCM) so that the sum of its elements is equal to 1. Each element (r,c) in the normalized GLCM is the joint probability occurrence of pixel pairs with a defined spatial relationship having gray level values r and c in the image. graycoprops uses the normalized GLCM to calculate properties. From the gray-level co-occurrence matrix glcm,Contrast,Homogeneity,Correlation,Energy were derived[4] as follow: Contrast: Returns a measure of the intensity contrast between a pixel and its neighbor over the whole image.Range = [0 (size(GLCM,1)-1)^2] Contrast is 0 for a constant image. The property Contrast is also known as variance and inertia. Contrast=
Correlation: Returns a measure of how correlated a pixel is to its neighbor over the whole image. Range = [-1 1] Correlation is 1 or -1 for a perfectly positively or negatively correlated image. Correlation=
Energy Returns the sum of squared elements in the GLCM. Range = [0 1] Energy is 1 for a constant image. Energy=
10
Homogeneity Returns a value that measures the closeness of the distribution of elements in the GLCM to the GLCM diagonal. Range = [0 1] Homogeneity is 1 for a diagonal GLCM. Homogeneity=
The feature vector was made from the earlier mentioned features and feed, used as an artificial neural network for classification which in this case was used as a multi layer perceptron (MLP) method. Feature selection
●
Using UTA[3]
Feature selection is the problem of choosing a subset of features ideally necessary to perform the classification task from a larger set of candidate features. There are several ways to determine the best subset of features. UTA is a simple method which is based on trained artificial neural network. In the basis of this method, average of one feature in all instances is calculated. Then, the selected feature in all input vectors is replaced by the calculated mean value. Then, the trained network is tested with the new features and data matrix . The comparison error was defined in our strategy as follow: E= (FP (new) + FN (new)) - (FP (old) + FN (old)) (14) Where, FP(old) is the false positive and FN(old) is the false negative using the whole features and FP(new) and FN(new) are those values when one of the feature are replaced by the mean value. There are three states in this way: (1) one input is considered more relevant if E is positive and higher according to the other features; (2) one input is ineffective if E is zero; (3)
11
one input is not only ineffective but also noisy and should be removed from the input vector if E is negative.
●
Using data clustering against each other
Plot the data clusters against each features and identify the features that can be clustered most efficiently and the standard deviation is low.
Figure 3 : Example Visualization of Good Cluster
Figure 4 : Example Visualization of Bad Cluster
12
●
Principle Component Analysis
The research done in [16] had used the principle component analysis for the feature selection. In this experiment they had extracted 80 features from the tea image. When there are number of features extracted , the computation cost is increased. Therefore they had used PCA to reduce the number of features. Principle Component Analysis is an effective technique which can be used to reduce the features and also can be used to retain the most substantial principle components. Data pre-processing
●
Image Segmentation
The acquired images consist of both the object and the background. Therefore a technique capable of removing the background from the images is needed.
●
Background Removing using K mean Clustering
Using Statistics and Machine Learning Toolbox™ of matlab we can segment colors in an automated fashion using the L*a*b* color space and K-means clustering [5]
13
Figure 4-a : Original Image
●
Figure 4-b: Segmented Image
Background Removing using Gray Level Thresholding
Using MATLAB image processing tool computes a global threshold (level) that can be used to convert an intensity image to a binary image with im2bw. level is a normalized intensity value that lies in the range [0, 1].The graythresh function uses Otsu's method, which chooses the threshold to minimize the intraclass variance of the black and white pixels. Multidimensional arrays are converted automatically to 2-D arrays using reshape. The graythresh function ignores any nonzero imaginary part of I. [level EM] = graythresh(I) returns the effectiveness metric, EM, as the second output argument. The effectiveness metric is a value in the range [0 1] that indicates the effectiveness of the thresholding of the input image. The lower bound is attainable only by images having a single gray level, and the upper bound is attainable only by two-valued images.
14
2.9 Knowledge base Learning
Here the main purpose is to investigate the recent and relevant applications of learning techniques in computer vision for food industry and how can them apply in tea sorting and grading process.Figure 6 indicates the basic configuration of a knowledge base learning system can be applied to any system. As shown in figure 6 the images of samples are quantitatively characterized by a set of features such as size, shape, color and texture. As discussed in feature extraction part a variety of different methods have been developed to measure size, shape, color, and texture features. These features are identical data used to represent the samples, which can also be used to form the training set. After the training set has been developed, learning algorithm extracts the knowledge base necessary to make decisions of test samples. Based on the knowledge, intelligent decision is made as output and fed back to the knowledge base at the same time, which generalizes the way that inspectors use to accomplish their tasks.
Figure 5 : Basic concept of a machine learning system
In literature, a variety of different learning techniques such as artificial neural network, statistical learning, fuzzy logic, and decision tree have been used for learning, which have been reviewed by [6]. The learning techniques are related to our project to do the classification in sorting/grading process. Classification need to ensure whether the captured particle is a tea leaf or dust. ●
Neural network
15
Figure 6 : structure of the artificial neural network.
Artificial neural network is a learning technique which behaves according to the theory of human brain information processing system. Artificial Neural Network gathers information by detecting and extracting the patterns and the relationship of the data set. An ANN is made with hundreds of single unit which is called perceptron. This single unit has weighted inputs, transfer function and one single output. A neural network consists of three main layers which are called input layer, hidden layer and output layer. The behavior of the whole network depends on the transfer functions of each neuron. The weights are adjustable hence a neural network is a parameterized system. During the training the inter unit connections are optimized until error of classification minimize and the classification becoming to a specified accuracy level. ANN technology allows the extension of computer vision technology into the areas of color, content, shape, and texture inspection at near-human levels of performance, and can provide the decision making and classification capabilities to succeed in these inspection tasks [9]. Currently the Artificial neural network has been used in food engineering for classification and prediction purposes. In literature, which has been reviewed in [6] has mentioned about food classification experiment results which were done using artificial neural networks. This classification has
16
done based on the quality of the foods. To analyze the quality, they had used quantitative measurements such as shape, color, texture etc. They had done this experiments under several food categories i.e fishery, fruit, grain, meat and vegetable. As the result of this experiment they present the following accuracy table.
Figure 7 : Summary of ANN application for classification of food product done in [6].
This study is related with our system because we also need to do a classification to separate tea leaves from the mixture. And also the quantitative features such as size, shape, color, texture are related with our process.
●
Support Vector machines
Support vector machine is a supervised learning method which is used in both classification and regression [19]. In SVM, it minimizes the empirical classification error and maximizes the geometric margin simultaneously. So the SVM is identified as the maximum margin classifier. The input vector is mapped to a higher dimensional space where a maximal separating hyperplane is constructed. There are two parallel hyperplanes are constructed on each side of the hyperplane that separate the data. . An assumption is made that the larger the margin or distance between these parallel hyperplanes the better the generalization error of the classifier will be [19].
17
Figure 8 : maximum margin hyper plane fo r a SVM with two class data set
Kernel of SVM: A kernel is a similarity function. This kernel function provides an alternative way to the basic machine learning concept. When considering some example classification system, instead of defining slew of features, define a kernel function to find similarity between data points. The rapidly used kernel functions are linear kernel, polynomial kernel, radial basis kernel and sigmoid kernel. In these popular kernel functions, RBF is the main kernel function because of following reasons [19]: 1. The RBF kernel nonlinearly maps samples into a higher dimensional space unlike to linear kernel. 2. The RBF kernel has less hyper parameters than the polynomial kernel. 3. The RBF kernel has less numerical difficulties. A study done by [18] have been used support vector machines to classify tea into different categories based on quantitative features i.e 64 color features and 16 wavelet features. In this experiment they had used principle component analysis to identify important features and then they had tested with the support vector machines. As the support vector machines cannot deal with noise and the outliers i.e, its performance will decrease sharply when the data set either contains outliers or was contaminated by noises they had used a variant of SVM which is called Fuzzy SVM to overcome the issue. In this study, they had used an advanced technique to calculate the out-of-sample performance of the proposed identification system: the
K-fold
stratified cross validation (SCV) technique. The following figure 10 shows their
18
proposed knowledge based learning system to categorized tea based on the identified quantitative measures.
Figure 9 : SVM Classification Architecture of the literature [18]
●
Bayesian Inference Technique
This technique has developed based on the Bayes theorem. An object of an image contains a range of colors than a single color. So this technique has the ability to utilizing a color data set containing a number of colors from the image and has used to find similar color data in another image and has provide useful information for classification[17]. The Bayesian inference technique improves on the use of averages by capturing more detailed information about the colours, and their distribution, in the image. It calculates proportions of various ranges of colour by dividing the colour space of the reference sample image into several regions. By calculating the reference region probabilities, the process enables one to evaluate which of the reference food images is the most likely match to the sample food image. The study done by [17] has used Bayesian inference technique to do the food classification. This classification has done using the average of CIE L*a*b* color space. Since this study is color based classification system and color based classification is a major part of our system this solution should be considerable while doing the experiments. 2.10 Hardware Used for Sorting
To identify the color, shape and texture of the particles in real time special hardware is required .Since the image acquisition and processing should be done in real time.
●
Image Sensor
19
It is important to capture the image in a required quality to perform the classification. Both line sensors as well as area sensors[14] are being used for this purpose. To obtain a higher frame rate the region of interest of the camera can be defined to cover a small range. For example in ―A machine vision system for high speed sorting of small spots on grains‖[14] a color VGA sensor which operates in near linear scale s used. The image sensor that was used is ―KAC-9628‖ manufactured by ―Eastman Kodak Co., Rochester, NY‖. It consists with 640x480 pixels of sensors and uses a color bayer filter to sense the Red , Green and Blue colors . Approximate spatial resolution is 0.06 mm per pixel. By reducing the region of interest of the camera it is operated in near line scanner mode. Each line consists 640 pixels. This line is divided to three parallel channels. From this line 3 different regions of interest are defined. For this particular application 124 pixels from each channel is used. Not only standard optical cameras, some optical sorting systems use X-ray and infrared sensors to acquire the image. But these images requires long processing time compared to standard optical cameras [12]
●
Clock
The FPGA can be clocked faster than the camera. But both camera and the FPGA is clocked to synchronize both devices. The image sensor is configured to reduce the clock frequency by dividing it by a factor of a power of 2. For example 48 MHz clock source can be used to clock the FPGA in 48 MHz and the image sensor is clocked in 12 MHz [14]. Therefore the mathematical operations can be performed while transferring pixel data from image sensor to the FPGA. ●
EEPROM EEPROM is used to store configuration data to be read by the FPGA at the system startup. ●
Data Transferring
Cameras boards use various color schemes to produce the output image. For example RGB [python 1300] , YCrCb color schemes. ●
Lighting System
A very important factor of image acquisition is lighting condition. Most of the existing sorting machines uses array of Light Emitting Diode as the source. A linear lense is used to defuse the light from the lightning source and to focus to the imaging point .
20
Colors such as cool white [14] as well as various color combinations are used when capturing the image[metaclorsorter]. To avoid the LED s being overheated an Aluminum heat sink is used sometimes with a fan to dissipate the heat[14]. For special requirements special lighting sources are used. For example in mineral sorting Xray light source is used [12]. ●
Processing Unit
In industrial optical sorting systems different kind of processing units are used depending on the application. For high speed sorting such as sorting grains in free fall, Field Programmable Gate Arrays are used [14] as the processing unit because the processing need to happen real time. Some other sorting systems such as industrial mineral sorting systems where minerals are being identified while they are on a convoy belt [12]industrial pcs are used as
the
processing component. When the development cost needed to be get into the account without going for complete hardware based solution like FPGA microprocessors have been used [8]. For example if a raspberry pi is used as a processing unit image processing libraries such as opencv can be used. But due to limited processing power the convoy belt method have to be used instead of free fall under gravity method.
21
Chapter 3: EXPERIMENTAL PROCEDURE 3.1 Preprocessing
First we collected the tea steam and leaf samples from different areas of country to analyses their color variation. Then we could identify that Matara tea has high brown color than Rakwana tea.
Figure 10 : Tea samples from different areas of the country
Then we plot the histograms of those tea samples which as shown below.
Histogram for tea Leaves from different area(Pelmadulla, Rakwana, Deniyaya and Balangoda )
YCbCr Color Space
Figure 11 : Y Component Histogram
22
Figure 12 : Cb Component Histogram
Figure 13 : Cr Component Histogram
23
RGB Color Space
Figure 14 : R Component Histogram
Figure 15 : G Component Histogram
Figure 16 : B Component Histogram
24
L*a*b Color Space
Figure 17 :L Component Histogram
Figure 18 : *a Component Histogram
Figure 19 : *b Component Histogram
25
HSV color Space
Figure 20 : H Component Histogram
Figure 21 : S Component Histogram
Figure 22 : V Component Histogram
26
By that we could identify main existing problem of current method which is hard coded the threshold values within the machine learning module. So for our system we try to built dynamically changing thresholds which could able to train time to time.
And also we could identify that based on color space which we are using the classification property of the tea samples are vastly vary. So we create a heuristic which we can classify tea stem and leaf if we could able to find the proper color space. So we separate tea stem and leaf samples of Pelmadulla tea. And the following steps are applied for data preprocessing.
Gathered samples are pasted on white background paper sample wise.
27
Crop the pasted samples to their bounding box. For this one we use moor neighbor algorithm. The Moore neighborhood of a pixel, P, is the set of 8 pixels which share a vertex or edge with that pixel. These pixels are namely pixels P1, P2, P3, P4, P5, P6, P7 and P8shown in
F igure 1 below. The Moore neighborhood (also known as the 8-neighborsor i ndirect neighbors) is an important concept that frequently arises in the literature.
Given a digital pattern i.e. a group of black pixels, on a background of white pixels i.e. a grid; locate a black pixel and declare it as your "start" pixel. (Locating a "start" pixel can be done in a number of ways; we'll start at the bottom left corner of the grid, scan each column of pixels from the bottom going upwards -starting from the leftmost column and proceeding to the right- until we encounter a black pixel. We'll declare that pixel as our " start" pixel.)
28
Next step is white balancing this is mainly for reducing the effect of background light while we are taking training sample at different background. We basically tried 3 method. Method 1
Taking the background RGB values and by assuming they are white and 255-Color value And add that value to all pixels
29
Method 2: taking the mean intensity of full i mage
Color = color * meanGray / meanColor
Method 3: taking the mean intensity of background of image
Color = color * meanGray / meanColor
Method one does not gave much accurate result so we choose method second
30
For removing the background we use following algorithm:
31
To convert an image in to binary we choose Otsu algorithm which involves iterating through all the possible threshold values and calculating a measure of spread for the pixel To convert image in to binary we Otsu algorithm which involves iterating through levels eachanside of the threshold, i.e.choose the pixels that either fall in foreground or allbackground. the possibleThe threshold calculatingvalue a measure forforeground the pixel levels aim is values to find and the threshold where of thespread sum of and each side of the threshold, theminimum. pixels that either fall in foreground or background. The aim is to background spreads i.e. is its find threshold value where the sum of foreground spreads its Thethe algorithm will be demonstrated using the simpleand 6x6background image shown below.is The histogram for the image is shown next to it. To simplify the explanation, only 6 grey scale minimum. levels are used.
32
The algorithm will be demonstrated using the simple 6x6 image shown below. The histogram for the image is shown next to it. To simplify the explanation, only 6 grey scale levels are use
And then for each sample extract the color component mean values color spaces such as above mentioned. Then plot the histogram stem and leaf for get the idea how they behave on those color spaces.
33
Histogram for Stem and leaf for Palmadulla tea Standard color spaces : Color mean of object
Sample Size: 270 YCbCr Color Space
Figure 23 : Y Component Histogram
Figure 24 : Y Component Histogram
34
Figure 25 : Cr Component Histogram
RGB Color Space
Figure 26 : R Component Histogram
Figure 27 : G Component Histogram
35
Figure 28 : B Component Histogram
HSV Color Space
Figure 29 : H Component Histogram
Figure 30 : S Component Histogram
36
Figure 31 : V Component Histogram
L*a*b Color Space
Figure 32 : L Component Histogram
Figure 33 : a* Component Histogram
37
Figure 34 : b* Component Histogram
Then again we try to optimize those results by taking mean of top 50 percentile of the color values. This is because stem may contain the partially burned area so those areas color is black and if we take the mean value those feature may destroy.
Histogram for Stem and leaf for Palmadulla tea Standard color spaces : Color sum of Top 50 percentile YCbCr Color Space
Figure 35 : Y Component Histogram
38
Figure 36 : Cb Component Histogram
Figure 37 : Cr Component Histogram
RGB Color Space
Figure 38 : R Co mponent Histogram
39
Figure 39 ; G Component Histogram
Figure 40 : B Component Histogram
So finally we tried to build a derived color components by using existing color components such as dividing, linear combinations. At this step we does not use any methodological approach we just observe the behavior of histograms before finding the methodological approach.
Histogram for Stem and leaf for Palmadulla tea Optimized color spaces Sample count: 272
40
Cr/Cb Color Space
Figure 41 : Cr/Cb Component Histogram
Figure 42 : 1.5*Cr-2*Cb Component Histogram
Feature selection methods weights for Y, Cr, Cb components
41
Figure 43 : 9.3628*Y+8.9859*Cr+8.6208*Cb Component Histogram
Figure 44 : -0.0015Y+ -0.0035*Cr+0.0003*Cb Component Histogram
42
Figure 45 : -0.3339*Y -0.3069*Cr -0.1662*Cb Component Histogram
Figure 46 : -0.0003*Y+ 0.0002*Cr+ 0.0027*Cb Component Histogram
Figure 47 : Multi linear regression to Y Cr Cb Components
43
Figure 48 : Multi linear regr ession to R G B Components
Figure 49 : H/S Component
Figure 50 : H-S Component Histogram
44
Figure 51 : b*Cb Component Histogram
Figure 52 : Cb-2*b3 Component Histogram
Figure 53 : Cb-2.5*b3-0.8*Cr Component Histogram
45
Figure 54 : -37.797*R-74.204*G+112*B Component Histogram
Figure 55 : 0.5687*R - 0.2813*G + 0.88*B Component Histogram
Figure 56 : R2 + B2 Component Histogram
46
Finally at the end of the data analysis part we come across to the decision based on the color space we could achieve significantly optimized result. Based on these result we build a SIMULINK models and tested it with real time data.
Figure 57:Single Particle Identification Block Diagram
And another analysis we done is to try to find the detect the motion of a particle. Basically we did this for ejecting particle at a correct place. So for that we tried following methods: 1) Detect Bounding box motion (Frame Differentiation)
47
Figure 58;Motion Detection Block Diagram
2) Intensity difference
Figure 59 : Object Detection Block Matching Diagram
Next thing we tested is that textural and shape features. For the textural features analysis we tried Gray Level Co-Occurrence Matrix features.GLCM_Features1 helps to calculate the features from the different GLCMs that are input to the function. The GLCMs are stored in a i x j x matrix, where n is the number of GLCMs calculated usually due to the different
48
orientation and displacements used in the algorithm. Usually the values i and j are equal to 'NumLevels' parameter of the GLCM computing function graycomatrix(). Under that we tested following features: ●
Autocorrelation:
●
Contrast
●
Correlation
●
Cluster Prominence
●
Cluster Shade
●
Dissimilarity
●
Energy
●
Entropy
●
Homogeneity
●
Maximum probability:
●
Sum of squares: Variance
●
Sum average
●
Sum variance
●
Sum entropy
●
Difference variance
●
Difference entropy
●
Information measure of correlation
●
Inverse difference (INV) is holomorphic
●
Inverse difference normalized (INN)
●
Inverse difference moment normalized
And also we tried following as shape factors: ●
Area
●
Bounding Box ratio(width/height)
●
Perimeter
●
Minimum rectangle ratio
●
Major Axis to Minor axis ratio
Following are some plots shows the result we got from above
49
Implementation on SIMULINK Case 1 : Width to height ratio of bounding box
Case 2 :Bounding box area of tea stem and leaf particles
50
Case 3 :Object pixel count of tea stem and leaf particles
3.1.2 Background removing The acquired images consist of both the object and the background. Therefore a technique
capable of removing the background from the images is needed. We used ‗Gray Level Thresholding‘ method to identify background and remove it.
3.1.3 White Balancing
To Collect data we captured series of images. While capturing images due to light condition and due to dynamic nature of image sensor captured colors may vary in image to image. To minimize that variation we did pixel selective white balancing. In pixel selective white balancing, pixels which should be white (white places in actual scene) are selected . If those places are not appear as white in image colors are changed by the image capturing process. By selecting those pixels and calculating color correction for those pixels color correction for image can be done. We used a white background to capture images. After identifying the background, color correction for image can be done using pixel selective white balancing method.
51
3.1.3 Data extraction
Color data of stem and leaves are extracted in RGB, HSV, YCrCb and Lab color spaces.
3.1.4 Data cleaning 3.1.4.1 Detect and remove noisy data
As a data cleaning we basically did the remove noisy data. The image acquiring devices and background light conditions introduce artifacts to the images. This results in degradation of the srcinal content of the image. So as explained in image processing techniques we did the white balancing and reduce the effect of color variations and when taking color value means we remove the background and get mean color in the object only. And also we have try to accurate result by instead of mean color value we try to take mean of top 50 percentile of color values. This is because some stems are half burned and there may have half black region but we need to categorize them to stem category.
3.1.4.2 Outliers
The data can have some waste materials like small plastic, nails added while processing to sorting step. So here we use Grubbs Test for deleting outliers with the given significant level.
52
3.1.5 Feature Selections Attribute Sampling: Select only a subset of the most important attributes from the data.
Here we selected the color space using pairwise clustering of different color components in different main color spaces and selected the most clearly clustered color space as Y Cb Cr. This is specially because all the color spaces are derived from RGB color space and if we feed all of them separately to ML models dependency in between attributes are increased. About YCbCr color space:
To extract the color features of the image we used YCbCr color space. About YCbCr Color Space : ●
Y: luminance
●
Cb : Blue Chromaticity
●
Cr : Red Chromaticity
Figure 60 : RGB Color Space
53
Figure 61 : YCbCr Color Space
Figure 62 : HSV Color Space
Figure 63 : L* a* b* Color Space
From above color space graphs we select YCbCr color space since it gives more clear clusters
54
Figure 64 : Shape Features
Here we select Axis Ratios, Bounding Box Ratio, Minimum Rectangle Ratio and object Area as the most important shape features
Figure 65 : Textural Features
Here we select Contrast, Correlation, Homogeneity only as textural features as the most important features
55
3.1.6 Data Transformation Create binary features : One-Hot-Encoding
Encode categorical integer features using a one-hot aka one-of-K scheme. The input to this transformer should be a matrix of integers, denoting the values taken on by categorical features. The output will be a sparse matrix where each column corresponds to one possible value of one feature. Overview of Sample Data Set
Train data set Attribute of the Data Set ●
CompY: Y component of the color space
●
CompCb : Cb component of the color space
●
CompCr : Cr component of the color space
●
Entropy :
A statistical measure of randomness that can be used to
characterize the texture of the input image. ●
Contrast : A measure of the intensity contrast between a pixel and its neighbor over the whole image.
●
Homogeneity : A value that measures the closeness of the distribution of elements in the gray-level co-occurrence matrix to the GLCM diagonal.
●
Correlation :A measure of how correlated a pixel is to its neighbor over the whole image.
●
Energy :The sum of squared elements in the gray-level co-occurrence matrix
●
MajorMinorAxisRatios : Ratio between major axis length and minor axis length
●
Roundness :Roundness of the particles
56
●
BoundingBoxArea : Area of the object cropped by the bounding box algorithm
●
BoundingBoxRatio : Height to width ratio of the cropped image from the bounding box algorithm
●
MinRectangleRatio : Ratio between minimum rectangle height and minimum rectangle width
●
ObjectArea : Area of the object
●
Label : Class Label. We used two class labels. 1 - tea leaf 0 – tea stemp
Derived Attributes
CompCr:CompCb : Ratio between Cr and Cb color components. CompCr:CompY: Ratio between Cr and Y color components. CompCb:CompY: Ratio between Cb and Y color components.
3.2 Testing With Different Machine Learning Model & Evaluation of the Results Evaluation Criteria
Basically when evaluate a machine learning model, following factors are considered. They are Predictive accuracy, Speed and scalability which explained by time to construct the model, time to use the model and efficiency in disk-resident databases, Robustness which explain by handling noise, missing values and irrelevant features, streaming data and Interpretability which explained by understanding and insight provided by the model
Evaluation technique that we used
In our problem we did binary classification using above mentioned machine learning models. Therefore we mainly focused on the accuracy of the model. And also used the ROC curve for model selection.
Weighting the Models
We first focused on the default attributes of the models. Then we change the default parameters of the models and checked the accuracy variance. As an example in random 57
forest model first we trained with the arbitrary number of trees (100 trees) and we plot the Out of Bagged error Vs number of trees. Then we were able to identify the error distribution while increasing the number of trees as shown in below figure.
Using this graph we selected number of trees as the Out-of-Bag Classification error is minimized. Likewise we use weighting and parameter changing in models SVM,ANN etc. This data set consists of class labels for each data tuple. Therefore we needed to apply the supervised learning approaches. There are three types of supervised learning i.e classification, regression, structured prediction. Our problem belongs to the classification type. Therefore we try out machine learning models which were used for classification. Classifiers also mainly divided into two types. 1.
Lazy Learners (Eg: K-Nearest Neighbor)
2.
Eager Learners (Eg: SVM,ANN,Random Forest etc)
Here we have focused more towards eager learners. Eager learners are capable of classifying instances quicker than lazy learners.
58
3.2.1 Artificial Neural Network
Figure 66 : ROC Curve of ANN
Confusion Matrix of ANN
3.2.2 Perceptron
59
Figure 67: ROC Curve for perceptron
Confusion Matrix of perceptron
3.2.3 Naive Bayes
Figure 68 : ROC Curve of Nive Bayesian
●
Confusion matrix of Naïve Bayesian
60
3.2.4 Support Vector Machine
In SVM, it minimizes the empirical classification error and maximizes the geometric margin simultaneously. So the SVM is identified as the maximum margin classifier. The input vector is mapped to a higher dimensional space where a maximal separating hyperplane is constructed. There are two parallel hyperplanes are constructed on each side of the hyperplane that separate the data. An assumption is made that the larger the margin or distance between these parallel hyperplanes the better the generalization error of the classifier will be. Kernel - Linear
Accuracy = 88.67%
61
ROC - SVM Linear kernel
Figure 69 : ROC Curve of SVM
Kernel - rbf
Accuracy : 92.45%
ROC - SVM- rbf kernel
62
3.2.5 Random Forest
Accuracy : 90.56%
Figure 70 : ROC Curve of Random Forest
3.3 Result Optimization 3.3.1 Feature Selection
To optimize the result we can use feature selection of the data set. For that we can use feature selection using following algorithms 1. Principle Component Analysis (PCA) 2. Filter Methods 3. Forward Selection 4. Backward Elimination 5. Bi-directional Search 6. MRMr 7. inffs 8. relieff 9. mutinffs 10. fsv 11. laplacian
63
12. mcfs 13. rfe 14. fisher And get the best accuracy level for each model 3.3.2 Derive New Attributes.
We can derive new attributes from the existing attributes. Sometimes this method helps to optimize the result because the derived attributes can be more dependable on class labels than existing attributes. 3.3.3 Outlier Analysis
Using the outlier analysis methods we can identify and treat to the outlier. For this we can using following methods 1.
K-medoid
2. Density based clustering 3. Ridge Regression 4. occam razor principle 5. Cross-Validation And get the best accuracy level for each model
3.3.4
Optimized Results
3.3.4.1 Artificial Neural Network
Five features are selected using relieff feature selection algorithm which gave maximum accurate features from feature selection methods mentioned in feature selection section.
Confusion Matrix
64
ROC Curve
3.3.4.2 Naive Bayes
Five features are selected using mutinffs feature selection algorithm which gave maximum accurate features from feature selection methods mentioned in feature selection section. Confusion Matrix
ROC Curve
65
3.3.4.2 Support Vector Machine
By Applying Principal Component Analysis SVM:Kernel - Linear
ROC Curve
66
Accuracy : 92.45% Increased Accuracy = 3.78%
SVM:Kernel - rbf
ROC Curve
67
Accuracy : 96.22% Increased Accuracy : 3.77%
3.3.4.3 Random Forest
By Applying Principal Component Analysis
68
Accuracy : 94.33% Increased accuracy : 3.77%
3.3.5 Final Algorithm of the Research Component
69
70
Chapter 4 : SYSTEM DESIGN 4.1 Component Diagram
Figure 71 : Component Diagram of the Sy stem
This diagram shows main components of the system and their interconnections. 4.1.1 Image Acquisition S ubsystem
Main purpose of this sub-system is capture images using the image sensor. Captured images will be send to data extracting module.
4.1.2 Data Extracting module
Get color data of tea leaves and stems from captured image is the main purpose of this module. Color data is converted to suitable color spaces by this module. Those data will be used by decision making module and machine learning module.
71
4.1.3 Decision Making Module
This module do the stem/leaf classification. Threshold values given from machine learning system is used to do the classification. After classification is done this module will decide to trigger or not to trigger actuators.
4.1.4 Actuator Controller
This module control actuators according to decision making module‘s commands.
4.1.5 Controlling Module
Controller is responsible for coordinating subsystems, change behavior of the system according to user inputs and monitor status of the system.
4.1.6 Human Interface Module
This module handle all interactions with the user. Get user inputs and communicate them to controller and show current status of the system to the user.
4.1.7 Machine learning System
Machine learning system Collects data from data extraction module find threshold values using training data set.
4.2 High Level Hardware Architecture
72
Figure 72 ; High Level Architecture Diagram
This diagram shows the high level hardware architecture of the system. MicroZed and IPC(Industrial PC) are main two hardware components of this system. Those two components communicate using TCP over Ethernet. MicroZed contains the Real Time Stem/Leaf Classification module. Fig. xx shows the detailed architecture of the Real Time Stem/Leaf Classification module implemented in MicroZed. In MicroZed there are 2 main modules Processing System(PS) and Programmable Logic(PL). Those two modules communicate using AXI bus. AXI is a protocol designed to do the communication between module in same System on Chip (SoC). PS and PL are in the same chip. PS is a general purpose CPU and PL is a FPGA. IPC contains the Controlling Module, Human machine interaction module and the Machine Learning Module. Fig xx shows the detailed architecture of those modules.
73
4.3 Detailed Architecture 4.3.1 Real Time Stem/Leaf Classification Module
Figure 73 : Logic Diagram of the Classification Module
This diagram shows where each module is implemented in ZYNQ7020 SoC and how external inputs and outputs connect to the system.
4.3.2 Controlling module
Figure 74 : Flow Diagram of Controlling Module
74
4.4 Data Flow Diagram
Figure 75 : Logic Diagram of H/W implementation
This diagram shows the path of data flow which is srcinated from the image sensor. Image sensor capture image frames and send row pixel data to the system. Those pixel data are used for real time data processing and do the non-real time machine learning. Raw pixel data taken from image sensor converted to AXI-4 video stream before doing any modifications to those data. AXI-4 video protocol is used to the video data transferring between IPs inside the FPGA. After the stream protocol conversion RAW pixel data will be converted to 24 bit RGB data using ‗Color filter array interpolation algorithm‘. This algorithm is used to create RGB data from Bayer Pattern color data. Then the RGB data will be converted to YCrCb 422 and to the custom color space which derived to do the stem/leaf classification. YCrCb 422 data will be send to RAM of the microZed via a DMA controller. Then those data will be transferred to Machine Learning module using TCP over Ethernet. YCrCb 422 format is used instead of RGB because it require 16bits for one pixel. After converting to YCrCb 422 minor color information will be lost. But this conversion is required otherwise bandwidth of Ethernet connection will be not enough to transfer all pixel data without missing them. This transferring has to be done real time because the RAM of MicroZed is not sufficient for large buffer size. Transfer frames will be stored in the IPC and will be used do the learning and find optimal threshold values to do the stem/leaf
75
classification. Machine learning task is non-real time task. Then those threshold values will be send to CPU of the MicroZed using TCP over Ethernet connection. CPU send those parameters to decision making module through Data Exchange Module which is located in FPGA part of the SoC via the AXI-Lite interface. Data Exchange Module send those values to Decision Making Module using their communication ports. Other destination of image data is the decision making module. RGB data converted to the custom color space and send to the decision making module to do the decision making process. Those pixel data will be used to check whether the capture frame is contain a object and the to decide that the object is a stem or a leaf. To identify object pixels and to take the decision, threshold values which are send from machine learning module will be used. After taking the decision signal will be sent to Actuators to activate them to or deactivate them based on the decision.
76
3.5 Decision Making Flowchart
Figure 76 : Decision Making Flow chart
This diagram shows the flowchart of decision making process. After arriving new pixel data to the decision making module it check whether that pixel is an object pixel or a background pixel based on threshold values. If values of all three channels are less than threshold values that pixel is considered as an object pixel. If it is not an object pixel discard the pixel and wait for the next pixel. If it is an object check whether it is a stem or a leaf based on threshold values.
If values of all channels are greater than respective threshold value pixel is
considered as a stem. If the pixel is a stem pixel increment stemPixel counter else increment leafPixel counter. This process continues until the last pixel of the frame arrives. After the last pixel is arrive stem percentage is calculated using stemPixel count and leafPixel Count. If the stem percentage is higher than the 100-Sensitivity. This sensitivity value can be configured by the user. If the sensitivity value is 50 all the objects which stem percentage is greater than 50% will be considered as a stem. This approach is used because of some stems are partially burned and some pixels of those stems have color values of leaves. If the user set the sensitivity to 100 this algorithm will consider object as a stem if there is at least one stem pixel in the object.After take the decision actuators will be activated if the object is a stem. Finally all the counters will be reset and start the process from the beginning for the new frame.
77
Chapter 5 : IMPLEMENTATION 5.1 Real time Stem/Leaf Classification Module 5.1.1 Hardware used
MicroZed Board
ZYNQ 7020 SoC 85k Logic Cells Dual Core Arm Processor
1 GB Ram Gigabit Ethernet
Python 1300 camera and Vision Carrier Card
78
1.3 Mega Pixel 1000 FPS (Maximum frame rate) Manual Focusing
5.1.2 Implementation Details
We used microzed board to implement frame capturing module, decision making module, communication module (to communicate with IPC) and actuator controller. SoC chip used in microzed board is ZYNQ7020. This SoC contain both FPGA (Programmable Logic) and CPU (Processing System) modules. To minimize the delay, frame capturing, decision making and actuator controller was implemented in FPGA. Because in FPGA tasks can be executed parallelly. Communication module which handle the communication between FPGA and the IPC (which contains HMI panel and machine learning module) was implemented in Processing System of the ZYNQ7020 SoC. To do the communication between MicroZed and the IPC Gigabit ethernet port of the Microzed is used. Theoretical speed of it is 1Gbps. Capturing images was done using Python 1300 camera module. That camera module was connected to microzed board using the Vision Carrier Card. Fig. xx shows the camera with the vision carrier card. 3rd Party IP cores are used to get raw pixel data from the image sensor and convert them to AXI4-video stream which contain RGB data of pixels. An IP core was designed to do color space conversion do the decision making. Fig xx shows the block view
79
of the decision making IP. Another IP was designed to do the communication between PS and the PL. That IP use AXI4-Lite protocol to communicate with the PS. Communication module implemented in PS write data to PL and read data from PL using address of that ip. 4 registers are used to write data to PL from PS. Those registers are used to send threshold values and other configuration data like monitor enabling to decision making module. 12 registers are used to read to from PL to PS. Those registers are used to read current threshold values and other configuration data from the decision making IP. All those IPs are written using verilog.
5.2 Machine Learning Module
80
Figure 77 : Logic Diagram of M L Module
5.2.1.Pre Processing Unit
First remove the blank frames and duplicate frames which are send through controlling unit. Duplicate frame is identified by frame differentiation and blank frame deleted by object detection using otusu algorithm. Then using Moore-Neighbor tracing boundary detection method, contours of the object was detected and cropped by its bounding box. This function was b ased on the ‗boundaries function‘ presented in the first edition of Digital Image Processing Using MATLAB, by Gonzalez, R. C., R. E. Woods, and S. L. Eddins, New Jersey, Pearson Prentice Hall, 2004 [8]. Fig. 1 shows the final result These images consist of both object and the background. For the calculations, it was needed to extract the color data pixels of the object. Therefore, the backgrounds of the images should be removed. For this, the background was identified through the Gray threshold level using ‗otsu‘ algorithm [9] and then the image was transformed into a binary image (background should be black). Next, the srcinal image was multiplied by all the pixel values by the corresponding pixel value of a binary image. Since the binary image background contains 0‘s pixel value, the background of resulting image will be 0 [10]. Fig.2 shows the resulting image. 81
The remove the noise using mean filtering.
Data Sample Separation
. Stems and Leafs data samples
5.2.3.Feature Extraction Unit
This unit is responsible for color space conversion and extracted the mean value of each frame. 5.2.4.Feature Selection and analytical unit
This unit is responsible of finding the most accurate threshold using histogram plot method.
82
83
5.3
Human Machine Interaction Module and Controlling Module
5.3.1 Hardware Used Industrial PC
84
Intel Atom 1.86 GHz 2 GB RAM 8‖ Touch Screen Windows 7 operating system
85
This module was implemented inside the Industrial PC unit. The role of this module is to build the communication between each system modules and the communication between user and the machine. This module consists of two sub modules. They are HMI Panel and the Communication module. HMI Panel : This sub module was implemented to build the interaction between machine and the machine operator. This interaction is done through an interface which is shown in below figure.
86
Figure 78 : UI of HMI Panel
The special characteristic of this machine is the machine operator can train the machine learning model and can derive thresholds value as specific to the factory. To do this the machine operator no need to have any engineering knowledge. He can simply perform this through the above interface. He can select the category of the tea using the drop down menu and he can assign the training time through this UI and need to click on start button. These user inputs are sent to the communication module. Then it calculates the number of frames that can capture from the camera according to the frame rate and send it to the FPGA module. Then machine operator need to enter stem and leaf samples within the assigned time by the machine. The entered samples are capture by the camera in the FPGA module and those captured images are sent to the communication module as binary image files through a TCP connection. After then those binary files are converted in to png image files and send them to the training module. After the training there are two thresholds are sent to the communication module. Then the communication module store this two thresholds values permanently and send it to the FPGA module.
87
After the training the machine operator can view the captured images by simply clicking on ―Display‖ button and also machine operator can view the statistics graphs by simply clicking on the ―Statistic‖ button.
88
Chapter 6: DISCUSSION From starting with the native which are very commonly used color spaces our experiments broaden to the extent to which is creating new hybrid color space which gave similar accuracy to the native multilayer neural network. So after that our experiment is went through the above mentioned textural and shape features for intending to create more optimized hybrid color texture and shape space. But fortunately we ended up with following low accuracies.
So as the 1st version of our product we limited our scope to hybrid color space until we found the new texture value.
A. Partially burned Tea particles
B. Light Color Tea Leaves
89
C. Low resolution images
As to minimize errors comes under A and B we use histogram plotting method and to remove the Future work: Tea Categorizing In tea industry , tea is categorized under different grading levels. Those grading levels are decided based on the size and the shape of the tea leaf. In current situation they use seiver is used to do this grading process and this is done by labourers. The current process is time consuming and required large space. And also it damage to the tea particles and it affect to the quality of the tea. Therefore as the solution we propose to do this process in 95% confident level using a computer vision based system.
90
Chapter 7: CONCLUSION As a main part of this project we proposed the research which is a new criterion for tea stem and leaf classification in the field of color image analysis.Through this research, the advantages of using a color pixels classification algorithm in a Hybrid Color Space with derived color components were presented. This new color space depends on the application. This introduced a new method which is a lightweight solution for implementation and more accurate than feed forward neural network. To increase the accuracy of result evaluation the characteristic curve of histogram was used instead of histogram overlapping calculation method which reduces the dependency on training sample and the accuracy level.
And as a production level we improve the accuracy of the system including using above mentioned new color space and various other image processing and image enhancement techniques and also we introduce dynamic thresholding by implementing separate training module to the current tea industry.
91
REFERENCES [1] Alireza Pazoki and Zohreh Pazoki (2011).Classification system for rain fed wheat grain cultivars using artificial neural network, in: proceeding of African Journal of Biotechnology Vol. 10(41), pp. 8031-8038, 3 August, 2011 . [2] Alireza Pazoki, Zohreh Pazoki and Behzad Sorkhilalehloo (2013).Rain Fed Barley Seed Cultivars Identification Using Neural Network and Different Neurons Number, in: proceeding of World Applied Sciences Journal 22.(5): 755-762, 2013 [3]. Ankur M Vyas ,Bijal Talati and Sapan Naik, Colour Feature Extraction Techniques of Fruits, International Journal of Computer Applications (0975 –8887), Sardar Vallabhbhai Patel Institute of technology,2013. [4] A. R. Pazoki , F. Farokhi , Z. Pazoki (2014).CLASSIFICATION OF RICE GRAIN VARIETIES USING TWO ARTIFICIAL NEURAL NETWORKS (MLP AND NEUROFUZZY), in: proceeding of The Journal of Animal & Plant Sciences. Pp.336-343. [5] Chang Chun1,Chen Yiqun1,Xiao Hongru1,Ren Shougang2,Song Weidong1(1.Nanjing Research Institute for Agriculture Mechanization,Ministry Agriculture,Nanjing,210014,China; 2.Nanjing Agriculturalof University,Nanjing,210095,China) (2013).Smart fresh tea sorter with neural network image analysis in Journal of Changshu Institute of Technology. [6]Cheng-Jin Du, Da-Wen Sun(2006). Learning techniques used in computer vision for food quality evaluation. [7] "Color-Based Segmentation Using K-Means Clustering - MATLAB & Simulink Example", In.mathworks.com, 2016. [Online]. Available: http://in.mathworks.com/help/images/examples/color-based-segmentation-using-k-meansclustering.html. [Accessed: 26- Aug- 2016]. [8] Desai, Vasishth P., and Arjav Bavarva. "Image Processing Method for Embedded Optical Peanut Sorting." International Journal of Image, Graphics and Signal Processing 8.2 (2016): 20. [9]Domenico, S., & Gary, W. (1994). Machine vision and neural nets in food processing and packaging—natural way combinations. In Food processing automation III—Proceedings of the FPAC conference [10]Du, C.-J., & Sun, D.-W. (2004). Recent developments in the applications of image processing techniques for food quality evaluation. Trends in Food Science & Technology, 15(5), 230–249. [11]Gunasekaran. S. (1996.) Computer vision technology for food quality assurance. Trends in Food Science & Technology , 7.
92
[12] "Global image threshold using Otsu's method - MATLAB graythresh", In.mathworks.com, 2016. [Online]. Available: http://in.mathworks.com/help/images/ref/graythresh.html. [Accessed: 26- Aug- 2016]. [13]Kolacz, Jacek. "Advanced sorting technologies and its potential in mineral processing." AGH Journal of Mining and Geoengineering 36.4 (2012): 39-48. [14] Pearson, Thomas. "High-speed sorting of grains by color and surface texture." Applied engineering in agriculture 26.3 (2010): 499-505. [15] "Properties Of Gray-Level Co-Occurrence Matrix - MATLAB Graycoprops". In.mathworks.com. N.p., 2016. Web. 26 Aug. 2016. [16]Shuihua Wang & Xiaojun Yang.(2015). Identification of Green, Oolong and Black Teas in China via Wavelet Packet Entropy and Fuzzy Support Vector Machine [17] S. Somatilake and A.N.Chalmers, Food Colour Measurement, BE(Hons) Engineering Research Project Report, Manukau Institute of Technology,2007. [18]Shuihua Wang & Xiaojun Yang.(2015). Identification of Green, Oolong and Black Teas in China via Wavelet Packet Entropy and Fuzzy Support Vector Machine [19] V. Vapnik (1995). The Nature of Statistical Learning Theory. NY: Springer-Verlag
93
