What are classifiers used for in machine learning?
Artificial Intelligence & Machine Learning
A classifier in machine learning is an algorithm that automatically orders or categorizes data into one or more of a set of “classes.” Moreover, one of the most common examples is an email classifier that scans emails to filter them by class label. Today let’s look at using a classifier to look at news content!
1.1 The goal:
The goal is to create a classifier, which could predict the category according to the news content.
1.2 The approaches I choose:
There are 2 stages, data processing and model training,
1. For data processing:
• Use jieba and regular expression to filter the keywords
• TF-IDF and 3-grams
2. For model training:
• Naive bayes
• RNN
2 Data processing
Before putting data into machine, I need to pre-process the data.
2.1 keywords:
Chinese sentence is a bit different from English, there is no blank between words(e.g., ”我很爱你.” means ”I love you very much.”).
• There are 2 problems:
1. The word ”很” means ”very much”, apparently it does not influence the meaning of the sentence. In English we can split words by spaces, however in Chinese we need a wiser way to get rid of these ”useless” words.
2. Another problem is: the formation of Chinese word is also different from English. For example, ”爱” means love when it appears alone, ”好” means good when it appears alone, however, when you combine them together to ”爱好”, it means ”hobby”. Hence we need to let computer know it sup poses to mean ”love”, rather than ”hobby”. Then split these words into English form(e.g. ‘我’,‘很’,‘爱’,’你‘. and ‘I’,’love’,’you’,’very’,’much’.).
Fortunately I don’t need to worry about these, there are several conve nient packages which could solve this problem. The most famous one is called: jieba. After applying it the sentence ”我很爱你.” will be splitted to a list: [”我”,”爱”,”你”]. (The regular expression is also used here to exclude the notation, so the period is gone.)
2.2 3-grams:
After getting the key-words, I applied 3-grams to the words list. • There are 2 advantages of applying 3-grams:
1. The vocabulary library will be enlarged, the machine will have more effective samples to learn.
2. 3-grams builds the connection between words, the machine will receive a meaningful phrase rather than a single word.
• The implementing detail:
– Say we have a list: [”我”,”爱”,”计算机”,”和”,”音乐”] (it means ”I love computer and music”), when applying n=3, the algorithm will combine 3 words together sequentially
– In this case, the algorithm will output.(Figure 1)
Figure 1: 3-grams
– When n=2, the algorithm will do the same thing but combine 2 words instead of 3.
– I decided to keep both 2-grams and 3-grams.
– In fact, after preprocess the data using jieba, eventually 2grams&3- grams will output a list contains only 3 elements: [’爱 计算机’, ’计算 机 音乐’, ’爱 计算机 音乐’].
– This is because the word ”我” and ”和” means ”Me” and ”And” separately, these words will not influence the meaning of the sentence. As you can see, after the previous steps we do get the core idea of the sentence: ”love both computer and music”
– Then I will input a series of data into the TF-IDF to get the frequency matrix.
– 3-grams code.(Figure 2)
2.3 TF-IDF:
There are 2 steps: Term-Frequency and Inverse Document Frequency • Term-Frequency:
Figure 2: 3-grams
– This part is simple, the lib I used is called sklearn, the package CountVectorizer will help me to count the frequency of a word appeared in the document. If a word appeared in the list once, the frequency will increment. The machine will consider the entire con tent lists as the corpus.
– This is what I get from above steps.(Figure 3)
Figure 3: TF
• Inverse Document Frequency:
– In this part, I still use the package in the same lib called TfidfTrans former, it will help me to transform data into IDF.
– The formula it used is(Use log to normalize):
– IDF = log
T he total number of documents
T otal number of documents including the word + 1
• Then perform TF x IDF, I will get the TF-IDF matrix.(Figure 4)
Figure 4: TF-IDF
3 DataSet
Before starting the training part, I need to talk about the dataset.
3.1 Dataset:
• I choosed a well-defined data set on:
– https://github.com/aceimnorstuvwxz/toutiao-multilevel-text-classfication dataset
– This repo has 5 data sets, I took the first four data sets.(aa, ab, ac, ad)
– Each data set contains around 600 thousands data, so I chose the first 3 as my training set and 50 percents data of the last one as my testing set.
• The original data set contains over 1000 categories, of course I don’t need that many categories. After some modification, moreover, I selected 13 categories as the main categories and combined some sub categories:
– 13 main categories:
- 1. edu
- 2. entertainment
- 3. fashion
- 4. finance
- 5. food
- 6. health
- 7. military
- 8. society
- 9. culture
- 10. sports
- 11. tech
- 12. travel
- 13. science
– Firstly, combined game and comic with entertainment.
– Secondly, combined digital with tech.
– Thirdly, combined history with culture.
– Fourthly combined politics with society.
– Lastly, I also rejected some subcategories that are not very common.
• After simplification, the size of each data set reduced around 15 percents, eventually I will have about 1.6 million data for training set and 0.3 million data for testing set.
4 Model Training
4.1 Naive Bayes:
• Implementation:
– The package I used is MultinomialNB in lib sklearn.
– By definition: ”The multinomial Naive Bayes classifier is suitable for classification with discrete features (e.g., word counts for text clas sification). The multinomial distribution normally requires integer feature counts. However, in practice, fractional counts such as tf-idf may also work.”
– More simply, that is Multinomial Na¨ıve Bayes consider a feature vector where a given term represents the number of times it appears or very often i.e. frequency.
– So the whole implementation is based on 2 assumption:
1. The data points are distributed independently. Not like weather prediction, news contents have no inner relationship. In Multi nomialNB the prior probability is a polynomial distribution
2. The features are independent as well. In my case, there are 13 feature vectors(categories), indeed they are independent.
– Moreover, those 2 assumption simplified the whole process. So each part in the Bayes formula can just simply multiply each other to get the outcome, instead of using the law of total probability to compute.
• Result:
– The result is quite promising, the accuracy is 86 percents.
– The training speed of naive bayes is very fast, even for this huge data set it still only takes 1 minute.
– The confusion matrix: (Figure5)
Figure 5: Confusion Matrix
4.2 LSTM:
• Implementation:
– The package I used is tensorflow2.0 and keras.
– At the beginning, I used the same process to parse the data (TF-IDF + 3-grams), however, it will output a csr matrix, which is not array. Transform csr matrix into array requires a lot of RAM when the size is huge (In my case, it requires 900GiB RAM). This is because although the shape of the matrix is (num data, vocabulary), the vocabulary of my data set is huge (millions of words).
– After trying several alternatives, I decided to give up using TF-IDF and using 3-grams only. The data process follows the similar steps, except at the embedding stage, I used Tokenizer.(Figure6)
Figure 6: Tokenizer
– The structure of network: (Figure7)
Figure 7: LSTM
– In case the parameter will be over billions, I used a smaller dataset.(8000 data for training and 1600 for testing). Moreover, the input parameter is 1,374,506.
• Result:
– The result is good as well; the accuracy is 78 percents. (Figure 8) – When epoch=3, it takes around 6 minutes to finish training.
Figure 8: Accuracy
5 Conclusion:
5.1 Conclusion
• As a conclusion, both solutions‘ performance is quite good. The reason why LSTM is weaker may because I used a much more smaller data set
comparing with Bayes(8k to 1.8 million).
- Moreover, I believe if LSTM received enough data for its vocabulary pool, the accuracy will above 90 percents.
- Furthermore, it is surprising that Naive Bayes could have such a good score.
- I had a lot of fun while implementing all of these stuffs. This project contents my curiosity and interest to machine learning, more importantly.
I implement everything I want, so far.
• About further goals:
– The accuracy should at least hit 95 percents, can it be applied to the actual news classification environment. Lastly, I will try to optimize the LSTM. (I think the limit of Naive Bayes is 86 percents, it is meaning less to add a larger data set)
– In conclusion, maybe we can talk about content recommend system further?
What are classifiers used for in machine learning?
