Using NLP to Create an ESG Dictionary The texts we are using for the whole project are 10-Ks and 10-Qs in 2020 and 2021. The reason why we pick the two-year frame is that we want to strike a balance between using more data and including only the up-to-date information that reflects companies’ current ESG commitment. Then, we preprocess these files so that only the relevant texts are kept. One special treatment we adopt is to exclude the section with more than 70% of content being figures.
10-K 10-Q Scraping
These sections are normally financial sheets such as balance sheets, statement of cash flows, etc. Companies usually have similar balance sheet structures so texts in these sections might not offer much information. Then, we perform standard NLP preprocessing techniques such as tokenizing, lemmatizing and removing stop words.
1. For governance dictionary, we would try to only include the sections that are related to board of directors announcement/information
2. Identify sections with too many figures (tweaking the 70% parameter) and see whether the performance is improved
Unigram + Linear Regression
The first approach we take is fitting a simple linear regression model for each word to understand its impact on E, S, G score individually. The impact is measured as the beta (slope) of the regression line. Specifically, we regress score on the indicator function of each word to measure that if a word exists in a company’s filings. What would be its impact on the score. We do this for each of 11 sectors and 3 score types. Another version is to use the tfidf scores of each word instead of the indicator function to account for the frequency of each term in each document and its number of existence across different companies’ filings. The words with the highest slopes are the ones that have the largest negative impact on E/S/G score, and vice versa.
While this approach does successfully generate some reasonable results by looking into the dictionary. It somehow fails to exclude some words without sentiment. The resulting dictionary seems not good enough.
With the unigram dictionary, we perform the same validation process as described in the next section on the out of sample data. The performance is not promising mainly for two reasons. First, the way we classify companies as good or bad depends on the number of occurrences of good and bad words in each company’s filings. The drawback of this methodology is that bad words may occur much more frequently than good words. This leads to the result that the counts of bad words exceed the counts of good words for most companies. Which then results in a huge skew toward prediction of “bad”. The second limitation is that unigram might not be as good in capturing sentiments as bigram and trigram. If we want to better classify out of sample companies, we should probably use bigram/trigram.
3. Try the same methodology on bigram/trigram, so we can compare the results with the next approach
4. Instead of working on each sector individually, do the same approach on all companies
Word Count/Validation
The validation process was divided into a series of subsequent steps. Firstly, companies were classified into two groups: good and bad companies. This was done according to the companies scores and the type of scores. Where companies in the top third classified as bad companies while the bottom third as good companies. This leaves us with a list of bad and a list of good companies for each score type to further conduct the analysis. Below are distribution graphs that will help visualize company scores for each type.
Next, we split the good and bad lists into 70% to train the model while the remaining 30% for validation. This leaves a total training set of 209 companies. And a validation set of 90 companies for each of the score types. To avoid any biases, the train and test splits randomly generated, and data reshuffled every time the model runs.
Following that, the training set labeled with both bad and good companies were fed to the scraping algorithm. That produced a dictionary of bigrams and trigrams for each score type. The dictionaries were first applied to the in sample data to understand the predictive power of the model. Below are the performance results of the in sample model:
After obtaining reasonable results from the training data, the dictionaries used on the out of sample data. To make sure that the out of sample model performance is consistent. And not random, the validation phase performed more than once with a randomly generated testing set. Below are the results of two runs on the out of sample data:
Finally, we present a sample (top 20 words in each score type) of the dictionaries that produced the above results. The dictionary shows a big improvement over the dictionaries we generated for each sector individually, which were what we presented during the midterm presentation. Most grams in this dictionary are relevant to each E, S, G topics intuitively.
5. Figure out a better way to classify companies than simply counting the number of good and bad bigram/trigram (this also applies to the previous approach) a. Assign different weights for each gram, based on the difference in its occurrence in goodies and baddies (e.g taking the ratio)
b. Figure out a reasonable threshold to label a gram as good vs bad (Currently taking the top and bottom 30% out of 200 grams as good and bad grams)
Words in Context
Words out of context do not imply a sentiment for any of the ESG scores by themselves. My goal is to analyze the sentences in which these words occur to see if there are average differences in sentiment. Furthermore, between good and bad companies. The first step was to sort the top 200 bigrams and trigrams to select those which were most common and shared between good and bad companies. This would ensure that we could analyze sentences from as many companies as possible and that we could reasonably compare results of the analyses.
For the analysis framework, we utilized the VADER framework since it does not require any training and is easily interpretable. To boost the power of the VADER framework, we added 20000 additional words and scores to its dictionary from the Federal Reserve Bank of San Francisco. This bank increased the number of financial sentiment-related words that the analyzer could identify and process as part of its scoring algorithm. This word bank was used by the FRBSF to create a sentiment index from news headlines, so we were fairly confident in its ability to accurately capture nuances in language.
Unfortunately, results so far remain inconclusive. After analyzing the shared ngrams across all sectors and score types, we were not able to show a significant difference in sentiment between sentences from good company filings and bad company filings. We even attempted to skew how distributed the words are between good and bad companies, i.e. we wanted to see if words that showed up more frequently in either good or bad companies (but still were in both) may have more of a sentiment bias one way or another. Again, results were not able to display any difference in the sentiment of good and bad companies.
