How AI is used in M&A? FORECASTING M&A TARGETS LEVERAGING ON MACHINE LEARNING
Artificial Intelligence & Machine Learning
Merger and Acquisition (M&A) prediction has been a challenging topic in the financial industry in the past a few decades. Yet seldom past studies have forecasted the takeover targets from the perspective of Machine Learning approaches. This project focuses on the implementation of Machine Learning models in predicting M&A targets within the universe of Russell 3000 Index and the construction of an equity portfolio based on the model results to capture the spike in the stocks’ prices caused by the acquired premiums. The predicted M&A target list with the highest F1 score and precision is generated by the Random Forest (RF) method, which gives Return on Investment (ROI) of 19.61% and successfully forecasts two takeover targets within the Russell 3000 Index in 2020 Q4.
1.1 Time-series Analysis of M&A Deals
Figure 1.1 shows the time series analysis of announced M&A deals happening in companies in Russell 3000 Index in the recent two decades. We can see that the deal count and deal volume peaked around the end of 2006 and the beginning of 2007, and then dropped sharply following the 2008 financial crisis. After that, the deal volume reached a higher peak around 2015 and 2016 due to the low interest rate environment by then and also the extra amount of cash held by the companies. Also, we can see that the deal count and deal volume have been very volatile, and they tend to be cyclical: over two thirds of the M&A deals actually happened within the first 2 quarters of the year.
As for 2020, COVID-19 has substantially reduced M&A activities in the 2nd quarter, but it was only a temporary effect as we can see the M&A deal and volume has started to recover in Q3. This can be partly due to the fact that the COVID-19 pandemic was actually a great opportunity for companies with strong balance sheets and sufficient credit facilities to exploit M&A hotspots. Also, with the buoyant stock prices of the firms in the industries like Technology and Healthcare, there’s more prospective buyers and sellers in the market. One interesting fact we found out is that the average takeover premium in 2020 is actually the highest in the past 10 years, and this high premium was actually driven up by a few large takeovers in the pharmaceutical industry: Alexion Pharmaceuticals, Inc acquired Portola Pharmaceuticals, Inc with an announced premium of 156% in July 2, 2020; Menarini Group paid $12.50 per share for Stemline Therapeutics, a sizeable premium on the company’s share price of $4.75; and Gilead Sciences acquired Immunomedics and Forty Seven with premiums both around 110% in 2020 Q1 and Q3.
Figure 1.1: Predicted M&A Targets by Industry
1.2 Deal Size Breakdown and Target Industry Summary
If we take a closer look of deal breakdown over the 2 decades (Figure 1.2), around half of the deals are within $500 million, and almost 90% of the deals are the small and medium sizes within 5 billion dollars; while there’s only 5% of the deals are greater than $10 billion, they account for most of the deal volume.
Figure 1.3 presents the target industry summary: the top 4 industries of M&A targets are Consumer-non cyclical, Financials, consumer cyclical and Technology. For the Technology industry, although it has a relatively large number of deals, its deal volume is actually quite low, which indicates there’s a lot of small M&A deals happening in the technology industry. On the contrary, the Financial Industry has the largest average deal volume. In general, Energy, Utilities and Basic Material are the sectors with the lowest deal volumes and fewest megadeals. As those industries continue to consolidate, the opportunities for M&A have been shrinking; in addition, the sheer size of investment also limits the number of buyers in the market, and even when the right-size deal is available in the market, the competitions are tense and the average takeover premiums are much higher than the other industries. In the Energy industry, the collapse of oil prices due to COVID-19 and the increasing negative reaction to environmental, social and government (ESG) issues at oil and gas companies was slowing down the M&A activity in 2020 Q2. There are only 8 deals in the Energy Industry so far this year, compared to last year and the year before that, which both have 21 deals.
Figure 1.2: Deal Size Breakdown
Figure 1.3: Target Industry Summary
The remainder of the paper is structured as follows: section 2 presents the data collection and analysis methods, including what macroeconomic and fundamental factors we chose as our independent variables and why we chose them; in section 3, we spend a fair amount of time on data processing including data cleaning and clustering, data transformation and several ways to balance our dataset as our raw dataset from Bloomberg is noisy and extremely imbalanced. In section 4, we introduce our feature selection methods; section 5 and 6 apply a series of Machine Learning models with logistic regression being the benchmark model, and then analyze their results by comparing the accuracy and precision of all models, followed by the portfolio construction and future improvement.
2. Dataset
Our dataset is the universe of Russell 3000 companies, spanning over 20 years from 2000 Q1 to 2020 Q3, dictated by the announcement date. We include both macroeconomic and fundamental factors as our explanatory variables, and all data was extracted from Bloomberg on a quarterly basis. In addition, we create an extra column labeling the signal shift of all the companies. The companies that become the takeover targets in the subsequent quarter are labeled 1 and the rest are labeled 0. For instance, if company XYZ is acquired by another firm in 2020 Q3, then we’ll label it as 1 in 2020 Q2; for all other cases, company XYZ will be labeled as 0.
We consider macroeconomic and fundamental factors in our model because macroeconomic factors reflect the economic trend, long-term market projections, and impact of fiscal and monetary measures, which can all potentially affect the amount of M&A deals in a quarter. An expanding, optimistic, fast growing and stable market usually has more M&A activity. Factors like real GDP and unemployment rate indicate the economic cycle, other variables include consumer confidence index and long term interest rate suggest the future outlook of the market, and the stability of the stock market is revealed by index like VIX and gold spot price, and market inflation is measured by Consumer Price Index. Some other factors that may indirectly affect M&A are also included in our model, for instance: the Dollar Index, Crude Oil price (Cushing, OK WTI Spot Price) and Dow Jones Industrial Average (DJIA) Index. These macroeconomic factors are also often interlinked.
Fundamental factors are the company specific attributes which are directly related to its M&A activity. In our model, we have market capitalization measures the size of the company; return on equity, profit margin and EBITDA measure the profitability of the company; there’s also company-valuation metric like earning per share and price multiples P/E, P/B, P/S; and some balance sheet items like current ratio, total assets and liability to indicate the health of a company’s financial status. And some common factors like stock price and total shares outstanding are also included.
Figure 2.1: Macroeconomic and Fundamental Factors list
3. Data Processing
3.1 Data Cleansing
Our dataset that we extracted from Bloomberg is extremely noisy as it contains missing values, NaNs and a lot of dashes. The raw data has 2,45,916 data entries with 46 features and 1 target variable. To deal with this issue, we came up with a condition that for every feature, whenever the sum of NaNs, dashes and missing values was equal to or greater than 30%, we would drop those columns entirely. After this procedure, we had 32 features and 1 target variable (See Figure 2.1).
This step is reasonable since these fundamental factors are all characterizing the operation of the enterprise so we can obtain a certain degree of compensation from other factors that have not been deleted without losing too much information. Also, since we may build a model to fill in the missing values, keeping them may lead to too much autocorrelation.
After these features were selected, since missing values were still present in the features that existed, we did an iterative imputation to fill out all the missing values in the feature columns; the methodology for which is described below.
3.1.1 Iterative Imputation
This approach is an extremely efficient technique to impute missing values. Iterative imputation refers to a process where each feature is modeled as a function of the other features, e.g. a regression problem where missing values are predicted. Each feature is imputed sequentially, one after the other, allow ing prior imputed values to be used as part of a model in predicting subsequent features.It is iterative because this process is repeated multiple times, allowing ever improved estimates of missing values to be calculated as missing values across all features are estimated.
The scikit-learn machine learning library provides the IterativeImputer class that supports iterative imputation.
3.2 Data Splitting
After performing iterative imputation for the missing values, we split the dataset into train and test data. Since the data spans from Q1 2000, we used the data from then to Q2 2020 as training data and the data in Q3 2020 is used for prediction.
We also performed a logic-check of the dataset by defining criteria for all the features. For all the rows, eg. for the price feature, whenever the price<0 or ROE>100, we deleted those corresponding rows. The same process was repeated with criteria defined on each of the features in the dataset. This resulted in 228966 rows and 32 columns in the training data set and 228966 rows and 32 columns in 2453 rows and 32 columns in the predicting data set. The dataset is now almost ready for modeling, however, we also wanted to standardize the data before we went ahead with modeling.
3.3 Standard Scaling
Standardizing the features around the center and 0 with a standard deviation of 1 is important when we compare measurements that have different units. Variables that are measured at different scales do not contribute equally to the analysis and might end up creating a bias.
For example, A variable that ranges between 0 and 1000 will outweigh a variable that ranges between 0 and 1. Using these variables without standardization will give the variable with the larger range weight of 1000 in the analysis. Transforming the data to comparable scales can prevent this problem. Typical data standardization procedures equalize the range and/or data variability.
We performed standard scaling of the dataset which is inclusive of both the training and testing data.
4. Feature Selection
After we finished the data cleaning, we now have the complete dataset to analyze. In this section, we adapted the methods in Professor Lopez De Prado’s Advances in Financial Machine Learning book to perform feature selection and feature importance.
4.1 SelectKBest Method
Feature selection is a technique where we choose those features in our data that contribute most to the target variable. In other words, we choose the best predictors for the target variable.
The classes in the sklearn.feature_selection module can be used for feature selection/dimensionality reduction on sample sets, either to improve estimates’ accuracy scores or to boost their performance on very high-dimensional datasets.
The main advantages of this method are that it
• Reduces Overfitting: Less redundant data means less possibility of making decisions based on redundant data/noise.
• Improves Accuracy: Less misleading data means modeling accuracy improves.
• Reduces Training Time: Less data means that algorithms train faster. • Independent of the predictive method used for modeling 9
The methodology of this is that it builds a contingency table that has a multivariate frequency distribution of the variables whose dependence is measured using a score function like Chi Squared statistic. The below table shows the features with their importance scores listed in descending order.
4.2 Hierarchical Clustering
We constructed a hierarchical clustering model on all the features in the training dataset. Hierarchical clustering is an algorithm that groups similar objects into groups called clusters and the similarities between objects are calculated by a distance function. There are different kinds of hierarchical clustering techniques and we used agglomerative hierarchical clustering techniques to analyze our data. This technique uses a bottom-up approach. The algorithm starts by treating each object as a singleton cluster. Next, pairs of clusters are successively merged until all clusters have been merged into one big cluster containing all objects. We used a complete linkage distance function to form clusters sequentially.
Complete linkage distance method is one of the most common functions used for clustering. The distance between two clusters is defined as the maximum value of all pairwise distances between the elements in cluster 1 and the elements in cluster 2. The main advantage of this method is that It tends to produce more compact clusters.
Apart from identifying clusters, this technique can also be used to identify unique and duplicates in the data.
The result is a tree-based representation of the objects, named dendrogram. The distance threshold on the y axis sets the similarity cutoff. The lower the value, the more similar the data instances have to be to belong to the same cluster. We set the cutoff as 0.1 for our dataset and from the dendrogram being low, it can be observed that the total liabilities feature has a value of less than 0.1. Hence, it can be concluded that it could be a duplicate and we deleted that feature alone from the dataset. In all, we have the remaining 31 features as listed in Table 1 that we could use for our modeling the training dataset. The next step we did was to build a baseline model using the training dataset of 31 features.
Figure 4.2: Dendrogram : Hierarchical Clustering
4.3 Feature Importance
After we trained our model using Logistic Regression, we wanted to better interpret the contribution of all the features towards the prediction. Hence, we decided to use the Shapley method to do this. SHAP values interpret the impact of having a certain value for a given feature in comparison to the prediction we would make if that feature took some baseline value.
The idea behind SHAPE feature importance is simple: Features with large absolute Shapley values are important. Since we want the global importance, we average the absolute Shapley values per feature across the data:
Next, we sort the features by decreasing importance and plot them. The following figure shows the SHAPE feature importance for the Logistic Regression trained before for predicting target acquisitions.
Figure 4.3: SHAP Feature Importance Plot: Logistic Regression
SHAP feature importance is an alternative to permutation feature importance. There is a big difference between both importance measures: Permutation feature importance is based on the decrease in model performance. SHAP is based on magnitude of feature attributions. SHAP feature importance measured as the mean absolute Shapley values. The Shapley value is the average marginal contribution of a feature value across all possible coalitions. The Shapley value method is based on the property that the difference between the prediction and the average prediction is fairly distributed among the feature values of the instance. The axioms – efficiency, symmetry, dummy, additivity – give the explanation a reasonable foundation to the theory. However, the main disadvantage of this technique is that it requires a lot of computing time.
From the above plot, it can be observed that the P/E is the most important feature that has the maximum impact on the predicted target acquisition.
While the feature importance plot is useful, it contains no information beyond the importance. For a more informative plot, we also plotted a summary plot below:
Figure 4.4: SHAP Summary Plot: Logistic Regression
The summary plot combines feature importance with feature effects. Each point on the summary plot is a Shapley value for a feature and an instance. The position on the y-axis is determined by the feature and on the x-axis by the Shapley value. The color represents the value of the feature from low to high. From the plot, it can be observed that a high level of the “P/E” value has a high and negative impact on the target variable.
Overlapping points are jittered in the y-axis direction. The features are ranked in descending order according to their importance. We have set a threshold as 0.2 for our analysis. This essentially means all the features that have mean SHAP value over 0.2 have been used to retrain the model.
5. Model Selection
5.1 Overview
This section aims to find an optimized combination of the machine learning model and imbalanced sampling method that has the best performance given the pre-processed dataset. The candidate machine learning models are selected based on the nature of our objective problem in addition to the strength and popularity of each algorithm in the quantitative finance industry. In addition to the benchmark model Logistic Regression, the candidate model list includes Naïve Bayes, Random Forest, Support Vector Machine, Ada Bayesian Reticulum, Ada Boost, and XG Boost.
As an overview, the model selection process can be divided into cross validation, model, and analysis. In specific, a cross validation technique for train-test data split is utilized. Within each iteration of the split, a version of the imbalanced sampling method is applied to the training dataset in order to enhance the training process for each model, while the testing dataset remains untouched so that the true performance of each trained model in each split will be captured.
Subsequently, a function that tunes the hyper-parameter of a selected model is applied to the training dataset and returns a selected machine learning algorithm with the best hyper-parameter for this iteration. After fitting the training dataset into the model, the untouched testing data split will be used for model prediction. The predicted result is then compared with the true target signal from the testing dataset, and the performance metrics are computed and recorded. The mean score of the metrics across all iterations will be used to evaluate the performance of each model and imbalanced sampling method combination. Hence, the best performing machine learning algorithm and under-sampling or over-sampling method will be applied in the actual prediction as described in the following section.
Figure 5.1: Procedure
5.2 Cross Validation
Since one of the major problems that challenges the project objective is the class imbalance of the target variable, an uneven training and testing split will discredit the performance metrics of the iteration. In order to make the data splitting reflective of the sampled dataset, the stratified k fold cross validation is used. In each iteration, the fold generates training and testing sets such that all contain the same class distribution as the sample does. Moreover, stratified k fold also preserves order dependencies in the dataset ordering. As a result, the fold method also minimizes information leakages.1 In order to have a close estimate of the model performance metrics to the sampled dataset, the amount of cross validation splits is set to 20 with the shuffling feature turned on.
In this project, the cross validation is set up as part of the model selection framework, which will apply one machine learning model and one imbalanced sampling method to compute the average performance metrics for further anal ysis. In each train-test split, only the training data split will be processed through the imbalanced sampling method selected by the current framework (as described in Imbalanced Sampling Method) and, subsequently, moved on to the model subsection (as described in Model).
5.3 Imbalanced Data
One of the major challenges to the project objective is the imbalanced class size. In order to increase the proportion of minority classes, all categorized takeover instances are generalized as one minority class. However, the proportion of takeover instances among Russell 3000 companies takes less than 3%. This makes the sample classes still severely imbalanced. This project selected 4 imbalanced sampling methods that are 3 versions of Near Miss under-sampling and 1 version of SMOTE oversampling. Under-sampling methods balance class size by reducing the majority class instances; over-sampling methods balance class size by increasing the minority class instances. The following paragraph will describe the mechanism of Near Miss and SMOTE in detail and compare between these sampling methods.
5.3.1 Near Miss
There are three versions of the Near Miss Method, which have similarities in algorithmic mechanics but differences in the selection of instances in the majority class. The differences are illustrated in the procedures below.
5.3.1.1 Procedures for Near Miss Version 1
Near Miss Version 1 selects instances from the majority class based on the smallest Euclidean distance to k closest instances of the minority class. In practice of this project, the default value of k=3 is used.
• Step 1: The Version 1 algorithm will find the distances between all instances of the majority class and the instances of the minority class.
• Step 2: Instances of the majority class for which the average distances to the k closest instances from the minority class are the smallest are selected.
• Step 3: The selected instances from the majority class will merge with the entire dataset of the minority class and form a new pandas data frame.
5.3.1.2 Procedures for Near Miss Version 2
Near Miss Version 2 selects instances from the majority class based on the farthest Euclidean distance to k farthest instances of the minority class. In practice of this project, the default value of k=3 is used.
• Step 1: The Version 1 algorithm will find the distances between all instances of the majority class and the instances of the minority class.
• Step 2: Instances of the majority class for which the average distances to the k farthest instances from the minority class are the smallest are selected.
• Step 3: The selected instances from the majority class will merge with the entire dataset of the minority class and form a new pandas data frame.
5.3.1.3 Procedures for Near Miss Version 3
The procedures for Near Miss Version 3 can be considered the combination of the first and the second version since it is selecting N largest nearest-neighbors out of the M nearest-neighbors. The algorithm starts by a phase of re-sampling. The parameter N corresponds to the number of neighbors selected which will be used in forming the subset in which the selection will be performed. 3 In the practice of this project, both values take the default parameters, which is M=3 and N=3.
• Step 1: The Version 1 algorithm will find the distances between all instances of the majority class and the instances of the minority class.
• Step 2: For each minority class instance, their M nearest-neighbors will be stored. Then, the majority class instances are selected for which the average distance to the N nearest-neighbors is the largest.
• Step 3: The selected instances from the majority class will merge with the entire dataset of the minority class and form a new pandas data frame.
5.3.1.4 Advantage and Disadvantages for Near Miss Under-sampling Advantages
• Same as any other under-sampling method, one of the most important features of Near Miss is reduction in time complexity. It can help improve run time and storage problems by reducing the number of training data samples especially when the training data set is huge.
Disadvantages
• Similar to any other under-sampling techniques, Near Miss can omit crucial information given its filtering nature of instances from the majority class based on the set up of the algorithm version and the selection of parameters. It can discard potentially useful information which could be important for building rule classifiers in the model training process.
• Bias might have been created as a result of selecting instances from the majority class based on a patterned technique. The sample chosen by Near Miss under-sampling may be a biased sample. And it will not be an accurate representation of the population. Thereby, resulting in inaccurate results with the actual test data set.
5.3.2 SMOTE
SMOTE representing Synthetic Minority Oversampling Technique is one of the most commonly used oversampling methods to solve the imbalance problem.2 It aims to balance class distribution by randomly increasing minority class examples by replicating them. SMOTE synthesizes new minority instances between existing minority instances. It generates the virtual training records by linear interpolation for the minority class. These synthetic training records are generated by randomly selecting one or more of the k-nearest neighbors for each example in the minority class. After the oversampling process, the data is reconstructed and several classification models can be applied for the processed data. Figure 1 below greatly illustrates the last procedure of the SMOTE oversampling method. In the practice of this project, the number of nearest neighbors used to construct synthetic samples is set up by the default value 5.
Figure 5.2: Analytics Vidhya
5.3.2.1 Procedures of SMOTE algorithm
• Step 1: Setting the minority class set A, for each x ∈ A, the k-nearest neighbors of x are obtained by calculating the Euclidean distance between x and every other instance in set A.
• Step 2: The sampling rate N is set according to the imbalanced pro portion. For each x ∈ A, N examples (i.e x1, x2, . . . xn) are randomly selected from its k-nearest neighbors, and they construct the set A1
• Step 3: For each example xk ∈ A1 (k=1, 2, 3. . . N), the following formula is used to generate a new example: x0 = x + rand(0, 1)∗ | x − xk | in which rand(0, 1) represents numbers sampled from a standard uniform distribution between 0 and 1.
5.3.2.2 Advantages and Disadvantages of SMOTE Oversampling Advantages
• The SMOTE over-sampling method tends to have better performance than general under-sampling methods. Differing from under-sampling methods which add filters to instances of the majority class and discard instances of the majority class that is unselected based on the algorithm technique, SMOTE over-sampling method does not delete data points from the majority class but adding instances to the minority class to reach the desired class balance ratio (usually 1-1). Since no information is lost given rise to the imbalanced sampling method, the dataset processed by the SMOTE method will provide as complete information as possible to the model training process.
Disadvantages
• The SMOTE over-sampling method will be quite problematic from the perspective of time-complexity. If we aim to transform the class balance
ratio to 1-1, the size of the training dataset is almost doubled since under stratified k fold, the proportion of the minority class in the training dataset is maintained the same as that in the sample. Consequently, SMOTE method posts severe challenges to run time and storage requirements.
• The SMOTE over-sampling method increases the likelihood of overfitting since it is replicating instances of the minority class. The effect of overfitting is quite significant since the amount of genuine minority class instances takes merely 3% of the total data points. And this means another 94% of the total data points will be synthetically created from the genuine minority class instances which takes merely 3% of the original to tal. As a result, the synthesized instances are over 31 times the amount of the original minority class instances. Thus, there exists a quite high risk of overfitting.
Figure 5.3: Comparison between oversampling and under-sampling
6. Model Introduction
6.1 overview
In addition to the benchmark model Logistic Regression, the candidate model list includes Naïve Bayes, Random Forest, Support Vector Machine, Ada Bayesian Reticulum, Ada Boost, and XG Boost. In the model selection framework, one of the machine learning algorithms will form a combination with one of the imbalanced sampling methods. Each combination of resampling method and model will go through the hyper-parameters tuning process using the RandomizedSearchCV function. In the practice of this project, the hyper-parameter range for each model is fitted into the RandomizedSearchCV function to find the best combination of the hyper-parameter for each permutation of model and imbalanced sampling method given the current train-test split. Consequently, the optimized hyper-parameter for the selected model will be fitted into the resampled training data for the current cross validation split. Then, the trained model will be used to predict the testing dataset of the current train-test split. The result will be used for computing performance metrics along with the actual target signals from testing data split. After all steps have been completed, the above procedures will be repeated for the next iteration. The performance metrics of each permutation of machine learning algorithms and imbalanced sampling methods is compounded and the mean metrics is calculated accordingly after all iterations have been finished for each permutation.
6.2 Logistic Regression
6.2.1 Introduction
Logistic regression is a widely used generalized regression method for classification problems that combine factors linearly and produce a linear boundary. Logistic regression converts the factors to a pseudo probability using a more complex cost function than simple linear regression defined as the “Sigmoid function,” which maps any real value into another value between 0 and 1.
Figure 6.1: Loss Function
The hypothesis of logistic regression tends to limit the cost function between 0 and 1. After applying the sigmoid function, the hypothesis function for logistic regression is as follows:
Our task now is to choose the best parameters θ in the equation above, given the current training set, in order to minimize errors. The procedure is similar to what we did for linear regression: define a cost function and try to find the best possible values of θby minimizing the cost function as follows:
The minimization will be performed by a gradient descent algorithm, whose task is to parse the cost function output until it finds the lowest minimum point. We expect our classifier to return a probability score between 0 and 1 and a test data point would be predicted as positive if the predicted probability is higher than a cutoff value.
6.2.2 Advantages and Disadvantages
Advantages:
• Logistic regression is easier to implement, interpret, and very efficient to train.
• It makes no assumptions about distributions of classes in feature space.
• It can easily extend to multiple classes(multinomial regression) and a natural probabilistic view of class predictions.
• Good accuracy for many simple data sets and it performs well when the dataset is linearly separable.
Disadvantages:
• The major limitation of Logistic Regression is the assumption of linearity between the dependent variable and the independent variables.
• It can only be used to predict discrete functions. Hence, the dependent variable of Logistic Regression is bound to the discrete number set.
• If the number of observations is lesser than the number of features, Logistic Regression should not be used, otherwise, it may lead to overfitting.
• It constructs linear boundaries.
6.3 Naive Bayes
Naïve Bayes classifiers are a family of “simple probabilistic based classifiers” based on the Bayes’ theorem, with the fundamental strong assumption that features are independent from each other.
The objective of a Naive Bayes algorithm is to measure the conditional probability of an event with predictor variables x1, x2, . . . , xn belonging to a particular class Ci.
On computing the above equation, we get the following equation:
However, the conditional probability P(xj|xj + 1, . . . , xn, Ci) sums down to P(xj|Ci) since each predictor variable is assumed to be independent in Naive Bayes so we arrive at the following final equation:
It would then classify a data point to the particular class with the highest conditional probability calculated by using the equation above.
Although Naïve Bayes is relatively simple and computationally efficient to implement, one of the major drawbacks is that its assumption would be too strong for applications in the finance world. More specifically, although we could apply a more sophisticated feature selection algorithm or Principal component analysis (PCA) to the company-specific features, the correlation between fundamental factors is hard to avoid, violating the key assumption of using Naïve Bayes Classifier.
6.4 Ada Bayesian
6.4.1 Introduction
there are two popular techniques for supervised learning, namely neural network and decision tree. And these two methods have recently been combined in a single construct.
This combined algorithm is called Ada Bayesian, and it is essentially constructing a neural network with a tree format. The key concept behind Ada Bayes is the gate function shown below, and that gate function allows the neural network to mimic a tree split.
The above graph shows a two-layer Ada Bayesian model, each leaf will represent a different probability, which in our problem is the probability of a company being taken over in the next quarter if this company falls into that specific leaf.
To be more specific, the following example will show exactly how this algorithm works.
For each data point fed into the model, it will first go through the door and produce an output which is a probability. In this case, we get the probability of 0.21, 0.21 will then be passed to the left node, and 0.79 will be passed to the right node. However, after each node gets its own input, the input will be further transformed using a Bayesian update function.Using the same logic, we are able to construct the neural network in a tree format.
6.4.2 Advantages and Disadvantages
Ada Bayesian model is able to prevent overfitting since the observations would be split across leaves and a Bayesian approach is allowed to the probability of the output. Besides, this algorithm uses the power of back-propagation so that the creation of a new leaf would automatically adjust the parameters of the rest of the tree. Lastly, the Ada Bayesian Model is much more explainable than the traditional neural network, in that we are able to monitor how each company was assigned to different leaves which is an essential part in financial questions.
However, it is hard to determine the structure of the tree and it is difficult to construct it. Besides, we cannot leverage a global gradient ascent method to optimize the hierarchical parameters.
6.5 Support Vector Machine
6.5.1 Introduction
The objective of the support vector machine algorithm is to find a hyperplane or set of hyperplanes in an N-dimensional space, where N is the number of features, as decision boundaries that help classify the data points. Data points falling on either side of the hyperplane can be attributed to different classes.
In our project, we aim to separate our data points into two distinct classes and there are many possible hyperplanes that could be chosen. We started by trying the Max Margin classifier, which tries to find a perfect plane that has the maximum margin, where the margin is the distance between the nearest point and this plane.
The search for the maximum margin separating hyperplane could be treated as a constrained optimization problem, aiming to maximize the margin under the constraints that all data points must lie on the correct side of the hyperplane:
After plugging in the definition of and adding re-scaling of w and b as an equality constraint, we obtain a new optimization problem as follows:
This new formulation is a quadratic optimization problem. The objective is quadratic, and the constraints are all linear, so it has a unique solution whenever a separating hyperplane exists.
Figure 6.4: Max-Margin Classifier
But this max margin method has an apparent drawback with application in the real market as there is no perfect hyperplane that correctly separates all positive outcomes from negatives given the existence of outliers and random noises. In this case, there is no solution to the optimization problems stated above. Therefore, we tried to fix this problem by allowing the constraints to be violated ever so slightly with the introduction of slack variables, while in return producing a more flexible and realistic decision boundary for our future predictions. This small variegated version of supporting vector machine is called Soft Margin classifier.
To maximize the performance of SVM on our dataset, we applied the cross validation method and different kernel functions to select the best parameter by maximizing F-1 scores.
6.5.2 Comparison
In this section, we will discuss the difference between logistic regression and SVM. The key difference between these two algorithms is logistic regression uses sigmoid loss function while SVM typically uses hinge loss functions. So that SVM tries to find the best margin separating different classes, while logistic regression tries to find the decision boundaries between classes.
Denote N as the number of features in our dataset, M as the number of samples in the dataset. If N is relatively large, logistic regression will typically have a better performance than SVM. However, if N is small while M is intermediate, it is more preferable to use SVM. Lastly, if N is small and M is large, the first thing needs to be done is trying to add more features to the dataset, otherwise, both models will encounter the problem of underfitting.
6.6 Random Forest
6.6.1 Introduction
Ensemble methods combine the results of multiple machine learning algorithms to obtain better predictive performance, aiming to produce a more predictive model with less variance, bias and improve the prediction outcome. The main principle behind ensemble methods is that a group of “weak learners” can come together to form a “strong learner”. The figure below provides an example for ensemble methods. The data to be modeled are the blue circles. We assume that they represent some underlying function plus random noise. Each individual learner is shown as a gray curve. Each gray curve (a weak learner) is a fair approximation to the underlying data. The red curve (the ensemble “strong learner”) can be seen to be a much better approximation to the underlying data.
There are two major types of ensemble methods:
Bagging: a parallel ensemble method that trains multiple trees using bootstrapped data to reduce variance and prevent overfitting (e.g. Random Forest)
Boosting: a sequential ensemble method. Similar to bagging, but it starts with a highly biased model and learns sequentially and reduce bias and variance earns sequentially and builds off previous trees (e.g. Gradient Boosting, AdaBoost)
The random forest is one of the bagging ensemble methods. It consists of multiple “decision trees” with bootstrapping on both sample space and feature space, which, in ensemble terms, corresponds to our weak learner, and would classify the data point according to the majority vote of the sub-trees.
We also applied cross-validation to tune the optimal parameters used for our model, using randomized search cross-validation to speed up the parameter tuning process given the complexity of fitting a random forest model. The table below shows the optimal set of parameters we obtained through cross validation by maximizing F-1 scores.
6.6.2 Advantages and Disadvantages
We consider Random Forest to be an effective model for the goal of our project for the following reasons: it could reduce variance compared to fit only on decision trees. Random forest is unlikely to overfit the data set because of the bootstrap aggregation process.
Random Forest is an effective model for our classification task for the following reasons:
• It is capable of handling large data sets with high dimensionality, enabling the incorporation of more features in the future,
• It is robust to outliers and can make a relatively accurate classification of them
• Each decision tree has a high variance, but low bias. As we average all the trees in random forest, we are averaging the variance as well, resulting in a low bias and moderate variance model.
6.7 Boosting
6.7.1 Comparison between Bagging and Boosting
Bagging and boosting are both ensemble methods in machine learning, in that they combine a set of weak learners using base algorithms and then create a stronger learner that has a better performance than a single learner. However, bagging and boosting are different in several aspects.
First of all, in terms of sampling, both methods randomly sample from the training sample, however each sample is equally likely to be included using bagging, while boosting methods will assign different weights to different samples based on the performance in the previous training.
On the other hand, bagging and boosting are different in the way they combine those single weak learners. For bagging, each single learner works in parallel, and gets the final prediction using weighted average or takes the majority vote. For the boosting method, the training process is sequential that each single learner will learn from the previous learners as shown in the graph below.
Besides, in the boosting method, some extra conditions are included to determine if a single learner should be kept or discarded. Finally, to predict the class of a new data, we apply the learners we trained to that data. In Bagging the result is obtained by averaging the responses . However, Boosting assigns different weights to different learners and then takes a weighted average of their estimates.
There is no absolute answer whether it is better to use bagging or boosting, it all depends on the dataset we used and the question settings we have. Firstly, both methods are similar to single learners so that it will normally provide a higher stability. However, if the single learner is suffering from low accuracy and bad performance, bagging will not give a much lower bias, but boosting using sequential optimizing may help in these circumstances. In other scenarios, if we are facing the overfitting problem, then bagging would be the better choice, its randomness helps avoid overfitting.
6.7.2 AdaBoost
Ada boost is one of the most famous algorithms in the boosting family. The main logic in this algorithm is that it sequentially optimizes the tree, and it assigns a larger weight to those previously wrongly classified samples, in this way it can learn from the previous training process and construct a stronger learner. The final prediction will be based on the weighted majority vote.
For each training round, the following procedure will be processed.
• Calculate distribution p by normalizing the weight vector w (the initial weights in w for the first round are 1N, where N represents the number of labeled examples)
• Grow a weak learner (decision tree) using the distribution p; return hypothesis h with prediction values for each example
• Calculate error term of h
• Assign β with 1−
• Update the weight vector to w = w ∗ β so that predictions with poor performance will have higher a weight and predictions with better performance will have a lower weight
Ada Boost has obvious advantages compared with other boosting methods that it is easy to visualize and understand, while we think interpret-ability is an essential part in dealing with financial questions. Besides, if the dataset is with low noise, Ada boost performs well in preventing overfitting. However, as stated above, Ada boost will assign larger weights to those wrongly-classified samples, so if we have a noisy dataset and with large amount of outliers, the model may put too much effort on trying to make these data points correctly classified, while looking at the whole picture, the algorithm is slow to converge and doesn’t perform as well as facing a low-noise data set. Lastly, Ada boost algorithm is not optimized for speed so that it is relatively slow and may require more computational power.
6.7.3 XG Boost
XG Boost stands for Extreme Gradient Boosting. The single learners in XGBoost are regression trees that are similar to decision trees while they use a continuous score assigned to each leaf. The learning process aims to minimize the overall score which is composed of the loss function.
XG Boost extracts the advantages of random forest and AdaBoost and adds some additional features:
• It uses sequential tree optimizing, that the learners will be able to learn from the previous learners.
• It utilizes gradient descent to tune the parameter.
• It uses parallel processing that it is much faster than AdaBoost
• it leverages a regularization parameter, so it can cope with the problem of overfitting. And it includes a learning rate parameter that increases its ability to generalize even further.
However, XG Boost is hard to understand and visualize so it has lower interpretability. Besides, there are more parameters for us to tune using XGBoost.
6.8 Analysis
After the model training process in each train-test split, the model prediction given the testing dataset will be compared with the actual target signal and the performance metrics within each iteration will be computed. As 20 stratified k folds of cross validations are completed, the result from each iteration is combined and the average value of the performance metrics will be computed. In the practice of this project, the major performance metrics include accuracy, precision, recall, F-1 score, and AUC (Area Under Curve) score. Each model and each imbalanced sampling method combination is applied to the model selection framework and obtain the performance metrics and mean for each combination for analytical purposes. Due to the nature of the project, precision, F-1 score, and AUC played a deterministic role in the model selection process. Specifically speaking, the best model after using the fine-tuned parameters for the project objective should have the best average precision score while keeping a good balance to accuracy. As a result, the best model is chosen not solely by the mean precision value, but also the F-1 score and AUC as well.
7. Portfolio Construction and Hedging
Based on our model results, we generated 2 portfolios in total. We chose Interactive Broker as our paper trading platform and started with NAV of 1 million dollars. The first portfolio was created on Sept 28th based on our benchmark model results, and we rebalanced the portfolio on Nov 5th by uploading the new takeover target list generated by our best approach– the RF method with Near Miss Version 3. Both portfolios are equally weighted, and we hedged the portfolios by shorting the same value of IWV (iShare Russell 3000 ETF) in order to reduce the systematic risk and preserve alpha.
The first portfolio had a negative return on investment of -2.75%, and our rebalanced portfolio was built on the ending NAV of the first portfolio– $972,496.10. On Dec 5, 2020, the NAV of the long position in the rebalanced portfolio was $1.16 million while the short position was -$2411.92, and the total return on investment of the rebalanced portfolio is 19.61%. Among the forecasted takeover targets list, we successfully predicted the MA target Endurance International Group Holdings, Inc (EIGI), an IT firm specializing in web hosting and design, which announced on Oct 30 that it has entered into a definitive merger agreement to be acquired by affiliates of Clearlake Capital Group L.P.. Its share price jumped from $6 per share to $9.5 per share since the MA announced date. Also, another company predicted in our MA target list– the International Game Technology (IGT), a multinational gambling company, announced on Dec 8, 2020 that it has sold its major business unit to Apollo Management.
Figure 6.2 and 6.3 show more detailed analysis of the rebalanced portfolio. Interestingly, the model predicted a lot of Airlines and Hospitality Recreation companies like American Airlines, Delta, Alaska and United airlines, and Marriott and Wyndham Hotels, etc. which intuitively make sense as those companies have weaker balance sheets which make them become ideal takeover targets. Figure x shows the stocks that contributed the most to the rebalanced portfolio returns. The top 4 positions are HA (Hawaiian Holdings), AAL (American Airlines Group), IHRT (iHeartMedia) and AMC Theaters, which are some of the companies that were hit the hardest by the COVID-19 pandemic, and their stock price haven’t yet fully recovered like some Technology and Financials firms. A side note here is that EIGI, the target firm that RF method successfully predicted is not shown as one of the largest contributor in the figure because we rebalanced the portfolio after its announced date, however, our model did successfully predict its MA in 2020 Q4 even though we didn’t fully capture its premium.
8. Conclusion and Future Improvement
In this paper we explored 6 Machine Learning methods to predict the MA tar gets. After analyzing and comparing each approach, the Random Forest model with data balancing method Near Miss Version 3 stood out because of its high precision and F1 score. We then incorporated the company list generated by the Random Forest model into the paper trading account and rebalanced our portfolio and got the final ROI of 19.61%. The Random Forest model also successfully captured two MA activities in 2020 Q4.
Further research could focus on several perspectives: first of all, in terms of data collection, we could replace DJIA Index with the Russell 3000 Index which is more related to the prediction and eliminate the impact of the price movements of the companies with high stock prices. We would also consider more fundamental factors that are company specific, for instance, the ESG factors. Secondly, since the raw dataset from Bloomberg is messy and noisy, some dynamic methods can be potentially applied to detect the outliers. Thirdly, we used quarterly data for our analysis but there is scope to check how results may improve if we used weekly or daily data for our work. We could also try transforming our features to higher degrees and explore non- linear Machine Learning techniques for modeling, for example, using splines or GAM. Alternative data like news feed along with traditional data could also be a great value add as it might contribute to a marginal improvement in predictive power.
Written By: Langkun Guo, Ranjini Rangamani, Ruoyu Chen, Songzhou Jiang & Zoe Wang
Presented to the Faculty of the Graduate School of Cornell University in partial fulfillment of the requirements for the Master of Engineering Degree and the Financial Engineering Concentration
Faculty Advisor: Marcos M López de Prado
Sponsor: Timothy Tien
