Machine Learning in Compiler Optimisation

1. Feature Engineering
a. Feature representation
- Static code: number and type of instruction (Arithmetic instructions, Memory
operations, Branch instructions, loop information, parallel information)
- Tree based and graph based approach: Tree built using features helped to
contrast cost function using neural network, graph based approached was used
in SVM where kernel is based on graph similarity metric.
- Dynamic features: Drawback of static code is it may contain information of
code segments that are rarely executed, this confused machine learning
models. loop iteration counts the cannot be decided at compile time, dynamic
memory and I/O behavior of the application as well as CPU load and thread
contention, control flows, frequently executed code regions, how many
instructions have been executed and of what types, and the number of cache
loads/stores as well as branch misses.
b. Reaction based features
- They then use the program “reactions” to predict the best available application
speedup.
c. Automatic feature generation
- the predictive model takes in a set of human-crafted features, program code is
used directly in the training data. Programs are fed through a series of neural
network based language models which learn how code correlates with the
desired optimization options.
d. Feature selection and dimensionality reduction
- Feature selection: Feature selection requires understanding how does a
particular feature affect the prediction accuracy. The methods that can be used
are Pearson correlation coefficient, mutual information, LASSO.
- Feature dimensionality reduction: It is important for efficiency of model and
curse of dimensionality problem (KNN). PCA linearly combines the original
features to construct new features. LDA is also a similar technique. The
alternative way is using autoencoders. The encoder tries to compress the
original input into a low-dimensional representation, while the decoder tries to
reconstruct the original input based on the low-dimension representations
generated by the encoder.

2. Machine Learning Models

a. Supervised Learning
- Regression: This technique has been used in various tasks, such as predicting
the program execution time input or speedup for a given input, or estimating
 power consumption and latency. Linear regression, SVM, ANN can be used
here. The later 2 requires more training data.
- Classification: Useful for predicting discrete values, such as choosing
compiler flags, #threads, loop unroll factors (how many times loop has to be
unrolled), algorithmic implementations etc. KNN, Decision tree are the
algorithms that can be used. KNN is not robust and relatively slow. Decision
tree can be easily visualized. Random forest can be used to avoid over-fitting.
SVM with rbf kernel can also be used.
- Deep Neural Network: The advantage of DNNs is that it can compactly
represent a significantly larger set of functions than a shallow network, where
each function is specialised at processing part of the input. This helps to
identify complex relationships between features.
b. Unsupervised learning
- Clustering: Data analysis, such as grouping profiling traces into clusters of
similar behavior. K-Means is one of the simplest and popular algorithms used.
- Principal component analysis (PCA) is a statistical method for unsupervised
learning. This method has been heavily used in prior work to reduce the
feature dimension.
- Autoencoders is used for discovering the efficient codings of input data in an
unsupervised fashion.
c. Online learning
- Evolutionary search: Evolutionary algorithms (EAs) like genetic algorithms,
genetic programming and stochastic based search have been employed. Three
are three key operations in an EA algorithm: selection, cross-over and
mutation. Selection is based on fitness. In cross over the offspring with good
fitness values are only created. Mutation reduces the problem of algorithm
getting stuck in local optimal.
- Reinforcement learning: Useful for exploring a large optimisation space,
runtime adaption, dynamic task scheduling where the optimal outcome is
achieved through a series of actions. RL has been used in prior research to
schedule RAM memory traffics, selecting software component configurations
at runtime , and configure virtual machines.
3. Deployment
- In the final step, the learned model is inserted into the compiler to predict the
best optimisation decisions for new programs.
4. Scope
a. Optimise sequential programs
- Early works for machine learning in compilers look at how, or if, a compiler
optimisation should be applied to a sequential program. Some of the previous
 studies build supervised classifiers to predict the optimal loop unroll factor or
to determine whether a function should be inlined.
- Evolutionary algorithms like generic search are often used to explore a large
design space, to solve the phase ordering problem (i.e. at which order a set of
compiler transformations should be applied) , determining the compiler flags
during iterative compilation , selecting loop transformations, and tuning
algorithmic choices, etc.
b. Optimise parallel programs
- The aim is given an already parallelized program, how to map the application
parallelism to match the underlying hardware to make the program runs as fast
as possible or be as energy-efficient as possible. ANN helps in searching
optimal number of threads and SVM helps in predict scheduling policy.
- Researchers have used machine learning to build classifiers to determine
which processor to use and at which clock frequency the processor should
operate. Others used regression techniques to build curve fitting models to
search for the sweat spot for work partitioning among processors or a trade-off
of energy and performance.
c. Other research problems
- To build the best model we need huge amount of data, the rich information in
large open source code bases could provide a powerful knowledge base for
training machine learning models to solve compiler optimisation problems,
and deep learning could be used as an effective tool to extract such knowledge
from massive program source code