Managing GPU memory constraints for gradient boosting models

Gradient boosting works by constructing hundreds of trees, find best split by eval gradient/hessian of features is the most expensive and time consuming task. It has been an active research area to do that in parallel on GPU. The follow papers provide detailed descriptions of GPU implementation and benchmarks on xgboost and lightgbm.

Mitchell R, Frank E. (2017) Accelerating the XGBoost algorithm using GPU computing. PeerJ Computer Science 3:e127 https://doi.org/10.7717/peerj-cs.127
Zhang, Si, Hsien, GPU acceleration for Large Scale Tree Boosting

With data size over 1-10 million range, I have observed GPU acceleration speeding up training time by up to 5-10x, while offering comparable accuracy, which is a big boost to data science work. However, the most significant constraint with the use of GPU for machine learning is GPU memory, which prevents modeling with larger data sets.

Know how much memory your GPU has

Whether you are on AWS, Google or Azure, chances are we have the same memory constraint based on Nvidia GPU, the most common hardware for machine learning is Tesla V100 with 16G GPU memory, which can fit the ballpark data size of around 10 million records. Actual milage would vary with data set and how you tuned it, more on this later.

It should be noted although multi-GPU option exists, it does nothing to alleviate per GPU memory constraint, since data set needs to be fit on a single GPU.

Earlier this year, Nvidia announced that all new Tesla V100 comes with 32G memory, which double the amount of memory per GPU. Effectively, it should also double the data set that can fit to around 20 million records. However, cloud vendors have not confirmed timeline. I have not seen it available yet.

Gradient Boosting parameters affecting GPU memory footprint

With implementation such as GPU version of xgboost, certain parameters affect memory allocation, therefore determines whether your data set will fit (otherwise you will get a memory error). The following parameters have an effect on whether your data set will fit on limited GPU memory, so you will need to budget available memory and allocate wisely. For example, you don't want to set parameters values unnecessarily high and causing over-allocation in certain dimensions.

Max Bin

Research in gradient boosting shows histogram method, which turns continuous feature value into discrete points for eval, can be equally effective, and it leads to more efficient implementation on GPU. Max_bin is a parameter that defines the maximum number of discrete points to evaluate the feature on.

Reducing from default value of 256 to 64, 32, or even 16 will reduce GPU memory required. You can run comparison tests on smaller data sets and compare evaluation metrics in order to determine impact on model performance.

Max depth, Boost round, Early Stopping

These parameters determines the number of trees to build and depth, which directly affects amount of memory allocated, so you want to put some thoughts into adjusting. Maybe start with an iterative approach and setting parameters how enough to fit the model first, then combine with parameter optimization, push the boundary of maximum memory utilization with fine tuning.

CPU Predictor

Even with GPU based training, you can still set CPU to be used for prediction, which reduces GPU memory required.

'predictor':'cpu_predictor',

Using smaller GPU to fit larger data set

Resolving Compiler issues with XgBoost GPU install on Amazon Linux

GPU accelerated xgboost has shown performance improvements especially on data set with large number of features, using 'gpu_hist' tree_method. More information can be found here:

https://peerj.com/articles/cs-127.pdf

http://xgboost.readthedocs.io/en/latest/gpu/

The installation on ubuntu based distribution is straight forward. Best results are obtained with latest generation of Nvidia GPU (AWS P3)
http://xgboost.readthedocs.io/en/latest/build.html#building-with-gpu-support

However, when compiling on Amazon Linux (including deep learning image), the following error is seen at "cmake" step:

cmake .. -DUSE_CUDA=ON

-- The C compiler identification is GNU 7.2.1

-- The CXX compiler identification is GNU 4.8.5

-- Check for working C compiler: /usr/bin/cc

-- Check for working C compiler: /usr/bin/cc -- works

-- Detecting C compiler ABI info

-- Detecting C compiler ABI info - done

-- Detecting C compile features

-- Detecting C compile features - done

-- Check for working CXX compiler: /usr/bin/c++

-- Check for working CXX compiler: /usr/bin/c++ -- works

-- Detecting CXX compiler ABI info

-- Detecting CXX compiler ABI info - done

-- Detecting CXX compile features

-- Detecting CXX compile features - done

-- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES) (found version "1.0")

-- Found OpenMP_CXX: -fopenmp (found version "3.1") 

...

CMake Error at /home/ec2-user/anaconda3/envs/python3/share/cmake-3.9/Modules/FindPackageHandleStandardArgs.cmake:137 (message):

  Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES) (found

  version "1.0")

This can be a frustrating error to correct, if focusing on troubleshooting OpenMP. Turns out, OpenMP version support is "embedded" in the compilers (gcc and g++). So different versions of compilers come with different versions of OpenMP implementation. In this case, the error message indicates OpenMP_C is version 1.0, while OpenMP_CXX is version 3.1. Xgboost GPU will not compile when versions mismatch.

The mismatch is a result of Amazon Linux comes with mismatched C and C++ compiler versions:

$ gcc --version

gcc (GCC) 7.2.1 20170915 (Red Hat 7.2.1-2)

$ g++ --version

g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)

To fix this problem, remove default older version of cmake, remove gcc (twice since it has two packages), reinstall gcc and gcc-c++:

sudo yum remove cmake

sudo yum remove gcc –y

sudo yum remove gcc –y

sudo yum install gcc48 gcc48-cpp –y

sudo yum install gcc-c++

Also reinstall cmake (using your preferred method)

conda install -c anaconda cmake

Now cmake finds matching versions for xgboost GPU compile to proceed:

-- Found OpenMP_C: -fopenmp (found version "3.1")

-- Found OpenMP_CXX: -fopenmp (found version "3.1")

The lessons learned here: Amazon Linux may not come perfectly set up, check compiler environment before installing new software, especially when GPU and parallel processing is involved.

AWS Kaggle Machine – turnkey data science “Lab in a Box”

The advancement in data science and machine learning has not only brought breakthroughs like AlphaGO, but also starting to have broad impact in our everyday lives (Airbnb uses data to predict traveler’s destinations). Gone are the days when data science is only accessible by those in the ivory tower with million dollar proprietary software, new trends have emerged:

• open source software and tools
• compute capacity at cloud scale, with dramatic cost reduction
• public data set, community based problem solving (Kaggle)

AWS provides both cost efficiency and scalable capabilities. It makes sense for data scientist to tap into the power of public cloud. An AWS image is developed here which:

• Automates the installation and configuration of a comprehensive set of open source data science tools
• Allows instance sizing based on needs
• Control cost (shut down or terminate when done, launch in a few minutes)

What it is

An AWS AMI which provides “data science server in a box” with current open source toolkit (RStudio, Jupyter Notebook, Anaconda, Xgboost…). Builds automatically, fully configured ready for use in less than five minutes.

How to build a Kaggle Machine

Using the community AMI named “kaggle machine”,  build your Kaggle Machine in AWS, with one of the following method. Note the AMIs are currently available in us-east-1 and us-west-2. For other regions, you can build your machine in the above two regions, and copy AMI across regions.

Build Kaggle Machine from AWS console
Launch EC2 instance, search for “Kaggle-machine” in Community AMIs, specify a key pair. After instance creation, add a Security Group which allows port 8787, 9999 and 22 for ssh.

Build Kaggle Machine using CloudFormation Stack
A CloudFormation template can be used to build Kaggle Machine and Security Group automatically. Download the template and use it to build your stack in us-east-1 or us-west-2. The template can be found at http://github.com/seanxwang/kagglemachine/.

How to use it

After instance creation, note public DNS name of the machine, from any client on the internet, access services by pointing your browser to:

Rstudio: http://:8787 (default ruser/ruser)

Jupyter: http://:9999 (default password jupyter)

Change the default password immediately. The EC2 instance runs on Ubuntu, you can ssh to it

Cost

The cost is based on AWS EC2 usage. You only pay hourly when instance running, shutdown the instance when done. When you are done with your project and no longer need data to be saved on server, terminate the instance.

The development of Kaggle Machine originates from the needs of data scientists participating in Kaggle challenges, hope it will be provide you a useful toolset as well.

Evaluating Amazon Machine Learning - in a Kaggle competition

Cloud changes IT in many ways. A new class of platform, database, messaging and app services have emerged to enable the rapid delivery of cloud native apps.  IT architecture can no longer be satisfied with delivering compute, network and storage. It must expand “up the stack”, putting more capabilities more rapidly into the hands of developers and business users.

A primary example of new IT capacities in demand is in the area of Big Data and Machine learning. With elasticity and on-demand computing, cloud has dramatically lowered the cost of entry. With emerging open source tool sets (e.g., Distributed Machine Learning Common, Jupyter, Anaconda, Python...), even individuals are now capable of performing analytics on large data sets, at a fraction of the cost of traditional methods (SAS grid).

To gain insight and bridge the gap between IT and data science community, I have experimented with Amazon Machine Learning (AML) service, comparing with custom built open source tool sets. By participating in a Kaggle competition, the results are also benchmarked in the real world.

The particular Kaggle competition I used has the goal of predicting hazard score using a dataset of property information. The hazard score to be predicted is a numeric value.

https://www.kaggle.com/c/liberty-mutual-group-property-inspection-prediction

Machine Learning “as a service” test

AWS has delivered a service that puts modeling and predictive analysis capabilities into the hands of a non-IT and non-data-scientist person. Its documentation provides sufficient information to build a model and perform analytics, and requires no prior modeling skills.

The first step is creating a data source. From input data, AWS infers attribute types and creates a schema, which can be further modified by user.

The second step is to build a ML model using the data source. Amazon supports only three Models (Binary classification, Multiclass classification, Numerical regression). Since the prediction result is a numeric value, the only Amazon model applicable is the regression model. Note that there are many more models, attribute selection techniques and sequencing variations than the three models offered by AWS, thus making data science equally an art than it is a science.

AWS’s built in regression model evaluation uses residual distribution to evaluate the model. In this particular case, the model has a tendency for negative residuals which indicates an overestimation (the actual target tends to be smaller than the predicted target). 

To further evaluate model’s performance, it is used to calculate Hazard score for the real data set in Kaggle competition. After the competition closed, the AWS ML model obtained a score of 0.343. Compared to all submission, it ranks 1830th (over a total of 2236). The winning submission scored 0.397.

Machine Learning “on a server” test

For comparison, a custom server is built on AWS infrastructure. A set of data science tools and libraries from the open source community are then deployed, with no additional cost. For the Kaggle competition, an emerging ML model called XGBoost is used (developed by Tianqi Chen, a PhD student at University of Washington). The resulting score is 0.392, which ranks 299th/2236.

For cost comparison, running evaluation on Amazon ML was quite expensive. I only ran a few times with a record size of 50000, and end up spending over $50. The cost of custom server is almost negligible, as the use of a mid-sized EC2 instance is quite adequate to run XGBoost Python code.

Amazon Machine Learning “as a service” delivers a very easy to use tool. It frees users from build, scale, and maintain machine learning infrastructure. However, in its current form, it is only suited to handle a narrow set of problems that matches the simple models provided. As large enterprises typically faces more sophisticated data analytical challenges, as those represented in Kaggle competitions, AML is of limited value to the data science community.

On the other hand, as data science is being revolutionized by open source, there seems to be huge opportunities for Amazon and AML to improve on.

I haven’t found much benchmarking work out there. Here are a couple of posts comparing AML with others including Google Prediction and Azure Machine Learning.

https://blog.onliquid.com/machine-learning-services-2/

http://www.kdnuggets.com/2015/05/machine-learning-wars-amazon-google-bigml-predicsis.html