Auto starting R Studio on AWS Deep Learning server

As an enhancement to machine learning servers built on AWS or Azure, it is often necessary to set up R development environment to meet the needs of data science community.

Adapt for your specific environment. Here we assume we to use AWS deep learning conda image (ubuntu). Specially we use "python3" virtual environment (source activate python3). One of the reasons to use this environment is it is already set up to run Jupyter Notebook (see auto start jupyter), we can therefore add an additional R kernel to it.  Then we have a consolidated image that can be offered to both Python and R users.

The easiest method to install R is using conda:
conda install r r-essentials

RStudio is a popular development environment. Follow instructions to install RStudio, for example:
sudo apt-get install gdebi-core
wget https://download2.rstudio.org/rstudio-server-1.1.419-i386.deb
sudo gdebi rstudio-server-1.1.419-i386.deb

The above procedure also sets up auto start of R studio server by adding /etc/systemd/system/rstudio-server.service. However, because the only available procedure installs RStudio with "sudo" into the default system environment, it cannot find R which has been installed into a different environment. As a result, RStudio fails to start with error indicating RStudio cannot find R.
rstudio-server verify-installation
Unable to find an installation of R on the system (which R didn't return valid output); Unable to locate R binary by scanning standard locations

This can be easily fixed by specifying the exact path to R for RStudio, replace path with your installation of R:
sudo sh -c 'echo "rsession-which-r=/home/ubuntu/anaconda3/envs/python3/bin/R" >> /etc/rstudio/rserver.conf'

Restart instance, now RStudio Server starts successfully. Login with Linux credential at:
"http:(server IP):8787"

AWS Kaggle Machine – turnkey data science “Lab in a Box”

The advancement in data science and machine learning has not only brought breakthroughs like AlphaGO, but also starting to have broad impact in our everyday lives (Airbnb uses data to predict traveler’s destinations). Gone are the days when data science is only accessible by those in the ivory tower with million dollar proprietary software, new trends have emerged:

• open source software and tools
• compute capacity at cloud scale, with dramatic cost reduction
• public data set, community based problem solving (Kaggle)

AWS provides both cost efficiency and scalable capabilities. It makes sense for data scientist to tap into the power of public cloud. An AWS image is developed here which:

• Automates the installation and configuration of a comprehensive set of open source data science tools
• Allows instance sizing based on needs
• Control cost (shut down or terminate when done, launch in a few minutes)

What it is

An AWS AMI which provides “data science server in a box” with current open source toolkit (RStudio, Jupyter Notebook, Anaconda, Xgboost…). Builds automatically, fully configured ready for use in less than five minutes.

How to build a Kaggle Machine

Using the community AMI named “kaggle machine”,  build your Kaggle Machine in AWS, with one of the following method. Note the AMIs are currently available in us-east-1 and us-west-2. For other regions, you can build your machine in the above two regions, and copy AMI across regions.

Build Kaggle Machine from AWS console
Launch EC2 instance, search for “Kaggle-machine” in Community AMIs, specify a key pair. After instance creation, add a Security Group which allows port 8787, 9999 and 22 for ssh.

Build Kaggle Machine using CloudFormation Stack
A CloudFormation template can be used to build Kaggle Machine and Security Group automatically. Download the template and use it to build your stack in us-east-1 or us-west-2. The template can be found at http://github.com/seanxwang/kagglemachine/.

How to use it

After instance creation, note public DNS name of the machine, from any client on the internet, access services by pointing your browser to:

Rstudio: http://:8787 (default ruser/ruser)

Jupyter: http://:9999 (default password jupyter)

Change the default password immediately. The EC2 instance runs on Ubuntu, you can ssh to it

Cost

The cost is based on AWS EC2 usage. You only pay hourly when instance running, shutdown the instance when done. When you are done with your project and no longer need data to be saved on server, terminate the instance.

The development of Kaggle Machine originates from the needs of data scientists participating in Kaggle challenges, hope it will be provide you a useful toolset as well.