Showing posts with label automation. Show all posts
Showing posts with label automation. Show all posts

Wednesday, January 31, 2018

Auto starting Jupyter Notebook on AWS Deep Learning server

Cloud and computing on demand is an increasingly powerful and cost effective combination of enabling technologies for data scientists. Further, utilizing machine learning servers such as those based on AWS deep learning AMIs can make a full suite of machine learning tools available in a matter of minutes.

Jupyter Notebook is a popular development interface for data analysis and model training. Currently, AWS has a published procedure for configuring, starting, and connecting to notebook server.
https://docs.aws.amazon.com/dlami/latest/devguide/setup-jupyter.html

However, setting up can be challenging, and repeating the above step each time an instance restarts is not ideal, especially when server is offered to the broad data science community.

Here is an alternative and enhancement to auto start notebook server.

Adapt for your specific environment. Here we assume we to use AWS deep learning conda image (ubuntu). Specially we install into "python3" environment (source activate python3).

Configure Jupyter Notebook

Similar to steps outlined here, configure Jupyter Notebook, which consists of:

Create key and cert. For example, in ~/.jupyter/ directory:
openssl req -x509 -nodes -days 11499 -newkey rsa:1024 -keyout "jupytercert.key" -out "jupytercert.pem" -batch

Create notebook password, copy generated string in .json file
jupyter notebook password

update ~/.jupyter/jupyter_notebook_config.py
c.NotebookApp.open_browser = False
c.NotebookApp.ip = '*'
c.NotebookApp.port = 8888
c.NotebookApp.password = sha1:xxx
c.NotebookApp.certfile = '/home/ubuntu/.jupyter/jupytercert.pem'
c.NotebookApp.keyfile = '/home/ubuntu/.jupyter/jupytercert.key'

Set up Auto Start Jupyter Notebook (virtualenv)

Setting up auto start is usually straightforward (for example, use /etc/rc.local). In this case, because the target environment is virtualenv. We don't want to auto start in the default python environement, or as root user. But we still want to use rc.local. Use the following 2 step process.

create a script /home/ubuntu/.jupyter/start_notebook.sh (note use of absolute path to invoke the executable) 
#!/bin/bash
source /home/ubuntu/anaconda3/bin/activate python3
/home/ubuntu/anaconda3/envs/python3/bin/jupyter notebook &


Edit /etc/rc.local and add the following, note we switch to ubuntu user, and invoke the startup script:
cd /home/ubuntu
su ubuntu -c "nohup /home/ubuntu/.jupyter/start_notebook.sh >/dev/null 2>&1 &"

The reason for this two step process is to be able to execute multiple commands (I didn't find effective ways to do that easily in rc.local)

User Access to Jupyter Notebook

Jupyter Notebook will always start automatically with instance. Without any additional set up, user can conveniently access Jupyter server at
"https:(server IP):8888"

Sunday, January 14, 2018

Azure automation with Logic App - passing variable in workflow

Similar to AWS Lambda, Azure Logic App can be used for automated workflow. However, clear documentation is harder to come by, with fewer working examples, and often lack of effective technical support.

In a workflow, it should be a common requirement to pass the output of one step to another step. The motivation to post this working solution, is there is no clear example that illustrates how exactly that is done. It should be learned in a few minutes, rather than hours of trial and error.

output from step 1

Using a simple two step workflow to illustrate, in step 1, we use an Azure Function App with a powershell script.  We can obtain a user email dynamically from Azure VM's user defined tag field.
$user_email = (Get-AzureRmVM -ResourceGroupName $resourceGroupName -Name $resourceName -ErrorAction $ErrorActionPreference -WarningAction $WarningPreference).Tags["user_email"]

More importantly, the obtained result needs to be sent to this rather odd "Out-File" structure. This is how variable can be passed in the workflow:
$result = $user_email | ConvertTo-Json
Out-File -Encoding Ascii -FilePath $res -inputObject $result


input to step 2

In a subsequent step, we can use the output of previous step, in this case, sending an email to VM's user per tag. This is best illustrated using the graphical interface of Logic App Designer:

Azure recognized a step generates an output, and make it available to be used for subsequent steps. The particular handle is shown as "Body" of Step 1 Function App, again, rather odd representation.

But it does work. And this simple mechanism is a much needed building block to construct complex features in a workflow.

Friday, May 6, 2016

AWS High Availability Gateway – Part 1 – Basic HA Model



Why this HA model
Gateway is used in AWS VPC to control egress traffic. In addition to NAT gateway, more feature rich gateway such as Gateway Transparent mode provides more sophisticated controls.

What is an optimal HA model for critical infrastructure components such as gateway? The AWS reference HA model for NAT and Gateway is rather dated. It uses script running on instances to ping each other for health, it has these potential shortcomings:
  • Depends on a continuous running shell script to monitor availability and perform failover.  If the process were to be terminated then no failover would occur.
  • Ping only provides limited indication of health
  • "split brain" scenario: when connectivity between the NAT instances fails (possibly due to Security Group) but each of them are still capable of connecting to the Internet, it is possible that each NAT instance will shut the other one down
  • Does not account for scenarios when an instance is terminated, the instance will not be recreated
Since ELB is not currently supported as a target for route tables, gateway instance must be defined as route target. When a gateway fails, the route target becomes a black hole. The first step towards HA is running gateway per AZ. The second step is detection of gateway failure in AZ and recover from it. The third step is provide dynamic failover during gateway recovery to minimize downtime. The HA model proposed has two parts:
1.       Basic HA model, with auto recovery of gateway
2.       Enhanced HA model, with dynamic route table failover during gateway recovery

The design and implementation of basic HA model is covered here. See part 2 for enhanced HA.

Design Overview

Health Monitoring and Auto Scaling
In cloud architecture, all instances should be behind an auto scaling group for resiliency. Here we leverage ASG to monitor gateway instance health. Auto Scaling health checks use the results of the EC2 status checks to determine the health status of an instance. Auto Scaling marks an instance as unhealthy if its instance status is any value other than running or its system status is impaired.
Therefore gateway is monitored based on AWS health monitoring for auto scaling instances. Customization is also supported.

Route Target and ENI
In this non-proxy design, internet access via default route, which is defined in a private route table per AZ. In a HA scenario, instances may get replaced, so the routing table entry will either 1) remain "persistent" outside the instance, or 2) updated to point to the new instance.
For the first option, what could be a persistent target for default route to point to? ELB would be an option, but it is not supported as a routing target. ENI is a network interface that can be detached and attached to instances so it can serve as the persistent target. Although there are some feature limitations and workarounds required, it is still proven to work.

Instance Recovery and Bootstrapping
Another feature that comes with ASG is automated recovery of instances. However, there are some limitations to ASG, for example, it cannot set instance attributes and it cannot attach ENI. Those are implemented via instance bootstrapping.

Implementation
The diagram shows an architectural view of the new HA model. Gateway is placed in a single instance ASG, with two interfaces. An ENI is attached to gateway instance, which provides persistence in route table even when a gateway instance fails (the ENI is reattached to a recovered instance).

For sample code, please refer to github repo:
There is a limitation with this HA model, when a gateway instance fails, recovery time may take up to 10-15 minutes (to build a new gateway, install and configure the appliance). During the time gateway is being rebuilt for that zone, traffic is black holed in the route table before ENI can be attached to a new gateway instance. See part 2 for enhanced HA.

Saturday, July 25, 2015

AWS automation – lessons with Cron set up

I spent quite a bit of time on what appeared to be a rather simple problem to solve, I figured posting it may save others time and frustration. This may appear trivial to a Linux admin, but these days we have people from various backgrounds, wrestling with scripts, Python, DevOps, and infrastructure as code in the “cloud” – it’s nice to share some common lessons along the way. 

I have some “monkey” jobs, which are Python scripts that runs from servers in AWS. To perform regular monitoring, generate custom CloudWatch metric, and raise alarms via SNS, Cron seems like a natural method.

But this is where I got the unexpected glitch. Python scripts runs like a charm from the command line, but Cron does not. The crontab job runs at specified interval (as seen in /var/log/cron), but no metric was received by CloudWatch. Unfortunately, this is where my Google search mislead to wrong directions (due to various posts about crontab misbehavior).  I’ve tried a number of things with no success, including running Cron as root, even rebuilding server.

Finally, I got back on track by focusing on getting more output from the job. Use the following to direct output to log file, note “2>&1” indicates that the standard error (2>) is redirected to the same file 
Crontab –
* * * * * /home/ec2-user/monkey.sh > /home/ec2-user/cron.log 2>&1

Among the output received, the message “socket.timeout: timed out” was clear indication of some sort of network problem. At that point, it is pretty obvious that it is an internet access issue - Cron does not pick up proxy setting.

There was another twist as I tried to set proxy for crontab. I tried setting environment variables in /etc/crontab, which is not supported for proxy. I then went on a detour to set proxy inside Python, way too complicated and unnecessary. It then occurred to me, all that is needed is to set proxy as the first step in crontab job. The last piece of the puzzle is to execute everything in one line (so it is one job run sequentially), like this
Crontab –
* * * * * source /etc/profile.d/proxy.sh & /home/ec2-user/monkey.sh > /dev/null 2>&1

The above job runs every minute, it sets up proxy first, and then does some monkey business.

Sunday, May 18, 2014

AWS automation – CloudFormation bootstrapping early lessons – Part 3

The sample template illustrates a simple bootstrapping scenario; it creates a windows helper instance, which is set up to execute a PowerShell script stored in external repository (S3). You may develop a number of scripts to build, manage, and monitor the VPC and associated resources, this method can be used to deploy those capabilities systematically and automatically.

Everything is built around “WindowInstance”, which has these main components associated with bootstrapping
  •  “UserData” section defines "cloud-init" bootstrapping, which performs the execution of cfn-init
  •   “Metadata” section is defined for "cfn-init" bootstrapping. It installs PowerShell script from S3 to the local directory, and defines the command to run powershell script with
  •  cfn-signal script is used to return the status of command execution back to CloudFormation with the use of a WaitConditionHandle

Note the instance is defined with IAM Instance Profile, which provides it necessary privilege to access external data store, and perform VPC operations.

This simple method works nicely for initial deployment. How to manage ongoing changes? In this simple model, we will pick up new configuration by launching a new instance. We can put the instance behind an auto-scaling group, by terminating the existing instance, a new one will spin up automatically, triggering the execution of updated configurations.

There is an alternative method to trigger updates without launching new instances. AWS has designed cfn-hup to assist with updates by polling the CloudFormation meta-data for changes, and then executes defined actions when a change is detected. Now, instead of recreating the stack and launching a new instance, an update of CloudFormation stack will kick of the configuration change on the running instance. Please see Peter Hancock’s “Updating your AWS bootstrap” for a nice explanation of the technique. 

See sample template below:

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description" : "CF bootstrapping template sample: windows instance running a powershell script, obtained from S3, note how cfn:init defines command to use option switch to run powershell",

  "Parameters" : {
    "KeyPairName" : {
      "Description" : "Name of an existing Amazon EC2 key pair",
      "Type" : "String",
 "Default" : "xxx"
    },
"WindowInstanceSubnet" : {
      "Description" : "Subnet ID to launch instance",
      "Type" : "String",
      "Default" : "subnet-xxx"
    },

    "WindowInstanceSGs": {
    "Description": "Comma-delimited list of Security Group IDs for instance",
    "Type": "CommaDelimitedList",
      "Default": "sg-xxx, sg-xxx"
    },
    "InstanceType" : {
      "Description" : "Amazon EC2 instance type",
      "Type" : "String",
      "Default" : "t1.micro",
      "AllowedValues" : [ "t1.micro", "m1.small", "m1.medium", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.medium", "c1.xlarge"]
    }
  },

  "Mappings" : {
    "AWSInstanceType2Arch" : {
      "t1.micro"   : { "Arch" : "64" },
      "m1.small"   : { "Arch" : "64" },
      "m1.medium"  : { "Arch" : "64" },
      "m1.large"   : { "Arch" : "64" },
      "m1.xlarge"  : { "Arch" : "64" },
      "m2.xlarge"  : { "Arch" : "64" },
      "m2.2xlarge" : { "Arch" : "64" },
      "m2.4xlarge" : { "Arch" : "64" },
      "c1.medium"  : { "Arch" : "64" },
      "c1.xlarge"  : { "Arch" : "64" }
    },
    "AWSRegionArch2AMI" : {
      "us-east-1"      : {"64" : "ami-dfcdc4b6"},
      "us-west-1"      : {"64" : "ami-c2cef187"},
      "us-west-2"      : {"64" : "ami-16197726"},
      "eu-west-1"      : {"64" : "ami-fde21e8a"},
      "ap-southeast-1" : {"64" : "ami-08f5a45a"},
      "ap-southeast-2" : {"64" : "ami-7377ee49"},
      "ap-northeast-1" : {"64" : "ami-514e3e50"},
      "sa-east-1"      : {"64" : "ami-35319228"}
    }
  },

  "Resources" : {
    "InstanceRole":{
      "Type":"AWS::IAM::Role",
        "Properties" : {
          "AssumeRolePolicyDocument" : {
            "Statement": [{
              "Effect" : "Allow",
              "Principal" : {
                "Service" : [ "ec2.amazonaws.com" ]
              },
              "Action" : [ "sts:AssumeRole" ]
            }]
          },
          "Path" : "/"
        }
    },
      
    "RolePolicies" : {
      "Type" : "AWS::IAM::Policy",
      "Properties" : {
        "PolicyName" : "VPCupdate",
        "PolicyDocument" : {
          "Statement" : [
{
            "Action" : [ "ec2:*" ],
            "Effect" : "Allow",
            "Resource" : "*"
},
{
            "Action" : [ "s3:*" ],
            "Effect" : "Allow",
            "Resource" : "*"
}
 ]
        },
        "Roles" : [ { "Ref" : "InstanceRole" } ]
      }
    },
      
    "InstanceProfile" : {
      "Type":"AWS::IAM::InstanceProfile",
      "Properties" : {
        "Path" : "/",
        "Roles" : [ { "Ref":"InstanceRole" } ]
      }
    },

  "WindowInstance": {
      "Type" : "AWS::EC2::Instance",
      "Metadata" : {
        "AWS::CloudFormation::Init" : {
          "config" : {
            "files" : {
              "C:\\cfn\\yourscript.ps1" : {
                "source" : "https://s3.amazonaws.com/your-cfn-repo/yourscript.ps1"
              }
            },
            "commands" : {
     "1-update" : {
  "command" : "powershell.exe -ExecutionPolicy Bypass -NoLogo -NonInteractive -NoProfile -File C:\\cfn\\yourscript.ps1"
              }
            }
            
          }
        }
      },
      "Properties": {
        "InstanceType" : { "Ref" : "InstanceType" },
        "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" },
                      { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
         "Tags":[
            {
                  "Key":"Name",
                  "Value":"WindowInstance"
            }
        ],
"IamInstanceProfile" : { "Ref" : "InstanceProfile" },
        "SubnetId" : { "Ref" : "WindowInstanceSubnet" },
        "SecurityGroupIds" : { "Ref" : "WindowInstanceSGs" },
        "KeyName" : { "Ref" : "KeyPairName" },
        "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
                ""
          ]]}}
        }
    },

    "WindowInstanceWaitHandle" : {
      "Type" : "AWS::CloudFormation::WaitConditionHandle"
    },

    "WindowInstanceWaitCondition" : {
      "Type" : "AWS::CloudFormation::WaitCondition",
      "DependsOn" : "WindowInstance",
      "Properties" : {
        "Handle" : {"Ref" : "WindowInstanceWaitHandle"},
        "Timeout" : "500"
      }
    }
  },

  "Outputs" : {
...
  }
}