Train Model Lambda

  1. Go to the AWS Console and under Services, select Lambda
  2. Go to the Functions Pane and select Create Function
  3. Author from scratch
  4. Or follow this link https://console.aws.amazon.com/lambda/home?region=us-east-1#/create?firstrun=true
  5. Parameters for the function:
    • Name: lambdaModelTrain
    • Runtime Python 3.6
    • Executing role: Use an existing role and select the role you created in the previous step (workshop-role) - Create function

This lambda function doesn’t need to receive any parameters, but it should return the resulting hyperparameter tunning optimization job name.

We will use that job name in the next lambda function to check status, the container it used and that the status is now “In Progress”.

Let’s copy this code into the editor.

Take special care of:

  • Line 9: role (the Role ARN of workshop-role, also in this link https://console.aws.amazon.com/iam/home?region=us-east-1#/roles/workshop-role)
  • Line 12: bucket_path (your bucket for the results)

    import json
    import boto3
    import copy
    from time import gmtime, strftime
    
    
    region = boto3.Session().region_name    
    smclient = boto3.Session().client('sagemaker')
    role = 'arn:aws:iam::1111111111:role/role/workshop-role'
    
    
    bucket_path='s3://bucket-name'
    prefix = "invoice-forecast"
    
    container = '811284229777.dkr.ecr.us-east-1.amazonaws.com/xgboost:latest'
    
    
    def lambda_handler(event, context):
            
    tuning_job_config = {
    "ParameterRanges": {
      "CategoricalParameterRanges": [],
      "ContinuousParameterRanges": [
        {
          "MaxValue": "1",
          "MinValue": "0",
          "Name": "eta"
        },
        {
          "MaxValue": "2",
          "MinValue": "0",
          "Name": "alpha"
        },
        {
          "MaxValue": "10",
          "MinValue": "1",
          "Name": "min_child_weight"
        }
      ],
      "IntegerParameterRanges": [
        {
          "MaxValue": "20",
          "MinValue": "10",
          "Name": "max_depth"
        }
      ]
    },
    "ResourceLimits": {
      "MaxNumberOfTrainingJobs": 5,
      "MaxParallelTrainingJobs": 5
    },
    "Strategy": "Bayesian",
    "HyperParameterTuningJobObjective": {
      "MetricName": "validation:mae",
      "Type": "Minimize"
    }
    }
      
    training_job_definition = \
      { 
        "AlgorithmSpecification": {
            "TrainingImage": container,
            "TrainingInputMode": "File"
        },
        "RoleArn": role,
        "OutputDataConfig": {
            "S3OutputPath": bucket_path + "/"+ prefix + "/xgboost"
        },
        "ResourceConfig": {
            "InstanceCount": 2,   
            "InstanceType": "ml.m4.xlarge",
            "VolumeSizeInGB": 5
        },
        "StaticHyperParameters": {
            "objective": "reg:linear",
            "num_round": "100",
            "subsample":"0.7",
            "eval_metric":"mae"
        },
        "StoppingCondition": {
            "MaxRuntimeInSeconds": 86400
        },
        "InputDataConfig": [
            {
                "ChannelName": "train",
                "DataSource": {
                    "S3DataSource": {
                        "S3DataType": "S3Prefix",
                        "S3Uri": bucket_path + '/train/',
                        "S3DataDistributionType": "FullyReplicated" 
                    }
                },
                "ContentType": "text/csv",
                "CompressionType": "None"
            },
            {
                "ChannelName": "validation",
                "DataSource": {
                    "S3DataSource": {
                        "S3DataType": "S3Prefix",
                        "S3Uri": bucket_path + '/validation/',
                        "S3DataDistributionType": "FullyReplicated"
                    }
                },
                "ContentType": "text/csv",
                "CompressionType": "None"
            }
            ]
        }
        
    tuning_job_name = prefix + strftime("%Y%m%d%H%M%S", gmtime())
        
    event["container"] = container
    event["stage"] = "Training"
    event["status"] = "InProgress"
    event['name'] = tuning_job_name
        
    smclient.create_hyper_parameter_tuning_job(HyperParameterTuningJobName = tuning_job_name,
                                           HyperParameterTuningJobConfig = tuning_job_config,
                                           TrainingJobDefinition = training_job_definition)
    
    # output
    return event
    
  • Hit Save

Test the lambda function

Create a test event with no parameters save and test. In your AWS console under SageMaker>Hyperparamter tunning jobs you should see the HPO runnning.

https://us-east-1.console.aws.amazon.com/sagemaker/home?region=us-east-1#/hyper-tuning-jobs