Invoking AWS Batch Service via API Gateway

Usually batch services are scheduled to execute periodically to complete high-volume, compute intensive, repetitive data jobs with minimal human interaction during the off peak time when computing resources are available. Batch service execution may fail due to some reason and Ops team may have to run run the service manually using API.

AWS Batch, a fully managed service, allows to run containerised batch services on AWS compute offerings, such as Amazon ECS, Amazon EKS, AWS Fargate, and Spot or On-Demand Instances at any scale. AWS Batch automatically provisions compute resources and optimises the workload distribution based on the quantity and scale of the workloads. You don’t need to install or manage batch computing software.

AWS Batch job can be scheduled to run using AWS Amazon EventBridge with the desired cron expression or executed based on events on AWS Amazon EventBridge. You have to set the AWS Batch as target for EventBridge rule. In case, you want to execute the batch service manually, there is no public API readily available. In this post, let’s see how to create a generic rest API on AWS API Gateway to execute a AWS Batch job on demand.

With AWS Batch, first you need to create compute environments, which consists of managed or unmanaged compute resources, within a new or existing VPC and associate it with a job queue. AWS Batch job will be submitted to a particular job queue. A job queue can be associated with one or more compute environments. Next, you need to define the job definitions that specify which Docker container images to run your jobs. Container images are stored in and pulled from container registries, which may exist within or outside of your AWS infrastructure.

To execute AWS Batch service, you need to submit a new AWS Batch job from a job definition to job queue. Parameters defined in SubmitJob request will override parameters defined in the job definition. For SubmitJob request complete schema, refer API. Best practice is not to override job definition parameter and just pass only the following parameters required to invoke the service.

{
  "jobName": "[name of the job]",
  "jobQueue": "[Name or ARN of the job queue where the job is submitted.]",
  "jobDefinition": "[Name or ARN of the job definition with or without the revision]"
}

Now, let’s see how to expose internal SubmitJob API on API Gateway. This is can be automated using the following CloudFormation template.

AWSTemplateFormatVersion: '2010-09-09'
Description: API Gateway to invoke AWS Batch Service

Parameters:
  ApiGatewayName:
    Type: String
    Default: submit-job-api
  ApiGatewayStageName:
    Type: String
    AllowedPattern: '[a-z0-9]+'
    Default: dev

Resources:
  ApiGatewayBatchRole:
    Type: AWS::IAM::Role
    Properties:
      Path: !Join [ "", [ "/", !Ref "AWS::StackName", "/" ] ]
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: AllowApiGatewayServiceToAssumeRole
            Effect: Allow
            Action:
              - 'sts:AssumeRole'
            Principal:
              Service:
                - apigateway.amazonaws.com
      Policies:
        - PolicyName: AWSBatchSubmitJob
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'batch:SubmitJob'
                Resource:
                  - !Join
                    - ''
                    - - 'arn:'
                      - !Ref AWS::Partition
                      - ':batch:'
                      - !Ref AWS::Region
                      - ':'
                      - !Ref AWS::AccountId
                      - ':job-definition/*'
                  - !Join
                    - ''
                    - - 'arn:'
                      - !Ref AWS::Partition
                      - ':batch:'
                      - !Ref AWS::Region
                      - ':'
                      - !Ref AWS::AccountId
                      - ':job-queue/*'

  RestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: !Ref ApiGatewayName
      Description: 'Batch Service API'
      DisableExecuteApiEndpoint: 'true'
      EndpointConfiguration:
        Types:
          - REGIONAL

  ApiGatewayResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId: !Ref RestApi
      ParentId: !GetAtt RestApi.RootResourceId
      PathPart: api

  VersionGatewayResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId: !Ref RestApi
      ParentId: !Ref ApiGatewayResource
      PathPart: v1

  SubmitJobGatewayResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId: !Ref RestApi
      ParentId: !Ref VersionGatewayResource
      PathPart: submitJob

  ApiGatewayCloudWatchLogsRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: api-gateway-cw-logs-role
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - apigateway.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: api-gateway-cw-logs-policy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:DescribeLogGroups'
                  - 'logs:DescribeLogStreams'
                  - 'logs:PutLogEvents'
                  - 'logs:GetLogEvents'
                  - 'logs:FilterLogEvents'
                Resource: '*'

  ApiGatewaySubmitMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      AuthorizationType: NONE
      HttpMethod: POST
      MethodResponses:
        - StatusCode: 200
          ResponseModels:
            application/json: Empty
        - StatusCode: 403
          ResponseModels:
            application/json: Error
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS
        Credentials: !GetAtt ApiGatewayBatchRole.Arn
        Uri: !Sub arn:aws:apigateway:${AWS::Region}:batch:path//v1/submitjob
        PassthroughBehavior: WHEN_NO_TEMPLATES
        RequestTemplates:
          application/json: !Sub |
            #set($inputRoot = $input.path('$')) 
              {
                "jobName": "$inputRoot.jobName",
                "jobQueue": "$inputRoot.jobQueue",
                "jobDefinition": "$inputRoot.jobDefinition"
              }
        IntegrationResponses:
          - StatusCode: 200
            ResponseTemplates:
              application/json: !Sub
                - |-
                  {
                  "message": "Successfully submitted the batch job"
                  }
                - { }
          - SelectionPattern: 403
            StatusCode: 403
            ResponseTemplates:
              application/json: !Sub
                - |-
                  #set($errorRoot = $input.path('$'))
                  {
                    "message": "$errorRoot"
                  }
                - { }
      ResourceId: !Ref SubmitJobGatewayResource
      RestApiId: !Ref RestApi

  ApiGatewayAccount:
    Type: 'AWS::ApiGateway::Account'
    Properties:
      CloudWatchRoleArn: !GetAtt ApiGatewayCloudWatchLogsRole.Arn

  ApiDeployment:
    DependsOn:
      - ApiGatewaySubmitMethod
    Type: 'AWS::ApiGateway::Deployment'
    Properties:
      RestApiId: !Ref RestApi
      StageName: !Ref ApiGatewayStageName

Outputs:
  ApiEndpoint:
    Description: "API Gateway endpoint URL in invoke AWA Batch SubmitJob API."
    Value: !Sub https://${RestApi}.execute-api.${AWS::Region}.amazonaws.com/${ApiGatewayStageName}

Once you created the stack using above template, you can access submit job API from API Gateway as below.

This post explained how to expose AWS Batch SubmitJob API on API Gateway with minimum effort. You can integrate this API with ITSM systems such as ServiceNow to make operational team life easy in case scheduled AWS batch job failed to execute due to unhandled exceptions.