S3

Amazon Simple Storage Service (Amazon S3), provides developers with secure, durable, highly-scalable object storage. Amazon S3 is easy to use, with a simple web service interface to store and retrieve any amount of data from anywhere on the web.

Gordon allow you to integrate your lambdas with S3 using their notification service. The idea is simple, every time an object get’s created/deleted, S3, will trigger a notification which you can route to three different services:

  • Lambda: Your code will be triggered each time something happens. (1 S3 event = 1 lambda executed)
  • SQS: S3 will create a message in a SQS queue every time something happens. Right now there is no any api for Lambdas to consume messages from a sqs queue, but because you need to define all bucket notification in the same place, we need to support this. In the future AWS will might support consuming SQS messages using lambda. (1 S3 event = 1 message in a queue)
  • SNS: S3 will send a message to a SNS Topic. You can subscribe as many lambdas as you want to this topic and process those messages individually (1 S3 event = N lambdas executed)

Note

As always, is not gordon’s business to create those sqs or sns resources. You should create them in advance. You can read Why in the FAQ

Limitations

This integration, has some limitations because how the AWS API is designed:
  • You must define all notifications for a bucket within the same integration.
  • Gordon will refuse to configure notifications in bucket if it already has some other notifications configured manually (this is a safe measure).

Anatomy of the integration

s3:

  { INTEGRATION_NAME }:
    bucket: { BUCKET_NAME }
    notifications:

      { NOTFICATION_ID }:
        lambda: { LAMBDA_NAME }
        queue: { QUEUE_NAME }
        topic: { TOPIC_NAME }
        events:
          - { EVENT_NAME }
        key_filters:
          prefix: { STRING }
          suffix: { STRING }

Properties

Integration Name

Name Key of the s3 map.
Required Yes
Valid types string
Max length 30
Description Name for your s3 integration. Try to keep it as short and descriptive as possible.

Bucket

Name bucket
Required Yes
Valid types string, reference
Max length 30
Description Name of the bucket source of the events

Notifications

Name notifications
Required Yes
Valid types list
Description List of notifications to configure.

Notification ID

Name id
Required Yes
Valid types string
Description Unique identifier for this notification

Notification Lambda

Name lambda
Required No
Valid types lambda-name, arn
Description Name of the lambda you want to notify

Note

Each notification can only configure one lambda, queue or topic.

You can reference lambdas by name

lambda: app.s3consumer

Or by their full arn:

lambda: arn:aws:lambda:eu-west-1:123456789:function:function-name

Notification Queue

Name queue
Required No
Valid types queue-name, map
Description Name of the queue you want to notify

Note

Each notification can only configure one lambda, queue or topic.

You can reference queues by name if they are in the same account than the bucket

queue: my-queue-name

If your queue is on a different account you can use the dictionary format:

queue:
  name: my-queue-name
  acount_id: 123456789

Notification Topic

Name topic
Required No
Valid types topic-name, map
Description Name of the topic you want to notify

You can reference topics by name if they are in the same account than the bucket

topic: my-topic-name

If your topic is on a different account you can use the dictionary format:

topic:
  name: my-topic-name
  acount_id: 123456789

Notification Events

Name events
Required Yes
Valid types list
Description List of events you want to make trigger a notification

The list of available events is the following:

  • s3:ObjectCreated:*
  • s3:ObjectCreated:Put
  • s3:ObjectCreated:Post
  • s3:ObjectCreated:Copy
  • s3:ObjectCreated:CompleteMultipartUpload
  • s3:ObjectRemoved:*
  • s3:ObjectRemoved:Delete
  • s3:ObjectRemoved:DeleteMarkerCreated
  • s3:ReducedRedundancyLostObject

Note

Remember that you can’t overlap events between notifications. So, if you for example subscribe a lambda to s3:ObjectCreated:*, you’ll not be able to subscribe any other notification to: s3:ObjectCreated:Put, s3:ObjectCreated:Post, etc…

Key Filters

Name key_filters
Required No
Valid types map
Description Map of filters you want to apply

Filters are optional to all notifications. The current AWS API only allows you to filter events by the key’s prefix and suffix. One notification can’t define more than one of each (prefix and suffix) and filters in a bucket can’t overlap one to each other.``prefix`` and suffix value and can be either a string or a references.

Full Example

s3:
  my_s3_integration:
    bucket: my_bucket_name
    notifications:

      lambda_on_create_cat:
        lambda: app.s3consumer
        events:
          - s3:ObjectCreated:*
        key_filters:
          prefix: cat_
          suffix: .png

      queue_on_remove_dog:
        queue: removed_dogs_queue
        events:
          - s3:ObjectRemoved:*
        key_filters:
          prefix: dog_

      topic_on_redundacy_lost:
        topic: redundacy_lost_topic
        events:
          - s3:ReducedRedundancyLostObject:*