S3¶
Amazon Simple Storage Service (Amazon S3), provides developers with secure, durable, highly-scalable object storage. Amazon S3 is easy to use, with a simple web service interface to store and retrieve any amount of data from anywhere on the web.
Gordon allow you to integrate your lambdas with S3 using their notification service. The idea is simple, every time an object get’s created/deleted, S3, will trigger a notification which you can route to three different services:
- Lambda: Your code will be triggered each time something happens. (1 S3 event = 1 lambda executed)
- SQS: S3 will create a message in a SQS queue every time something happens. Right now there is no any api for Lambdas to consume messages from a sqs queue, but because you need to define all bucket notification in the same place, we need to support this. In the future AWS will might support consuming SQS messages using lambda. (1 S3 event = 1 message in a queue)
- SNS: S3 will send a message to a SNS Topic. You can subscribe as many lambdas as you want to this topic and process those messages individually (1 S3 event = N lambdas executed)
Note
As always, is not gordon’s business to create those
sqs
orsns
resources. You should create them in advance. You can read Why in the FAQ
Limitations¶
- This integration, has some limitations because how the AWS API is designed:
- You must define all notifications for a bucket within the same integration.
- Gordon will refuse to configure notifications in bucket if it already has some other notifications configured manually (this is a safe measure).
Anatomy of the integration¶
s3: { INTEGRATION_NAME }: bucket: { BUCKET_NAME } notifications: { NOTFICATION_ID }: lambda: { LAMBDA_NAME } queue: { QUEUE_NAME } topic: { TOPIC_NAME } events: - { EVENT_NAME } key_filters: prefix: { STRING } suffix: { STRING }
Properties¶
Integration Name¶
Name | Key of the s3 map. |
Required | Yes |
Valid types | string |
Max length | 30 |
Description | Name for your s3 integration. Try to keep it as short and descriptive as possible. |
Bucket¶
Name | bucket |
Required | Yes |
Valid types | string , reference |
Max length | 30 |
Description | Name of the bucket source of the events |
Notifications¶
Name | notifications |
Required | Yes |
Valid types | list |
Description | List of notifications to configure. |
Notification ID¶
Name | id |
Required | Yes |
Valid types | string |
Description | Unique identifier for this notification |
Notification Lambda¶
Name | lambda |
Required | No |
Valid types | lambda-name , arn |
Description | Name of the lambda you want to notify |
Note
Each notification can only configure one lambda
, queue
or topic
.
You can reference lambdas by name
lambda: app.s3consumer
Or by their full arn
:
lambda: arn:aws:lambda:eu-west-1:123456789:function:function-name
Notification Queue¶
Name | queue |
Required | No |
Valid types | queue-name , map |
Description | Name of the queue you want to notify |
Note
Each notification can only configure one lambda
, queue
or topic
.
You can reference queues by name if they are in the same account than the bucket
queue: my-queue-name
If your queue is on a different account you can use the dictionary format:
queue:
name: my-queue-name
acount_id: 123456789
Notification Topic¶
Name | topic |
Required | No |
Valid types | topic-name , map |
Description | Name of the topic you want to notify |
You can reference topics by name if they are in the same account than the bucket
topic: my-topic-name
If your topic is on a different account you can use the dictionary format:
topic:
name: my-topic-name
acount_id: 123456789
Notification Events¶
Name | events |
Required | Yes |
Valid types | list |
Description | List of events you want to make trigger a notification |
The list of available events is the following:
s3:ObjectCreated:*
s3:ObjectCreated:Put
s3:ObjectCreated:Post
s3:ObjectCreated:Copy
s3:ObjectCreated:CompleteMultipartUpload
s3:ObjectRemoved:*
s3:ObjectRemoved:Delete
s3:ObjectRemoved:DeleteMarkerCreated
s3:ReducedRedundancyLostObject
Note
Remember that you can’t overlap events between notifications. So, if you for example subscribe a lambda to s3:ObjectCreated:*
, you’ll not
be able to subscribe any other notification to: s3:ObjectCreated:Put
, s3:ObjectCreated:Post
, etc…
Key Filters¶
Name | key_filters |
Required | No |
Valid types | map |
Description | Map of filters you want to apply |
Filters are optional to all notifications. The current AWS API only allows you to filter events by the key’s prefix
and suffix
. One notification can’t define
more than one of each (prefix
and suffix
) and filters in a bucket can’t overlap one to each other.``prefix`` and suffix
value and can be either a string
or a references
.
Full Example¶
s3:
my_s3_integration:
bucket: my_bucket_name
notifications:
lambda_on_create_cat:
lambda: app.s3consumer
events:
- s3:ObjectCreated:*
key_filters:
prefix: cat_
suffix: .png
queue_on_remove_dog:
queue: removed_dogs_queue
events:
- s3:ObjectRemoved:*
key_filters:
prefix: dog_
topic_on_redundacy_lost:
topic: redundacy_lost_topic
events:
- s3:ReducedRedundancyLostObject:*