How to upload files to S3 using a Presigned Upload URL

Build a API using AWS API Gateway and you'll be disappointed to find out that it's not possible to send requests containing large bodies because the endpoint will timeout before the request completes.

Files such as images, large documents, and videos become impossible to upload simply by POSTing directly to your back-end API.

Instead, you have to upload them from the client, bypassing API Gateway. But you can't simply allow clients to upload files whenever they want because then, malicious users will upload bogus files that will cause your AWS charges to accumulate.

In this post, I'll show you how to solve this problem by using a feature called Presigned POST.

Uploading Directly

Users can select files if you use the HTML input element with the type attribute set to file.

<form onSubmit={handleSubmit}>
  <input type="file" name="profilePicture" />
  <button type="submit">Upload</button>
</form>

In your handleSubmit function, you'll serialize the FormData object and POST it somewhere.

async function handleSubmit(e) {
  const formData = new FormData();
  formData.append("profilePicture", e.target.profilePicture.files[0]);
  await fetch("https://yourapi.io", {
    method: "POST"
  });
}

Presigned Post Endpoint

First, you'll create an endpoint that accepts a GET request.

If you're using AWS Lambda, your handler will look something like:

import boto3
import json

def lambda_handler(event, context):
    file_key = event["queryStringParameters"]['file-key']
    client = boto3.client('s3')
    presigned_post = s3.generate_presigned_post(Bucket='my-bucket', Key=file_key, ExpiresIn=900)

    return {
        "statusCode": 200,
        "body": json.dumps(presigned_post),
    }

On line 4, I'm getting the file key, which is typically the file's name and prefix that you want to store on S3, from the query string.

Of course, in real world usage, you'll also perform validations on the file key to ensure that users aren't able to mess your bucket up.

Note: To get query strings, you'll need to enable Lambda Proxy integration for your resource.

Then, I'm generating the presigned post using this file key, with an expiration date that's 900 seconds in the future.

The result of the generate_presigned_post will be a dictionary in the form {url: string, fields: dict}

You're simply going to send this information to the client.

Using The Presigned Post

Before uploading the file, you'll issue a GET to the previous endpoint to retrieve the presigned post information.

async function getPresignedPost(fileKey) {
  const response = await fetch(`https://your-presigned-post-endoint.yourdomain.dev?file-key=${fileKey}`);
  const presignedPost = await response.json();
  return presignedPost;
}

async function handleSubmit(e) {
  const theFile = e.target.profilePicture.files[0];
  const fileKey = `user-uploads/${theFile.name}`

  const {url: uploadUrl, fields: presignedFields} = await getPresignedPost(fileKey);
  const formData = new FormData();
  Object.entries(presignedFields).forEach(([k, v]) => {
    formData.append(k, v);
  });
  formData.append('file', theFile);
  await fetch(uploadUrl, {
    method: "POST",
    body: formData
  });
}

The fields object contains some metadata that you'll need to submit along with the file you're uploading itself. So, you simply append them to the FormData payload.

After that, you append the actual file to the payload as well.

Once you POST this information to the URL that AWS provided you with, your file should appear in the target S3 bucket under the file key that you chose.

Summary

As you saw, there are only 3 steps to take in order to securely upload files directly yo S3 from your front-end client:

1. Create an endpoint to generate the presigned post data.
2. Retrieve the presigned post data from the server before uploading a file.
3. POST your file along with all the metadata from AWS to the URL that AWS provided you with.

Cheers! Subscribe to my free newsletter if you haven't already so that you can be notified of new posts like this one.