Presigned URLs and multipart uploads
Letting a client send and fetch large files directly, without opening the bucket and without passing every byte through your servers.
5 minute lesson
Keep the bytes off your servers
A user uploads a 2 GB video. If it goes through your application, a server is busy for minutes passing bytes along, and every byte travels twice. The client should send it straight to object storage. But the bucket is private, and the client must not get your credentials.
- Through the appClient to app to storagea server is tied up for the whole upload
- DirectClient to storagethe app only gives permission
Permission in a link
- Ask
The client tells your application that it wants to upload a video. The application checks that this user may do so.
- Clientmay I upload?
- Applicationchecks the user
- Sign
The application creates a presigned URL: a link to one object, for one operation, valid for ten minutes, signed with the application's credentials. Creating it needs no call to the storage service.
- Applicationsigns a link
- Clientgets the link
- Upload
The client sends the bytes to that link. The storage service checks the signature and the expiry and accepts the file. Your servers never see the video.
- Clientsends 2 GB
- Object storageaccepts it
An application gives a browser a presigned URL to upload the object videos/881.mp4, valid for 10 minutes.
What can the browser do with it?
- Upload that one object within the 10 minutes
- Upload any object to the bucket for 10 minutes
- Read and delete the other objects of the bucket
- Upload that one object at any time from now on
Show the answer
- Upload that one object within the 10 minutes
The signature covers the object's key, the operation and the expiry. Change any of them and the signature no longer fits. So the link is good for exactly this upload, and only for ten minutes. Everything else in the bucket stays as private as before.
A presigned URL allows one operation on one object until it expires. Nothing more.
A link is a key
Whoever holds a presigned URL can use it. The storage service does not ask who is sending. That makes it safe to hand to a browser and unsafe to hand around.
- Make it short-lived: minutes for an upload, not days.
- Sign one object and one operation, never a whole bucket.
- Keep it out of logs, and out of pages that others can see.
Large files go in parts
A single request of 2 GB that fails at 95% starts again from zero. A multipart upload cuts the file into numbered parts. The parts are sent separately, several at a time, and the storage service joins them at the end.
- Startget an upload id
- Partssend in parallelretry only what failed
- Completejoin the parts
A 5 GB file is sent as a multipart upload of 100 parts. Part 63 fails because the connection drops for a moment.
What has to be sent again?
- Part 63 and nothing else
- Parts 63 to 100, the rest
- All 100 parts, from the start
- Nothing, as 99 parts suffice
Show the answer
- Part 63 and nothing else
Each part is an upload of its own, with its own number. The 99 that arrived are stored and wait. Only part 63 is repeated, and the upload is completed once all parts are there. With a single request of 5 GB, the same hiccup would have thrown away everything that was already sent.
In a multipart upload a failure costs one part, not the whole file.
How can a browser upload a file straight into a private bucket, and what makes that safe?
Show the answer
The application checks the user and then signs a presigned URL: a link that allows one operation on one object for a short time. The browser sends the file to that link, and the storage service verifies the signature and the expiry. The application's credentials never leave the server. It is safe as long as the link is short-lived, narrow and kept private, because anyone who holds it can use it.
Recap
- A presigned URL lets a client reach one object for one operation and a short time, without credentials.
- Whoever holds the link can use it. Keep it short-lived, narrow and private.
- Multipart uploads send a large file in parts, in parallel, and retry only the part that failed.