Confirm every upload on the server

How to do it

A dropped connection, a faulty proxy or a bug in a resumed upload can store a damaged file that looks fine until someone opens it. RFC 9530 defines the Content-Digest header for the bytes of one message, with sha-256 and sha-512 as active algorithms and MD5 and SHA-1 deprecated. Object storage takes its own checksum headers, such as x-amz-checksum-sha256 on S3, and refuses bytes that do not match. Resumable uploads check each part and then the whole file. When the client says the upload is done, or storage sends its event, read the stored object. Compare its size with your limit. Find its type from the first bytes, and checksum it with SHA-256. Store all three in your files table, and refuse a file whose SHA-256 differs from the client's. Compare it again after every copy, as Restore files and their records together says. Only then mark it ready for the next step. The attacks this blocks, and how to test for them, are in check every uploaded file. Delete pending files that were never confirmed within 24 hours.

Examples

JavaScript example

import { fileTypeFromBuffer } from 'file-type';

const ALLOWED_TYPES = new Set(['image/jpeg', 'image/png', 'application/pdf']);

export async function confirmUpload(firstBytes, storedSize, maxBytes) {
  const detected = await fileTypeFromBuffer(firstBytes);
  if (storedSize > maxBytes) return { status: 'rejected', reason: 'too-large' };
  if (!detected || !ALLOWED_TYPES.has(detected.mime)) return { status: 'rejected', reason: 'type-not-allowed' };
  return { status: 'awaiting-scan', mime: detected.mime };
}

Python example

import magic

ALLOWED_TYPES = {"image/jpeg", "image/png", "application/pdf"}

def confirm_upload(first_bytes: bytes, stored_size: int, max_bytes: int) -> dict:
    detected = magic.from_buffer(first_bytes, mime=True)
    if stored_size > max_bytes:
        return {"status": "rejected", "reason": "too-large"}
    if detected not in ALLOWED_TYPES:
        return {"status": "rejected", "reason": "type-not-allowed"}
    return {"status": "awaiting-scan", "mime": detected}

How to check

Upload a text file named photo.jpg with a content type of image/jpeg. The file never leaves pending, and the API answers with a problem response naming the type. Upload a file with the checksum of other bytes, and storage or the server refuses it. With the right checksum, the files table holds the same SHA-256.

Sources

• OWASP File Upload Cheat Sheet
• RFC 9530 (Digest Fields)

Related rules

• APP-24 Check files in the browser before they upload in Web app interface
• MED-1 Define each kind of upload once in Files and media
• MED-41 Upload straight to storage with a signed address in Files and media
• MED-42 Make large uploads resumable in Files and media
• MED-6 Restore files and their records together in Files and media
• APS-43 Check every uploaded file in Application security
• MED-9 Accept the photo formats phones save in Files and media