Agent Wranglers

Files and uploads Optional

Store the same file once per owner

When users upload the same file often, store it once per owner under its SHA-256 hash and count the references.

How to do it

Hash the bytes on the server after upload, never trusting a hash the client sends. Look up the hash within the same owner only. Matching across owners tells an attacker that someone else holds a given file. Point each file record at the shared object and count references. Delete the object only when the count reaches zero, inside the same transaction that removes the last record.

Example

import { createHash } from 'node:crypto';
import { createReadStream } from 'node:fs';

export async function sha256OfFile(path) {
  const hash = createHash('sha256');
  for await (const chunk of createReadStream(path)) hash.update(chunk);
  return hash.digest('hex');
}

export const contentKey = (shopId, sha256) => `files/shops/${shopId}/blobs/${sha256}`;
import hashlib

def sha256_of_file(path: str) -> str:
    with open(path, "rb") as f:
        return hashlib.file_digest(f, "sha256").hexdigest()

def content_key(shop_id: str, sha256: str) -> str:
    return f"files/shops/{shop_id}/blobs/{sha256}"

How to check

One owner uploads the same file twice, and storage holds one object with a count of 2. Deleting one record keeps the object, and deleting the second removes it. A second owner uploading the same file gets their own object.

For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers api design playbook: https://agentwranglers.se/playbook/api-design/files-and-uploads/dedupe-by-hash.