Design Cloud File Storage and Sync

Desktop / mobile client p. 60

Deep dives p. 60

• Chunking and dedup: split files into 4 MB (or content-defined, variable-size) chunks; identify each by its hash. The client sends the list of hashes, the server replies which chunks it lacks, and only those are uploaded. Identical chunks across files and users are stored once. Content-defined chunking (rolling hash) keeps boundaries stable when bytes are inserted, so edits to a large file re-upload only a few chunks.
• Metadata consistency: the file tree, version history and chunk lists live in a strongly consistent store (sharded SQL keyed by user/namespace). A new version is committed in one transaction only after all its chunks exist, so readers never see half-uploaded files.
• Sync protocol: every namespace has a monotonically increasing journal/cursor. A device tells the server its last cursor; the server returns the changes since then (long-poll or WebSocket for push). Local changes are batched and applied with optimistic version checks.
• Conflicts: if two devices modify the same version offline, the second commit is rejected; the client creates a "conflicted copy" (Dropbox) or merges (Docs-style OT/CRDT for collaborative documents).
• Durability and cost: replicate chunks across zones or use erasure coding; background scrubbing verifies checksums; move cold versions to cheaper tiers; reference counting + garbage collection deletes chunks no longer referenced by any version.
• Large files and flaky networks: resumable multi-chunk upload, parallel chunk transfers, compression, delta encoding, and LAN sync between devices on the same network.