System Design: A URL Shortener With Real Numbers

Start by pinning down what the system must do, because every later number depends on it.

Functional:

• Create a short link for a long URL and return the short code.
• Redirect a short code to its long URL.
• Optionally: custom aliases, expiry dates, editing or deleting a link, and click analytics.

Non-functional:

• Redirects must be fast and highly available. A broken short link in a printed poster or a sent email cannot be fixed by the person who sees it.
• Codes should be short and hard to guess, so private links (shared documents, unlisted pages) cannot be found by enumeration.
• Creating links can be slower and less available than following them, since it is a far rarer operation.

The asymmetry is the main design driver. Reads (redirects) vastly outnumber writes (new links), and almost every design choice below optimises the read path at some cost to the write path. Writes are not free, though: every redirect can also produce a click record, which is a write. That tension is the subject of Read-Heavy vs Write-Heavy System Design.

Estimates are only as good as their assumptions, so state them first. These are illustrative, not measurements:

• 100 million new links a month.
• 100 redirects per link created (a 100:1 read-to-write ratio).
• Links kept for 5 years.
• About 500 bytes per stored link (the code, a long URL averaging about 200 bytes, owner, timestamps and index overhead).
• Peak traffic 5 times the average.
• The 10 million most popular links cached, at about 600 bytes each.

The script that turns them into numbers:

1import math2# Assumptions (state them; change them for your system)3new_per_month = 100_000_0004read_write = 1005years = 56row_bytes = 500            # code, long URL (avg ~200 B), owner, timestamps, index overhead7peak_factor = 58cache_links = 10_000_0009cache_bytes_per_link = 60010 11sec_month = 30 * 24 * 360012w = new_per_month / sec_month13r = w * read_write14total = new_per_month * 12 * years15print(f"writes/s avg {w:,.1f}  peak {w*peak_factor:,.0f}")16print(f"reads/s  avg {r:,.0f}  peak {r*peak_factor:,.0f}")17print(f"links after {years}y: {total:,}")18print(f"storage: {total*row_bytes/1e12:,.2f} TB")19print(f"cache: {cache_links*cache_bytes_per_link/1e9:,.1f} GB")20for n in (6, 7, 8):21    space = 62 ** n22    print(f"base62 len {n}: {space:,} codes = {space/total:,.0f}x the 5-year total; P(new random code collides at {total:,} used) = {total/space:.4%}")23# birthday: expected first collision ~ sqrt(pi/2 * N)24N = 62 ** 725print(f"len7 expected codes before first collision (birthday): {math.sqrt(math.pi/2*N):,.0f}")26# bandwidth for redirects27resp_bytes = 50028print(f"redirect egress at peak: {r*peak_factor*resp_bytes*8/1e6:,.0f} Mbit/s")

Its output:

• Writes: 100,000,000 ÷ 2,592,000 seconds in a 30-day month ≈ 38.6 per second, about 193 per second at peak.
• Reads: 38.6 × 100 ≈ 3,858 per second, about 19,290 per second at peak.
• Storage: 100 million × 12 months × 5 years = 6 billion links, × 500 bytes ≈ 3 TB.
• Cache: 10 million × 600 bytes ≈ 6 GB, which fits in memory on a single cache node.
• Redirect bandwidth at peak: 19,290 × 500 bytes ≈ 77 Mbit/s.

What these numbers decide: a few hundred writes a second is easy for one database, so the write path needs no special scaling. Around 19,000 reads a second at peak is where a cache pays off. And 3 TB is large enough that you should plan how to partition the data before you need to.

Two endpoints cover the core:

• POST /links with a JSON body {"url": "https://..."} returns 201 and {"code": "6ZY3LCH", "short_url": "https://sho.rt/6ZY3LCH"}.
• GET /{code} returns a redirect to the long URL, or 404 if the code does not exist.

The stored record is small: code (primary key), url, created_at, and optionally owner_id, expires_at and a disabled flag for abuse takedowns. The only query on the hot path is a lookup by primary key, so almost any database works. A relational database is a good default while the data fits on one machine. At 3 TB you need a plan for partitioning: shard by a hash of the code, which spreads random codes evenly, or use a distributed key-value store that does this for you.

Make link creation idempotent if clients retry it. Otherwise a timeout during POST /links can produce two codes for one request. An idempotency key on the request solves that; see Idempotency Keys.

A code is a string over an alphabet, usually base62 (0-9, A-Z, a-z). The length sets the key space:

• 6 characters: 62⁶ ≈ 56.8 billion codes, 9 times the 6 billion links expected in 5 years.
• 7 characters: 62⁷ ≈ 3.52 trillion codes, 587 times the 5-year total.
• 8 characters: 62⁸ ≈ 218 trillion codes.

There are four common ways to choose a code:

• Counter plus base62: encode an auto-incrementing ID. There are no collisions, but codes are predictable: anyone can try the next one, which exposes private links and reveals your growth rate.
• Random codes: draw characters from a cryptographic random source. They are unguessable, but they can collide.
• Hash of the URL: take the first characters of a hash of the long URL. The same URL always gets the same code, which deduplicates, but it merges different users' links (and their analytics and deletions) and still needs a collision check, because truncated hashes collide.
• Pre-allocated key ranges: each server takes a block of counter values from a coordinator and uses them locally. No per-write coordination is needed, but the codes are sequential within a block, so they are guessable.

Random codes need collision handling, and the birthday bound shows why it cannot be skipped. With 7-character codes, the first collision is expected after only about 2.35 million links (√(π/2 × 62⁷)). Collisions are certain long before the key space fills. They stay rare per insert, though: once 6 billion codes are in use, a new random code clashes with probability 6 × 10⁹ ÷ 3.52 × 10¹² ≈ 0.17%. So the design is a unique constraint on code plus a retry: about one insert in 600 needs a second draw.

Here is the create and redirect logic, using SQLite so it runs anywhere. In production the table lives in your main database and the dictionary is a shared cache.

1import secrets, sqlite32from urllib.parse import urlsplit3 4ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"5CODE_LEN = 76MAX_TRIES = 57 8db = sqlite3.connect(":memory:", isolation_level=None)9db.execute("CREATE TABLE links (code TEXT PRIMARY KEY, url TEXT NOT NULL, created_at TEXT DEFAULT CURRENT_TIMESTAMP)")10cache = {}11 12def new_code():13    return "".join(secrets.choice(ALPHABET) for _ in range(CODE_LEN))14 15def shorten(url):16    parts = urlsplit(url)17    if parts.scheme not in ("http", "https") or not parts.netloc:18        raise ValueError("only absolute http(s) URLs can be shortened")19    for _ in range(MAX_TRIES):20        code = new_code()21        try:22            db.execute("INSERT INTO links (code, url) VALUES (?, ?)", (code, url))23            return code24        except sqlite3.IntegrityError:   # code already taken: draw another25            continue26    raise RuntimeError("no free code after retries; increase CODE_LEN")27 28def resolve(code):29    if code in cache:30        return cache[code]31    row = db.execute("SELECT url FROM links WHERE code = ?", (code,)).fetchone()32    url = row[0] if row else None33    cache[code] = url                   # negative results are cached too34    return url35 36def redirect(code):37    url = resolve(code)38    if url is None:39        return 404, {}40    # 302: browsers come back each time, so analytics and edits keep working41    return 302, {"Location": url, "Cache-Control": "private, max-age=60"}

Three details matter. shorten accepts only absolute http and https URLs, which rejects javascript: and data: links that would turn the shortener into an attack tool. It relies on the database's primary-key constraint to detect a collision, and draws a new code when the insert fails, instead of checking first and inserting second. Check-then-insert is a race between concurrent requests. And resolve caches misses too, so repeated requests for a nonexistent code do not reach the database.

The tests confirmed each path. A valid URL got a 7-character code and a 302 with a Location header, and an unknown code got 404. A javascript: URL was rejected. With the alphabet shrunk to two letters and the length to two characters (four possible codes), the first four links succeeded through retries. The fifth failed with "no free code after retries", which is the signal to lengthen codes.

RFC 9110 defines four redirect codes that matter here:

• 301 Moved Permanently: the resource has a new permanent URI. A 301 is heuristically cacheable: without explicit cache headers, a cache may store it and reuse it.
• 302 Found: the resource is temporarily at another URI, and the client ought to keep using the original. A 302 is not on the RFC's list of heuristically cacheable codes, so it is only cached when headers say so.
• 307 and 308: the temporary and permanent equivalents that forbid changing the request method. For 301 and 302, the RFC notes that a user agent may change POST to GET for historical reasons. 308 is heuristically cacheable like 301.

For a shortener, the choice is about who sees the second click. With a 301, a browser may cache the redirect and go straight to the destination next time, skipping your server. That saves load, but you lose the click in analytics, and editing or disabling the link will not reach browsers that cached it. With a 302, every click reaches you, so analytics, edits, expiry and abuse takedowns keep working, at the cost of serving every request.

The implementation above returns 302 with Cache-Control: private, max-age=60. That allows a browser to reuse the redirect for a minute, so a quick double-click is served locally, while changes take effect within a minute. If you never edit links and do not need per-click analytics, 301 with a long max-age gives the lowest load.

At about 19,000 redirects a second at peak, the database should not serve every lookup. Put a cache in front with the cache-aside pattern: look up the code in the cache, and on a miss read the database and store the result.

Link popularity is usually very uneven: a small share of links (a viral post, a campaign) takes most of the clicks. That is why caching 10 million hot links, about 6 GB, can absorb most of the traffic. How much it absorbs depends on your real distribution, so measure the hit rate rather than assuming one.

Three cache details matter for a shortener:

• Cache misses (negative caching). Bots probe random codes. Without cached 404s, every probe reaches the database. Cache misses with a short TTL.
• Invalidation on edit and takedown. When a link is edited or disabled, delete its cache entry immediately. Otherwise a phishing link you disabled keeps redirecting until the entry expires.
• The cold-cache failure. If the cache cluster restarts empty, all 19,000 requests a second fall through to the database at once. Either size the database to survive that, or coalesce concurrent misses for the same code so only one query runs.

The cache is also where latency is won: an in-memory lookup in the same data centre is far faster than a database query, and a redirect is a response the user waits on before the real page even starts loading.

Analytics turns every read into a write, and doing that write synchronously is the easiest way to make redirects slow. An UPDATE links SET clicks = clicks + 1 on every request creates hot rows. A viral link becomes one row updated thousands of times a second, and those updates contend for the same lock.

Instead, make the redirect emit an event and return immediately. The redirect service appends a click record (code, timestamp, and whatever attributes you need) to a queue or log. A separate consumer aggregates them in batches into an analytics store, for example counts per link per hour. The redirect's latency no longer depends on the analytics database, and an analytics outage loses or delays counts instead of breaking links.

The boundary is accuracy. Asynchronous counts lag behind, and if the queue drops events under overload, counts are low. Decide what you promise: approximate real-time counts and exact daily totals is a common split. Also decide what you store. IP addresses and user agents can be personal data under privacy laws such as the GDPR, so collect only what you need, keep it only as long as you need it, and aggregate early.

A public shortener will be used to disguise malicious links. Plan for it from the start:

• Validate and screen on creation. Accept only http and https, and check new URLs against a URL reputation or blocklist service before issuing a code.
• Rate-limit creation per user and per IP, since bulk creation is how spam campaigns use shorteners.
• Support takedowns. A disabled flag checked on every redirect, plus immediate cache invalidation, lets you kill a link in seconds.
• Consider an interstitial page for links reported as suspicious, so users see the destination before going there.

For scale and failure, the estimates point to where effort goes:

• Database: at 3 TB, partition by a hash of the code. Read replicas help if the cache hit rate is low, but every replica must receive takedowns quickly.
• Cache loss: covered above; it is the most likely cause of a sudden database overload.
• Region failure: redirects should keep working when one region is down. Replicate link data to every region that serves redirects, and accept that link creation may be briefly unavailable or slower during a failover.

The redirect path is what users notice, so give it the strictest availability target and keep it simple: cache, primary-key lookup, redirect, event.

Work through the design with the two scripts in this guide:

1. Change the assumptions in estimate.py to 1 billion new links a month and a 1,000:1 read ratio, and recompute writes, reads, storage and cache. 2. From the new numbers, decide whether a single database can still take the writes, and how many cache nodes the hot set needs. 3. Compute the 5-year link total under the new assumptions and the smallest base62 code length that keeps the per-insert collision probability below 1% at the end. 4. Run shortener.py with a two-letter alphabet and two-character codes, and confirm that creation fails cleanly once all four codes are used. 5. Change redirect to return 301 and explain which features stop working for browsers that cached it. 6. Add a disabled column and make redirect return 410 for disabled links, invalidating the cache entry when a link is disabled.

Pass criterion: your estimates follow from the assumptions with the arithmetic shown, your chosen code length meets the 1% bound, and a disabled link stops redirecting immediately in your test.