async/await Explained: How One Thread Serves a Hundred Waiting Tasks

📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram

Originally published on software-engineer-blog.com.

Every async tutorial starts at the syntax. Start one level below it instead.

A hundred requests are in flight on your server. Every one of them is waiting on the database. Now take away their hundred threads, and a single question is left standing: where does each half-finished function keep its place?

That question is all async / await really answers. Everything else — the keywords, the coloured-function arguments, the library churn — is downstream of it.

A thread is a stack, and a stack is a bookmark

Ask what a thread actually buys you and the answer is disappointingly concrete: a stack. That stack exists to hold two things — your local variables, and the line you are on. That is it. When the CPU switches away from your thread and comes back, the stack is how it knows where "back" was.

Look at a perfectly ordinary handler:

def get_order(order_id):
    started = time.time()           # a local
    row = db.query(order_id)        # ← 200 ms of waiting
    elapsed = time.time() - started # a local
    return render(row, elapsed)

Time it honestly and it spends maybe two milliseconds computing and two hundred milliseconds waiting. For that entire 200 ms the thread is doing nothing at all. It is not slow; it is parked. And it is holding a full stack open the whole time, purely so that when db.query returns, something remembers that started was 1726... and that the next line to run is the one assigning elapsed.

That is the bill nobody itemises. On Linux a pthread reserves 8 MB of address space for its stack by default (ulimit -s), committed lazily page by page; runtimes that choose their own are usually in the ~1 MB neighbourhood. Round it to a megabyte and the arithmetic is brutal:

100 parked requests → ~100 MB of memory held open to remember three variables and a line number.

You are not paying for work. You are paying for bookkeeping, at the price of a data structure built for something far more general.

await is a suspension point, not "go faster"

Here is the misreading that survives most tutorials: await does not make the database answer sooner. It does not add a thread. It does not parallelise anything.

await marks a suspension point — the line your function is allowed to stop on.

async def get_order(order_id):
    started = time.time()
    row = await db.query(order_id)   # ← stops HERE
    elapsed = time.time() - started
    return render(row, elapsed)

When control reaches that await, three things happen in order:

1. The function stops, mid-body, with two lines still unexecuted.
2. It registers interest: wake me when the row is ready.
3. It hands control back to whoever is running it.

Step 3 is the one that pays. The thread is not parked on your function any more — it is free, immediately, to go run somebody else's.

But step 1 leaves a debt. A function that stopped halfway still has a started value and a resume point, and those have to live somewhere.

Where the state goes: a frame on the heap

This is the whole trick, and it is smaller than it sounds.

Instead of keeping a megabyte-sized stack alive to hold three values, the runtime copies the locals plus the resume point into a small object on the heap — a coroutine frame. A few hundred bytes. In CPython you can hold one in your hand:

async def get_order(order_id): ...

coro = get_order(42)      # nothing has run yet
print(type(coro))         # <class 'coroutine'>
print(coro.cr_frame.f_locals)  # the locals live here, on the heap

Nothing executed when you called it. Calling an async function does not run it — it builds the bookmark.

A stack is a room you rent to hold your place. A coroutine frame is a slip of paper with your place written on it. Both remember the same two facts. One costs a megabyte of reserved address space; one costs the size of the variables you actually have.

100 parked requests → ~30 KB of heap frames, rather than ~100 MB of stacks. Same information. Three orders of magnitude apart.

One thread, a ready queue of bookmarks

Now the loop. One thread holds a ready queue of those frames and does something almost boringly simple, forever:

1. Take the next ready frame off the queue.
2. Run it up to its next await.
3. Put it aside; take the next one.
4. When an I/O completes, push that frame back onto the ready queue.

Step 4 is where the operating system earns its keep. The loop is not spinning and checking each socket in turn — it blocks in one epoll_wait-style call that says wake me when any of these three hundred sockets is ready, and the kernel hands back exactly the ones that are.

async def main():
    # a hundred bookmarks, one thread
    await asyncio.gather(*(get_order(i) for i in range(100)))

gather does not start a hundred threads. It creates a hundred frames, and every one of them runs on the same thread, up to its first await, in turn.

Resuming: the line after the await

When your row finally arrives, the frame rejoins the ready queue, the loop reopens it, and execution continues on the line after the await — with started exactly as you left it.

It did not restart. It did not re-run time.time(). It was parked, with no thread held open to wait for it.

That is the entire mechanism. A suspension point, a bookmark on the heap, and a loop that reopens bookmarks in the order they become ready.

The three catches, which are the whole interview

Everything above is the part tutorials cover. These three are the part they skip — and they are exactly what a good interviewer probes.

1. You bought concurrency on waiting, never on computing. There is still one thread. If two tasks both need to hash a password, they do it one after the other. async moved the waiting out of the way; it did not add a core. CPU-bound work needs processes, threads, or a native extension that releases the GIL — not await.

2. One blocking call freezes everyone. This is the failure mode that actually reaches production. A single synchronous call inside an async function — requests.get, time.sleep, a driver that is secretly synchronous, a 40 ms JSON parse — does not just stall that task. It holds the only thread, so every other task on the loop stops too, including ones whose data arrived ten milliseconds ago.

async def handler():
    row = await db.query(1)      # fine: yields
    blob = requests.get(url)     # 🔴 blocks the ENTIRE loop
    return blob

The fix is to push it off the loop rather than pretend it is async:

blob = await asyncio.to_thread(requests.get, url)

3. Nothing is pre-empted. The OS can interrupt a thread anywhere. The event loop cannot. Scheduling here is cooperative: a task keeps the thread until it chooses to yield, and it only ever chooses at an await. A task that never awaits never lets anyone else run — a while True: with no suspension point inside it is not a busy task, it is a stopped server.

The mental test for any async code you write is one question: between here and the next await, how long is this thread mine?

The same mechanism, one floor up: serving LLMs

This is not a backend-trivia pattern. It is the reason a single modest Python process can sit in front of a GPU cluster and hold thousands of open chat connections — and it is also the reason people misdiagnose their inference latency.

An LLM request is almost entirely waiting. A completion that streams for eight seconds involves a few milliseconds of your code and eight seconds of the model producing tokens somewhere else. Thread-per-request would mean one megabyte-class stack per open conversation, parked, doing nothing. The gateway in front of a model is the purest possible async workload: thousands of bookmarks, one loop, near-zero CPU.

Streaming makes the suspension point visible. When you iterate a token stream, every async for is an await — the loop hands the thread away between tokens, so one process can fan hundreds of concurrent streams out to hundreds of clients:

async def stream(prompt):
    async for token in client.completions(prompt, stream=True):
        yield token            # suspends between tokens; loop stays free

But catch #1 sets a hard ceiling on what async can fix. Async gives you concurrency on waiting, and the two numbers everyone tunes — TTFT (time to first token) and TPOT (time per output token) — are not waiting. They are GPU work. No amount of await shortens them. What actually raises throughput is continuous batching: the server merges newly arrived requests into the running batch at every decoding iteration, rather than making them queue behind a finished batch. Note the symmetry — that scheduler is doing the same thing the event loop does, at the level of a token instead of a syscall: keep a queue of half-finished work, advance each one step, never hold an expensive resource open for something that is merely waiting.

And catch #2 is where real inference gateways bleed. Tokenising a 100 KB prompt, a synchronous embedding call, a hefty json.loads of a tool-call payload — each is pure CPU on the loop thread, and each stalls every in-flight stream, not just its own. The symptom is maddening: the model is fast, the GPU is idle, and p99 latency is terrible. The cause is a blocking call inside an async handler.

The verdict

You never needed a hundred threads. You needed a hundred bookmarks and one thread to turn the pages.

• A thread is a stack, and a stack's whole job is to remember your locals and your line.
• await is where the runtime takes that memory and writes it on a slip of paper instead.
• One loop runs the slips, each up to its next suspension point, and resumes each exactly where it stopped.
• It buys you waiting, not computing — and it holds only as long as every task keeps yielding.

Reach for it when your service is I/O-bound: an API gateway, a scraper, a websocket fan-out, a model proxy. Reach for processes when the work is genuinely CPU-bound. And whichever you choose, the diagnostic question stays the same: who is holding this thread, and when do they give it back?

References and further reading

On what a thread actually costs: the stack

• M. Kerrisk, The Linux Programming Interface (No Starch Press, 2010) — chapters 29–33 on POSIX threads: what a thread owns privately (its stack) versus what it shares, and why the per-thread stack is the dominant per-thread cost.
• pthread_create(3) — Linux man-pages — states that a thread's stack size defaults to the RLIMIT_STACK soft limit (commonly 8 MB on Linux) and that the stack is allocated per thread: man7.org — pthread_create(3)
• A. Silberschatz, P. Galvin, G. Gagne, Operating System Concepts (Wiley, 10th ed., 2018) — chapter 4 on threads and chapter 5 on CPU scheduling: the context switch, and the pre-emptive model that the event loop deliberately does not use.

On await as a suspension point, and where the state lives

• Y. Selivanov, PEP 492 — Coroutines with async and await syntax (Python, 2015) — the language-level definition: await suspends execution of the coroutine and yields control, and calling a coroutine function returns a coroutine object rather than running it: peps.python.org — PEP 492
• G. van Rossum, PEP 3156 — Asynchronous IO Support Rebooted: the "asyncio" Module (Python, 2012) — the design of the event loop itself: the ready queue, callbacks on I/O completion, and the single-threaded execution model: peps.python.org — PEP 3156

On the loop and non-blocking I/O

• M. Kerrisk, The Linux Programming Interface (No Starch Press, 2010) — chapter 63 on alternative I/O models: select, poll and epoll, the syscall the loop actually blocks in while every task is suspended.
• The Node.js Event Loop — Node.js docs — the same single-threaded loop described phase by phase, for the runtime where this model is the default rather than a choice: nodejs.org — event loop

On the three catches: blocking, cooperation, and what concurrency is not

• Developing with asyncio — Python docs — the explicit warning that blocking CPU-bound or I/O-bound calls must never be called directly from a coroutine because they block the whole event loop, plus to_thread / run_in_executor as the escape hatch: docs.python.org — developing with asyncio
• Don't Block the Event Loop — Node.js docs — the same hazard stated as an operational rule, with the latency consequences for every other connection: nodejs.org — don't block the event loop
• R. Pike, Concurrency Is Not Parallelism (Heroku Waza talk, 2012) — the distinction catch #1 rests on: dealing with many things at once is a structural property, and it is not the same as doing many things at once: go.dev — talk

On the LLM-serving section: why async cannot shorten TTFT

• G. Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI, 2022) — introduces iteration-level scheduling, where the server revisits its batch at every decoding step instead of waiting for the slowest request to finish: usenix.org — Orca
• W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP, 2023) — the vLLM paper: continuous batching plus paged KV-cache memory, and the throughput ceiling that request-level concurrency alone cannot lift: arxiv.org — 2309.06180

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the reel: async/await: how ONE thread serves a hundred waiting tasks