asyncio for LLM calls: what the event loop is really doing

An LLM call takes one to ten seconds, and almost all of that time your program is doing nothing: it sent a request and is waiting for bytes to come back. If you need to embed 2,000 chunks or grade 300 eval answers, calling the API one at a time means waiting 2,000 times in a row.

Here's that cost, with time.sleep(0.2) standing in for a 0.2-second API call:

import time

def call_llm(prompt):
    time.sleep(0.2)              # waiting on the network
    return f"answer to {prompt}"

start = time.perf_counter()
answers = [call_llm(p) for p in ["q1", "q2", "q3", "q4", "q5"]]
print(len(answers), "answers in",
      round(time.perf_counter() - start, 1), "s")

5 answers in 1.0 s

One second for five calls, and the CPU was idle the whole time. asyncio lets one thread overlap all of that waiting. But it's also the source of the most confusing Python bugs: "coroutine was never awaited", a whole server freezing because of one requests.get, and rate-limit storms. All of them make sense once you can see the event loop.

Part I · The core idea

Chapter 1: Concurrency is overlapping waits

The problem

Two words people mix up:

• Parallelism: doing several things at the same instant, on several CPU cores. Java threads on a multi-core machine can do this.
• Concurrency: making progress on several things in overlapping time, even with one worker, by switching between them whenever one is stuck waiting.

asyncio gives you concurrency with one thread. It does not make CPU work faster.

Details, built on the model

The model also predicts when asyncio won't help: if the chef must chop (CPU work such as tokenising a huge file or running a local model), there's no waiting to overlap. Chopping five onions takes five times as long no matter how cleverly you walk around. For that you need more chefs: processes, or threads around libraries that release Python's lock (NumPy and PyTorch do).

Chapter 2: A function that can pause

The problem

For the chef to walk away from a pot, a function must be able to stop halfway, remember where it was, and continue later. Normal functions can't. Python has a kind of function that can: the generator, any function containing yield.

Details, built on the model

def steps():
    print("  step A")
    yield "paused after A"
    print("  step B")
    yield "paused after B"

g = steps()                # nothing runs yet: a closed book
print(next(g))
print(next(g))

  step A
paused after A
  step B
paused after B

Calling steps() didn't print anything; it just made the book. That's exactly why this common bug happens:

import asyncio

async def fetch():
    return "data"

c = fetch()                # only creates the coroutine (a book)
print(type(c).__name__)
c.close()                  # tidy up so Python doesn't warn
print(asyncio.run(fetch()))

coroutine
data

Calling an async def function only creates a coroutine object. Nothing runs until something drives it, which is either await from another coroutine or the event loop. Forgetting await gives the "coroutine was never awaited" warning.

Chapter 3: Build an event loop in 20 lines

The problem

Who walks between the pots? The event loop: a loop that holds a queue of paused tasks and resumes whichever is ready. Let's build one from generators so it's not a black box.

Details, built on the model

import heapq, time

def run(tasks):
    start = time.perf_counter()
    queue = [(0.0, i, t) for i, t in enumerate(tasks)]
    heapq.heapify(queue)
    while queue:
        wake, i, task = heapq.heappop(queue)
        delay = wake - (time.perf_counter() - start)
        if delay > 0:
            time.sleep(delay)        # nothing ready: wait
        try:
            secs = next(task)        # resume to next yield
            now = time.perf_counter() - start
            heapq.heappush(queue, (now + secs, i, task))
        except StopIteration:
            pass
    return round(time.perf_counter() - start, 1)

def fake_llm(name):
    print(f"  {name}: request sent")
    yield 0.2                        # "wake me in 0.2 s"
    print(f"  {name}: answer back")

print("took", run([fake_llm(f"q{i}") for i in range(1, 6)]), "s")

  q1: request sent
  q2: request sent
  q3: request sent
  q4: request sent
  q5: request sent
  q1: answer back
  q2: answer back
  q3: answer back
  q4: answer back
  q5: answer back
took 0.2 s

Five 0.2-second waits finished in about 0.2 seconds on one thread. All five "request sent" lines print first, because each task runs until its yield and hands control back.

Now the critical edge case, predicted by the model: what if a task does slow work without yielding? The loop can't take control back, because the loop only runs between yields.

def blocker():
    print("  blocker: calling time.sleep(0.5)")
    time.sleep(0.5)                  # never yields
    yield 0

print("took", run([blocker(), fake_llm("q1")]), "s")

  blocker: calling time.sleep(0.5)
  q1: request sent
  q1: answer back
took 0.7 s

q1 couldn't even send its request until the blocker was done. In a real asyncio server, calling requests.get or time.sleep inside async def does exactly this: every user's request freezes. That's cooperative scheduling: tasks have to give control back voluntarily, and asyncio only works if every wait is an awaitable one.

Part II · Real asyncio

Chapter 4: Tasks and gather

The problem

Now map real asyncio onto our loop. asyncio.sleep(0.2) is our yield 0.2. await passes the pause up to the loop. What starts things concurrently?

Details, built on the model

import asyncio, time

async def call_llm(prompt):
    await asyncio.sleep(0.2)          # a non-blocking wait
    return f"answer to {prompt}"

async def one_by_one(prompts):
    return [await call_llm(p) for p in prompts]

async def together(prompts):
    return await asyncio.gather(*(call_llm(p) for p in prompts))

prompts = [f"q{i}" for i in range(1, 6)]
for fn in (one_by_one, together):
    t = time.perf_counter()
    res = asyncio.run(fn(prompts))
    print(fn.__name__, res[0], round(time.perf_counter() - t, 1), "s")

one_by_one answer to q1 1.0 s
together answer to q1 0.2 s

Errors: by default, if one coroutine in gather raises, gather raises that error, and you lose the other results. For batch jobs you usually want every result, with failures as values:

async def flaky(i):
    await asyncio.sleep(0.01)
    if i == 2:
        raise RuntimeError("429 rate limited")
    return i

async def main():
    rs = await asyncio.gather(*(flaky(i) for i in range(4)),
                              return_exceptions=True)
    print(rs)

asyncio.run(main())

[0, 1, RuntimeError('429 rate limited'), 3]

If a blocking library has no async version, the model says: move it off the chef's hands. asyncio.to_thread(func, ...) runs it on a worker thread and gives the loop an awaitable to wait on.

Chapter 5: Limits and timeouts

The problem

Firing 2,000 requests at once hits the provider's rate limit (HTTP 429), and one hung request can stall a batch forever. Two tools fix this.

Details, built on the model

import asyncio, time

async def call_llm(sem, prompt, stats):
    async with sem:                       # take a burner
        stats["now"] += 1
        stats["peak"] = max(stats["peak"], stats["now"])
        await asyncio.sleep(0.1)
        stats["now"] -= 1
        return prompt

async def main():
    sem = asyncio.Semaphore(3)
    stats = {"now": 0, "peak": 0}
    t = time.perf_counter()
    await asyncio.gather(*(call_llm(sem, i, stats)
                           for i in range(9)))
    print("peak in flight:", stats["peak"],
          "| took", round(time.perf_counter() - t, 1), "s")

asyncio.run(main())

peak in flight: 3 | took 0.3 s

Nine 0.1-second calls with three burners take three rounds: about 0.3 seconds, never more than 3 in flight. Now the timer:

async def hung():
    await asyncio.sleep(10)

async def main():
    try:
        await asyncio.wait_for(hung(), timeout=0.2)
    except TimeoutError:
        print("gave up after 0.2 s; the task was cancelled")

asyncio.run(main())

gave up after 0.2 s; the task was cancelled

In production you combine all three: a semaphore sized below your rate limit, a timeout per call, and return_exceptions=True (or retries) so one failure doesn't throw away the batch.

All the mental models, in one place