9.7

Workers, the advisory lock, and the cost of a conversation

With a real model the ceiling is the model server, so add workers for availability, take the thread lock in Postgres, and price a conversation from measured token counts.

Traffic doubles, and you start a second uvicorn worker. Two requests for the same thread now reach two processes. An in-process lock no longer stops them, and both runs write to one checkpoint chain. The second question is what the extra worker buys you, and what each conversation costs.

When you finish this section, you can run several workers with one Postgres lock per thread. You can find the resource that limits throughput, and estimate the price of 1,000 conversations from token counts that you measured.

One run per thread, across processes

Advisory lock: a Postgres lock on a number that your code chooses, held by a session and released when the session ends. The service hashes the thread id into that number with hashtextextended.

pythonthe thread lock: a Postgres advisory lock on one dedicated connectionFull log
KEY = "select {fn}(hashtextextended(%s, 0))"
…
    async def acquire(self, thread_id):
        if thread_id in self.held:
            return False
        self.held.add(thread_id)
        if await self.call("pg_try_advisory_lock", thread_id):
            return True
        self.held.discard(thread_id)
        return False

Three choices in that class matter.

  • Each worker holds its locks on one connection of its own, outside the pool. With locks on pool connections, 20 runs would hold all 20 connections. Their checkpoint writes would then wait for a 21st connection.
  • An advisory lock is reentrant inside one session, so a second request in the same worker would get the lock again. The held set refuses it first.
  • When a worker dies, Postgres ends its session and frees its locks. No lease or timer is needed.

We measured the race test on the Mac in each of the six Qwen runs of the suite, three per model. It starts two uvicorn processes, A and B, on one database. The result was the same in 6 of 6 runs. Raw runs

  • One run to A and one to B on the same thread, at once: one 200, one 409, and one human message.
  • Two runs sent to A on the same thread: one 200 and one 409.
  • A run sent to B while A held the thread: 409.
  • A killed with SIGKILL while it held the thread, then a run sent to B: 200. The lock was free as soon as the process died.

Where the ceiling is

We measured the service under load on the Mac with Qwen3 4B, one run of each cell. Each user had one thread and sent four requests in order. They were an order question, a FAQ question, a refund with the amount and the reason, and the approval. All users started at once. The load generator ran on the same Mac with a read timeout of 120 s, and each worker had a pool of 20 connections. Requests went to the workers in turn. Raw runs

As a baseline, one user alone ran 5 sessions of 4 requests. A run took a median of 3.0 s, and the user finished 0.269 runs each second. Leaked test servers held 66 Postgres connections during this baseline, so treat it as a rough reference.

WorkersUsers at onceRequestsClient timeouts409 on approveRuns each secondMedian s per run95th percentile sPostgres connections, max
11040020.25936.051.111
21040020.24238.348.615
41040020.25835.646.726
1501574820.208164.7192.522
2501634640.216152.5188.735
4501594910.220145.6181.448

Figure 9.7 puts the three measurements side by side.

Fig. 9.10workers against the model server, Qwen3 4B on the Macchart
Workers against the model server, Qwen3 4B on the Mac Three panels. Left: completed runs each second for 1 worker and 10 users 0.259, 1 worker and 50 users 0.208, 2 workers and 10 users 0.242, 2 workers and 50 users 0.216, 4 workers and 10 users 0.258, 4 workers and 50 users 0.22. Middle: median and 95th percentile seconds per run in each cell, against a median of 3.01 s for one user alone, who finished 0.269 runs each second. Right: output tokens each second from Ollama with 1 to 8 requests at one time, 49, 22, 18, 19. 10 users at once 50 users at once one user alone 0 0.1 0.2 0.3 0.4 1 2 4 runs finished each second uvicorn workers 0 50 100 150 200 250 1 2 4 seconds per run dot median, bar to p95 0 20 40 60 1 2 4 8 Ollama tokens each second requests at one time
More uvicorn workers did not raise throughput, because every run waited for the one Ollama server. The left panel is runs finished each second, by workers. The middle panel is seconds per run, with a dot at the median and a bar to the 95th percentile. The right panel is Ollama alone, with 1 to 8 requests at once.

Throughput did not move. Ten users finished 0.242 to 0.259 runs each second, and fifty users 0.208 to 0.220, with 1, 2, or 4 workers. One user alone finished 0.269. Every extra user only waited longer. A median run took 3.0 s alone, 36 to 38 s with ten users, and 146 to 165 s with fifty.

The wait was in front of the model. In a probe of Ollama alone, eight requests sent at once came back one after another, about one second apart. One request alone produced 45 to 50 output tokens each second. With 4 or 8 requests at once, all of them together produced 17 to 20. Each of those requests reported 0.45 to 1.34 s of load_duration, against 0.13 s alone. We did not find the cause of that drop.

Postgres was far from its limit. The busiest cell had 48 connections, with 4 workers, and each run wrote about 20 KB of checkpoint rows.

Fifty users broke the clients first. In each 50-user cell, 46 to 49 of the 50 sessions ended with ReadTimeout. The stream sent no byte for 120 s while the run waited for the model.

The runs did not stop with them. In the 1-worker cell the server ended 107 runs as done, 29 as interrupted, and 19 as cancelled at shutdown. The clients saw 102 done and 5 interrupted. With an SSE comment line every 15 s, the same 1-worker cell lost no session. It had 0 read timeouts in 200 requests, at 0.231 runs each second. Section 9.1 has that run and its code.

Our position: on one model server, extra uvicorn workers buy you availability and nothing else. Measure the model server alone before you size the service, and add model capacity for throughput. The 409 column is the model again: in each cell 1 to 4 users got no refund proposal, so their approve had nothing to resume.

The cost of a conversation

A price per conversation needs three numbers: the model calls in a conversation, the input tokens, and the output tokens. Each must come from the provider usage of every call, as section 9.4 showed. The spans missed a fifth of the input tokens there.

We measured the 20 evals on the Mac, with token counts from a callback on every model call. Claude Haiku 5.5 ran once. The Qwen rows are the first run of each model. The Qwen3 4B row counts only calls inside spans, because its usage log was not on yet. Case 9 calls no model, so each row has 19 conversations. Raw runs

ModelModel calls per conversation, median and maxInput tokens, median and meanOutput tokens, median and meanModel seconds per conversation, median
Claude Haiku 5.52, 141,989, 4,004214, 3944.01
Qwen3 14B2, 12992, 1,79960, 11312.97
Qwen3 4B, spans only2, 81,166, 1,34450, 661.26

The counts differ by tokenizer as well as by behaviour. The first model call of one FAQ question had 912 input tokens on Claude Haiku 5.5. It had 557 on Qwen3 4B and 454 on Qwen3 14B. The two Qwen models share a tokenizer, so the gap between them is their chat templates.

The whole Haiku suite used 64 calls, 76,076 input tokens, and 7,493 output tokens. At the list price that was $0.011353. That is the money that left. The price of 1,000 such conversations follows from the per-conversation counts and the list prices of 2026-11-14, standard tier, no caching.

Model$ per MTok, input and output$ per 1,000 conversationsThe costliest conversation
Claude Haiku 5.50.10, 0.500.61$0.0028
Claude Haiku 4.51.00, 5.006.06$0.0276
Claude Sonnet 5.52.00, 10.0012.11$0.0552
Claude Sonnet 4.63.00, 15.0018.17$0.0829
Claude Opus 5.54.00, 20.0024.22$0.1105

Only the first row is measured. The others put Haiku 5.5 counts at other prices. Another model makes other calls, and the pricing page says that Sonnet 4.6 and earlier use an older tokenizer that gives about 30% fewer tokens. Read those rows as estimates. The mean includes the runaway and the six-turn thread, which are the two costliest conversations. Your traffic sets the mix.

For the local models the price is the machine. Qwen3 14B spent a median of 12.97 s of model time per conversation, and Qwen3 4B 1.26 s. The load test above shows how many of those the Mac serves at once.

LangChain (Python): langchain, langchain-core, langgraph, deepagents, langsmith 1.5.0 · c2f4036 · 2026-11-11 · Edition 2026.11