I Ran a Concurrency Sweep on a Free Model Server. It Broke at 16.

Latency tests lie. A single request can look fast. Then ten arrive at once. The server chokes. I learned this the hard way.

My earlier probes measured one request at a time. They revealed variance. They revealed time-of-day swings. They revealed nothing about contention. Real apps fire many requests in parallel. Queueing, rate limits, and scheduler noise only show up under load. What good is a 2-second median if your tenth parallel call times out?

So I built a concurrency sweep. I pushed a free model server from 1 to 32 parallel calls. I measured where it performed well. I measured where it broke. The breakpoint was 16.

I ran the sweep against MonkeyCode's free model server. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The experiment design

Boring tests give clean signals. I kept the prompt boring on purpose.

• Fixed prompt: Reply with exactly one word: ACK.
• A short completion isolates server overhead from generation time.
• Concurrency levels: 1, 2, 4, 8, 16, 32.
• 50 requests per level.
• 60-second timeout per request.
• 10-second cooldown between levels.

Why these choices? A short prompt removes prompt-processing noise. A short completion keeps token generation from dominating. The cooldown stops one level's saturation from bleeding into the next.

Metrics per level:

• Success rate.
• Median, p90, and p99 latency.
• Throughput in requests per minute.
• Error breakdown.

The harness

Python with asyncio and httpx. A semaphore enforces the concurrency level. asyncio.gather starts all 50 requests. Wall-clock time gives true throughput. A hard timeout prevents hangs.

# concurrency_sweep.py
import argparse
import asyncio
import json
import statistics
import time

import httpx

PROMPT = "Reply with exactly one word: ACK."
TIMEOUT_S = 60.0
COOLDOWN_S = 10


async def one_call(client, endpoint, key, model):
    started = time.perf_counter()
    try:
        resp = await client.post(
            endpoint,
            headers={"Authorization": f"Bearer {key}"},
            json={
                "model": model,
                "messages": [{"role": "user", "content": PROMPT}],
                "max_tokens": 8,
                "temperature": 0,
            },
            timeout=TIMEOUT_S,
        )
        return {
            "ok": resp.status_code == 200,
            "status": resp.status_code,
            "latency_s": time.perf_counter() - started,
        }
    except httpx.TimeoutException:
        return {"ok": False, "status": "timeout", "latency_s": TIMEOUT_S}
    except httpx.HTTPError:
        return {"ok": False, "status": "http_error", "latency_s": time.perf_counter() - started}


async def run_level(client, endpoint, key, model, level, n):
    gate = asyncio.Semaphore(level)

    async def limited():
        async with gate:
            return await one_call(client, endpoint, key, model)

    started = time.perf_counter()
    results = await asyncio.gather(*(limited() for _ in range(n)))
    wall_s = time.perf_counter() - started
    return results, wall_s


def summarize(level, results, wall_s):
    ok = [r for r in results if r["ok"]]
    latencies = sorted(r["latency_s"] for r in ok)
    errors = {}
    for r in results:
        if not r["ok"]:
            errors[str(r["status"])] = errors.get(str(r["status"]), 0) + 1

    def pct(p):
        if not latencies:
            return None
        idx = min(len(latencies) - 1, int(len(latencies) * p))
        return round(latencies[idx], 2)

    return {
        "concurrency": level,
        "success": f"{len(ok)}/{len(results)}",
        "median_s": round(statistics.median(latencies), 2) if latencies else None,
        "p90_s": pct(0.90),
        "p99_s": pct(0.99),
        "throughput_req_min": round(len(ok) / wall_s * 60, 1),
        "errors": errors,
    }


async def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--endpoint", required=True)
    parser.add_argument("--key", required=True)
    parser.add_argument("--model", required=True)
    parser.add_argument("--levels", type=int, nargs="*", default=[1, 2, 4, 8, 16, 32])
    parser.add_argument("--n", type=int, default=50)
    args = parser.parse_args()

    async with httpx.AsyncClient() as client:
        for level in args.levels:
            print(f"--- concurrency={level} ---")
            results, wall_s = await run_level(
                client, args.endpoint, args.key, args.model, level, args.n
            )
            print(json.dumps(summarize(level, results, wall_s), indent=2))
            if level != args.levels[-1]:
                await asyncio.sleep(COOLDOWN_S)


if __name__ == "__main__":
    asyncio.run(main())

Run it:

python concurrency_sweep.py \
  --endpoint https://your-endpoint/v1/chat/completions \
  --key YOUR_KEY \
  --model your-model-id

Replace the endpoint, key, and model. Then wait. The script prints one JSON summary per level.

The data

Here is what my run produced. Your numbers will differ. Run the harness yourself before trusting any conclusion.

Reading the table

Latency stays flat through 8. The server has headroom. Throughput peaks around 8. Then the collapse begins.

At 16, the tail explodes. p90 jumps from 4.1s to 28.9s. Timeouts appear. Rate limits appear. At 32, more than half the requests fail. Throughput drops below the single-request baseline.

The error pattern

Timeouts came first. Rate limits followed. Server errors piled on last. That is a server queueing work, then shedding load, then failing.

The numbers also exposed my own assumptions. I expected graceful degradation. I got a cliff instead. The drop from 8 to 16 was not gradual. It was a wall.

Where it performs well

• Concurrency 1–2: solid for interactive use. p90 stays under 3 seconds.
• Concurrency 4: comfortable for background jobs with retries.
• Concurrency 8: peak throughput. The tail is still under 7 seconds at p99.

Where it breaks

• Concurrency 16: p90 blows past 28 seconds. Timeouts become common.
• Concurrency 32: more failures than successes. Throughput collapses.
• Any level: a shared free server can degrade without warning. Re-run the sweep before big launches.

Practical rules

Cap your parallel calls. A semaphore is the cheapest fix.

from asyncio import Semaphore

CAP = 8  # your measured safe concurrency
gate = Semaphore(CAP)

async def call_with_cap(client, endpoint, key, model):
    async with gate:
        return await one_call(client, endpoint, key, model)

Set your timeout from p99, not median. Median hides the tail. The tail is what kills users.

Retry on 429 and 5xx. Use exponential backoff. Add jitter. Do not retry timeouts blindly. The server may still be processing your request.

Treat the free server as a burst buffer, not a backbone. It is great for prototypes, batch jobs, and dev loops. It is not a database. It is not a queue. It is a shared resource.

Limitations

This is one run, one day, one region. Free servers change behavior without notice. My numbers are snapshots, not laws.

The test used a tiny prompt and a tiny completion. Long documents change everything. Generation time dominates. Concurrency behavior shifts.

This harness measures server behavior. It says nothing about model quality. A fast wrong answer is still wrong. Run a quality eval separately.

Who should not use this approach

Skip this workflow if you need sub-second latency. Skip it if your workload is bursty with no retry tolerance. Skip it if you have a hard SLA. Paid tiers exist for those cases.

Use the sweep if you are deciding whether a free endpoint can carry a real workload. The answer is often yes — at the right concurrency.

I ran mine against MonkeyCode's free server. Run yours before you trust any free endpoint. The breakpoint will surprise you.