Should Concurrency Limits Be Set per Model, Endpoint, API Key, or Customer?

Overview

Use layered limits. The provider’s account-level ceiling protects WaveSpeedAI capacity, while your application still needs per-customer fairness, model-specific protection, and queue controls. A single global limit lets one expensive workflow consume slots needed by every other tenant.

Assign each limit a job

Read the current account documentation and runtime responses to determine which provider constraints apply. Do not infer that limits are global, per key, per endpoint, or independently allocated unless the current source states so. Multiple API keys can separate credentials, but they should not be treated as evidence of additional capacity.

Inside your system, add:

per-customer limits to prevent noisy-neighbor behavior;
per-model limits for slow or costly endpoints;
per-endpoint limits when operations have different resource profiles;
global headroom for retries, support tests, and priority work.

Use a token bucket for request rate and a semaphore or worker pool for concurrency. Queue accepted work instead of firing until 429 responses appear. Record queued, running, completed, and dead-letter states separately.

Choose limits from service goals

Small products can start with one queue and conservative tenant caps. Larger systems need weighted fairness, priority classes, and capacity-aware routing. Measure queue wait, task duration, rejection rate, and slot utilization by model.

Keep the promise realistic

Your customer plan should allocate a share of capacity, not mirror the provider’s theoretical maximum. Leave operational headroom and update limits when task duration or model mix changes.