Rate limits & concurrency
The platform enforces multiple limits for fairness and stability. Hitting one returns 429 rate_limit_exceeded.
Limit types
| Type | Description |
|---|---|
| Rate (RPM) | Requests per minute cap |
| Throughput (TPM) | Tokens per minute cap |
| Concurrency | Concurrent in-flight requests cap |
| Quota | Optional monthly spend cap per key |
Hitting a limit
- Returns
429 rate_limit_exceeded; theretry-afterheader suggests how long to wait. - Not billed: rate-limited requests incur no charge.
Pre-authorization
To prevent overspending, the platform conservatively reserves an amount before each request, settles against actual usage afterward, and releases the difference. Reservations are not final balance — only actual usage counts.
Recommendations
- Implement exponential backoff and honor
retry-after. - Contact support to raise limits for high-concurrency workloads.