Budgets, caps, and quotas · Shipping AI Systems

Every LLM product has a wallet, and the wallet has a bottom. The stories of Rs 200,000 overnight bills come from projects that had no limits — one loop, one script, one Facebook share turned a working app into a bankruptcy in eight hours. This section is the three hard limits that keep that from happening. Every AI product ships with all three, or it does not ship at all.

The three limits

Every LLM service, no matter how small, needs:

A total budget cap — a hard ceiling on monthly spend. If you cross it, the service stops calling the model.
A per-user quota — a limit on how much any single user can consume.
A per-request cap — a limit on the size of any single call.

These layers are complementary. The per-request cap stops single-call abuse. The per-user quota stops a single account from abusing 10,000 calls. The budget cap catches everything that slips past the first two.

Skip any one and you have a hole. Ship all three and the worst-case bill is bounded.

Layer 1 — Total budget cap

The strongest limit, and the easiest to skip because it feels like giving up on your product.

At the provider level. Both Anthropic and OpenAI let you set a monthly spending cap in the dashboard. Do this before writing a line of code. It is the fire door that guarantees your service stops when the wallet empties, regardless of any bug in your own code.

Anthropic Console → Billing → Usage limits.
OpenAI dashboard → Billing → Usage limits.

Set the cap to a number you can genuinely afford to lose. For a small project, this is often “$50/month” or “$200/month” — deliberately not “$5,000.”

At the app level. Complement the provider cap with your own tracking. Log every call’s cost estimate to a database. A cron job (or middleware) checks the running total and stops accepting new calls when the budget is reached. Show users a clear message: “Daily budget reached; service will resume at midnight NPT.”

Two layers means one failing doesn’t sink you.

Layer 2 — Per-user quota

Some users, some scripts, some bots will send far more requests than average users. Cap how much any single user can consume.

A minimal in-Python quota using a simple key-value store (Redis):

import redis
import time

r = redis.Redis()

DAILY_CALL_LIMIT = 100
DAILY_TOKEN_LIMIT = 50_000


def check_and_record_usage(user_id: str, tokens_used: int) -> bool:
    """Return True if within quota; False if the user has exceeded it today."""
    day = time.strftime("%Y-%m-%d")

    calls_key = f"quota:{user_id}:{day}:calls"
    tokens_key = f"quota:{user_id}:{day}:tokens"

    calls = int(r.get(calls_key) or 0)
    tokens = int(r.get(tokens_key) or 0)

    if calls >= DAILY_CALL_LIMIT or tokens + tokens_used > DAILY_TOKEN_LIMIT:
        return False

    r.incr(calls_key)
    r.incrby(tokens_key, tokens_used)
    r.expire(calls_key, 86_400 * 2)
    r.expire(tokens_key, 86_400 * 2)
    return True

In your endpoint:

@app.post("/summarise")
def summarise(req: SummariseRequest, user_id: str = Depends(get_current_user)):
    if not check_and_record_usage(user_id, estimated_tokens(req)):
        raise HTTPException(
            status_code=429,
            detail="Daily quota reached. Please try again tomorrow.",
        )
    # ... normal handling

Two dimensions of quota: call count (prevents runaway loops) and token consumption (prevents one user submitting huge inputs many times). Track both.

Free tier vs. paid tier. In a real product, you likely have two quota tiers: a small free-tier daily limit and a larger paid-tier limit. The check is the same; only the number changes based on the user’s plan.

Layer 3 — Per-request cap

The last line of defense: no single call can exceed a defined size.

MAX_INPUT_CHARS = 10_000
MAX_OUTPUT_TOKENS = 500

@app.post("/summarise")
def summarise(req: SummariseRequest):
    if len(req.text) > MAX_INPUT_CHARS:
        raise HTTPException(400, "Text exceeds maximum length.")

    response = client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=MAX_OUTPUT_TOKENS,   # hard cap on output length
        messages=[{"role": "user", "content": build_prompt(req)}],
    )
    ...

Input cap — reject oversized input at the boundary. Ten thousand characters is more than any legitimate summarisation needs; a 5MB payload is almost always abuse or a bug.
Output cap — max_tokens in the API call. This is not just for cost; long outputs also make responses slow and reduce quality. Set it low unless you specifically need long outputs.

What “unusual” looks like

Once your quotas and caps are in place, log the near-misses. A user who hits 95% of the daily quota on three consecutive days is either genuinely heavy-use (worth reaching out about a paid tier) or bot behavior (worth investigating).

if calls > DAILY_CALL_LIMIT * 0.9:
    log_event("quota_near_limit", user_id=user_id, calls=calls)

Small habit; large payoff over months.

Rate limits, briefly

The slowapi library (introduced in Section 1.2) handles rate limits cleanly:

from slowapi import Limiter
from slowapi.util import get_remote_address

limiter = Limiter(key_func=get_remote_address)

@app.post("/summarise")
@limiter.limit("10/minute")     # by IP
def summarise(request: Request, req: SummariseRequest):
    ...

Better: rate-limit by user ID once you have authentication:

def get_user_key(request):
    return request.state.user_id

limiter = Limiter(key_func=get_user_key)

Ten requests per minute per user is a reasonable default for interactive AI services. Increase for legitimate high-volume use cases with authentication.

Putting it together — the layer stack

For an endpoint that calls a model, the guard sequence looks like this:

@app.post("/summarise")
@limiter.limit("10/minute")
def summarise(req: SummariseRequest, user_id: str = Depends(get_current_user)):
    # Layer 3 — per-request
    if len(req.text) > MAX_INPUT_CHARS:
        raise HTTPException(400, "Text too long.")

    # Layer 2 — per-user quota
    if not check_and_record_usage(user_id, estimated_tokens(req)):
        raise HTTPException(429, "Daily quota reached.")

    # Layer 1 — total budget (in middleware, before the request even arrives)
    # handled by budget_middleware() above

    response = client.messages.create(
        model=DEFAULT_MODEL,
        max_tokens=MAX_OUTPUT_TOKENS,
        messages=[...],
    )
    ...

Every LLM-facing endpoint should look approximately like this. Every one.

Communicate the limits

When a user hits a limit, be honest with them:

“You have reached your daily limit of 100 messages. It resets at midnight NPT.”
“This service is over its monthly budget. It will resume on the 1st. Sorry.”
“Your input is too long (12,000 chars). Please shorten to 10,000 or fewer.”

Users tolerate limits when they understand them. Silent failures produce distrust. Mysterious errors produce anger. A clear, friendly quota message keeps the relationship intact.

Check your understanding

Quick check

—

A team ships a chatbot with 'we'll set limits if we need them.' A friend shares it on Facebook and it goes viral overnight. What is the honest expected outcome?

The chatbot works fine; more users is good news.
Without a total budget cap, per-user quota, or per-request cap, one viral moment can burn through the provider account balance in hours. The team wakes up to a Rs 200,000 invoice (or worse: locked-out API access after their card was declined), and the beautiful chatbot is now a cautionary tale. The 'we'll set limits if we need them' plan is the plan that ends businesses. Every LLM-facing endpoint ships with all three layers, or it does not ship.
They should have hosted on a cheaper provider.
They should have used a smaller model.