Calling models from code

docs.scrimba.com

The last chapter built a list of messages. This chapter sends that list to a real model from Python and gets an answer back. That round trip is the smallest complete loop of building with these models, and almost every feature you will write sits on top of it.

You type a prompt, your code sends it somewhere, and text comes back. It feels like the model lives in your program. It does not, and seeing where it actually runs explains the cost, the speed, and the quirks you will meet.

So picture what is physically happening. The model is a huge set of parameters sitting on the provider's servers, on hardware you will never see. When you call it, your code sends an ordinary web request over the internet carrying your messages.

The provider runs the prediction loop on their machine and sends the generated tokens back to you. A model call is renting a few moments on someone else's computer. That framing makes sense of a lot: latency is the model generating tokens one at a time on their end, cost is them charging for that compute, and the whole thing is a network call, with everything that implies.

The examples use the official OpenAI library. Other providers differ in details, but the shape is nearly the same everywhere: you send a list of messages, you get a message back. That sameness is your insurance against lock-in, since the same code can point at an open model you host or rent elsewhere, often with only a change of base URL (Open and closed models covers that choice).

You call a library function, a few seconds pass, and text comes back. The library makes it feel local, but nothing about the model is on your machine.

The move to understand is what that function is wrapping. Your code packages the request as JSON, opens an HTTPS connection to the provider, and waits while their hardware runs the prediction loop and streams tokens back. The SDK (the provider's software library, like openai) is a thin convenience layer over one POST request.

Knowing that the wire format underneath is plain JSON pays off in practice. When a call behaves oddly, you can inspect the exact request and response. When you want a feature the SDK has not exposed yet, you can send the field by hand.

The whole thing being a network call is what drives the cost, latency, and error handling for the rest of this chapter. The examples use OpenAI, but the request shape, a list of messages in, a message out, is close to identical across providers.

A model call looks like a function call and behaves like a remote procedure over flaky infrastructure. That mismatch is where production problems come from, so it is worth holding the real picture from the start.

Underneath the SDK, every call is one HTTPS POST: a JSON body goes up, the provider runs inference on their hardware, and tokens come back, optionally streamed over the open connection. The SDK (the provider client library) buys you auth, retries, typing, and streaming parsing, and it costs you a layer of abstraction over churn you do not control. Treat it as a wrapper you can see through, not a black box.

Two consequences set up the rest of the chapter. First, this is a network call billed by token, so cost, latency, idempotency, and failure handling are design concerns, not afterthoughts you bolt on later. Second, the SDK and its field names are the most volatile surface you depend on, so the durable move is to isolate the provider-specific parts behind your own boundary. The examples use OpenAI; the request shape is close to universal, but the wrapper around it is exactly what you do not want spreading through your codebase.

The request

Install the library with pip install openai, create a client, and call it. The client reads your API key from an environment variable, so the key never appears in your code.

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment

MODEL = "gpt-4o-mini"  # the one line you change to swap models

response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a concise assistant. Answer in one sentence."},
        {"role": "user", "content": "Why is the sky blue?"},
    ],
)

Two pieces are required: model, the name of the model you want to run, and messages, the list of role-tagged messages from the last chapter. Everything else has a default.

Notice the model name lives in a single constant. In a field where a better, cheaper model lands every few months, that is deliberate. Keep the model name in one place so swapping models is a one-line change. This is the churn rule in practice: keep the volatile specifics where you can find them, not scattered through your code.

JunoThe request A call needs only two things: model and messages. The model name belongs in a single constant, because a cheaper or better one shows up every few months and you do not want to go hunting for it. Took me a while to learn that one the slow way, after a model got renamed and I found it pasted in nine files.

Two fields are required, model and messages, and everything else has a default. The model name in a constant is not a style point: it is the one volatile detail you want in one place, because a cheaper or better model arrives often enough that hardcoding it across files turns a swap into a hunt.

Now look one level down. That create call serialises to a JSON body and POSTs it. Spelled out, the wire format is the literal request the provider receives:

import httpx, os

# the same request the SDK sends, by hand
resp = httpx.post(
    "https://api.openai.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"},
    json={
        "model": "gpt-4o-mini",
        "messages": [
            {"role": "system", "content": "You are a concise assistant."},
            {"role": "user", "content": "Why is the sky blue?"},
        ],
    },
    timeout=20,
)
data = resp.json()  # same shape the SDK parses into objects

You will reach for the SDK in real code: it handles auth, retries, and typing for you. But seeing that the body is a plain dict makes the rest concrete. The URL and field names here are OpenAI's, and the exact endpoint and keys vary by provider, but the move, JSON in, JSON out, is the same everywhere.

JunoThe request Required fields are model and messages, nothing else. The SDK create call is one JSON POST under the hood, so when something looks wrong you can drop to the wire and inspect the literal body. Keep the model name in a constant: it is the part that churns, and you want it in one place.

Required fields are model and messages. The interesting question is not what they are but where they live in your code, because model is the single most volatile value in the request and the SDK is the most volatile dependency around it.

The create call serialises to a JSON POST. That matters because it means the request is data you own, not magic the SDK performs. You can build the body, log it, diff it across runs, or send it by hand when the SDK lags a new provider feature. Knowing the wire format is what lets you debug a call the abstraction is hiding from you.

The production habit is to route every call through one narrow function of your own:

def complete(messages, *, model=DEFAULT_MODEL, **params):
    # one choke point: swap providers, add logging, change retries here
    return client.chat.completions.create(model=model, messages=messages, **params)

That choke point is where a model swap becomes a one-line change and where provider-specific churn stays contained instead of leaking into every caller. The endpoint path and field names shown are OpenAI's; another provider renames the route and some keys, so the value of the boundary is that the rename touches one file, not the whole codebase. Pin the model version too: "latest" silently moving underneath you is the same regression risk covered in How LLMs work, now in your request body.

JunoThe request Only model and messages are required, and both deserve to live behind a boundary you control. Route every call through one function: that is where a model swap is one line, where logging goes, and where provider churn stops before it reaches your callers. Pin the model version while you are at it, because "latest" moving underneath you is a regression you did not deploy and cannot see.

Reading the response

The reply comes back wrapped in some structure. The text you want is one path in, but the rest of the structure is worth knowing, because it tells you what happened.

choice = response.choices[0]

print(choice.message.content)   # the reply text
print(choice.finish_reason)     # "stop"  -> the model finished on its own
print(response.usage)           # Usage(prompt_tokens=24, completion_tokens=18, total_tokens=42)

Three things in there matter. message.content is the text, sitting inside choices[0] because the API can return more than one option, though you almost always ask for one and read the first.

finish_reason tells you why the model stopped. "stop" means it finished its thought, while "length" means it ran into your token cap and got cut off mid-answer. Checking that field is how you detect a truncated reply in code rather than by eye.

And usage reports the tokens the call spent, split into the prompt you sent and the completion it wrote. That usage object is your bill, itemised. Every call tells you exactly what it cost in tokens, which is the number to watch as you build.

JunoReading the response The reply text is at response.choices[0].message.content, but two neighbours matter too. finish_reason tells you why it stopped: "stop" means done, "length" means it hit your token cap and got cut off. And response.usage is your bill in tokens, prompt and completion split out, so you never have to guess what a call cost.

Three fields carry the signal. choices[0].message.content is the text. finish_reason is why the model stopped: "stop" is a clean finish, "length" means it hit your token cap and got truncated, so checking it is how you catch a cut-off reply in code instead of shipping half an answer. And usage is the token count, prompt and completion separated.

The move that pays off here is turning usage into real money. The usage object is per-call billing data, so you can compute cost on the spot:

# provider prices are per-token; check current rates, they change
PROMPT_RATE = 0.15 / 1_000_000      # dollars per prompt token
COMPLETION_RATE = 0.60 / 1_000_000  # dollars per completion token

u = response.usage
cost = u.prompt_tokens * PROMPT_RATE + u.completion_tokens * COMPLETION_RATE
print(f"{u.total_tokens} tokens, ${cost:.6f}")

Prompt and completion are usually priced differently, and completion tokens often cost more, so the split matters when you optimise. Log the cost per call and you can see which features are expensive before the monthly invoice does the telling. The field names here are OpenAI's; other providers expose the same counts under slightly different keys, so read the shape, not the exact attribute.

JunoReading the response Read three fields: content for the text, finish_reason to catch a "length" truncation in code, and usage for tokens. Turn usage into dollars on the spot, prompt and completion priced separately, and log it per call. You will spot the expensive feature long before the invoice does.

The response carries three fields you act on: content, finish_reason, and usage. The one builders skip and regret is finish_reason. A "length" value means the model hit your max_tokens and the output is truncated, often as malformed JSON or a half-finished sentence that your downstream parser then chokes on. Treat any value other than "stop" as a failed call and handle it, do not assume content is complete.

The usage object is your unit-economics telemetry, and it deserves to be wired into observability, not printed and forgotten. Prompt and completion tokens are priced separately, completion usually higher, so emit both per call tagged by feature:

u = response.usage
log.info("llm_call", feature="invoice_extract", model=MODEL,
         prompt_tokens=u.prompt_tokens, completion_tokens=u.completion_tokens,
         finish_reason=response.choices[0].finish_reason)

With that in place, cost per feature and per user becomes a query, not a guess, and a runaway prompt that quietly tripled in size shows up as a metric instead of a surprise charge. Watch the asymmetry: input tokens are cheap and parallel to process, output tokens are the expensive, serial part covered in How LLMs work, so a feature that emits long replies costs and lags more than its prompt size suggests. The attribute names are OpenAI's and shift by provider, so log the counts behind your own boundary rather than scattering response.usage.prompt_tokens through the app.

JunoReading the response Treat any finish_reason that is not "stop" as a failed call: a "length" truncation hands your parser broken JSON and you will chase the bug downstream. Wire usage into your logs tagged by feature, prompt and completion split, so cost per feature is a query and a ballooning prompt is a metric, not a billing surprise. Output tokens are the expensive serial part, so a chatty feature costs more than its prompt suggests.

The parameters worth knowing

Beyond model and messages, two optional settings come up constantly.

temperature is the randomness dial from How LLMs work, made concrete. Low values (near 0) keep the model close to its top picks: focused and repeatable, what you want for extraction and classification. Higher values (around 0.7 to 1) let it reach for less likely tokens: more varied and creative, and more prone to wander. You are tuning how boldly the model samples, nothing more.

max_tokens caps how many tokens the reply can run to. It protects you from a runaway answer and a runaway bill. Set max_tokens too low and the model gets cut off, which is exactly the "length" finish reason from the last section, so leave real headroom.

response = client.chat.completions.create(
    model=MODEL,
    messages=messages,
    temperature=0.2,  # focused and consistent
    max_tokens=300,   # cap the reply length
)

JunoThe parameters worth knowingtemperature tunes how boldly the model samples: low for focused, repeatable answers, higher for varied, creative ones. max_tokens caps the reply length to protect your bill, but set too low it triggers the "length" cutoff, so leave headroom. For extraction and classification, reach for a low temperature every time.

Two parameters carry most of the weight: temperature and max_tokens. The mechanics are in How LLMs work; the move here is matching the value to the task instead of trusting whatever default ships.

Think of temperature as a per-task setting, not a global one. Pick it from what the task needs:

TEMPERATURE_BY_TASK = {
    "extraction": 0.0,      # one right answer, want it every time
    "classification": 0.0,  # pick a label, no creativity wanted
    "summary": 0.3,         # mostly faithful, a little phrasing freedom
    "brainstorm": 0.9,      # variety is the whole point
}

For anything you parse afterwards, structured data, a category, a number pulled from a document, go to 0 and stop reasoning about randomness. For drafting and ideation where variety is the value, go high.

max_tokens is a ceiling, not a target: it caps spend and runaway replies, but set it under what the task needs and you get the "length" truncation from the last section, which is worse than a slightly larger bill. Size it to the longest legitimate answer plus headroom. These two field names are stable across providers; some rename max_tokens, so confirm the key when you switch.

JunoThe parameters worth knowing Set temperature per task, not once globally: 0 for anything you parse afterwards, high for drafting where variety is the point. max_tokens is a safety ceiling, so size it to the longest real answer plus headroom; set it too tight and you trade a small bill saving for a truncated reply. Stop trusting the default the SDK ships.

temperature and max_tokens look like dials and behave like policy decisions with cost and reliability attached. The depth here is in how they interact with everything else you ship.

Treat temperature as part of your reliability contract. For any output you parse or store, run at 0 so the sampling-driven variance from How LLMs work is as small as the provider allows. It is not perfectly deterministic even at 0, so do not build exact-match caching or byte-identical tests on top of it, but it is the floor you want for structured work. Reserve higher temperatures for surfaces where variety is the product, and treat that variance as something your evals measure across runs, not a single pass you eyeball.

max_tokens is a latency and cost lever, not only a safety net. Output tokens are the serial, expensive part of generation, so capping length bounds both your worst-case bill and your worst-case wait.

The trap is sizing it by eye: too tight truncates valid answers into the "length" failure that hands your parser broken output, too loose lets a degenerate loop run up cost and latency before anything stops it. Size it to the longest legitimate answer plus margin, per task, and log when you hit the cap so a feature that keeps truncating becomes visible. Both parameter names are OpenAI's and a couple of providers rename them, so set them behind the boundary function rather than at every call site.

JunoThe parameters worth knowingtemperature is a reliability contract: 0 for anything you parse, higher only where variance is the product, and never exact-match cache on top of 0 because it is not byte-deterministic. max_tokens bounds your worst-case cost and latency, so size it to the longest real answer plus margin and log when you hit it, because a feature that keeps truncating is a bug hiding as a parameter. Set both behind your boundary, not at every call site.

Streaming

By default you wait for the whole answer to finish generating, then it arrives at once. Since the model produces tokens one at a time, a long reply means a long wait staring at nothing.

Streaming sends each token down to you the moment it is generated, so text appears right away and fills in live, the way chat apps show a reply typing itself out.

stream = client.chat.completions.create(
    model=MODEL,
    messages=messages,
    stream=True,
)

for chunk in stream:
    piece = chunk.choices[0].delta.content
    if piece:
        print(piece, end="", flush=True)

The total generation time is the same either way; the model is not faster. Streaming only changes when you see the tokens, surfacing them as they are produced instead of holding them back until the end.

You handle it by iterating over chunks and printing each delta of text. The trade-off is a little more code for a much better wait. Streaming changes when you see tokens, not how fast they arrive.

JunoStreaming The model generates tokens one at a time, so by default a long reply is a long silent wait. Streaming forwards each token the moment it lands, so the answer fills in live, which is why chat apps feel responsive. Same total time, you only see the tokens sooner by iterating over chunks and printing each delta.

Streaming forwards each token as the model produces it, so the user sees text immediately instead of waiting for the full reply. Total generation time is unchanged; what drops is time to first token, which is most of what "feels fast" means.

The clean version handles two things the toy loop skips: chunks with no text, and accumulating the full reply while you display it.

stream = client.chat.completions.create(model=MODEL, messages=messages, stream=True)

full = []
for chunk in stream:
    delta = chunk.choices[0].delta
    piece = delta.content
    if piece is None:          # role-only and final chunks carry no text
        continue
    full.append(piece)
    print(piece, end="", flush=True)

reply = "".join(full)          # the complete message, for history or storage

The empty delta check matters: the first chunk often carries the role and no text, and the final chunk can carry a finish_reason with empty content. Skip the None and you avoid printing "None" into your output.

Accumulating into full gives you the whole message to append to history, since streaming displays text but does not save it for you. One trade-off to know: a streamed call does not return a usage block by default, so if you need token counts you request them explicitly or count them yourself. Streaming exists across providers, but the chunk shape, delta, content, where finish_reason lands, is provider-specific.

JunoStreaming Streaming cuts time to first token, not total time. Guard for the empty delta: the first chunk is often role-only and the final one carries finish_reason with no text, so skip None or you print "None" into the reply. Accumulate the pieces yourself for history, and remember a streamed call skips usage unless you ask for it.

Streaming trades a simpler response for a better latency profile: you cut time to first token, but you give up the single clean response object and take on assembling state as chunks arrive. Worth it for anything a human watches, rarely worth it for a backend call you parse whole.

The details that bite in production are at the chunk boundaries. The first chunk is typically role-only, the final chunk carries finish_reason with empty content, and you must accumulate text yourself because nothing hands you the assembled message:

full, finish = [], None
for chunk in stream:
    choice = chunk.choices[0]
    if choice.delta.content:
        full.append(choice.delta.content)
    if choice.finish_reason:        # arrives on the last chunk
        finish = choice.finish_reason
reply = "".join(full)
if finish == "length":              # truncation still happens mid-stream
    handle_truncated(reply)

Three production realities:

• finish_reason still arrives at the end of a stream, so the "length" truncation does not go away when you stream, you only notice it later, and you check it before trusting the assembled text.
• A stream can break mid-flight: the connection drops after ten tokens of forty, leaving you a partial reply, so streamed calls need their own partial-failure handling and are awkward to retry cleanly.
• Streamed responses usually omit usage, so you pass a flag to include it or tokenize the assembled text yourself for cost tracking.

The SSE framing and chunk schema are provider-specific, so keep stream parsing behind the same boundary as the rest, and let callers see your assembled result, not raw chunks.

JunoStreaming Streaming buys time to first token and costs you the clean response object, so use it where a human is watching and skip it for backend parsing. The final chunk still carries finish_reason, so a "length" truncation is real mid-stream, check it before trusting the text. And a stream can die after ten tokens of forty, so handle the partial reply and keep the provider-specific chunk parsing behind your boundary.

Every call is independent

This is the most important behaviour in the chapter, and it follows straight from the context window lesson. The API is stateless: each request stands completely on its own. The provider's server does not remember your previous call. It runs the prediction on exactly the messages you sent, returns the result, and keeps nothing about the exchange for next time.

So the conversation is not stored anywhere by the model or the API. It lives in your code. If you want turn two to know what happened in turn one, you keep the running list of messages yourself and send the whole thing again. A back-and-forth chat is an illusion you create by resending an ever-growing history on every call.

That is also why a long conversation costs more per message over time: each call re-sends all the prior turns as input tokens, so the prompt keeps growing even when the user's new question is short.

history = [
    {"role": "system", "content": "You are a friendly travel assistant. Keep answers short."},
]

def chat(user_text):
    history.append({"role": "user", "content": user_text})

    # the ENTIRE history goes up every time; the server remembers nothing
    response = client.chat.completions.create(model=MODEL, messages=history)
    reply = response.choices[0].message.content

    history.append({"role": "assistant", "content": reply})  # keep the reply for the next turn
    return reply

chat("I have a weekend in Lisbon. What should I see?")
chat("Which of those is best for kids?")  # only works because history carries turn one

The second question makes sense only because the first question and its answer are still in history and get sent along. You are the memory. This one idea, that you assemble and resend the full context every time, underlies conversation, and later it underlies tool use, retrieval, and agents too. They are all variations on deciding what goes into that messages list before each independent call.

JunoEvery call is independent The API is stateless: each request runs on exactly the messages you send, and the server forgets the exchange the moment it replies. A conversation lives in your code, not the model, so you keep the growing message list and resend all of it every turn. That is why long chats cost more over time, each call re-sends every prior turn as input tokens.

The API is stateless: the server runs on exactly the messages you send and keeps nothing afterward. The conversation lives in your code, and you resend the whole history every turn to fake continuity, which is why a long chat costs more per message even when the new question is short.

That growth is a problem you manage, not one you can ignore, because the history eventually outgrows the context window. The first move is a sliding window: keep the system message plus the most recent turns.

def trim(history, keep_recent=10):
    system = history[:1]               # always keep the system message
    recent = history[1:][-keep_recent:]  # newest turns only
    return system + recent

This is cheap and bounds your cost, but it silently drops older turns, so the model forgets facts from early in the conversation that the user still expects it to know. When that matters, the next step up is summarising the dropped turns into one short message instead of deleting them outright. Which you pick depends on the feature: a support bot that references the original issue needs the summary, a quick Q and A is fine with the window. The statelessness itself is universal across providers; only the message format around it varies.

JunoEvery call is independent Stateless means the server forgets after every reply, so you resend the whole history and pay for it as it grows. Trim with a sliding window of recent turns plus the system message to bound cost, but know it silently drops early facts the user still expects you to remember. When that bites, summarise the dropped turns instead of deleting them.

The API is stateless: you own the conversation and resend it every call. The naive version, append forever and send the lot, fails two ways at scale: it eventually overflows the context window, and it makes every turn more expensive and slower than the last, since input grows without bound.

So history management is a budgeting problem with three standard strategies, each failing in its own way:

• A sliding window keeps the most recent N turns: cheap and bounded, but it silently drops early facts the user still references.
• Summarisation compresses old turns into a running summary: holds more history per token, but loses detail and can bake its own errors into every later turn.
• Retrieval stores turns externally and pulls back only the relevant ones: scales to long histories, but depends on your index returning the right pieces and adds a lookup.

Pick per use case and instrument it, because each one will eventually surface the wrong context.

def build_messages(system, history, user_msg, budget_tokens=6000):
    msgs = [system]
    running = count_tokens(system) + count_tokens(user_msg)
    for turn in reversed(history):           # newest first
        t = count_tokens(turn)
        if running + t > budget_tokens:
            break                             # older turns fall off the budget
        msgs.insert(1, turn)
        running += t
    msgs.append(user_msg)
    return msgs

Budget by token count, not turn count, since turns vary wildly in size and a turn-count window blows your context budget the moment someone pastes a document. The strategy is provider-independent; the tokenizer you count with is not, so count with the model's own. This is the same context-construction work that prompting and RAG build on, now on the conversation itself.

JunoEvery call is independent Stateless means history management is your budgeting problem, and "append forever" overflows the window and inflates every turn's cost. Pick a strategy by use case: a sliding window is cheap but drops early facts, summarisation holds more but bakes in its own errors, retrieval scales but leans on your index. Budget by token count not turn count, because one pasted document blows a turn-count window instantly.

When things go wrong

A model call is a network call to someone else's servers, so it fails in all the ways network calls fail. Design for it from the start.

• Errors and rate limits. Providers cap how many requests you can send in a window. Hit the cap and the call fails with a rate-limit error. The standard response is to wait and retry, lengthening the wait after each failure (this is called backoff) so you do not hammer a busy service.
• Timeouts. A call can hang. Set a timeout so one slow request does not freeze your app.
• Keys stay on the server. Your API key is a password that spends your money. Never put the API key in browser code. Anyone can open dev tools and read it. The browser talks to your backend, and your backend, holding the key, talks to the model.
• Assume any answer can be wrong. Even a successful call can return a hallucination or malformed output, so the next chapters are about making the result something your code can trust.

def ask_model(messages):
    try:
        response = client.chat.completions.create(
            model=MODEL,
            messages=messages,
            timeout=20,  # give up after 20 seconds
        )
        return response.choices[0].message.content
    except Exception as err:
        print("Model call failed:", err)
        return "Sorry, something went wrong. Please try again."

JunoWhen things go wrong A model call is a network call to someone else's servers, so expect rate limits, timeouts, and plain failures, and wrap every call in error handling with retry-and-backoff. Keep your API key on the server, never in the browser, because it spends real money and dev tools will read it. And treat even a successful reply as possibly wrong, which the next chapters help you handle.

A model call fails the way network calls fail: rate limits, timeouts, transient server errors. The move is to catch the specific failures and retry the ones worth retrying, instead of wrapping everything in a bare except that hides real bugs.

Catch typed exceptions and apply backoff, waiting longer after each failure so you stop hammering a busy service:

import time
from openai import RateLimitError, APITimeoutError, APIError

def ask_model(messages, retries=3):
    for attempt in range(retries):
        try:
            r = client.chat.completions.create(model=MODEL, messages=messages, timeout=20)
            return r.choices[0].message.content
        except (RateLimitError, APITimeoutError, APIError) as err:
            if attempt == retries - 1:
                raise
            wait = 2 ** attempt          # 1s, 2s, 4s: exponential backoff
            print(f"retry {attempt + 1} after {err}")
            time.sleep(wait)

The exception types tell you what to do: RateLimitError and APITimeoutError are transient and worth retrying, while a bad request (malformed messages, unknown model) is your bug and should fail loudly, not retry. Doubling the wait each attempt keeps you from piling onto a service that is already struggling. The SDK retries some failures for you, so check its defaults before adding your own layer.

Keep the key server-side, and treat a successful reply as possibly wrong, which the next chapters address. The exception class names here are OpenAI's; other SDKs raise their own, so map them at your boundary.

JunoWhen things go wrong Catch typed exceptions, not a bare except: RateLimitError and APITimeoutError are transient and worth a retry, a bad request is your bug and should fail loudly. Back off exponentially, doubling the wait, so you stop hammering a struggling service. Check whether the SDK already retries before you add your own layer, and keep the key server-side.

Failure handling is where this stops being a function call and becomes distributed-systems work. The call fails in known ways, rate limits, timeouts, transient 5xx errors, and the unsafe part is not the retry, it is retrying without thinking about what a retry costs.

The trap is idempotency (whether running an operation twice has the same effect as running it once). A naive retry after a timeout can double-bill you: the first request may have completed and generated tokens on the provider's side, the response never reached you, so your retry pays for a second full generation. For a read-style call that is wasted money, for anything with a side effect it is a duplicated action. Where the provider supports an idempotency key, send one so a retry is recognised as the same request; where it does not, make the surrounding operation safe to repeat.

import random, time
from openai import RateLimitError, APITimeoutError, APIError

RETRYABLE = (RateLimitError, APITimeoutError, APIError)

def complete(messages, *, retries=3, **params):
    for attempt in range(retries):
        try:
            return client.chat.completions.create(messages=messages, model=MODEL, **params)
        except RETRYABLE as err:
            if attempt == retries - 1:
                raise
            time.sleep((2 ** attempt) + random.random())  # backoff + jitter

Three more production realities:

• Jitter, the random fraction added to each wait, stops a fleet of clients from retrying in lockstep and re-spiking the rate limit (the thundering-herd problem).
• Connection reuse matters under load: create one client and share it, so connections are pooled instead of paying a fresh TLS handshake per call, and run independent calls concurrently rather than in a slow serial loop.
• Classify before you retry: a RateLimitError is transient, a malformed-request error is your bug and retrying it wastes time and money on a call that will never succeed.

The exception classes and the idempotency mechanism are provider-specific, so this whole policy belongs behind the boundary function, where you map each provider's errors to your own retry decision once. The key stays server-side, and a successful reply is still untrusted until validated, which structured output and the safety chapter handle.

JunoWhen things go wrong The retry is the dangerous part: a naive retry after a timeout can double-bill, because the first call may have generated tokens you never received, so use an idempotency key or make the operation safe to repeat. Add jitter to your backoff so a fleet of clients does not retry in lockstep and re-spike the limit, reuse one pooled client, and run independent calls concurrently. Classify first, retry transient errors and fail loudly on your own bad requests, and keep the whole policy behind one boundary so each provider's quirks map to your retry decision in one place.