Dedicated endpoints

Calling an endpoint

import os
from openai import OpenAI

# GRIDPORT_ENDPOINT_ID is the endpoint's id (ep_…), shown in the console.
client = OpenAI(
    base_url=f"https://gridport.ai/v1/endpoints/{os.environ['GRIDPORT_ENDPOINT_ID']}",
    api_key=os.environ["GRIDPORT_API_KEY"],   # a key of the endpoint's project
)
resp = client.chat.completions.create(
    model="zai-org/GLM-5.3",   # the model the endpoint serves
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

An endpoint answers at /v1/endpoints/<endpoint id>/ followed by chat/completions, completions, responses, messages or embeddings, so the OpenAI SDKs take the gateway’s /v1/endpoints/<endpoint id> as their base URL. The console’s Copy URL gives the chat completions address.

A client that cannot change its path, such as the Anthropic SDK, keeps its usual base URL and sends the endpoint’s id in the x-tf-endpoint-id header instead:

client = anthropic.Anthropic(base_url="https://gridport.ai", api_key=os.environ["GRIDPORT_API_KEY"],
    default_headers={"x-tf-endpoint-id": os.environ["GRIDPORT_ENDPOINT_ID"]})

Use an API key of the endpoint’s project and set model to the model the endpoint serves. Another project’s key or another model gets 404 model_not_found, a deleted endpoint 404 not_found. Cross-model fallbacks (tf.routing.fallback_models) are refused with 400 on an endpoint, and the response cache and the default output limit of serverless requests do not apply to it.

Endpoint state and cold start

A request the endpoint cannot serve right now is answered at once instead of being held open. The x-tf-endpoint-state header says why, and Retry-After when to try again:

503  x-tf-endpoint-state: cold_start    Retry-After: 5    sleeping; this request asked it to wake
503  x-tf-endpoint-state: waking        Retry-After: 5    a replica is starting
503  x-tf-endpoint-state: scaling       Retry-After: 5    its capacity is changing
503  x-tf-endpoint-state: maintenance   Retry-After: 30   recovering; billing is paused
503  x-tf-endpoint-state: wake_blocked  Retry-After: 60   sleeping; no GPU is free in its pool yet
402  x-tf-endpoint-state: wake_blocked                    sleeping; waking it cannot be funded
                                                          (insufficient_balance or budget_exceeded)
503  x-tf-endpoint-state: unavailable                     its deployment failed or did not answer
503                                                       stopped, or its prepaid window expired

An endpoint whose minimum replicas is zero sleeps after its idle time (Scaling & capacity in the console). The first request after that asks it to wake. The OpenAI SDKs retry a 503 twice by default, which can be shorter than a cold start, so keep retrying in your own code until x-tf-endpoint-state is gone; the console shows how long the wake has been waiting. A 402 wake_blocked is not retried: the pending wake proceeds once the balance or the budget allows it.

Rate limits

An endpoint is your own capacity, so the shared per-model limits do not apply to it. Your organization’s tier limits (org_rpm, org_tpm), the project’s caps and the key’s own limits still do, and the x-ratelimit headers report them. Under a capacity contract the contract’s terms take the place of the organization’s and the project’s limits: the response carries x-tf-capacity-contract and x-tf-capacity-tier, the x-ratelimit headers show the contract’s figures, and only the key’s own limits are counted. Rate limits

Billing

Before an endpoint starts, a prepaid window — 15 minutes of its GPUs by default — is reserved from paid or contract credit; trial and promotional credit do not count, and the organization’s and the project’s budgets apply. Once the endpoint is ready, its GPUs accrue continuously, to the millisecond, at the offer’s GPU-hour rate, idle or not; the charge is taken from the reservation and the window is renewed while the endpoint runs. If it cannot be renewed, the endpoint stops serving until it can, and that time is not billed.

Startup, model load, maintenance, sleeping and stopped time are not billed, and requesting a stop ends billing at once. Token usage is recorded in the request log at no additional charge. Every charge is listed under Hourly charges with its period, GPU count and rate, and posted to the ledger.

Offers and regions

An endpoint is created from an offer that GridPort has published for your organization. The offer fixes the model and its version, the GPUs per replica, the GPU-hour price, the maximum replicas and the pool of nodes it runs on; creating an endpoint takes the offer, a name and a replica count. GET /projects/{id}/endpoints/offers in the console API lists the offers you can use, each with the regions its nodes are in. If your organization’s contract restricts it to regions, only offers with nodes in those regions are listed and replicas are placed only there.

With no offer published yet, the console’s Dedicated endpoints page lets you request one with your model and capacity requirements. Dedicated Endpoints