Batch Inference¶

Batch inference takes many requests in one go and answers them later. You submit the whole job, check on it from time to time, and collect the results once it has finished. FreeInference runs the requests in the background, a few at a time per model, and gives way to interactive traffic.

There are three ways to submit a batch: an inline HTTP form, OpenAI-compatible file batches and Anthropic Message Batches. The latter two work with the official OpenAI and Anthropic SDKs; no separate FreeInference SDK is needed:

Way to submit	How	Results
Inline	`POST /v1/batches` with a `requests` array	Paged results or a JSONL export
From a file (OpenAI Batch API)	Upload JSONL, then create a batch with its `input_file_id`	Download an output file and an error file
Anthropic Message Batches	`POST /v1/messages/batches`	Read a JSONL result stream

A batch request can be a chat completion (/v1/chat/completions), an embedding (/v1/embeddings), an independent Responses request (/v1/responses), or an Anthropic message (in a Message Batch). FreeInference runs each request itself, exactly as it runs the same request sent directly. It does not hand the batch to a provider’s own batch service.

Availability¶

Account access: Batch creation is restricted to internal accounts and administrators unless the operators enable it for other account tiers. An account without batch access receives 403 when creating a batch.

Inline chat-completion batches are the base feature. The rest of this page also covers extensions that FreeInference’s operators turn on separately:

batches from an uploaded file, and the /v1/files upload;
Anthropic Message Batches;
embedding and Responses requests in a batch;
completion webhooks.

This page describes how these extensions behave once they are on. It does not mean they are already available on FreeInference. While one is off, calls that would create something with it are refused and the message says it is not enabled:

Call	Answer while the extension is off
Uploading a file to `/v1/files`, or creating a batch with `input_file_id`	`400`, “File batches are not enabled”
An inline batch whose `url` is `/v1/embeddings` or `/v1/responses`	`400`, the error says `/v1/chat/completions` is the only endpoint batches serve
Creating a Message Batch	`403`, “Message Batches are not enabled”
Setting a completion webhook	`400`, “Batch webhooks are not enabled”

If an extension is turned off again later, batches you already submitted with it keep running, and you can still read, download, cancel and delete them. Operators can find the switches and the rest of the server-side reference in the HybridInference developer documentation on batches.

When to use it¶

Use a batch for a large job that does not need its answers right away: an evaluation run, labelling or summarizing a dataset, generating synthetic data, or embedding a corpus. Compared with sending the requests yourself:

One call submits the whole job, up to 50,000 requests by default. You do not hold thousands of connections open, and you can disconnect once it is submitted.
FreeInference paces and retries the work. Batch requests run under their own concurrency limit, not your interactive one, and transient errors are retried for you, so you do not need your own rate-limit or retry logic.
The results wait for you. A finished batch and its results are kept for 7 days by default.

Send the requests yourself when you need answers within seconds or need streaming. A batch has no fixed turnaround. Its requests start when their models have room, and any that have not started when its completion window closes (24 hours by default) expire. Batch requests are not discounted either: each one counts against your daily quota exactly as the same request sent directly would.

How batches run¶

The create call does not run inference. Inline requests are checked before the batch is stored. File batches are validated in the background before any request runs; Message Batches validate each request separately. The sections below explain each protocol’s status and validation results.
Paced, behind interactive traffic. Each model runs a limited number of batch requests at once, and fewer while it is busy with interactive requests. Batches take turns, so a large batch does not hold up a smaller one submitted after it. Depending on how busy its models are, a batch can finish in minutes or take hours. Requests run in no particular order. Their results are listed in the order you submitted them.
A separate concurrency limit, shared model capacity. Batch requests do not use your interactive concurrent-request limit. They have their own per-account limit, set by your account tier (see Limits). A running batch therefore does not use up your interactive slots, and your interactive requests do not consume your personal batch quota. Batch and interactive requests still run on the same models, though. Batch work yields to interactive traffic, so a model under heavy load runs fewer batch requests, and each batch request takes model capacity like any other request.
Like your own requests. Each request is served exactly as if you had sent it to its endpoint yourself, under your account, with the same models, routing and price. Each one counts toward your daily quota, which batch and interactive requests share: a large batch can use up the quota your interactive requests also draw on.
Retried when the error may pass. A request that fails with a timeout, a 429 or a 5xx is retried with backoff, up to eight attempts in all. A request rejected as invalid (any other 4xx) fails at once.
Checked against your key every time. Before each request runs, FreeInference checks the API key that created the batch again:
- if the key has been revoked or has expired, or the account was suspended or deleted, the batch fails;
- if your daily quota is used up, your batches pause until it renews at 00:00 UTC;
- if your account no longer has access to batch inference, your batches wait until it has again.

Keep the key you created a batch with until the batch finishes. The batch runs under that key, and the completion window keeps running while a batch is paused or waiting.

Managing a batch is a different matter. Checking on, listing, reading, downloading, cancelling and deleting your batches work with any valid API key of the same account, not only the key that created the batch. They also keep working while your daily quota is used up or your account can no longer create batches, so you can always collect and remove what you submitted. Only creating a batch (and uploading an input file) needs batch access, and only creating a batch is refused once the daily quota is used up. A batch that belongs to another account reads as not found.

Endpoints¶

Every endpoint is under the base URL https://freeinference.org/v1 and takes your FreeInference API key, as Authorization: Bearer hyi-your-api-key (or X-API-Key; see the API Headers Reference).

Request	What it does
`POST /v1/batches`	Create a batch, inline or from a file
`GET /v1/batches/{batch_id}`	Check on a batch
`GET /v1/batches/{batch_id}/results`	Read its results, a page at a time
`GET /v1/batches/{batch_id}/output.jsonl`	Download every result of a finished batch as JSONL
`POST /v1/batches/{batch_id}/cancel`	Cancel it
`DELETE /v1/batches/{batch_id}`	Delete it, with its results
`GET /v1/batches`	List your batches
`POST /v1/files`, `GET /v1/files`, `GET /v1/files/{file_id}`, `GET /v1/files/{file_id}/content`, `DELETE /v1/files/{file_id}`	Batch input and output files
`/v1/messages/batches` and its sub-paths (also under `https://freeinference.org/anthropic`)	Anthropic Message Batches
`PUT`, `GET`, `DELETE /v1/batch_webhook`; `GET /v1/batch_webhook/events`	Completion webhook

The examples read your key from an environment variable:

export FREEINFERENCE_API_KEY="hyi-your-api-key"

Inline batches¶

An inline batch carries its requests in the create call itself. It needs nothing but HTTP: no SDK, no file upload. The next sections show how to create one, check on it, read and download its results, and do all of that from a Python script.

Create a batch¶

POST /v1/batches with the requests inline:

curl https://freeinference.org/v1/batches \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: eval-42-run-1" \
  -d '{
    "requests": [
      {
        "custom_id": "q-1",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
          "model": "glm-5.3-flash",
          "messages": [{"role": "user", "content": "Summarize the plot of Hamlet in two sentences."}]
        }
      },
      {
        "custom_id": "q-2",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
          "model": "qwen3.6-35b",
          "messages": [{"role": "user", "content": "Translate \"good morning\" into French."}],
          "max_tokens": 50
        }
      }
    ],
    "metadata": {"project": "eval-42"}
  }'

The Idempotency-Key header is optional. It makes the call safe to retry; see Submitting safely twice.

Field	Required	Meaning
`requests`	Yes	The requests, at least one
`requests[].custom_id`	Yes	Your name for the request: 1–512 characters, unique within the batch. Its result carries it back
`requests[].method`	No	`POST`, the default and the only method accepted
`requests[].url`	No	`/v1/chat/completions` (the default), or, where enabled, `/v1/embeddings` or `/v1/responses`. Every request of one batch must use the same `url`
`requests[].body`	Yes	The request body, exactly as you would send it to that endpoint. `stream` must not be `true`
`metadata`	No	Up to 16 string key-value pairs for your own use; keys up to 64 characters, values up to 512

One inline batch can use several models. Name each one by a model ID from Available Models. Auto models (auto, auto-<tier>) cannot be used in a batch. If you name a model by an alias, the request runs on the model that alias pointed to when you created the batch.

FreeInference checks every request before it stores anything, and refuses the whole batch with 400 if any request is invalid. The error message names the request at fault, for example requests[1] (custom_id 'q-2'): Model 'no-such-model' not found. As elsewhere in the API, a model your key may not use is reported exactly like one that does not exist.

The answer is the new batch:

{
  "id": "batch_6f1c0a7e2b8d4e55a1c3f9d2e4b6a8c0",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "errors": null,
  "input_file_id": null,
  "completion_window": "24h",
  "status": "validating",
  "output_file_id": null,
  "error_file_id": null,
  "created_at": 1792929600,
  "in_progress_at": null,
  "expires_at": 1793016000,
  "finalizing_at": null,
  "completed_at": null,
  "failed_at": null,
  "expired_at": null,
  "cancelling_at": null,
  "cancelled_at": null,
  "request_counts": {"total": 2, "completed": 0, "failed": 0},
  "metadata": {"project": "eval-42"},
  "usage": {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0},
  "cost_usd": 0.0,
  "cost_unknown_attempts": 0,
  "results_expires_at": null,
  "error": null
}

The other endpoints take the batch’s id; the examples below read it from $BATCH_ID. Timestamps are Unix seconds, and expires_at is when the completion window closes. An inline batch has no files, so its file fields are null. usage totals the tokens of the requests that have finished. Some fields are not in OpenAI’s batch object:

cost_usd: what the batch’s finished requests have counted against your daily quota so far. It is an accounting figure, not a bill;
cost_unknown_attempts: attempts whose cost could not be known, and so are not in cost_usd (see Costs);
results_expires_at: once the batch has finished, when it and its results will be deleted;
error: why the whole batch failed.

Check on a batch¶

GET /v1/batches/{batch_id} returns the batch object again:

curl https://freeinference.org/v1/batches/$BATCH_ID \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

Poll it until status is completed, failed, expired or cancelled, the four final statuses:

Status	Meaning
`validating`	Stored; no request has started yet. A batch from a file stays `validating` while its file is checked
`in_progress`	Requests are running
`finalizing`	Batches from a file only: every request has finished and the output files are being written. It lasts a moment
`completed`	Every request has finished, successfully or not. Read each one’s outcome in the results
`failed`	The batch stopped because the key or account that created it is no longer valid; its `error` says so (`owner_unauthorized`). Requests that had not started fail with `batch_failed`. A batch from a file also fails, with nothing run, when its file is invalid; `errors` then lists the problems
`expired`	The completion window closed before every request had started. Those requests expire with `batch_expired`; the others keep their results
`cancelling`	You cancelled the batch while requests were running; they are finishing
`cancelled`	Cancelled. Requests that had not started never will

request_counts.total is the number of requests in the batch. request_counts.completed counts the requests that succeeded, and request_counts.failed those that failed, including the ones a failed or expired batch never ran. Cancelled requests are in neither.

The completion window only stops requests from starting. Requests already running when it closes finish, so a batch can reach its final status a little after expires_at.

Read the results¶

GET /v1/batches/{batch_id}/results pages through the batch’s requests in the order you submitted them, each with its outcome:

curl "https://freeinference.org/v1/batches/$BATCH_ID/results?limit=100" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

{
  "object": "list",
  "data": [
    {
      "id": "batch_req_1041",
      "custom_id": "q-1",
      "status": "completed",
      "response": {
        "status_code": 200,
        "request_id": "req_2c9f6a1d8e7b4c03a5f1e2d3c4b5a697",
        "body": {
          "id": "chatcmpl-9b1e",
          "object": "chat.completion",
          "model": "glm-5.3-flash",
          "choices": [
            {
              "index": 0,
              "message": {"role": "assistant", "content": "Prince Hamlet ..."},
              "finish_reason": "stop"
            }
          ],
          "usage": {"prompt_tokens": 16, "completion_tokens": 120, "total_tokens": 136}
        }
      },
      "error": null
    },
    {
      "id": "batch_req_1042",
      "custom_id": "q-2",
      "status": "failed",
      "response": null,
      "error": {"code": "invalid_request", "message": "...", "status": 400}
    }
  ],
  "has_more": false,
  "first_id": "batch_req_1041",
  "last_id": "batch_req_1042"
}

limit is 1–1000 (default 100). When has_more is true, ask for the next page with after set to this page’s last_id:

curl "https://freeinference.org/v1/batches/$BATCH_ID/results?limit=100&after=batch_req_1042" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

Every request of the batch is listed, finished or not. A request that is still queued or running has no response yet; if it has already failed once and is being retried, its error shows what went wrong last time. So read the results once the batch has reached a final status, or read them again later. This endpoint works for batches from a file too.

Result `status`	Meaning
`queued`	Waiting to run, or waiting to be retried
`running`	Running now
`completed`	Succeeded. `response.body` is the response, as the endpoint would have returned it (a chat completion, an embedding list or a Responses object)
`failed`	Failed, or never ran because the batch failed. `error` says why
`cancelled`	Never ran: the batch was cancelled
`expired`	Never ran: the completion window closed first

An error has a code and a message, plus the HTTP status behind it when there was one:

`error.code`	Meaning
`invalid_request`	The request was rejected as invalid (`400`, `422`)
`model_not_found`	The model is gone, or your key may no longer use it (`404`)
`forbidden`	The request was refused (`403`)
`rate_limited`	Still rate-limited after every retry (`429`)
`timeout`	Still timing out after every retry (`504`)
`service_unavailable`	No provider could serve it after every retry (`503`)
`upstream_error`	Another server error, after every retry
`result_too_large`	The response was larger than a batch result may be (4 MiB by default)
`batch_failed`	Not run: the batch failed
`batch_expired`	Not run: the completion window closed first

Download the results as JSONL¶

Once a batch has reached a final status, GET /v1/batches/{batch_id}/output.jsonl downloads all of its results as one JSONL file, one line per request in submission order, successes and failures alike. Each line has the shape of a line of an OpenAI batch output file (id, custom_id, response, error):

curl -o results.jsonl https://freeinference.org/v1/batches/$BATCH_ID/output.jsonl \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

Before the batch has finished, the answer is 409.

Python example¶

This script submits an inline batch, waits for it to finish and collects every result, using requests. It needs no OpenAI or Anthropic SDK:

import os
import time

import requests

API = "https://freeinference.org/v1"
session = requests.Session()
session.headers["Authorization"] = f"Bearer {os.environ['FREEINFERENCE_API_KEY']}"


def call(method, path, **kwargs):
    response = session.request(method, API + path, timeout=120, **kwargs)
    response.raise_for_status()
    return response.json()


prompts = ["Summarize the plot of Hamlet in two sentences.", "Name three prime numbers."]
batch = call(
    "POST",
    "/batches",
    # Reuse the same key if you retry this call, so a retry cannot submit the job twice.
    headers={"Idempotency-Key": "hamlet-primes-run-1"},
    json={
        "requests": [
            {
                "custom_id": f"q-{i}",
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": {
                    "model": "glm-5.3-flash",
                    "messages": [{"role": "user", "content": prompt}],
                },
            }
            for i, prompt in enumerate(prompts)
        ]
    },
)

# Wait for a final status.
while batch["status"] not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(30)
    batch = call("GET", f"/batches/{batch['id']}")

# Page through every result.
results, after = [], None
while True:
    params = {"limit": 1000}
    if after:
        params["after"] = after
    page = call("GET", f"/batches/{batch['id']}/results", params=params)
    results.extend(page["data"])
    if not page["has_more"]:
        break
    after = page["last_id"]

for result in results:
    if result["status"] == "completed":
        print(result["custom_id"], result["response"]["body"]["choices"][0]["message"]["content"])
    else:
        print(result["custom_id"], result["status"], result["error"])

Batches from a file (OpenAI SDK)¶

Available where file batches are enabled.

This is OpenAI’s own Batch API flow, so the official OpenAI SDKs work with it unchanged. Write one request per line to a JSONL file, upload it with purpose=batch, then create a batch that names the file:

{"custom_id": "q-0", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "glm-5.3-flash", "messages": [{"role": "user", "content": "Summarize the plot of Hamlet in two sentences."}]}}
{"custom_id": "q-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "glm-5.3-flash", "messages": [{"role": "user", "content": "Name three prime numbers."}]}}

The rules for the file:

each line is {"custom_id", "method": "POST", "url", "body"}, with url equal to the batch’s endpoint and a custom_id unique in the file;
every line names the same model. A file batch runs one model; use several files, or an inline batch, to mix models;
at most 50,000 lines, none longer than 4 MiB, and the file at most 32 MiB;
completion_window must be "24h", and endpoint one of /v1/chat/completions, /v1/embeddings or /v1/responses.

The create call answers at once with validating while FreeInference reads the file. If any line is invalid (not JSON, a repeated custom_id, another url or model, a body the endpoint would refuse), the whole batch is failed and none of it runs. errors.data then lists the problems with their line numbers. Otherwise the batch moves to in_progress. Once every request has finished, it is finalizing for a moment while its result files are written. Then it reaches its final status with:

output_file_id: one line per request that succeeded;
error_file_id: one line per request that failed, was cancelled or expired.

Either one is null when it would be empty. A batch in which everything succeeded has no error file, one in which nothing succeeded has no output file, and a batch that failed validation has neither. Check before downloading.

A complete script with the official OpenAI Python SDK:

import json
import os
import time

from openai import OpenAI

client = OpenAI(
    base_url="https://freeinference.org/v1",
    api_key=os.environ["FREEINFERENCE_API_KEY"],
)

# 1. Write the requests, one per line. Every line uses the same model.
prompts = ["Summarize the plot of Hamlet in two sentences.", "Name three prime numbers."]
with open("requests.jsonl", "w", encoding="utf-8") as f:
    for i, prompt in enumerate(prompts):
        line = {
            "custom_id": f"q-{i}",
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {"model": "glm-5.3-flash", "messages": [{"role": "user", "content": prompt}]},
        }
        f.write(json.dumps(line) + "\n")

# 2. Upload the file and create the batch.
with open("requests.jsonl", "rb") as f:
    upload = client.files.create(file=f, purpose="batch")
batch = client.batches.create(
    input_file_id=upload.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"project": "eval-42"},
    # The SDK retries failed calls on its own; the key keeps a retry from creating a second batch.
    extra_headers={"Idempotency-Key": f"eval-42-{upload.id}"},
)

# 3. Wait for a final status.
while batch.status not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(30)
    batch = client.batches.retrieve(batch.id)
print(batch.status, batch.request_counts)

if batch.errors and batch.errors.data:  # the input file was invalid; nothing ran
    for problem in batch.errors.data:
        print(f"line {problem.line}: {problem.message}")


def read_lines(file_id):
    """Download a result file, or nothing when the batch has none (file_id is None)."""
    if file_id is None:
        return []
    text = client.files.content(file_id).text
    return [json.loads(line) for line in text.splitlines() if line]


# 4. Successes, then everything else.
for line in read_lines(batch.output_file_id):
    print(line["custom_id"], line["response"]["body"]["choices"][0]["message"]["content"])
for line in read_lines(batch.error_file_id):
    # A request that got an HTTP error carries it in response.body.error;
    # one that never ran (cancelled, expired, batch failed) carries it in error.
    problem = line["error"] or line["response"]["body"]["error"]
    print(line["custom_id"], problem["code"], problem["message"])

The paged /results endpoint and the JSONL export also work for a batch from a file, if you prefer them to the two files.

Batch files¶

/v1/files holds only batch files. It is not a general Files API: a file cannot be referenced from a chat, Messages or Responses request.

Request	What it does
`POST /v1/files`	Upload a JSONL input file (`multipart/form-data` with a `file` part and `purpose=batch`)
`GET /v1/files`	List your files: uploads (`purpose` `batch`) and result files (`batch_output`)
`GET /v1/files/{file_id}`	Return one file object
`GET /v1/files/{file_id}/content`	Download its bytes
`DELETE /v1/files/{file_id}`	Delete it for good

Retention. An uploaded input file is kept for 30 days by default. The output and error files of a batch are kept as long as the batch itself, 7 days after it finishes by default, and go when the batch is deleted.
Fixed bytes. A result file has the same bytes every time you download it. Its size and SHA-256 (sent as the ETag) are fixed when the batch ends.
Deleting. Deleting an input file does not affect a batch already created from it. A deleted output or error file is not written again: the batch keeps naming it, but downloading it then answers 404. The results themselves stay readable through /results and the JSONL export until the batch is deleted.
Limits. You may keep up to 512 MiB of uploaded files and upload up to 60 files per hour, by default. Over the storage limit an upload answers 413; over the hourly limit, 429.

Anthropic Message Batches¶

Available where Anthropic Message Batches are enabled.

/v1/messages/batches follows Anthropic’s Message Batches API, so the official Anthropic SDKs work against it. The same routes are also served under /anthropic, as /anthropic/v1/messages/batches, matching the base URL used for Anthropic-compatible clients.

A complete script with the official Anthropic Python SDK:

import os
import time

import anthropic

client = anthropic.Anthropic(
    base_url="https://freeinference.org/anthropic",
    api_key=os.environ["FREEINFERENCE_API_KEY"],
)

prompts = ["Summarize the plot of Hamlet in two sentences.", "Name three prime numbers."]
batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"q-{i}",
            "params": {
                "model": "glm-5.3-flash",
                "max_tokens": 512,
                "messages": [{"role": "user", "content": prompt}],
            },
        }
        for i, prompt in enumerate(prompts)
    ],
    # The SDK retries failed calls on its own; the key keeps a retry from creating a second batch.
    extra_headers={"Idempotency-Key": "hamlet-primes-msgbatch-1"},
)

# Wait until the batch has ended; results are available only then.
while batch.processing_status != "ended":
    time.sleep(30)
    batch = client.messages.batches.retrieve(batch.id)
print(batch.request_counts)

for entry in client.messages.batches.results(batch.id):
    result = entry.result
    if result.type == "succeeded":
        text = "".join(block.text for block in result.message.content if block.type == "text")
        print(entry.custom_id, text)
    elif result.type == "errored":
        print(entry.custom_id, "errored:", result.error.error.type, result.error.error.message)
    else:  # "canceled" or "expired": the request never ran
        print(entry.custom_id, result.type)

How Message Batches differ from the OpenAI-shaped batches above:

The anthropic-version header is required. The SDK sends it for you. 2023-06-01 and 2026-10-21 are accepted. A batch keeps the version it was created with, and each of its requests is checked against that version’s rules. anthropic-beta is refused.
Checking. Some things are checked at once, and a problem with any of them refuses the whole batch: each custom_id must be 1–64 letters, digits, _ or -, and unique, and the batch must stay within the request-count and body-size limits. Each request’s params is then checked on its own. A request FreeInference cannot run as written, including one naming a model you may not use, becomes an errored result while the others still run. A parameter is never silently dropped: when a model’s route cannot carry a parameter (extended thinking, cache control or structured output on a non-Anthropic backend, for example), that request errors instead.
Progress. request_counts show every request as processing until the whole batch has ended; results_url appears then. Results can be read only once the batch has ended.
Cancelling and deleting. Cancelling makes the batch canceling until its running requests finish. A Message Batch can be deleted only once it has ended (400 before).
One turn per request. A result that asks for a tool call is for your program to carry out and send back in a new request; FreeInference never runs your tools. Server tools, Files API sources, MCP servers and containers are refused.
Separate listing. GET /v1/messages/batches lists your Message Batches. An ID from one API reads as not found on the other.
Your account, no workspaces. Message Batches belong to your account, like every batch here.

Embeddings and Responses in a batch¶

Available where embedding and Responses batches are enabled.

/v1/embeddings and /v1/responses can be batched like chat completions, inline or from a file. Set url (inline) or endpoint (file) to the endpoint, and give each request the body you would send to it. One batch targets one endpoint.

A batched embedding request accepts what /v1/embeddings accepts. A batched Responses request must be independent: it runs once, on its own, and is not stored. So:

omit store or set it to false;
leave out previous_response_id, conversation, background and stream;
hosted tools, file inputs and any other field FreeInference would have to drop are refused.

Submitting safely twice¶

A network error during a create call leaves you not knowing whether the batch was created. Send an Idempotency-Key header with the create call (any of the three APIs) to make it safe to retry:

the same key with the same request returns the batch the first call created, instead of creating another. This works even if your daily quota has since run out or your account has lost batch access;
the same key with a different request answers 409;
after that batch has been deleted, the key answers 410.

A key is 1–255 printable characters of your choosing, such as a job name and a run number. It belongs to your account, so any of your API keys can replay it, and a replay never changes which key the batch runs under. A key is remembered until a day after its batch ends. The OpenAI and Anthropic SDKs do not send this header for you: pass it with extra_headers, as in the examples above. Without it, an SDK’s automatic retry of a create call can submit the batch twice.

Costs¶

Batch requests cost the same as the same requests sent directly. There is no batch discount. Every attempt at a request is recorded with what it cost, at the same point and price as an interactive request, and a batch is never charged a second time on top. cost_usd adds up the attempts whose cost is known.

Sometimes an attempt’s result is lost after the request was sent, for example when the request timed out. The provider may still have run it. Its cost is then unknown: it is counted in cost_unknown_attempts and left out of cost_usd rather than counted as zero. A retried request may therefore have run more than once. A batch whose cost_unknown_attempts is not 0 may have cost somewhat more than cost_usd says.

Cancel a batch¶

POST /v1/batches/{batch_id}/cancel stops a batch and returns it:

curl -X POST https://freeinference.org/v1/batches/$BATCH_ID/cancel \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

Requests that have not started are cancelled; running ones finish and keep their results. The batch is cancelling until they do, then cancelled. A batch from a file that is cancelled while its file is still being checked ends cancelled with nothing run. Cancelling a batch that has already finished, or is finalizing, returns it unchanged. With the SDKs, call client.batches.cancel(batch_id) or client.messages.batches.cancel(batch_id).

Delete a batch¶

DELETE /v1/batches/{batch_id} deletes a batch with its requests and results, and with the output and error files of a batch from a file:

curl -X DELETE https://freeinference.org/v1/batches/$BATCH_ID \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

{"id": "batch_6f1c0a7e2b8d4e55a1c3f9d2e4b6a8c0", "object": "batch.deleted", "deleted": true}

A batch can be deleted in any status, but not while one of its requests is running: the answer is then 409. Cancel the batch, wait for its running requests to finish, then delete it. A Message Batch can be deleted only once it has ended (DELETE /v1/messages/batches/{batch_id}).

FreeInference also deletes a finished batch on its own, 7 days after it finished by default (results_expires_at says when), so collect its results before then.

List your batches¶

GET /v1/batches lists your batches, inline and from files, newest first:

curl "https://freeinference.org/v1/batches?limit=20" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"

limit is 1–100 (default 20). The answer has the shape of a page of results, with batch objects in data; for the next page, pass after set to the previous page’s last_id. Message Batches are listed by GET /v1/messages/batches.

Batches in the console¶

The Batches page of the console, linked from your dashboard, lists your batches from all three APIs with their progress, tokens and known cost. From it you can cancel and delete batches, download results and, where webhooks are enabled, set up your completion webhook.

Completion webhooks¶

Available where batch webhooks are enabled.

Instead of polling, you can have FreeInference call an HTTPS endpoint of yours when a batch ends. Register one endpoint for all your batches on the console’s Batches page, or with the API:

curl -X PUT https://freeinference.org/v1/batch_webhook \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/hooks/freeinference-batches"}'

The first time you set a webhook, the answer shows its signing secret, whsec_…, once; store it. Changing the URL later keeps the secret. Send "rotate_secret": true with the URL to get a new secret. GET /v1/batch_webhook shows the webhook, DELETE /v1/batch_webhook removes it, and GET /v1/batch_webhook/events lists recent deliveries.

When a batch ends, after its results can be read, FreeInference POSTs an event: batch.completed, batch.failed, batch.expired or batch.cancelled for an OpenAI-shaped batch, message_batch.ended for a Message Batch. The body names the batch only, never a prompt or a result:

{"id": "evt_…", "object": "event", "type": "batch.completed", "created_at": 1793016000, "data": {"id": "batch_6f1c…"}}

Fetch the results with your API key. When you receive a delivery:

verify its signature. Deliveries are signed following the Standard Webhooks specification (webhook-id, webhook-timestamp and webhook-signature headers), so its libraries can verify them with your whsec_… secret;
answer quickly with a 2xx. A delivery that does not get one is retried with backoff, up to eight attempts;
deduplicate on webhook-id. The same event can arrive more than once.

The URL must be HTTPS and reach a public address; private, loopback and link-local addresses are refused.

Limits¶

Limit	Default
Requests in one batch	50,000
Size of the create request’s body, and of an input file	32 MiB
One line of an input file	4 MiB
One request’s stored result	4 MiB
Unfinished batches per account (`validating`, `in_progress`, `finalizing` or `cancelling`)	10
Unfinished requests across all your batches	100,000
Batch requests of yours running at once, by account tier	free 8, pro 16, internal 64
Completion window: how long a batch’s requests have to start	24 hours
How long a finished batch and its results are kept	7 days
How long an uploaded input file is kept	30 days
Uploaded files you may keep, and upload per hour	512 MiB; 60

These are the defaults, and FreeInference may change them. Split a larger job into several batches. The role-based concurrency limit is counted per gateway process; several processes can run more requests in total, subject to the shared per-model limits. The completion window of a batch from a file and of a Message Batch is always 24 hours. For an inline batch it is FreeInference’s setting, and a completion_window you send is ignored.

Errors¶

Errors from creating a batch (and uploading a file):

Status	When
`400`	The body is not valid JSON or not a batch; a request is invalid, uses an unknown, unavailable or auto model, or repeats a `custom_id`; the batch holds too many requests; the file named by `input_file_id` is not found, not ready or not `purpose=batch`; an extension is not enabled; the `Idempotency-Key` is malformed; or a Message Batch lacks `anthropic-version` or sends `anthropic-beta`
`401`	The API key is missing, invalid or expired
`403`	Your account may not create batches; or Message Batches are not enabled
`404`	Batch inference is unavailable at the moment
`409`	The `Idempotency-Key` was already used for a different request
`410`	The batch this `Idempotency-Key` created has been deleted
`413`	The body or file is larger than the size limit, or an upload would exceed your file storage
`429`	You already have the most unfinished batches or batch requests allowed; your daily quota is used up until it renews at 00:00 UTC; or you have uploaded the most files allowed this hour

Errors from managing batches:

Status	When
`400`	Reading results: `after` is not a result ID such as `batch_req_42`. A Message Batch: reading the results of, or deleting, one that has not ended
`401`	The API key is missing, invalid or expired
`404`	The batch or file does not exist, was deleted, or belongs to another account; or batch inference is unavailable at the moment. The message says which
`409`	Deleting: one of the batch’s requests is still running. Downloading `output.jsonl`: the batch has not finished

Checking on, reading, downloading, cancelling and deleting a batch never answer 429 for a used-up daily quota.

Coming from OpenAI’s or Anthropic’s batch APIs¶

Requests run on FreeInference, not a provider’s batch service. Each one is served like an interactive request, at the same price: there is no batch discount, and turnaround depends on how busy the models are, not on a provider’s batch queue.
The limits are FreeInference’s (see Limits), lower than the providers’ own: 50,000 requests and 32 MiB per batch by default.
The completion window stops requests from starting. Requests already running when it closes finish, so a batch can end after expires_at.
Extras of FreeInference’s own: the inline form, the paged /v1/batches/{batch_id}/results, the JSONL export, DELETE /v1/batches/{batch_id}, Idempotency-Key, and the fields cost_usd, cost_unknown_attempts, results_expires_at and error.
OpenAI file batches run one model per file. Only purpose=batch files exist, and they serve batches only.
Responses requests in a batch are independent: store: true, previous_response_id, conversation, background and hosted tools are refused.
Message Batches belong to your account; there are no workspaces. anthropic-beta is refused, as are server tools, Files API sources, MCP servers and containers.