# Batch Inference

Batch inference takes many requests in one go and answers them later. You
submit the whole job, check on it from time to time, and collect the results
once it has finished. FreeInference runs the requests in the background, a few
at a time per model, and gives way to interactive traffic.

There are three ways to submit a batch: an inline HTTP form, OpenAI-compatible
file batches and Anthropic Message Batches. The latter two work with the
official OpenAI and Anthropic SDKs; no separate FreeInference SDK is needed:

| Way to submit | How | Results |
|---|---|---|
| [Inline](#inline-batches) | `POST /v1/batches` with a `requests` array | [Paged results](#read-the-results) or a [JSONL export](#download-the-results-as-jsonl) |
| [From a file](#batches-from-a-file-openai-sdk) (OpenAI Batch API) | Upload JSONL, then create a batch with its `input_file_id` | Download an output file and an error file |
| [Anthropic Message Batches](#anthropic-message-batches) | `POST /v1/messages/batches` | Read a JSONL result stream |

A batch request can be a chat completion (`/v1/chat/completions`), an
embedding (`/v1/embeddings`), an independent Responses request
(`/v1/responses`), or an Anthropic message (in a Message Batch). FreeInference
runs each request itself, exactly as it runs the same request sent directly.
It does not hand the batch to a provider's own batch service.

## Availability

> **Account access:** Batch creation is restricted to internal accounts and
> administrators unless the operators enable it for other account tiers.
> An account without batch access receives `403` when creating a batch.

**Inline chat-completion batches** are the base feature. The rest of this page
also covers extensions that FreeInference's operators turn on separately:

- batches from an uploaded file, and the `/v1/files` upload;
- Anthropic Message Batches;
- embedding and Responses requests in a batch;
- completion webhooks.

This page describes how these extensions behave once they are on. It does not
mean they are already available on FreeInference. While one is off, calls that would
create something with it are refused and the message says it is not enabled:

| Call | Answer while the extension is off |
|---|---|
| Uploading a file to `/v1/files`, or creating a batch with `input_file_id` | `400`, "File batches are not enabled" |
| An inline batch whose `url` is `/v1/embeddings` or `/v1/responses` | `400`, the error says `/v1/chat/completions` is the only endpoint batches serve |
| Creating a Message Batch | `403`, "Message Batches are not enabled" |
| Setting a completion webhook | `400`, "Batch webhooks are not enabled" |

If an extension is turned off again later, batches you already submitted with
it keep running, and you can still read, download, cancel and delete them.
Operators can find the switches and the rest of the server-side reference in
the [HybridInference developer documentation on batches](https://github.com/HarvardMadSys/hybridInference/blob/main/docs/developer/batches.md).

## When to use it

Use a batch for a large job that does not need its answers right away: an
evaluation run, labelling or summarizing a dataset, generating synthetic data,
or embedding a corpus. Compared with sending the requests yourself:

- **One call submits the whole job**, up to 50,000 requests by default. You do
  not hold thousands of connections open, and you can disconnect once it is
  submitted.
- **FreeInference paces and retries the work.** Batch requests run under their
  own concurrency limit, not your interactive one, and transient errors are
  retried for you, so you do not need your own rate-limit or retry logic.
- **The results wait for you.** A finished batch and its results are kept for
  7 days by default.

Send the requests yourself when you need answers within seconds or need
streaming. A batch has no fixed turnaround. Its requests start when their
models have room, and any that have not started when its completion window
closes (24 hours by default) expire. Batch requests are not discounted either:
each one counts against your daily quota exactly as the same request sent
directly would.

## How batches run

- **The create call does not run inference.** Inline requests are checked before
  the batch is stored. File batches are validated in the background before any
  request runs; Message Batches validate each request separately. The sections
  below explain each protocol's status and validation results.
- **Paced, behind interactive traffic.** Each model runs a limited number of
  batch requests at once, and fewer while it is busy with interactive requests.
  Batches take turns, so a large batch does not hold up a smaller one submitted
  after it. Depending on how busy its models are, a batch can finish in minutes
  or take hours. Requests run in no particular order. Their results are listed
  in the order you submitted them.
- **A separate concurrency limit, shared model capacity.** Batch requests do
  not use your interactive concurrent-request limit. They have their own
  per-account limit, set by your account tier (see [Limits](#limits)). A
  running batch therefore does not use up your interactive slots, and your
  interactive requests do not consume your personal batch quota. Batch and
  interactive requests still run on the same models, though. Batch work yields to
  interactive traffic, so a model under heavy load runs fewer batch requests,
  and each batch request takes model capacity like any other request.
- **Like your own requests.** Each request is served exactly as if you had sent
  it to its endpoint yourself, under your account, with the same models,
  routing and price. Each one counts toward your **daily quota**, which batch
  and interactive requests share: a large batch can use up the quota your
  interactive requests also draw on.
- **Retried when the error may pass.** A request that fails with a timeout, a
  `429` or a `5xx` is retried with backoff, up to eight attempts in all. A
  request rejected as invalid (any other `4xx`) fails at once.
- **Checked against your key every time.** Before each request runs,
  FreeInference checks the API key that created the batch again:
  - if the key has been revoked or has expired, or the account was suspended
    or deleted, the batch fails;
  - if your daily quota is used up, your batches pause until it renews at
    00:00 UTC;
  - if your account no longer has access to batch inference, your batches wait
    until it has again.

**Keep the key you created a batch with until the batch finishes.** The batch
runs under that key, and the completion window keeps running while a batch is
paused or waiting.

Managing a batch is a different matter. Checking on, listing, reading,
downloading, cancelling and deleting your batches work with **any valid API
key of the same account**, not only the key that created the batch. They also
keep working while your daily quota is used up or your account can no longer
create batches, so you can always collect and remove what you submitted. Only
creating a batch (and uploading an input file) needs batch access, and only
creating a batch is refused once the daily quota is used up. A batch that
belongs to another account reads as not found.

## Endpoints

Every endpoint is under the base URL `https://freeinference.org/v1` and takes
your FreeInference API key, as `Authorization: Bearer hyi-your-api-key` (or
`X-API-Key`; see the [API Headers Reference](api_headers.md)).

| Request | What it does |
|---|---|
| `POST /v1/batches` | [Create a batch](#create-a-batch), inline or [from a file](#batches-from-a-file-openai-sdk) |
| `GET /v1/batches/{batch_id}` | [Check on a batch](#check-on-a-batch) |
| `GET /v1/batches/{batch_id}/results` | [Read its results](#read-the-results), a page at a time |
| `GET /v1/batches/{batch_id}/output.jsonl` | [Download every result](#download-the-results-as-jsonl) of a finished batch as JSONL |
| `POST /v1/batches/{batch_id}/cancel` | [Cancel it](#cancel-a-batch) |
| `DELETE /v1/batches/{batch_id}` | [Delete it](#delete-a-batch), with its results |
| `GET /v1/batches` | [List your batches](#list-your-batches) |
| `POST /v1/files`, `GET /v1/files`, `GET /v1/files/{file_id}`, `GET /v1/files/{file_id}/content`, `DELETE /v1/files/{file_id}` | [Batch input and output files](#batch-files) |
| `/v1/messages/batches` and its sub-paths (also under `https://freeinference.org/anthropic`) | [Anthropic Message Batches](#anthropic-message-batches) |
| `PUT`, `GET`, `DELETE /v1/batch_webhook`; `GET /v1/batch_webhook/events` | [Completion webhook](#completion-webhooks) |

The examples read your key from an environment variable:

```bash
export FREEINFERENCE_API_KEY="hyi-your-api-key"
```

## Inline batches

An inline batch carries its requests in the create call itself. It needs
nothing but HTTP: no SDK, no file upload. The next sections show how to create
one, check on it, read and download its results, and do all of that from a
Python script.

## Create a batch

`POST /v1/batches` with the requests inline:

```bash
curl https://freeinference.org/v1/batches \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: eval-42-run-1" \
  -d '{
    "requests": [
      {
        "custom_id": "q-1",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
          "model": "glm-5.3-flash",
          "messages": [{"role": "user", "content": "Summarize the plot of Hamlet in two sentences."}]
        }
      },
      {
        "custom_id": "q-2",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
          "model": "qwen3.6-35b",
          "messages": [{"role": "user", "content": "Translate \"good morning\" into French."}],
          "max_tokens": 50
        }
      }
    ],
    "metadata": {"project": "eval-42"}
  }'
```

The `Idempotency-Key` header is optional. It makes the call safe to retry; see
[Submitting safely twice](#submitting-safely-twice).

| Field | Required | Meaning |
|---|---|---|
| `requests` | Yes | The requests, at least one |
| `requests[].custom_id` | Yes | Your name for the request: 1–512 characters, unique within the batch. Its result carries it back |
| `requests[].method` | No | `POST`, the default and the only method accepted |
| `requests[].url` | No | `/v1/chat/completions` (the default), or, [where enabled](#availability), `/v1/embeddings` or `/v1/responses`. Every request of one batch must use the same `url` |
| `requests[].body` | Yes | The request body, exactly as you would send it to that endpoint. `stream` must not be `true` |
| `metadata` | No | Up to 16 string key-value pairs for your own use; keys up to 64 characters, values up to 512 |

One inline batch can use several models. Name each one by a model ID from
[Available Models](models.md). Auto models (`auto`, `auto-<tier>`) cannot be
used in a batch. If you name a model by an alias, the request runs on the
model that alias pointed to when you created the batch.

FreeInference checks every request before it stores anything, and refuses the
whole batch with `400` if any request is invalid. The error message names the
request at fault, for example
`requests[1] (custom_id 'q-2'): Model 'no-such-model' not found`. As elsewhere
in the API, a model your key may not use is reported exactly like one that does
not exist.

The answer is the new batch:

```json
{
  "id": "batch_6f1c0a7e2b8d4e55a1c3f9d2e4b6a8c0",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "errors": null,
  "input_file_id": null,
  "completion_window": "24h",
  "status": "validating",
  "output_file_id": null,
  "error_file_id": null,
  "created_at": 1792929600,
  "in_progress_at": null,
  "expires_at": 1793016000,
  "finalizing_at": null,
  "completed_at": null,
  "failed_at": null,
  "expired_at": null,
  "cancelling_at": null,
  "cancelled_at": null,
  "request_counts": {"total": 2, "completed": 0, "failed": 0},
  "metadata": {"project": "eval-42"},
  "usage": {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0},
  "cost_usd": 0.0,
  "cost_unknown_attempts": 0,
  "results_expires_at": null,
  "error": null
}
```

The other endpoints take the batch's `id`; the examples below read it from
`$BATCH_ID`. Timestamps are Unix seconds, and `expires_at` is when the
completion window closes. An inline batch has no files, so its file fields are
`null`. `usage` totals the tokens of the requests that have finished. Some
fields are not in OpenAI's batch object:

- `cost_usd`: what the batch's finished requests have counted against your
  daily quota so far. It is an accounting figure, not a bill;
- `cost_unknown_attempts`: attempts whose cost could not be known, and so are
  not in `cost_usd` (see [Costs](#costs));
- `results_expires_at`: once the batch has finished, when it and its results
  will be deleted;
- `error`: why the whole batch failed.

## Check on a batch

`GET /v1/batches/{batch_id}` returns the batch object again:

```bash
curl https://freeinference.org/v1/batches/$BATCH_ID \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

Poll it until `status` is `completed`, `failed`, `expired` or `cancelled`, the
four final statuses:

| Status | Meaning |
|---|---|
| `validating` | Stored; no request has started yet. A batch from a file stays `validating` while its file is checked |
| `in_progress` | Requests are running |
| `finalizing` | Batches from a file only: every request has finished and the output files are being written. It lasts a moment |
| `completed` | Every request has finished, successfully or not. Read each one's outcome in the results |
| `failed` | The batch stopped because the key or account that created it is no longer valid; its `error` says so (`owner_unauthorized`). Requests that had not started fail with `batch_failed`. A batch from a file also fails, with nothing run, when its file is invalid; `errors` then lists the problems |
| `expired` | The completion window closed before every request had started. Those requests expire with `batch_expired`; the others keep their results |
| `cancelling` | You cancelled the batch while requests were running; they are finishing |
| `cancelled` | Cancelled. Requests that had not started never will |

`request_counts.total` is the number of requests in the batch.
`request_counts.completed` counts the requests that succeeded, and
`request_counts.failed` those that failed, including the ones a failed or
expired batch never ran. Cancelled requests are in neither.

The completion window only stops requests from *starting*. Requests already
running when it closes finish, so a batch can reach its final status a little
after `expires_at`.

## Read the results

`GET /v1/batches/{batch_id}/results` pages through the batch's requests in the
order you submitted them, each with its outcome:

```bash
curl "https://freeinference.org/v1/batches/$BATCH_ID/results?limit=100" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

```json
{
  "object": "list",
  "data": [
    {
      "id": "batch_req_1041",
      "custom_id": "q-1",
      "status": "completed",
      "response": {
        "status_code": 200,
        "request_id": "req_2c9f6a1d8e7b4c03a5f1e2d3c4b5a697",
        "body": {
          "id": "chatcmpl-9b1e",
          "object": "chat.completion",
          "model": "glm-5.3-flash",
          "choices": [
            {
              "index": 0,
              "message": {"role": "assistant", "content": "Prince Hamlet ..."},
              "finish_reason": "stop"
            }
          ],
          "usage": {"prompt_tokens": 16, "completion_tokens": 120, "total_tokens": 136}
        }
      },
      "error": null
    },
    {
      "id": "batch_req_1042",
      "custom_id": "q-2",
      "status": "failed",
      "response": null,
      "error": {"code": "invalid_request", "message": "...", "status": 400}
    }
  ],
  "has_more": false,
  "first_id": "batch_req_1041",
  "last_id": "batch_req_1042"
}
```

`limit` is 1–1000 (default 100). When `has_more` is `true`, ask for the next
page with `after` set to this page's `last_id`:

```bash
curl "https://freeinference.org/v1/batches/$BATCH_ID/results?limit=100&after=batch_req_1042" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

Every request of the batch is listed, finished or not. A request that is still
`queued` or `running` has no `response` yet; if it has already failed once and
is being retried, its `error` shows what went wrong last time. So read the
results once the batch has reached a final status, or read them again later.
This endpoint works for batches from a file too.

| Result `status` | Meaning |
|---|---|
| `queued` | Waiting to run, or waiting to be retried |
| `running` | Running now |
| `completed` | Succeeded. `response.body` is the response, as the endpoint would have returned it (a chat completion, an embedding list or a Responses object) |
| `failed` | Failed, or never ran because the batch failed. `error` says why |
| `cancelled` | Never ran: the batch was cancelled |
| `expired` | Never ran: the completion window closed first |

An `error` has a `code` and a `message`, plus the HTTP `status` behind it when
there was one:

| `error.code` | Meaning |
|---|---|
| `invalid_request` | The request was rejected as invalid (`400`, `422`) |
| `model_not_found` | The model is gone, or your key may no longer use it (`404`) |
| `forbidden` | The request was refused (`403`) |
| `rate_limited` | Still rate-limited after every retry (`429`) |
| `timeout` | Still timing out after every retry (`504`) |
| `service_unavailable` | No provider could serve it after every retry (`503`) |
| `upstream_error` | Another server error, after every retry |
| `result_too_large` | The response was larger than a batch result may be (4 MiB by default) |
| `batch_failed` | Not run: the batch failed |
| `batch_expired` | Not run: the completion window closed first |

## Download the results as JSONL

Once a batch has reached a final status, `GET
/v1/batches/{batch_id}/output.jsonl` downloads all of its results as one JSONL
file, one line per request in submission order, successes and failures alike.
Each line has the shape of a line of an OpenAI batch output file (`id`,
`custom_id`, `response`, `error`):

```bash
curl -o results.jsonl https://freeinference.org/v1/batches/$BATCH_ID/output.jsonl \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

Before the batch has finished, the answer is `409`.

## Python example

This script submits an inline batch, waits for it to finish and collects every
result, using [`requests`](https://requests.readthedocs.io/). It needs no
OpenAI or Anthropic SDK:

```python
import os
import time

import requests

API = "https://freeinference.org/v1"
session = requests.Session()
session.headers["Authorization"] = f"Bearer {os.environ['FREEINFERENCE_API_KEY']}"


def call(method, path, **kwargs):
    response = session.request(method, API + path, timeout=120, **kwargs)
    response.raise_for_status()
    return response.json()


prompts = ["Summarize the plot of Hamlet in two sentences.", "Name three prime numbers."]
batch = call(
    "POST",
    "/batches",
    # Reuse the same key if you retry this call, so a retry cannot submit the job twice.
    headers={"Idempotency-Key": "hamlet-primes-run-1"},
    json={
        "requests": [
            {
                "custom_id": f"q-{i}",
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": {
                    "model": "glm-5.3-flash",
                    "messages": [{"role": "user", "content": prompt}],
                },
            }
            for i, prompt in enumerate(prompts)
        ]
    },
)

# Wait for a final status.
while batch["status"] not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(30)
    batch = call("GET", f"/batches/{batch['id']}")

# Page through every result.
results, after = [], None
while True:
    params = {"limit": 1000}
    if after:
        params["after"] = after
    page = call("GET", f"/batches/{batch['id']}/results", params=params)
    results.extend(page["data"])
    if not page["has_more"]:
        break
    after = page["last_id"]

for result in results:
    if result["status"] == "completed":
        print(result["custom_id"], result["response"]["body"]["choices"][0]["message"]["content"])
    else:
        print(result["custom_id"], result["status"], result["error"])
```

## Batches from a file (OpenAI SDK)

> Available where file batches are [enabled](#availability).

This is OpenAI's own Batch API flow, so the official OpenAI SDKs work with it
unchanged. Write one request per line to a JSONL file, upload it with
`purpose=batch`, then create a batch that names the file:

```json
{"custom_id": "q-0", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "glm-5.3-flash", "messages": [{"role": "user", "content": "Summarize the plot of Hamlet in two sentences."}]}}
{"custom_id": "q-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "glm-5.3-flash", "messages": [{"role": "user", "content": "Name three prime numbers."}]}}
```

The rules for the file:

- each line is `{"custom_id", "method": "POST", "url", "body"}`, with `url`
  equal to the batch's `endpoint` and a `custom_id` unique in the file;
- every line names **the same model**. A file batch runs one model; use
  several files, or an inline batch, to mix models;
- at most 50,000 lines, none longer than 4 MiB, and the file at most 32 MiB;
- `completion_window` must be `"24h"`, and `endpoint` one of
  `/v1/chat/completions`, `/v1/embeddings` or `/v1/responses`.

The create call answers at once with `validating` while FreeInference reads the
file. If any line is invalid (not JSON, a repeated `custom_id`, another `url`
or model, a body the endpoint would refuse), the whole batch is `failed` and
none of it runs. `errors.data` then lists the problems with their line numbers.
Otherwise the batch moves to `in_progress`. Once every request has finished, it
is `finalizing` for a moment while its result files are written. Then it
reaches its final status with:

- `output_file_id`: one line per request that succeeded;
- `error_file_id`: one line per request that failed, was cancelled or expired.

**Either one is `null` when it would be empty.** A batch in which everything
succeeded has no error file, one in which nothing succeeded has no output file,
and a batch that failed validation has neither. Check before downloading.

A complete script with the official [OpenAI Python SDK](https://github.com/openai/openai-python):

```python
import json
import os
import time

from openai import OpenAI

client = OpenAI(
    base_url="https://freeinference.org/v1",
    api_key=os.environ["FREEINFERENCE_API_KEY"],
)

# 1. Write the requests, one per line. Every line uses the same model.
prompts = ["Summarize the plot of Hamlet in two sentences.", "Name three prime numbers."]
with open("requests.jsonl", "w", encoding="utf-8") as f:
    for i, prompt in enumerate(prompts):
        line = {
            "custom_id": f"q-{i}",
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {"model": "glm-5.3-flash", "messages": [{"role": "user", "content": prompt}]},
        }
        f.write(json.dumps(line) + "\n")

# 2. Upload the file and create the batch.
with open("requests.jsonl", "rb") as f:
    upload = client.files.create(file=f, purpose="batch")
batch = client.batches.create(
    input_file_id=upload.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"project": "eval-42"},
    # The SDK retries failed calls on its own; the key keeps a retry from creating a second batch.
    extra_headers={"Idempotency-Key": f"eval-42-{upload.id}"},
)

# 3. Wait for a final status.
while batch.status not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(30)
    batch = client.batches.retrieve(batch.id)
print(batch.status, batch.request_counts)

if batch.errors and batch.errors.data:  # the input file was invalid; nothing ran
    for problem in batch.errors.data:
        print(f"line {problem.line}: {problem.message}")


def read_lines(file_id):
    """Download a result file, or nothing when the batch has none (file_id is None)."""
    if file_id is None:
        return []
    text = client.files.content(file_id).text
    return [json.loads(line) for line in text.splitlines() if line]


# 4. Successes, then everything else.
for line in read_lines(batch.output_file_id):
    print(line["custom_id"], line["response"]["body"]["choices"][0]["message"]["content"])
for line in read_lines(batch.error_file_id):
    # A request that got an HTTP error carries it in response.body.error;
    # one that never ran (cancelled, expired, batch failed) carries it in error.
    problem = line["error"] or line["response"]["body"]["error"]
    print(line["custom_id"], problem["code"], problem["message"])
```

The paged [`/results`](#read-the-results) endpoint and the
[JSONL export](#download-the-results-as-jsonl) also work for a batch from a
file, if you prefer them to the two files.

## Batch files

`/v1/files` holds only batch files. It is not a general Files API: a file
cannot be referenced from a chat, Messages or Responses request.

| Request | What it does |
|---|---|
| `POST /v1/files` | Upload a JSONL input file (`multipart/form-data` with a `file` part and `purpose=batch`) |
| `GET /v1/files` | List your files: uploads (`purpose` `batch`) and result files (`batch_output`) |
| `GET /v1/files/{file_id}` | Return one file object |
| `GET /v1/files/{file_id}/content` | Download its bytes |
| `DELETE /v1/files/{file_id}` | Delete it for good |

- **Retention.** An uploaded input file is kept for 30 days by default. The
  output and error files of a batch are kept as long as the batch itself, 7
  days after it finishes by default, and go when the batch is deleted.
- **Fixed bytes.** A result file has the same bytes every time you download it.
  Its size and SHA-256 (sent as the `ETag`) are fixed when the batch ends.
- **Deleting.** Deleting an input file does not affect a batch already created
  from it. A deleted output or error file is not written again: the batch keeps
  naming it, but downloading it then answers `404`. The results themselves stay
  readable through [`/results`](#read-the-results) and the
  [JSONL export](#download-the-results-as-jsonl) until the batch is deleted.
- **Limits.** You may keep up to 512 MiB of uploaded files and upload up to 60
  files per hour, by default. Over the storage limit an upload answers `413`;
  over the hourly limit, `429`.

## Anthropic Message Batches

> Available where Anthropic Message Batches are [enabled](#availability).

`/v1/messages/batches` follows Anthropic's Message Batches API, so the official
Anthropic SDKs work against it. The same routes are also served under
`/anthropic`, as `/anthropic/v1/messages/batches`, matching the base URL used
for Anthropic-compatible clients.

A complete script with the official [Anthropic Python SDK](https://github.com/anthropics/anthropic-sdk-python):

```python
import os
import time

import anthropic

client = anthropic.Anthropic(
    base_url="https://freeinference.org/anthropic",
    api_key=os.environ["FREEINFERENCE_API_KEY"],
)

prompts = ["Summarize the plot of Hamlet in two sentences.", "Name three prime numbers."]
batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"q-{i}",
            "params": {
                "model": "glm-5.3-flash",
                "max_tokens": 512,
                "messages": [{"role": "user", "content": prompt}],
            },
        }
        for i, prompt in enumerate(prompts)
    ],
    # The SDK retries failed calls on its own; the key keeps a retry from creating a second batch.
    extra_headers={"Idempotency-Key": "hamlet-primes-msgbatch-1"},
)

# Wait until the batch has ended; results are available only then.
while batch.processing_status != "ended":
    time.sleep(30)
    batch = client.messages.batches.retrieve(batch.id)
print(batch.request_counts)

for entry in client.messages.batches.results(batch.id):
    result = entry.result
    if result.type == "succeeded":
        text = "".join(block.text for block in result.message.content if block.type == "text")
        print(entry.custom_id, text)
    elif result.type == "errored":
        print(entry.custom_id, "errored:", result.error.error.type, result.error.error.message)
    else:  # "canceled" or "expired": the request never ran
        print(entry.custom_id, result.type)
```

How Message Batches differ from the OpenAI-shaped batches above:

- **The `anthropic-version` header is required.** The SDK sends it for you.
  `2023-06-01` and `2026-10-21` are accepted. A batch keeps the version it was
  created with, and each of its requests is checked against that version's
  rules. `anthropic-beta` is refused.
- **Checking.** Some things are checked at once, and a problem with any of them
  refuses the whole batch: each `custom_id` must be 1–64 letters, digits, `_`
  or `-`, and unique, and the batch must stay within the request-count and
  body-size limits. Each request's `params` is then checked on its own. A request FreeInference cannot run as written, including
  one naming a model you may not use, becomes an `errored` result while the
  others still run. A parameter is never silently dropped: when a model's route
  cannot carry a parameter (extended thinking, cache control or structured
  output on a non-Anthropic backend, for example), that request errors instead.
- **Progress.** `request_counts` show every request as `processing` until the
  whole batch has `ended`; `results_url` appears then. Results can be read only
  once the batch has ended.
- **Cancelling and deleting.** Cancelling makes the batch `canceling` until its
  running requests finish. A Message Batch can be deleted only once it has
  ended (`400` before).
- **One turn per request.** A result that asks for a tool call is for your
  program to carry out and send back in a new request; FreeInference never runs
  your tools. Server tools, Files API sources, MCP servers and containers are
  refused.
- **Separate listing.** `GET /v1/messages/batches` lists your Message Batches.
  An ID from one API reads as not found on the other.
- **Your account, no workspaces.** Message Batches belong to your account,
  like every batch here.

## Embeddings and Responses in a batch

> Available where embedding and Responses batches are [enabled](#availability).

`/v1/embeddings` and `/v1/responses` can be batched like chat completions,
inline or from a file. Set `url` (inline) or `endpoint` (file) to the endpoint,
and give each request the body you would send to it. One batch targets one
endpoint.

A batched embedding request accepts what `/v1/embeddings` accepts. A batched
Responses request must be **independent**: it runs once, on its own, and is not
stored. So:

- omit `store` or set it to `false`;
- leave out `previous_response_id`, `conversation`, `background` and `stream`;
- hosted tools, file inputs and any other field FreeInference would have to
  drop are refused.

## Submitting safely twice

A network error during a create call leaves you not knowing whether the batch
was created. Send an `Idempotency-Key` header with the create call (any of the
three APIs) to make it safe to retry:

- the same key with the same request returns the batch the first call created,
  instead of creating another. This works even if your daily quota has since run
  out or your account has lost batch access;
- the same key with a different request answers `409`;
- after that batch has been deleted, the key answers `410`.

A key is 1–255 printable characters of your choosing, such as a job name and a
run number. It belongs to your account, so any of your API keys can replay it,
and a replay never changes which key the batch runs under. A key is remembered
until a day after its batch ends. The OpenAI and Anthropic SDKs do not send
this header for you: pass it with `extra_headers`, as in the examples above.
Without it, an SDK's automatic retry of a create call can submit the batch
twice.

## Costs

Batch requests cost the same as the same requests sent directly. There is no
batch discount. Every attempt at a request is recorded with what it cost, at the
same point and price as an interactive request, and a batch is never charged a
second time on top. `cost_usd` adds up the attempts whose cost is known.

Sometimes an attempt's result is lost after the request was sent, for example
when the request timed out. The provider may still have run it. Its cost is
then *unknown*: it is counted in `cost_unknown_attempts` and left out of
`cost_usd` rather than counted as zero. A retried request may therefore have
run more than once. A batch whose `cost_unknown_attempts` is not `0` may have
cost somewhat more than `cost_usd` says.

## Cancel a batch

`POST /v1/batches/{batch_id}/cancel` stops a batch and returns it:

```bash
curl -X POST https://freeinference.org/v1/batches/$BATCH_ID/cancel \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

Requests that have not started are cancelled; running ones finish and keep
their results. The batch is `cancelling` until they do, then `cancelled`. A
batch from a file that is cancelled while its file is still being checked ends
`cancelled` with nothing run. Cancelling a batch that has already finished, or
is `finalizing`, returns it unchanged. With the SDKs, call
`client.batches.cancel(batch_id)` or `client.messages.batches.cancel(batch_id)`.

## Delete a batch

`DELETE /v1/batches/{batch_id}` deletes a batch with its requests and results,
and with the output and error files of a batch from a file:

```bash
curl -X DELETE https://freeinference.org/v1/batches/$BATCH_ID \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

```json
{"id": "batch_6f1c0a7e2b8d4e55a1c3f9d2e4b6a8c0", "object": "batch.deleted", "deleted": true}
```

A batch can be deleted in any status, but not while one of its requests is
running: the answer is then `409`. Cancel the batch, wait for its running
requests to finish, then delete it. A Message Batch can be deleted only once it
has ended (`DELETE /v1/messages/batches/{batch_id}`).

FreeInference also deletes a finished batch on its own, 7 days after it
finished by default (`results_expires_at` says when), so collect its results
before then.

## List your batches

`GET /v1/batches` lists your batches, inline and from files, newest first:

```bash
curl "https://freeinference.org/v1/batches?limit=20" \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY"
```

`limit` is 1–100 (default 20). The answer has the shape of a page of results,
with batch objects in `data`; for the next page, pass `after` set to the
previous page's `last_id`. Message Batches are listed by
`GET /v1/messages/batches`.

## Batches in the console

The **Batches** page of the console, linked from your dashboard, lists your
batches from all three APIs with their progress, tokens and known cost. From it
you can cancel and delete batches, download results and, where webhooks are
enabled, set up your completion webhook.

## Completion webhooks

> Available where batch webhooks are [enabled](#availability).

Instead of polling, you can have FreeInference call an HTTPS endpoint of yours
when a batch ends. Register one endpoint for all your batches on the console's
**Batches** page, or with the API:

```bash
curl -X PUT https://freeinference.org/v1/batch_webhook \
  -H "Authorization: Bearer $FREEINFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/hooks/freeinference-batches"}'
```

The first time you set a webhook, the answer shows its signing secret,
`whsec_…`, **once**; store it. Changing the URL later keeps the secret. Send
`"rotate_secret": true` with the URL to get a new secret.
`GET /v1/batch_webhook` shows the webhook, `DELETE /v1/batch_webhook` removes
it, and `GET /v1/batch_webhook/events` lists recent deliveries.

When a batch ends, after its results can be read, FreeInference POSTs an
event: `batch.completed`, `batch.failed`, `batch.expired` or `batch.cancelled`
for an OpenAI-shaped batch, `message_batch.ended` for a Message Batch. The body
names the batch only, never a prompt or a result:

```json
{"id": "evt_…", "object": "event", "type": "batch.completed", "created_at": 1793016000, "data": {"id": "batch_6f1c…"}}
```

Fetch the results with your API key. When you receive a delivery:

- **verify its signature.** Deliveries are signed following the
  [Standard Webhooks](https://www.standardwebhooks.com/) specification
  (`webhook-id`, `webhook-timestamp` and `webhook-signature` headers), so its
  libraries can verify them with your `whsec_…` secret;
- **answer quickly with a `2xx`.** A delivery that does not get one is retried
  with backoff, up to eight attempts;
- **deduplicate on `webhook-id`.** The same event can arrive more than once.

The URL must be HTTPS and reach a public address; private, loopback and
link-local addresses are refused.

## Limits

| Limit | Default |
|---|---|
| Requests in one batch | 50,000 |
| Size of the create request's body, and of an input file | 32 MiB |
| One line of an input file | 4 MiB |
| One request's stored result | 4 MiB |
| Unfinished batches per account (`validating`, `in_progress`, `finalizing` or `cancelling`) | 10 |
| Unfinished requests across all your batches | 100,000 |
| Batch requests of yours running at once, by account tier | free 8, pro 16, internal 64 |
| Completion window: how long a batch's requests have to start | 24 hours |
| How long a finished batch and its results are kept | 7 days |
| How long an uploaded input file is kept | 30 days |
| Uploaded files you may keep, and upload per hour | 512 MiB; 60 |

These are the defaults, and FreeInference may change them. Split a larger job
into several batches. The role-based concurrency limit is counted per gateway
process; several processes can run more requests in total, subject to the
shared per-model limits. The completion window of a batch from a file and of a
Message Batch is always 24 hours. For an inline batch it is FreeInference's
setting, and a `completion_window` you send is ignored.

## Errors

Errors from creating a batch (and uploading a file):

| Status | When |
|---|---|
| `400` | The body is not valid JSON or not a batch; a request is invalid, uses an unknown, unavailable or auto model, or repeats a `custom_id`; the batch holds too many requests; the file named by `input_file_id` is not found, not ready or not `purpose=batch`; an extension is [not enabled](#availability); the `Idempotency-Key` is malformed; or a Message Batch lacks `anthropic-version` or sends `anthropic-beta` |
| `401` | The API key is missing, invalid or expired |
| `403` | Your account may not create batches; or Message Batches are not enabled |
| `404` | Batch inference is unavailable at the moment |
| `409` | The `Idempotency-Key` was already used for a different request |
| `410` | The batch this `Idempotency-Key` created has been deleted |
| `413` | The body or file is larger than the size limit, or an upload would exceed your file storage |
| `429` | You already have the most unfinished batches or batch requests allowed; your daily quota is used up until it renews at 00:00 UTC; or you have uploaded the most files allowed this hour |

Errors from managing batches:

| Status | When |
|---|---|
| `400` | Reading results: `after` is not a result ID such as `batch_req_42`. A Message Batch: reading the results of, or deleting, one that has not ended |
| `401` | The API key is missing, invalid or expired |
| `404` | The batch or file does not exist, was deleted, or belongs to another account; or batch inference is unavailable at the moment. The message says which |
| `409` | Deleting: one of the batch's requests is still running. Downloading `output.jsonl`: the batch has not finished |

Checking on, reading, downloading, cancelling and deleting a batch never answer
`429` for a used-up daily quota.

## Coming from OpenAI's or Anthropic's batch APIs

- **Requests run on FreeInference, not a provider's batch service.** Each one is
  served like an interactive request, at the same price: there is no batch
  discount, and turnaround depends on how busy the models are, not on a
  provider's batch queue.
- **The limits are FreeInference's** (see [Limits](#limits)), lower than the
  providers' own: 50,000 requests and 32 MiB per batch by default.
- **The completion window stops requests from starting.** Requests already
  running when it closes finish, so a batch can end after `expires_at`.
- **Extras of FreeInference's own:** the inline form, the paged
  `/v1/batches/{batch_id}/results`, the JSONL export,
  `DELETE /v1/batches/{batch_id}`, `Idempotency-Key`, and the fields `cost_usd`,
  `cost_unknown_attempts`, `results_expires_at` and `error`.
- **OpenAI file batches** run one model per file. Only `purpose=batch` files
  exist, and they serve batches only.
- **Responses requests in a batch are independent:** `store: true`,
  `previous_response_id`, `conversation`, `background` and hosted tools are
  refused.
- **Message Batches** belong to your account; there are no workspaces.
  `anthropic-beta` is refused, as are server tools, Files API sources, MCP
  servers and containers.

## See also

- [Available Models](models.md) — the model IDs a batch can use
- [API Headers Reference](api_headers.md) — authentication headers
- [IDE & Coding Agent Integrations](integrations.md) — send chat requests
  directly with the OpenAI SDK or curl
- [Quick Start](quickstart.md) — create an API key
- [HybridInference developer documentation: Batch Inference](https://github.com/HarvardMadSys/hybridInference/blob/main/docs/developer/batches.md)
  — the server-side reference behind this page