Batches

Run many chat completions asynchronously, in exchange for more flexible capacity and routing that favors cheaper providers. OpenAI-compatible Batches API.

Batches don't run in real time — the /v1/batches endpoints create and monitor jobs, but the actual work happens in the background and completes within 24 hours (completion_window). For the end-to-end flow, see Batch workflow.

Base path: /v1/batches Auth: Bearer token or x-api-key. Create requires inference.chat (or a scope that includes it, such as inference.*). List/get/cancel require any inference scope.

Create a batch

POST /v1/batches

Request

Example:

curl https://api.melious.ai/v1/batches \
  -H "Authorization: Bearer sk-mel-<YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-<FILE_UUID>",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h",
    "metadata": {"run": "nightly-classifier-2026-04-22"}
  }'

Response

{
  "id": "batch_3f1c6c2e-8a5b-4d7e-9c0a-1b2d3e4f5a6b",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "input_file_id": "file-9d8c7b6a-5f4e-4d3c-b2a1-0f9e8d7c6b5a",
  "completion_window": "24h",
  "status": "validating",
  "output_file_id": null,
  "error_file_id": null,
  "created_at": 1699999999,
  "in_progress_at": null,
  "expires_at": 1700086399,
  "finalizing_at": null,
  "completed_at": null,
  "failed_at": null,
  "cancelled_at": null,
  "request_counts": { "total": 0, "completed": 0, "failed": 0 },
  "metadata": {"run": "nightly-classifier-2026-04-22"},
  "errors": []
}

request_counts.total is 0 until a worker picks the batch up and loads its lines. errors is a list; a batch that ends in failed or expired carries the reason there, for example [{"code": "insufficient_balance", "message": "…"}].

Status values

• validating — created, waiting for a worker. Your file was already validated at upload.
• in_progress — lines are running.
• finalizing — all lines done, writing the output and error files.
• completed — done. Results are in output_file_id; error_file_id is set if any line failed.
• failed — stopped before the end, for example because a line could not be paid for. errors holds the reason. Lines that finished are in output_file_id.
• expired — the 24-hour window passed. Unfinished lines fail with batch_expired. The output file exists only if a worker finalized the batch.
• cancelling — you called cancel; lines already running finish.
• cancelled — cancel completed.

List batches

GET /v1/batches?limit=20&after=<cursor>

Paginated list of your batches.

Response:

{
  "object": "list",
  "data": [ /* batch objects */ ],
  "first_id": "batch_...",
  "last_id": "batch_...",
  "has_more": true
}

Retrieve a batch

GET /v1/batches/{batch_id}

Returns the batch object. Poll this for status; request_counts updates as work progresses.

Cancel a batch

POST /v1/batches/{batch_id}/cancel

Cancels a batch in validating or in_progress; any other status returns 400. The response is the batch object with status cancelling — never cancelled — and cancelled_at set.

What happens next depends on whether a worker had started the batch. If it had, lines already running finish, the remaining lines fail with batch_cancelled, the output file is written, and the status becomes cancelled. If no worker had started it, nothing runs, there is no output file, and the status becomes cancelled when the 24-hour window ends.

Input file format (JSONL)

One request per line. Each line needs custom_id, method, url, and body:

{"custom_id": "q1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "<MODEL_ID>", "messages": [{"role": "user", "content": "..."}]}}
{"custom_id": "q2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "<MODEL_ID>", "messages": [{"role": "user", "content": "..."}]}}

custom_id must be unique within the file and is echoed into the output so you can match responses back. File IDs look like file-<uuid>, batch IDs like batch_<uuid>.

body is a chat completion request. Batch reads these fields and ignores everything else: model, messages, temperature, max_tokens (or max_completion_tokens), top_p, frequency_penalty, presence_penalty, stop, tools, tool_choice, response_format, seed, n, logprobs, top_logprobs. reasoning_effort and stream have no effect.

Don't add a flavor suffix to model IDs in a batch file. Batch lines already run under batch routing, which prefers lower-cost providers, and the model ID is used exactly as written.

Output file format

One line per input line, in input order. Match rows by custom_id; the id field is the same on every line of a batch.

{"id": "batch_req_<BATCH_UUID>", "custom_id": "q1", "response": {"status_code": 200, "body": {"id": "chatcmpl-...", "object": "chat.completion", "created": 1700000000, "model": "<MODEL_ID>", "choices": [{"index": 0, "message": {"role": "assistant", "content": "..."}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 12, "completion_tokens": 40, "total_tokens": 52}}}, "error": null}
{"id": "batch_req_<BATCH_UUID>", "custom_id": "q2", "response": null, "error": {"code": "context_length_exceeded", "message": "..."}}

response.body is a trimmed chat completion: id, object, created, model, choices (index, message with role, content, and tool_calls, finish_reason), and usage (prompt_tokens, completion_tokens, total_tokens). It has no billing_cost, environment_impact, or _meta.

Failed lines appear twice: in the output file with error set, and in error_file_id. Don't add the two files together.

Errors are per-row for ordinary request failures — a single bad request doesn't fail the whole job. Insufficient balance stops the batch: when a request cannot be paid for, every line that hadn't started yet is marked failed with the insufficient_balance code and the batch ends in failed. Lines that already completed keep their results in the output file.

Limits

Defaults. Operations can raise them per deployment.

Pricing and rate limits

How a line is paid depends on your plan. On a concurrency plan, a plan without API energy coverage, or with no plan, every line is billed in credits — a positive energy balance doesn't fund it. Before a line runs, Melious holds the price of its input tokens plus 1,024 output tokens, at the model's most expensive provider. max_tokens doesn't change the hold. When the line finishes you pay for the tokens actually used and the rest of the hold is released. If the hold can't be covered, the line fails with insufficient_balance and the batch stops.

See Pricing and Rate limits.

Errors

At upload (POST /v1/files with purpose: "batch"), the file is refused when:

• it is larger than 100 MiB (104,857,600 bytes), has more than 50,000 lines, or a line is longer than 1 MiB.
• it is empty, not UTF-8, or a line is not valid JSON.
• a line is missing custom_id (string), method ("POST"), url ("/v1/chat/completions"), body (object), body.model or body.messages.
• a custom_id repeats within the file.
• your stored batch files would exceed 1 GiB.

At create (POST /v1/batches):

• 400 — endpoint is not /v1/chat/completions; completion_window is not "24h"; metadata breaks the 16-pair / 64-character key / 512-character value rule; the file's purpose is not batch; a line's url is not the batch endpoint; you already have 10 batches in validating, in_progress, or finalizing.
• 403 — your key lacks inference.chat.
• 404 — input_file_id does not exist or belongs to another account.

While running, failures are per line and appear in the output and error files, not as HTTP errors from these endpoints:

• insufficient_balance — the line could not be paid for. The batch stops, remaining lines get the same code, and the batch ends in failed.
• batch_cancelled, batch_expired — the line never ran because you cancelled or the 24-hour window ended.
• any other code — that line failed; the rest of the batch continues. See Errors.

Related

Files for uploading input and downloading output • Batch workflow for the full lifecycle • Routing for the :batch flavor.