Get API Key
Melious

Guides

Batch workflow

Upload a JSONL file, kick off a batch, download the results

For non-critical work, batching is usually the right move — cheaper routing, outside the per-minute rate limit, and no need to hold open thousands of concurrent requests.

The pattern is: write a JSONL file of requests, upload it, create a batch, poll for completion, download the output.

Full example

This Python script takes a list of prompts, submits them, and returns the results indexed by your custom ID. One file, end to end.

import json
import time
import httpx

API = "https://api.melious.ai/v1"
KEY = "sk-mel-<YOUR_API_KEY>"
HEAD = {"Authorization": f"Bearer {KEY}"}


def run_batch(prompts: dict[str, str], model: str = "<MODEL_ID>") -> dict[str, str]:
    # 1. Build JSONL — one request per line, with a custom_id we'll use to match results
    lines = [
        json.dumps({
            "custom_id": cid,
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {"model": model, "messages": [{"role": "user", "content": prompt}]},
        })
        for cid, prompt in prompts.items()
    ]
    jsonl = "\n".join(lines).encode()

    # 2. Upload as a file with purpose=batch
    upload = httpx.post(
        f"{API}/files",
        headers=HEAD,
        files={"file": ("requests.jsonl", jsonl, "application/jsonl")},
        data={"purpose": "batch"},
    ).json()
    input_file_id = upload["id"]

    # 3. Create the batch
    batch = httpx.post(
        f"{API}/batches",
        headers={**HEAD, "Content-Type": "application/json"},
        json={
            "input_file_id": input_file_id,
            "endpoint": "/v1/chat/completions",
            "completion_window": "24h",
        },
    ).json()
    batch_id = batch["id"]
    print(f"batch {batch_id} created ({batch['status']})")

    # 4. Poll until the batch reaches an end state
    while True:
        time.sleep(30)
        status = httpx.get(f"{API}/batches/{batch_id}", headers=HEAD).json()
        counts = status["request_counts"]
        print(f"  status: {status['status']} "
              f"({counts['completed']} completed, {counts['failed']} failed, {counts['total']} total)")
        if status["status"] in ("completed", "failed", "expired", "cancelled"):
            break

    if not status["output_file_id"]:
        raise RuntimeError(
            f"batch ended in status {status['status']} with no output: {status['errors']}"
        )

    # 5. Download the output JSONL and parse
    output_file_id = status["output_file_id"]
    body = httpx.get(f"{API}/files/{output_file_id}/content", headers=HEAD).text

    results = {}
    for line in body.strip().splitlines():
        entry = json.loads(line)
        if entry["error"]:
            results[entry["custom_id"]] = f"ERROR: {entry['error']['message']}"
        else:
            choice = entry["response"]["body"]["choices"][0]
            results[entry["custom_id"]] = choice["message"]["content"]
    return results


if __name__ == "__main__":
    prompts = {
        f"q{i}": f"In one sentence, why did Hanseatic city #{i} matter?"
        for i in range(10)
    }
    answers = run_batch(prompts)
    for cid, text in answers.items():
        print(f"{cid}: {text[:80]}")

Five steps, each mapping to one endpoint:

  1. JSONL build — each line is a chat completion request body (the fields batch reads are listed in the reference), wrapped with a custom_id you choose.
  2. POST /v1/files — upload the JSONL with purpose=batch.
  3. POST /v1/batches — create the job pointing at the file and the target endpoint.
  4. GET /v1/batches/{id} — poll until status is one of completed, failed, expired, or cancelled.
  5. GET /v1/files/{id}/content — download the output, match by custom_id.

When to pick batch

Good fits:

  • Nightly classification, summarization, or extraction over a large set.
  • Backfills and reprocessing of historical data.
  • Evaluation runs.

Bad fits:

  • Anything user-facing in real time.
  • Workloads with hard latency SLAs shorter than the completion_window.
  • Small runs (say, under 50 requests) — the per-minute rate limit is fine for those and you skip the upload/download dance.

Handling partial failures

Batches don't fail atomically — for ordinary request failures, a single bad request returns an error for that row, and the rest still run. The output JSONL has error set for failed rows and response set for successes, both keyed by your custom_id.

One failure that stops the whole batch is insufficient balance: when a request cannot be paid for, the lines that hadn't started are marked failed with insufficient_balance, and the batch ends in failed. Lines that already completed keep their results. Top up, then run the unfinished lines as a new batch.

Check each failed row's error before resubmitting it. Retry only temporary failures, and include only those rows in the new batch — see Errors.

Cost

Batch isn't cheaper per token today — there's no discount applied to batched requests. The savings come from batch routing, which prefers lower-cost providers, and from being outside the per-minute realtime rate limit. On a concurrency plan, a plan without API energy coverage, or with no plan, lines are billed in credits. See Pricing.

Gotchas

  • 100 MiB input cap (104,857,600 bytes). Split larger runs into several batches.
  • custom_id must be unique within a file and is echoed into the output. Use something you can match back reliably — UUIDs are fine, sequential IDs are fine.
  • Chat completions only. endpoint must be /v1/chat/completions, and every line's url must be the same. A file with another url fails at upload; a batch for another endpoint fails at create. Send embeddings, images, and audio to their endpoints directly.

Batches reference • Files reference • Routing for the :batch flavor.

On this page

Full exampleWhen to pick batchHandling partial failuresCostGotchasRelated