Melious
Inference

Batch API

Process large volumes of requests asynchronously with OpenAI-compatible batch processing

Batch API

Process large volumes of requests asynchronously with up to 24-hour completion windows. The Batch API is fully OpenAI-compatible and supports chat completions, embeddings, images, and audio.

Overview

Batch processing is ideal for:

  • Large-scale data processing - Process thousands of requests efficiently
  • Non-time-sensitive workloads - Background processing, data enrichment
  • Cost optimization - Better resource utilization for bulk operations

Batch jobs complete within 24 hours. For real-time responses, use the standard endpoints.

Endpoints

EndpointMethodDescription
/v1/batchesPOSTCreate a new batch job
/v1/batchesGETList your batch jobs
/v1/batches/{batch_id}GETGet batch job status
/v1/batches/{batch_id}/cancelPOSTCancel an in-progress batch

Create Batch

POST /v1/batches

Create a new batch processing job.

from openai import OpenAI

client = OpenAI(
    api_key="sk-mel-your-api-key-here",
    base_url="https://api.melious.ai/v1"
)

# First, upload your batch input file
batch_input = client.files.create(
    file=open("batch_requests.jsonl", "rb"),
    purpose="batch"
)

# Create the batch job
batch = client.batches.create(
    input_file_id=batch_input.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"project": "data-enrichment"}
)

print(f"Batch ID: {batch.id}")
print(f"Status: {batch.status}")
import OpenAI from 'openai';
import fs from 'fs';

const client = new OpenAI({
  apiKey: 'sk-mel-your-api-key-here',
  baseURL: 'https://api.melious.ai/v1'
});

// Upload batch input file
const batchInput = await client.files.create({
  file: fs.createReadStream('batch_requests.jsonl'),
  purpose: 'batch'
});

// Create the batch job
const batch = await client.batches.create({
  input_file_id: batchInput.id,
  endpoint: '/v1/chat/completions',
  completion_window: '24h',
  metadata: { project: 'data-enrichment' }
});

console.log(`Batch ID: ${batch.id}`);
console.log(`Status: ${batch.status}`);
# First upload your batch input file
curl https://api.melious.ai/v1/files \
  -H "Authorization: Bearer sk-mel-your-api-key-here" \
  -F purpose="batch" \
  -F file="@batch_requests.jsonl"

# Then create the batch job
curl https://api.melious.ai/v1/batches \
  -H "Authorization: Bearer sk-mel-your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-abc123",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h",
    "metadata": {"project": "data-enrichment"}
  }'

Request Parameters

ParameterTypeRequiredDescription
input_file_idstringYesID of uploaded JSONL file with batch requests
endpointstringYesTarget endpoint (see supported endpoints below)
completion_windowstringYesTime window: 24h
metadataobjectNoCustom key-value pairs (max 16 keys)

Supported Endpoints

EndpointDescription
/v1/chat/completionsChat completions
/v1/embeddingsVector embeddings
/v1/images/generationsImage generation
/v1/audio/transcriptionsSpeech-to-text

Input File Format

The input file must be a JSONL (JSON Lines) file where each line is a request object:

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "user", "content": "Hello!"}]}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "user", "content": "What is AI?"}]}}
{"custom_id": "request-3", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-oss-120b", "messages": [{"role": "user", "content": "Explain quantum computing"}]}}

Request Object Fields

FieldTypeRequiredDescription
custom_idstringYesYour unique identifier for tracking
methodstringYesAlways POST
urlstringYesTarget endpoint path
bodyobjectYesRequest body (same as regular API)

List Batches

GET /v1/batches

List your batch jobs with pagination.

batches = client.batches.list(limit=10)

for batch in batches.data:
    print(f"{batch.id}: {batch.status} ({batch.request_counts})")
curl "https://api.melious.ai/v1/batches?limit=10" \
  -H "Authorization: Bearer sk-mel-your-api-key-here"

Query Parameters

ParameterTypeDefaultDescription
limitinteger20Max results (1-100)
afterstringnullCursor for pagination

Response

{
  "object": "list",
  "data": [
    {
      "id": "batch_abc123",
      "object": "batch",
      "endpoint": "/v1/chat/completions",
      "status": "completed",
      "created_at": 1699999999,
      "completed_at": 1700000000,
      "request_counts": {
        "total": 100,
        "completed": 95,
        "failed": 5
      }
    }
  ],
  "has_more": false
}

Get Batch Status

GET /v1/batches/{batch_id}

Retrieve the current status of a batch job.

batch = client.batches.retrieve("batch_abc123")

print(f"Status: {batch.status}")
print(f"Progress: {batch.request_counts.completed}/{batch.request_counts.total}")

if batch.status == "completed":
    # Download results
    output = client.files.content(batch.output_file_id)
    print(output.text)
curl https://api.melious.ai/v1/batches/batch_abc123 \
  -H "Authorization: Bearer sk-mel-your-api-key-here"

Response

{
  "id": "batch_abc123",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "status": "completed",
  "input_file_id": "file-input123",
  "output_file_id": "file-output456",
  "error_file_id": "file-error789",
  "created_at": 1699999999,
  "in_progress_at": 1699999999,
  "completed_at": 1700000000,
  "expires_at": 1700086400,
  "request_counts": {
    "total": 100,
    "completed": 95,
    "failed": 5
  },
  "metadata": {
    "project": "data-enrichment"
  }
}

Batch Statuses

StatusDescription
validatingInput file being validated
failedValidation failed
in_progressRequests being processed
finalizingResults being compiled
completedAll requests processed
expiredBatch expired (24h limit)
cancellingCancellation in progress
cancelledBatch was cancelled

Cancel Batch

POST /v1/batches/{batch_id}/cancel

Cancel an in-progress batch job.

batch = client.batches.cancel("batch_abc123")
print(f"Status: {batch.status}")  # "cancelling"
curl -X POST https://api.melious.ai/v1/batches/batch_abc123/cancel \
  -H "Authorization: Bearer sk-mel-your-api-key-here"

Requests already in progress may still complete. Only pending requests are cancelled.

Output File Format

The output file is also JSONL format:

{"id": "response-1", "custom_id": "request-1", "response": {"status_code": 200, "body": {"id": "chatcmpl-abc", "choices": [...]}}}
{"id": "response-2", "custom_id": "request-2", "response": {"status_code": 200, "body": {"id": "chatcmpl-def", "choices": [...]}}}

Output Object Fields

FieldTypeDescription
idstringResponse ID
custom_idstringYour original custom_id
response.status_codeintegerHTTP status code
response.bodyobjectFull API response

Complete Example

Prepare Input File

Create batch_input.jsonl:

import json

requests = [
    {
        "custom_id": f"req-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-oss-20b",
            "messages": [{"role": "user", "content": f"Summarize topic {i}"}],
            "max_tokens": 100
        }
    }
    for i in range(100)
]

with open("batch_input.jsonl", "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")

Upload and Create Batch

from openai import OpenAI
import time

client = OpenAI(
    api_key="sk-mel-your-api-key-here",
    base_url="https://api.melious.ai/v1"
)

# Upload input file
input_file = client.files.create(
    file=open("batch_input.jsonl", "rb"),
    purpose="batch"
)

# Create batch
batch = client.batches.create(
    input_file_id=input_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

print(f"Batch created: {batch.id}")

Monitor Progress

while True:
    batch = client.batches.retrieve(batch.id)
    counts = batch.request_counts

    print(f"Status: {batch.status}")
    print(f"Progress: {counts.completed}/{counts.total}")

    if batch.status in ["completed", "failed", "cancelled", "expired"]:
        break

    time.sleep(30)  # Check every 30 seconds

Download Results

if batch.status == "completed":
    # Get successful results
    output = client.files.content(batch.output_file_id)
    with open("batch_output.jsonl", "w") as f:
        f.write(output.text)

    # Get errors (if any)
    if batch.error_file_id:
        errors = client.files.content(batch.error_file_id)
        with open("batch_errors.jsonl", "w") as f:
            f.write(errors.text)

    print("Results downloaded!")

Error Handling

Error CodeDescription
RESOURCE_NOT_FOUNDBatch ID doesn't exist
AUTH_INVALID_API_KEYInvalid API key
VALIDATION_INVALID_VALUEInvalid parameter
BATCH_ALREADY_COMPLETEDCannot cancel completed batch

Best Practices

  1. Use meaningful custom_ids - Makes it easy to correlate results
  2. Monitor progress - Check status periodically, don't poll too frequently
  3. Handle partial failures - Some requests may fail; check error file
  4. Set appropriate timeouts - 24h is the maximum window
  5. Use metadata - Tag batches for easier tracking

See Also

Streaming

Real-time streaming responses using Server-Sent Events (SSE)

Tools API

Execute AI-powered tools for web search, document processing, and more

On this page

Batch APIOverviewEndpointsCreate BatchRequest ParametersSupported EndpointsInput File FormatRequest Object FieldsList BatchesQuery ParametersResponseGet Batch StatusResponseBatch StatusesCancel BatchOutput File FormatOutput Object FieldsComplete ExamplePrepare Input FileUpload and Create BatchMonitor ProgressDownload ResultsError HandlingBest PracticesSee Also