Matched or beat the strongest frontier baseline on all five published benchmarks, with 30–140× lower serving cost. This page is the record behind that line: for each benchmark, the dataset, the metric, the frontier model actually run on the same items, the cost basis on both sides, and what the numbers do not show.
The path tested is stated per row: the intent row ran the
specialist directly from the benchmark harness; embedding and reranking
called the direct endpoints; ICD-10 and invoice ran the product path through
auto via the eval API. Quality is
shown relative to the strongest frontier baseline run on that benchmark
(= 100); the underlying metric is named per row. The cost ratio is that
baseline's measured cost divided by Pareta's for the same items.
| Task | Dataset | Metric | Pareta scorefrontier = 100 | Frontier compared | Cost ratio | Path tested | Date / version |
|---|---|---|---|---|---|---|---|
| Intent classification | Banking77 test split | macro-F1 | 113 | Claude Opus 4.7 | 88× lower | Specialist-path (benchmark harness) | May 2026 |
| Text embedding | CUAD retrieval queries | nDCG@10 | 109 | OpenAI text-embedding-3-large | 32× lower | Direct endpoint (/v1/embeddings) | 2026-07-10 |
| ICD-10 coding | De-identified inpatient discharge summaries from a credentialed clinical dataset, ICD-10-CM | micro-F1 | 106 | GPT-5.5 | 138× lower | Product path (auto, eval API) | 2026-07-06 |
| Invoice extraction | katanaml real invoices | field-level F1 | 103 | Claude Opus 4.7 | 142× lower | Product path (auto, eval API) | July 2026 |
| Document reranking | CUAD retrieval queries | nDCG@10 | 101 | Gemini 3.5 Flash | 31× lower | Direct endpoint (/v1/rerank) | 2026-07-09 |
One convention for every row, so the five can sit on one axis without averaging unlike metrics.
What was run, against what, and how each number on the table was produced.
auto uses for that task in production, without end-to-end routing, verification, or frontier escalation. Text embedding and document reranking called the direct endpoints /v1/embeddings and /v1/rerank, which are the production path for those workloads. ICD-10 coding and invoice extraction ran the product path — auto through the eval API — including routing and verification. None of the rows include route-mix or escalation statistics, so no row measures the blend of specialist and frontier serving that a live workload sees.X-Pareta-Frontier-Would-Have-Cost header uses the same tokens × list-price arithmetic but is an estimate, because the frontier model is not run; in these benchmarks it was.auto, and the per-unit rates for /v1/embeddings and /v1/rerank, as listed on the pricing page. Where a run predates the published token rates, the recorded cost derivation for that run is used instead; every figure carries a recorded derivation.auto applies to the task in production: the task's prompt and output contract, constrained decoding where the contract defines it (response_format with json_schema), and, for classification, the retrieved few-shot examples the serving path injects. Frontier baselines run zero-shot with the same label schema or output contract through the vendor API. For the intent row the benchmark harness's own 8-shot retrieval was used; the production path retrieves shots with Pareta's own embedder and scored 93.6 macro-F1 on the same 3,080 items in August 2026.| Metric | Rows | How it is scored |
|---|---|---|
| macro-F1 | Intent classification | One label per item, compared exactly to the gold label; an output that is not one of the labels counts as wrong. F1 is computed per class across the 77 intents and averaged without weighting, so rare intents count as much as common ones. |
| nDCG@10 | Text embedding · Document reranking | Each query's candidate pool is ranked — by cosine similarity of the embeddings, or by the reranker's relevance score — and scored against graded human relevance judgments derived from CUAD's expert clause annotations: 2 for a chunk that contains at least 90% of the annotated clause span, 1 for a substantial partial overlap, 0 otherwise. nDCG@10 is averaged over queries. |
| micro-F1 | ICD-10 coding | The predicted set of ICD-10-CM codes for each note is compared to the gold set. True positives, false positives and false negatives are pooled across all 150 notes before F1 is computed, so frequent codes weigh more than rare ones. Codes are compared without the decimal point. |
| Field-level F1 | Invoice extraction | Each header field is compared to gold after normalization. Line items are matched to gold rows without regard to order and then scored per sub-field; missing and hallucinated rows count against the score. |
These results answer one question — how the specialist performs on the task — and leave others open.
auto serves a request with a frontier model on a live workload, or the blended cost that results. Not yet published. Every response does carry the exact amount billed (X-Pareta-Billed) and an estimated frontier comparison (X-Pareta-Frontier-Would-Have-Cost); per-request serving path, escalation, and the serving revision ride as X-Pareta-Served-By, X-Pareta-Escalated, and X-Pareta-Route-Version.auto applies in production — result checks and frontier escalation when a specialist misses the bar — is exercised only by the two product-path rows (ICD-10 coding, invoice extraction); the harness and direct-endpoint rows do not exercise it, and no row reports route mix or escalation rate.The grading contracts that scored these benchmarks are the
ones the evaluation API applies to your data. Upload representative items
from the workload you plan to deploy — an evaluation set is 5–50 items —
name the frontier models to compare, and Pareta scores every contender,
auto and each frontier baseline, with the same scorer on the
same items, before you move live traffic.
json_schema for structured tasks).auto and the frontier baselines you name on the same items. Evaluation calls are metered like any other request: specialist rates where a Pareta specialist answers, the provider's list price for each frontier call.Changes to the published set are recorded here.