How much does it cost to index a 10-K with Claude vs. a local LLM?

The practical question was simple: if an investment research system indexes a full 10-K, is it worth paying for a frontier model to write section summaries? And if so, which model?

The first cut of this benchmark, on Lumentum's fiscal 2025 10-K, compared three backends — a local Gemma 4 model, Opus 4.7 via the Claude Code CLI, and Opus 4.7 via the direct Anthropic API. Direct Opus was the clear cloud winner. Then I widened the lineup to all five backends, including Claude Haiku 4.5 and Sonnet 4.6. The new ranking surprised me. A month later I ran the whole thing again across eight filings with proper statistics, and the answer changed shape one more time.

This note records both runs. The summaries that come out of the indexer are not quoted back to the user — they are navigation aids. When the system answers a question, it returns raw filing text. So the benchmark is not "which model writes better prose?" It is "which model makes retrieval better, and at what cost?"

V1 benchmark — five backends on a single 10-K

The Lumentum 10-K triggered 12 summarization calls — every section above 2,000 tokens. We sent the same 12-call workload through each backend with identical prompts and max_tokens=256.

Haiku was the fastest backend by a wide margin and roughly one-twentieth the cost of Opus. On the small retrieval test set it scored 8 / 8 — one above Opus — because it happened to use the word "concentration" in a Note 17 summary where Opus paraphrased the same idea. That gap should not be read as Haiku being smarter. It is one-query noise on a sample of eight. V2 below settles the question.

The local model held its own for free at the margin but lost on factual specificity: it caught the 115% effective tax rate without the $153.1M UK valuation-allowance release that caused it, and it missed named geopolitical risks (Russia-Ukraine) that every Claude model wrote out.

Why the CLI path lost

Claude Code is useful when the task needs an agent — tools, file edits, terminal work, code review, multi-step debugging. Batch summarization is not that task.

The CLI route launched claude --print once per section. Each subprocess paid startup cost and carried Claude Code's own system context. The benchmark showed roughly 377,000 reported input tokens across 12 calls for the CLI path, versus roughly 47,000 input tokens through direct API calls. Most of the CLI context was cached, so the headline token count did not bill at normal input-token rates, but it still added 30% to cost and 8x to wall time for identical underlying work.

For a tight inner loop of plain LLM calls, use the SDK directly.

Quality at each price point

The income tax note and the debt note are the cleanest places to see what each price tier buys.

In the income tax note, gemma4 named the 115% effective rate but did not connect it to the $153.1M UK valuation-allowance release. Every Claude model captured both. Haiku and Sonnet hit the same headline numbers Opus did — $172.1M pre-tax loss, $198.0M tax benefit, 115.04% effective rate, $153.1M UK release. Opus alone added cross-period context ($440.8M total valuation allowance, down $49.6M year over year) and Sonnet alone called out the Subpart F / GILTI inclusions driving the rate.

In the debt note, all three Claude models pulled the same anchor facts: $2.573B total debt, both convertible note series with principal, interest rate, maturity, and conversion price. Opus added the trustee identity. Sonnet added the year-over-year delta. Haiku was essentially equal in factual content.

The picture from V1: Haiku captures the figures that matter for navigation. Sonnet adds comparative reasoning. Opus adds incidental detail. For 2-3 sentence summaries written at max_tokens=256, the incremental value above Haiku is small.

V2 — eight filings, 48 queries, proper statistics

The single-filing eight-query test in V1 left a real question open. Is the Haiku-beats-Opus result a finding or a coincidence? In May I reran the benchmark across eight 10-Ks spanning tech, finance, retail, cyber, infrastructure, services, and clean energy (LITE, AMD, AMZN, BLK, CRWD, CRWV, CAR, BE) with 48 queries — 24 specific-factual, 16 semantic-paraphrased, 8 negative tests — and bootstrap 95% confidence intervals.

I also added a second retrieval mode. V1 used pure keyword scoring over title and summary and section text. V2 kept that but also tested an LLM-reasoning retrieval step, where Sonnet 4.6 reads a tree of section titles and summaries and selects the most relevant node ids. The same reasoning model was used across every indexing backend, so the only variable was which model wrote the summaries.

The headline number, Recall @ 1 across all 48 queries with LLM-reasoning retrieval:

Every backend lands at exactly 0.71 with identical confidence intervals. There is no statistically meaningful difference between local Gemma 4 and frontier Opus when you measure end-to-end retrieval correctness. The V1 finding that Haiku "outperformed" Opus was noise on a small query set. The V1 finding that Haiku matches Opus at a fraction of the cost survives.

The cost of running V2 in full was $8.28: $5.93 across all 32 filing-by-backend index builds, $1.86 for 192 LLM-reasoning retrieval calls, $0.49 for a Sonnet judge that read every retrieved section and ruled on relevance.

Where the real lift comes from

The interesting structure shows up when you split queries by class.

Specific-factual queries — questions that map cleanly to a section title, like "what does the filing say about income taxes?" — went from R @ 1 of 0.67 with keyword retrieval to R @ 1 of 1.00 with LLM-reasoning retrieval. Perfect, across every backend. The reasoning step is doing all the work and the summaries are just navigation aids.

Semantic-paraphrased queries — questions worded differently from the filing, like "how is the company exposed to US-China trade friction?" — stalled at R @ 1 of 0.38 across every backend, with or without LLM reasoning. This is the hardest class, and no amount of indexing-LLM spend fixes it. The reasoning LLM cannot pattern-match "trade friction" to a Risk Factors section without more aggressive query expansion. Opus shows a faint edge on keyword retrieval here (0.19 vs 0.12) because its longer summaries contain more incidental keyword matches, but the LLM-reasoning step erases the gap.

Negative tests — questions about topics not in the filing — were correctly rejected 100% of the time by keyword search and only 50% of the time by LLM-reasoning retrieval. The reasoning LLM often picks the closest tangentially related section instead of saying "this isn't in the document." That is a prompt-tuning fix, not a backend differentiation.

The shape of the answer: keyword-to-reasoning retrieval lifts Recall @ 1 from roughly 0.56 to 0.71 across every backend. That is a 15-percentage-point improvement. Going from gemma4 to Opus at indexing time lifts it by zero percentage points. The summaries are navigation aids. The retrieval LLM is the brain. Spend money where the brain is.

What I would ship

For Clarion's own research stack, the default has shifted from "Opus by direct API" to "Haiku by direct API, plus an LLM-reasoning retrieval step."

Three operational levers behind that table:

The summarization threshold matters more than model choice. In this test, sections above 2,000 tokens triggered LLM calls. Raising that threshold reduces call count and cost directly, and on this evidence does less retrieval damage than dropping a tier of model.

The max_tokens=256 cap truncated most Opus and Sonnet summaries mid-sentence. Moving to 512 would roughly triple output cost but capture full thoughts. For Haiku at $0.04 per filing, that is not a real cost decision.

For semantic queries — the class that broke this pipeline — the next move is query expansion at retrieval time, not a smarter indexing model.

Methodology

V1. Source filing: Lumentum Holdings fiscal 2025 Form 10-K, filed August 19, 2025. Pipeline: PageIndex-style SEC RAG. The indexer converts SEC HTML into structured markdown, builds a heading tree, stores short sections as raw text, summarizes sections above 2,000 tokens, and writes a JSON index for query-time retrieval. Workload: 12 sections exceeded the summarization threshold. Prompt: identical across all backends, asking for 2-3 sentence summaries with key topics, named entities, specific figures, and notable dates. Each backend re-indexed from the identical source HTML so the prompts seen were byte-identical. Retrieval: deterministic keyword scoring (title=5, summary=3, text=1) over an eight-query test set with known target sections.

V2. Corpus: 8 10-Ks (LITE, AMD, AMZN, BLK, CRWD, CRWV, CAR, BE). Queries: 48 — 24 specific-factual auto-generated from each filing's LLM-summarized sections, 16 semantic hand-curated with deliberately different wording, 8 negative one-per-filing. Backends: same gemma4, Haiku 4.5, Sonnet 4.6, Opus 4.7 setups as V1; dropped the Claude Code CLI. Retrieval modes: keyword (same as V1) and LLM-reasoning (Sonnet 4.6 reads a tree of titles and 180-character summary previews and selects up to five node ids). Ground truth: source-section auto-assignment for specific queries; hand-curated expected node ids for semantic; Sonnet judge (212 calls, $0.49) rules on every top-3 section that ground truth did not already validate. Metrics: Recall @ 1, Recall @ 3, MRR with 95% bootstrap confidence intervals (1000 resamples).

Hardware and software: Apple Silicon Mac, Ollama 0.5.x, Python 3.13, anthropic==0.45.x, Claude Code CLI 2.1.143. Pricing: Anthropic list pricing at the time of the May 2026 test. Model pricing and model quality will change, so the durable result is not the exact dollar figure. The durable result is the shape of the tradeoff: indexing-model choice barely moves retrieval correctness when an LLM-reasoning step sits on top.

Sources

• Original benchmark writeup with full results, full sample summaries, and all V2 query-class tables: GitHub Gist
• Source filing lookup: SEC EDGAR company search for Lumentum Holdings
• PageIndex-style retrieval concept: PageIndex