A label isn't a record.
open weights, closed weights, same spread of failure
Open or closed is a label, not a risk rating
Open models trail the frontier by about four months and are showing up everywhere. I split my 20-model census by weights. The label barely predicted failure. The model did.
On 4 October Chamath Palihapitiya replied "He's right." to Satya Nadella's warning that companies on closed models may "pay for intelligence twice": once in fees, once by handing over what they know. The open-weights case is now a boardroom case.
Chamath's 18 September deep dive makes the commercial version: open-weight models sit within roughly four months of the best publicly evaluated closed models, and leading companies use both, open for customisation and control, closed when they need the top capability. I agree with the framing. I'm adding the axis it leaves out.
Most of the argument is about price, control and capability. Almost none of it is about a fourth question: how does the model fail? That's the one I can check.
| What the market measures | Number | Who, and when |
|---|---|---|
| Capability lag, open behind closed frontier | about 4 months | Epoch AI, 29 May 2026 |
| Chinese models' share of Hugging Face downloads | 41% (a plurality) | Hugging Face, 17 Mar 2026 |
| Open share of enterprise AI use | 11%, down from 19% | Menlo Ventures, 495 US decision-makers, Nov 2025 |
| Open share of OpenRouter token usage | about one-third | OpenRouter and a16z, Dec 2025 |
The worst model on the chart is closed
My census runs 20 models on the same cases: 9 open-weight, 11 closed. Take Constraint Decay at Depth, where a rule set early in a long conversation gets silently dropped.
Split every mode by weights
Date arithmetic is the only mode where the open group's average model is reliably worse: 19% against 5%. One of six clearing zero is the kind of result I'd want to see twice before I built a policy on it. In my census, the gap between the two licence types is smaller than the gap inside each one.
The case for choosing by record
| Choose by label | Choose by record | |
|---|---|---|
| Money | Pay for the whole category, or exclude it | Pay for the specific model that clears your cases |
| Risk | A licence type stands in for a failure rate | The failure rate on your workload is the evidence |
| Advantage | The decision ages with each release | Rerun the same cases on the next release |
Two objections. Open weights keep your data in-house, which is Nadella's point and a real reason to pick them. Pick them for that, then test the one you picked. And my open group skews older, with a median release in March 2025 against June for the closed group. Qwen3 235B, released July 2025, failed none of the basis-demand cases. Which is the argument again: the model, not the label.
The first step costs a week, not a programme. Take the three failure modes your workload can least afford, write thirty cases for each, and run your open and closed candidates through the same set before the licence conversation starts.
Open or closed tells you who holds the weights. It doesn't tell you what the model does with your rule on turn forty.
When your team picked its last model, was it chosen by the label or by a record? Which would survive a board question?
- Epoch AI — Open models lag state-of-the-art closed models by 4 months (29 May 2026)
- Hugging Face — State of Open Source, Spring 2026 (17 Mar 2026)
- Menlo Ventures — 2025 State of Generative AI in the Enterprise (Dec 2025)
- OpenRouter and a16z — State of AI: an empirical 100 trillion token study (Dec 2025)
- Social Capital — Open vs. Closed deep dive, as summarised in Chamath Palihapitiya's LinkedIn article (18 Sep 2026); I read the summary, not the 99-page report
- Yahoo Finance — Chamath Palihapitiya backs Satya Nadella (5 Oct 2026)
- ModelCensus — the 20-model census, per-mode rates with intervals
Every figure here describes something measured and committed. See the measurements · read the method