🔍 Read the full analysis: Why The Best AI Test Result Didn’t Become My News Pipeline’s Model on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer reports that a test of 71 AI configurations found GPT-6 Astra produced the highest-scoring news articles, while GPT-6 Luna was selected for production at roughly one-hundredth of Astra’s reported per-article cost. The test also found that two fallback models fabricated quotes or failed output checks; both were removed from the news route on October 7.
Thorsten Meyer says he chose GPT-6 Luna to write articles in his news pipeline, even though GPT-6 Astra earned the highest scores in his test of 71 model configurations. In results published October 13, Meyer reported that Astra scored 4.00 out of 5 in both English and German, but cost about $0.12 per article, compared with about $0.0012 for Luna, which scored 3.67 in the robustness run.
The test ran from September 8 to October 13, 2026, using the same production prompt and real news sources. Meyer tested API, subscription, free-tier and locally run open-weight configurations. Each source was used to produce an English and a German article, which were graded by GPT-6.1 Sol and Claude Opus 5.5 on source fidelity, fabrication, clarity, news value, structure and polish, with German language quality also assessed.
Meyer used two initial sources: a 6,340-character product filing and a 619-character political news item. He said short sources are common in production, so the thinner item was the more representative test. A later robustness run covered 10 additional articles for the strongest configurations, mostly using short sources. Meyer said the robustness results were more reliable than the initial comparison and treated small gaps in the two-article test as noise.
In the robustness run, Astra scored 3.93 across 12 articles, with a reported 95% bootstrap interval of 3.83–4.02. Luna scored 3.67, with an interval of 3.59–3.76. Meyer reported that Astra’s standard-test source-fidelity score was 5.00, while Luna’s was 4.71 in the robustness run; every Luna article passed the pipeline’s output-validity check. He said Luna cost about $0.0013 per article in recent production and took a median of 17 seconds, versus roughly 50 seconds for Astra.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Why the Pipeline Uses Luna
The results illustrate a practical distinction between the highest test score and a model suited to a high-volume workflow. Based on Meyer’s reported figures, Astra delivered stronger writing scores, but Luna’s much lower per-article cost and consistent valid outputs made it his production choice. That trade-off matters to publishers weighing quality against recurring cost and processing time; it does not establish that Luna is the best choice for other newsrooms or prompts.
The test also highlights the risk of measuring only the primary model. Meyer said two fallback systems in his route produced serious failures: one generated invented quotes, while another produced no valid articles in its four attempts. A backup model can become a source of fabricated material if a pipeline routes work to it without equivalent checks. Meyer’s account makes fallback quality and output validation part of the editorial reliability question, not merely an engineering detail.
These are results from one publisher’s task and evaluation setup, not an independent assessment of every model or news use. The reported scores depend on the selected sources, prompt, judges and validity rules. They offer a case for testing models against actual newsroom work rather than treating general leaderboard rankings as a direct guide to editorial performance.
As an affiliate, we earn on qualifying purchases.
How Meyer Ran the Test
Meyer framed the project around a specific operational question: which model should turn the sources used by his sites into finished articles? He used his production prompt rather than a generic writing prompt, created editions in English and German, and required outputs to pass the same repair step used by the pipeline before counting them as valid.
The scoring combined judgments from two AI evaluators that both received the original source. Meyer said the judges rated seven areas, including whether articles stayed faithful to the source and avoided fabrication. He also reported a robustness run of 10 more articles for leading configurations, with 95% bootstrap intervals. He cautioned that the two-source initial test was too small to treat differences below about 0.3 points as meaningful, making the broader run central to his final ranking.
His account also compares model performance across tasks. Meyer said Claude Opus 5.5 performed strongly on a coding benchmark but scored 3.52 in the article-writing robustness run, below Luna. That comparison supports his narrower point: coding rankings did not predict writing performance in this particular test.
“For a high-volume task, the right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”
— Thorsten Meyer, in the October 13 report
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Scores
The findings come from one publisher’s workflow and a limited set of source material. The initial comparison used two sources, while the follow-up added 10 articles for leading configurations. Meyer reported confidence intervals for the robustness run, but the results do not establish how the models would perform across other topics, languages, prompts or editorial standards.
The article scores were assigned by AI judges, not described as a blinded review by human editors. The report does not establish whether human reviewers would rank the articles in the same order, or how often a newsroom would need to correct factual or stylistic issues before publication. It also does not provide a full independent audit of the scoring process or the reported costs.
Meyer said GPT-4.1 nano and gpt-oss-120b were removed from the route after the tests, but the supplied account ends with an incomplete description of the replacement arrangement. It states that Luna now runs first and begins to name a second Luna configuration, without specifying the complete fallback order. The status of any other backup models is not clear from the report.
AI language model API subscription
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Next Pipeline Check
Meyer’s immediate change, according to his report, was to remove GPT-4.1 nano and gpt-oss-120b from the news route on October 7. Luna became the primary production model. The report does not specify a complete revised failover chain, so readers cannot tell which model would take over if Luna were unavailable or failed a production check.
The next useful evidence would be continued production results: whether Luna maintains its reported article quality across a larger range of real sources, whether its costs and response times remain similar, and how often editors or automated checks reject its output. The report does not announce a date for a further test or provide a schedule for updating the benchmark.
For now, Meyer presents the decision as a workflow-specific choice, not a universal model ranking. The key open question is whether the reported balance of cost, speed and source fidelity holds over sustained use, especially on the short and varied sources his pipeline commonly handles.
cost-effective AI writing assistant
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
Why did Meyer choose GPT-6 Luna instead of GPT-6 Astra?
Meyer said Astra earned higher article scores, but cost about $0.12 per article. Luna scored 3.67 in the robustness run, passed the pipeline’s validity check in every tested article and cost about $0.0013 per article in recent production. He chose Luna for the reported balance of quality and cost in his workflow.
How many AI configurations were tested?
Meyer reported testing 71 configurations between September 8 and October 13, 2026. The configurations included API, subscription, free-tier and locally run open-weight models.
What happened with the fallback models?
Meyer said GPT-4.1 nano scored poorly and generated invented quotes flagged by the evaluators. He said locally run gpt-oss-120b produced zero valid articles out of four and also generated named quotes not present in the source. Both were removed from the news route on October 7.
Did higher reasoning effort improve Luna’s articles?
Not reliably in Meyer’s tests. He reported that the score differences across six reasoning-effort settings fell within the test’s noise, while the maximum setting cost about 12 times as much and took about 12 times as long as the no-effort setting.
Does this test show which model is best for every newsroom?
No. The test used Meyer’s own prompt, sources, language editions and AI judges. Its findings describe one production workflow; they do not establish how the models would rank under other newsroom requirements or editorial review.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
