🔍 Read the full analysis: ChatGPT Made Consultants More Effective At Data Science Tasks on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial involving Boston Consulting Group consultants found that those given GPT-4 and brief training scored higher on coding, statistics and prediction tasks than a group using Google and online resources. After ChatGPT was removed, the researchers found no evidence that the treated group performed better on technical questions, suggesting the measured gains depended on access to the tool.
A randomized trial at Boston Consulting Group (BCG) found that consultants given GPT-4 and 15 to 20 minutes of training scored higher than a comparison group on coding, statistics and prediction tasks, but researchers found no evidence of an advantage when participants later answered technical questions without ChatGPT. Published October 26 in Nature Human Behaviour, the study tests whether generative AI helps workers complete tasks beyond their existing skills—and whether that performance carries over after the tool is removed.
The study allocated 986 BCG consultants between a ChatGPT group and a control group offered Google and resources such as Stack Overflow and Khan Academy. The main analysis covered 487 participants who submitted work on both assigned tasks. Participants completed two of three 90-minute exercises: cleaning and combining datasets in Python, evaluating statistical models, or predicting how predictable international football matches are.
Compared with the control group, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. The researchers benchmarked performance against 44 BCG data scientists who did the same tasks without AI. The consultants using ChatGPT still scored below that expert group on average across all three tasks. The study reports that the ChatGPT group finished coding and prediction faster, while the time difference on statistics was not significant.
After the work tasks, participants answered five questions on technical topics without ChatGPT. The authors report no evidence that the ChatGPT group performed better than the control group on this follow-up. They also report that the AI group became more confident that ChatGPT could solve problems it could not actually solve; the authors characterize this confidence finding as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Performance Gains Did Not Carry Over
The results distinguish doing a task with AI from learning to do it independently. For employers, that difference affects how they interpret productivity gains: a worker may deliver stronger work with a chatbot without having acquired the underlying technical skill. The findings do not show that AI cannot support learning in other conditions, but this trial found no evidence of a measurable advantage on its short test once ChatGPT was unavailable.
That matters for work that depends on checking, explaining or repeating technical analysis. If employees rely on AI output without being able to evaluate it, errors may be harder to catch—particularly when the model produces a plausible but incorrect answer. The study’s exploratory finding about increased confidence in ChatGPT’s abilities adds a reason to take verification seriously, though it does not establish how often that confidence would lead to workplace mistakes.
The evidence is specific to the tasks, workers and training tested. It does not establish that every use of ChatGPT produces temporary gains, or that no form of AI-assisted training can build lasting skills. It does suggest companies should measure unaided competence separately from AI-supported output when making claims about workforce development.
Products Worth Considering
Logitech MX Master 4 Ergonomic Wireless Mouse with Haptic Feedback, Graphite
LG UltraGear 32-Inch QHD 180Hz Curved Gaming Monitor
Apple MacBook Pro 16.3-inch with PROLED XDR Display, Silver
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The experiment took place in late March and early April 2024 and was preregistered before data collection, on March 13, 2024. Participants in the treatment group received brief instruction on prompting ChatGPT for technical problems and then used GPT-4. The control group received instruction in using search engines and online learning resources instead.
BCG data scientists and OpenAI’s Economic Impacts Research team designed the tasks so ChatGPT could be useful but could not independently provide correct answers to every question. That design meant simply copying a chatbot response was not enough to complete the exercises. The researchers also compared participant scores with those of 44 BCG data scientists, who completed the tasks without AI.
The paper describes the effect as temporary upskilling: the AI helped participants perform work outside their skill set while they had access to it, but the follow-up did not show that they retained an advantage without it. The study’s main analysis included fewer than half of the 986 people originally allocated, because 573 began the survey and 487 submitted both tasks.
““Generative AI and the temporary upskilling of knowledge workers””
— The study’s authors
Limits of the Follow-Up Test
The follow-up consisted of five technical questions completed without ChatGPT. The supplied study summary does not establish whether performance would change after longer training, repeated practice, or a more extensive assessment of retained skills. It also does not show how the measured task results translate into the quality or productivity of consultants’ work on real client assignments.
The main analysis included 487 participants, compared with 986 people allocated to the trial. The available material does not give a full account of why participants did not complete both tasks or whether attrition affected the results. The finding about greater confidence in ChatGPT’s abilities is described by the authors as exploratory, so it should not be treated as a definitive measure of workplace risk.
The study was conducted at one consulting firm and tested a particular model, training session and set of tasks. The evidence therefore does not establish the same effects for other employers, workers, AI systems or training programs.
Testing Skills Beyond AI Access
The reported paper was published in Nature Human Behaviour on October 26, 2026. The source material does not identify a scheduled follow-up study or a further milestone. The immediate question for employers is whether similar results hold in longer-running training and in day-to-day work, where employees may use AI repeatedly and supervisors can review the output.
Further evidence could test whether additional instruction improves unaided performance, how long any gains last, and whether workers become better at identifying incorrect AI responses. Until those questions are answered, the trial supports a limited conclusion: in these tasks, access to ChatGPT improved measured performance, while the short unaided follow-up showed no evidence of a corresponding skill advantage.
Key Questions
What did the BCG study find?
Consultants using ChatGPT scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than a control group using Google and online resources. On average, they still scored below BCG data scientists who completed the tasks without AI.
Did participants retain the skills after ChatGPT was removed?
On a follow-up of five technical questions completed without ChatGPT, the authors found no evidence that the ChatGPT group did better than the control group. The test was limited, so the result does not answer whether longer or different training could produce lasting gains.
Who took part in the experiment?
The trial allocated 986 BCG consultants between two groups. The main analysis included 487 who submitted work on both assigned tasks. A separate benchmark consisted of 44 BCG data scientists who completed the tasks without AI.
Does the study prove ChatGPT cannot teach data science?
No. It tested a particular set of tasks, a brief training session and a short follow-up. The findings indicate that the performance gains measured in this trial did not appear on the unaided questions; they do not rule out lasting learning from other approaches.
Source: ThorstenMeyerAI.com
