生成式AI对高技能工作的影响-基于三项软件开发者领域实验的报告总结

The Effects of Generative AI on High Skilled Work:

Evidence from Three Field Experiments with

Software Developers

*

Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe,

Leon Musolff, Sida Peng, and Tobias Salz

September 2024

Abstract

This study evaluates the impact of generative AI on software developer produc-

Pennsylvania. Peng: Microsoft. Salz: MIT. We are grateful to Avi Goldfarb, Shane Greenstein, Anton

Korinek, Ethan Mollick, as well as the participants of the IIOC2024 conference and the AI, Cognition, and

the Economy Workshop. Thanks also for data help from employees at Microsoft, GitHub, and Accenture:

Phillip Coppney, Wen Du, Ya Gao, Lizzie Redford, Ryan J. Salva, Daniel A. Schocke, Amanda Silver, An-Jen

over 80% of U.S. jobs and that AI task coverage is notably higher for occupations that

require advanced degrees. The ability of generative AI to perform tasks required in such

high-skilled occupations — allowing it to assist doctors in diagnosing diseases, lawyers

in drafting legal documents, and software engineers with code development — has led

to predictions of substantial productivity gains from the adoption of such technologies

(Baily, Brynjolfsson, and Korinek 2023). Others, however, are less optimistic about such

productivity gains (Acemoglu 2024).

to make necessary complementary investments (Bresnahan 2024; Brynjolfsson, Rock, and

Syverson 2021) make it currently difficult to empirically assess whether or not optimism

ers, using software developers as an example. We analyze three large-scale randomized

controlled trials in a real-world environment. These experiments randomly assigned

access to Copilot, a coding assistant developed by GitHub in collaboration with Ope-

nAI, to just under five thousand software developers at Microsoft, Accenture, and an

anonymous Fortune 100 electronics manufacturing company (henceforth Anonymous

Company). These experiments were run as part of the ordinary course of business at

these companies to decide whether or how extensively to adopt these technologies, and

the companies kindly shared the resulting data with us.

raises task completion for more recent hires and those in more junior positions but not for

developers with longer tenure and in more senior positions. Prior work has shown that

1

lead to further improvements in quality or decrease the cost of inference and training.

2

in the top decile of human performance, performs at human level in a database of coding challenges that

measure programming logic and proficiency, and can write entire programs for simple video games from

several lines of instructions (Bubeck et al. 2023) Copilot is used by 1.3 million subscribers and more than

2023). Our results extend this finding by showing that even when workers are performing

tasks according to their tenure or seniority, generative AI increases productivity more for

lower-ability workers.

weight on periods with larger differences in treatment status. We make these choices

because our analysis must confront challenges related to statistical power, despite the

large number of software developers in the experiments. These challenges arise due to

large variation in measured outcomes and factors that reduce the take-up and duration

of the three experiments.

The experiment at Microsoft started before ChatGPT and

conducted in controlled lab-like experiments (Peng et al. 2023; Vaithilingam, Zhang, and

Glassman 2022, Campero et al. 2022, Noy and Zhang 2023). In a lab-in-the-field experi-

ment on consultants employed by Boston Consulting Group, Dell’Acqua et al. 2023 find

that productivity on 18 tasks designed to mimic the day-to-day work at a consulting

company increased by 12%-25%. Evidence from these experiments generally suggests

significant productivity effects of generative AI. The exception is Vaithilingam, Zhang,

and Glassman 2022, which did not find a statistically significant difference in completion

time. A second common observation from these studies is that generative AI has the

largest effect on the least productive and least experienced workers (e.g., Noy and Zhang

2023).

vational studies by studying the impact of generative AI using a field experiment in an

actual workplace setting. To date, there is still a dearth of experimental studies exam-

ining the effect of generative AI in a field setting. In a notable exception, Brynjolfsson,

Li, and Raymond 2023 find that an AI-based conversational assistant increases the pro-

ductivity of customer chat support agents by 14%. Our study complements theirs by

examining a field experiment with high-skilled and highly paid knowledge workers, a

group that is particularly relevant given the prediction that high-skilled jobs will be most

affected by this technology. Although we examine a different part of the skill distribu-

4

and employees in more junior roles.

GitHub Copilot, Amazon CodeWhisperer, and Replit Ghostwriter. In our study, we ex-

amine the effects of one of these tools, GitHub Copilot. GitHub Copilot was developed

by GitHub in partnership with OpenAI. The development of Copilot involved combining

advanced machine learning techniques and natural language processing. A substantial

amount of code from public GitHub repositories was used to train Copilot. This exten-

sive dataset allowed the AI model to learn from real-world coding practices, patterns,

and styles across various programming languages and frameworks.

text and generates relevant code snippets, comments, and documentation. It can au-

tocomplete code that developers might manually type or suggest snippets they would

crosoft, Accenture, and Anonymous Company. In the Microsoft and Accenture experi-

ments, one group of developers (the treated group) was randomly assigned to be able to

access GitHub Copilot, whereas the other group (the control group) did not have access

to the tool for a period of seven (Microsoft) or four (Accenture) months. In the Anony-

mous Company experiment, all users gained access to the tool over a period of two

months, but access dates were randomized, with some teams gaining access six weeks

before others.

Microsoft The experiment at Microsoft started in the first week of September 2022, in-

volving a sample size of 1,746 developers primarily located in the United States. Of these

developers, 50.4% were randomly selected to receive access to Github Copilot. Random-

Copilot as a productivity-enhancing tool and outlining its potential impact on their cod-

ing tasks (see Figure 6 in Appendix). Beyond this email, treated participants did not

receive any specific instructions regarding their workload or workflow to ensure they

use GitHub Copilot in their natural work environment. Control group participants did

not receive any communication as part of the study.

The experiment ended on May 3rd,

2023, as growing awareness of AI-assisted coding tools led the control group participants

to seek access to Copilot.