没有合适的资源?快使用搜索试试~ 我知道了~
AI对开发者生产力的影响:来自GitHub Copilot的证据.docx
1.该资源内容由用户上传,如若侵权请联系客服进行举报
2.虚拟产品一经售出概不退款(资源遇到问题,请及时私信上传者)
2.虚拟产品一经售出概不退款(资源遇到问题,请及时私信上传者)
版权申诉
0 下载量 167 浏览量
2024-04-17
12:48:15
上传
评论
收藏 1.97MB DOCX 举报
温馨提示
![preview](https://dl-preview.csdnimg.cn/89154500/0001-8f13a183c261d59b302acd223d986e08_thumbnail-wide.jpeg)
![preview-icon](https://csdnimg.cn/release/downloadcmsfe/public/img/scale.ab9e0183.png)
试读
19页
AI对开发者生产力的影响:来自GitHub Copilot的证据.docx
资源推荐
资源详情
资源评论
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![docx](https://img-home.csdnimg.cn/images/20210720083331.png)
![vsix](https://img-home.csdnimg.cn/images/20210720083646.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![docx](https://img-home.csdnimg.cn/images/20210720083331.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![](https://csdnimg.cn/release/download_crawler_static/89154500/bg1.jpg)
1
The Impact of AI on Developer Productivity:
Evidence from GitHub Copilot
Sida Peng,
1∗
Eirini Kalliamvakou,
2
Peter Cihon,
2
Mert Demirer
3
1
Microsoft Research, 14820 NE 36th St, Redmond, USA
2
GitHub Inc., 88 Colin P Kelly Jr St, San Francisco, USA
3
MIT Sloan School of Management, 100 Main Street Cambridge, USA
∗
To whom correspondence should be addressed; E-mail: sidpeng@microsoft.com.
Abstract
Generative AI tools hold promise to increase human productivity. This paper presents re-
sults from a controlled experiment with GitHub Copilot, an AI pair programmer. Recruited
software developers were asked to implement an HTTP server in JavaScript as quickly as
possible. The treatment group, with access to the AI pair programmer, completed the task
55.8% faster than the control group. Observed heterogenous effects show promise for AI
pair programmers to help people transition into software development careers.
Introduction
Artificial intelligence (AI) applications hold promise to increase human productivity. A va-
riety of AI models have demonstrated human-level capabilities in fields ranging from natural
language understanding to image recognition [Zhang et al., 2022]. As these systems are de-
ployed in the real-world, how do they change labor productivity? While there is a growing
literature studying perceptions of AI tools, how people use them, and their implications for
security and education [Nguyen and Nadi, 2022, Barke et al., 2022, Finnie-Ansley et al., 2022,
Sandoval et al., 2022] there has been little research on productivity impacts of AI-powered tools
arXiv:2302.06590v1
[cs.SE]
13 Feb 2023
![](https://csdnimg.cn/release/download_crawler_static/89154500/bg2.jpg)
2
in professional contexts, cf. [Mozannar et al., 2022, Vaithilingam et al., 2022, Ziegler et al., 2022].
The potential productivity impacts of AI have major implications for the labor market and
firms, including changes in employment, skills, and firm organization [Raj and Seamans, 2018,
Agrawal et al., 2019].
This paper studies the productivity effects of AI tools on software development. We present
a controlled trial of GitHub Copilot, an AI pair programmer that suggests code and entire func-
tions in real time based on context. GitHub Copilot is powered by OpenAI’s generative AI
model, Codex [Chen et al., 2021]. In the trial, programmers were tasked and incentivized to
implement an HTTP server in JavaScript as quickly as possible. The treated group had access
to GitHub Copilot and watched a brief video explaining how to use the tool. The control group
did not have access to GitHub Copilot but was otherwise unconstrained, i.e., they were free to
use internet search and Stack Overflow to complete the task.
The performance difference between treated and control groups are statistically and practi-
cally significant: the treated group completed the task 55.8% faster (95% confidence interval:
21-89%). Developers with less programming experience, older programmers, and those who
program more hours per day benefited the most. These heterogeneous effects point towards
promise for AI-pair programmers in support of expanding access to careers in software devel-
opment.
The paper proceeds as follows. We first describe the design of the controlled trial and
provide summary statistics. We then present the results. We conclude by a discussion on im-
plications of the study for productivity research on AI-powered tools, its limitations, and future
research directions on the broader economic impacts of AI-driven productivity.
![](https://csdnimg.cn/release/download_crawler_static/89154500/bg3.jpg)
3
Study Design
We conducted a controlled experiment to measure the productivity impact of using GitHub
Copilot in programming tasks. The experiment began on May 15, 2022 and ended on June 20,
2022, right before GitHub Copilot became generally available. We recruited 95 professional
programmers through Upwork, a freelancing platform. Participation in the experiment was
advertised on Upwork as a job posting, looking to recruit freelancer developers. Figures 1 and
2 show (respectively) the job posting and the contract that was sent to participants to sign, in
accordance with Upwork’s policies. Once participants signed the contract, they were randomly
split into control and treatment groups.
Figure 3 shows the instructions sent to each group through email. The treated group was
instructed to watch a 1-minute video introducing them to GitHub Copilot. In addition to the
instructions, they also received an automated email with installation instructions for GitHub
Copilot once granted access to the tool. We verify from telemetry after the experiment that all
participants from the treated group have configured GitHub Copilot and accepted recommenda-
tions other than five who did not finish the sign up and thus started the experiment without the
GitHub Copilot. Both treated and control groups were instructed to complete an entry survey to
provide demographic information such as age, gender, location, and educational background.
Before we began recruitment, we received approval for the study from the Microsoft Research
Ethics Review Board.
Participants were instructed to write an HTTP server in JavaScript—the treatment group
would use GitHub Copilot to complete the task, while the control group would not. Besides the
use of GitHub Copilot in the treated group, participants were unconstrained in their software
development —they could use any sources of information as they normally do, such as internet
search and Stack Overflow.
![](https://csdnimg.cn/release/download_crawler_static/89154500/bg4.jpg)
4
We calculated two metrics as a measure of performance for each group: task success and
task completion time. Task success was measured as the percentage of participants in a group
that adequately completed the task. Task completion time was measured as the time from start
to end of the task. Using a standardized task provides us with precise measures of performance
as it is difficult to measure productivity of software developers.
To administer the task, we used GitHub Classroom, a platform for teachers to issue and
grade coding assignments. In this way, we accurately measured the timing and completion for
each participant. The instructions gave participants a link to a particular GitHub Classroom
instance with a single assignment referencing a template repository. When joining the assign-
ment, participants received a personal copy of the template repository, with the task description
(shown in Figure 4) and a skeleton codebase for participants to build upon. The creation date
and time of that personal copy created a timestamp. Each participant’s repository was private
to them and visible to the researchers conducting the experiment—but not to other participants.
We included a test suite in the repository, comprising twelve checks for submission correct-
ness. If a submission passes, all twelve tests we counted are successfully completed. Partici-
pants could see the tests but were unable to alter them.
When participants committed and pushed their changes to GitHub, GitHub Classroom ran
the test suite on their submission and reported the number of passing tests. Participants could
push as often as they pleased, automatically logging a timestamp each time. The time elapsed
between the timestamp of repository creation and the timestamp of the first commit to success-
fully pass all 12 tests was counted as the participant’s task completion time.
The full history of test suite runs is visible on each repository, enabling researchers to ob-
serve partial results for participants that did not fully complete the task. The participants’ final
compensation is calculated based on their time to completion and the scale we had previously
shared with them (shown in Figure 1).
剩余18页未读,继续阅读
资源评论
![avatar-default](https://csdnimg.cn/release/downloadcmsfe/public/img/lazyLogo2.1882d7f4.png)
![avatar](https://profile-avatar.csdnimg.cn/68ef26bd67034c68b8d314222b3e4014_weixin_41429382.jpg!1)
百态老人
- 粉丝: 2112
- 资源: 2万+
![benefits](https://csdnimg.cn/release/downloadcmsfe/public/img/vip-rights-1.c8e153b4.png)
下载权益
![privilege](https://csdnimg.cn/release/downloadcmsfe/public/img/vip-rights-2.ec46750a.png)
C知道特权
![article](https://csdnimg.cn/release/downloadcmsfe/public/img/vip-rights-3.fc5e5fb6.png)
VIP文章
![course-privilege](https://csdnimg.cn/release/downloadcmsfe/public/img/vip-rights-4.320a6894.png)
课程特权
![rights](https://csdnimg.cn/release/downloadcmsfe/public/img/vip-rights-icon.fe0226a8.png)
开通VIP
上传资源 快速赚钱
我的内容管理 展开
我的资源 快来上传第一个资源
我的收益
登录查看自己的收益我的积分 登录查看自己的积分
我的C币 登录后查看C币余额
我的收藏
我的下载
下载帮助
![voice](https://csdnimg.cn/release/downloadcmsfe/public/img/voice.245cc511.png)
![center-task](https://csdnimg.cn/release/downloadcmsfe/public/img/center-task.c2eda91a.png)
安全验证
文档复制为VIP权益,开通VIP直接复制
![dialog-icon](https://csdnimg.cn/release/downloadcmsfe/public/img/green-success.6a4acb44.png)