【免费】软件工程众包系统文献综述_软件工程文献综述格式资源-CSDN文库

软件工程

需积分: 0 194 浏览量 2023-04-09 10:08:33 上传评论收藏 1.2MB PDF 举报

资源推荐

资源详情

资源评论

The Journal of Systems and Software 153 (2019) 200–219

Contents lists available at ScienceDirect

The Journal of Systems and Software

journal homepage: www.elsevier.com/locate/jss

A systematic literature review on crowdsourcing in software

engineering

Aslı Sarı

, Ay ¸s e Tosun

, Gülfem I ¸s ıklar Alptekin

a , ∗

Department of Computer Engineering, Galatasaray University, Istanbul, Turkey

Department of Computer Engineering, Istanbul Technical University, Istanbul, Turkey

a r t i c l e i n f o

Article history:

Received 20 September 2018

Revised 6 April 2019

Accepted 8 April 2019

Available online 9 April 2019

Keywords:

Crowdsourcing

Crowdsourcing in software engineering

Systematic literature review

Empirical software engineering

a b s t r a c t

Background: Crowdsourcing outsources a task to large groups of people by open call format, and it re-

cently plays signiﬁcant role for software practitioners.

Aim: The purpose of this study is to conduct a comprehensive overview on crowdsourcing in software

engineering (CSE), concerning business models, tools, platforms, software development processes, and

software economics.

Method: We conducted a systematic literature review on CSE. We identiﬁed 158 relevant studies and 6

secondary studies. We further reviewed 67 primary studies that passed our quality assessment criteria.

We deﬁned 10 research questions and synthesized different approaches used in primary studies regarding

each question.

Results: Majority of studies report the application of crowdsourcing for coding and testing tasks. Crowd-

sourcing follows a unique methodology in which project planning, task speciﬁcation and deployment

have more emphasis. There is not enough literature on effort estimation approaches in CSE and associ-

ated cost factors. Complexity of the task and its expected duration play signiﬁcant role in estimation.

Conclusions: Future studies should focus more on economic models, experience reports, speciﬁc software

development methodologies, and strategic pricing mechanism for CSE.

Introduction

Crowdsourcing is an emerging paradigm that outsources the

tasks to large group of people by means of open call format in

order to utilize collective intelligence. The term was coined by J.

Howe and M. Robinson ( Howe, 2010 ); it deﬁnes crowdsourcing as

outsourcing a work to undeﬁned, networked people in the form

of open call. Crowdsourcing process contains several actors and

their interactions such that crowdsourcing requester (client) sub-

mits the task request, the crowd performs crowdsourced tasks and

crowdsourcing platform executes crowdsourcing requests as broker

( Vukovic, 2009 ). In recent years, more and more companies have

relied on crowdsourcing; seeing its beneﬁts in terms of build-

ing software systems in a cost and time-eﬃcient way, and liabil-

ity, quality and law enforcement ( Naik, 2016 ). Accordingly, various

software design and development models are proposed for crowd-

sourcing in software engineering (CSE).

∗

Corresponding author.

E-mail addresses: asli.SARI@ogr.gsu.edu.tr (A. Sarı), tosunay@itu.edu.tr (A. To-

sun), gisiklar@gsu.edu.tr (G.I. Alptekin).

Several literature reviews and surveys have been conducted to

identify crowdsourcing terminology, roles, responsibilities, and ap-

plications of crowdsourcing to wide range of research areas. Six

literature reviews on CSE are discovered that summarize process

models, platforms and features ( Ambreen and Naveed, 2016; Hos-

seini et al., 2015; Mao et al., 2017; Morschheuser et al., 2017;

Thuan et al., 2016; Stol et al., 2017 ).

These earlier secondary studies focus on the general deﬁni-

tions and main actors in CSE. Except Mao et al. (2017) , none

of them conducted a systematic literature review (SLR) on this

topic. In the most comprehensive SLR related to CSE ( Mao et al.,

2017 ), the authors aggregate the existing evidence on the crowd-

sourcing applications with respect to the software process areas

and commercial platforms in case studies. Mao et al. (2017) state

CSE theories and models, task decomposition and remuneration

are among the open issues that need further investigation in

the future studies. In our SLR, we complement earlier reviews,

and systematically aggregate the characteristics of CSE with new

research questions covering open issues. We introduce the re-

search questions, namely RQ1, RQ2, RQ5, RQ6, RQ7, RQ8, and RQ9,

to summarize existing business models, technological platforms

https://doi.org/10.1016/j.jss.2019.04.027

A. Sarı, A. Tosun and G.I. Alptekin / The Journal of Systems and Software 153 (2019) 200–219 201

used for management of crowdsourcing activities, effort estima-

tion models cost drivers used in effort estimation, award deter-

mination mechanisms, and crowd selection and task decomposi-

tion strategies. Although the tools that assist CSE are studied in

details by Mao et al. (2017) , in our research we associate the as-

sisted tools with assisted task, award mechanism and supported

crowd selection strategy. We also compared our ﬁndings with

Mao et al. (2017) on the software development models (RQ3), and

software process areas that the crowdsourcing has been utilized

(RQ4). We also discuss the ﬁndings in earlier secondary studies

and compare our results aggregated through a systematic literature

review methodology. We believe our SLR provides a more com-

prehensive overview on the crowdsourcing in software engineering

tasks.

The paper is organized as follows: Section 2 presents prelim-

inaries of software business models and crowdsourcing by giving

related works. Six secondary studies on CSE are also examined

in Section 2 . Section 3 presents the steps of our SLR methodol-

ogy, whereas Section 4 reports the ﬁndings for each of our re-

search questions. A comprehensive discussion and threats to va-

lidity about our research questions are given in Section 5 . Finally,

Section 6 concludes the paper by introducing recommendations,

implications and open issues.

2. Preliminaries and related work

2.1. Business models in software development industry

Business models in software industry consists of ﬁve main char-

acteristics according to the framework proposed in Schief (2013) :

Strategy, Revenue, Upstream, Downstream and Usage . Although there

are other studies deﬁning business models with a list of com-

ponents, Schief (2013) states that these proposed characteristics

cover and align with the existing literature. The ﬁrst characteris-

tic, Strategy, deﬁnes the competitive advantage of the offered prod-

uct/service by the company in terms of value proposition, invest-

ment model, value chain for sustainability, integration to vertical

value chains, and cooperating partners. For instance, decisions such

as “the produced software will have higher quality than competi-

tors, be easier to maintain, use less resources, and prioritize se-

curity concerns of the users” are related to the Strategy of the

software company. The second characteristic, Revenue , deﬁnes the

amount of sold products, who pays for the product, revenue shar-

ing mechanisms, and the pricing model. The pricing model has an

association with the other characteristic, Upstream , during value

creation, i.e., how the product will be developed, whether it is

going to be an internal manufacturing or outsourcing, what ac-

tivities/processes are internally implemented, and key cost drivers

( Schief, 2013 ). Key cost drivers are deﬁned as Research & De-

velopment, Sales & Marketing, Services, Hardware and others in

Schief (2013) . Particularly in our study, we are concerned with the

research and development models in the software industry. For ex-

ample, the development costs or the number of resources may be

lower when an open source model is integrated into the prod-

uct development, whereas if a crowdsourcing model is preferred,

some processes may be internally managed and completed with

fewer resources while the others are done with collaboration to

the crowd workers. Depending on the crowdsourced task, one or

more processes may be externally developed with lower personnel

costs, and higher quality solutions could even be achieved through

the crowd collaboration and competition. The other two character-

istics of the business model framework, namely Downstream and

Usage, are concerned with the target market, distribution areas,

how the software is offered and used, etc. ( Schief, 2013 ). Due to

its differences in terms of research and development activities, re-

source model, pricing model and so on, crowdsourcing signiﬁcantly

changes the business models and strategies in the software indus-

try ( Howe, 2010 ).

2.2. Crowdsourcing

Crowdsourcing has been applied to various domains, such as

labelling large datasets ( Welinder and Perona, 2010 ), creative and

design-based applications ( Mao et al., 2017 ), and translation of text

to different languages ( Zaidan and Callison-Burch, 2011 ). Crowd-

sourcing utilizes open call format in order to offer tasks of dif-

ferent companies to a large group of people/committee. The com-

munity decides whether it performs tasks by attending competi-

tions or not. Crowdsourcing requestor speciﬁes and submits task

descriptions and performs management operations during crowd-

sourced progress. Beside this, the crowd achieves goals of the

released tasks individually or collaboratively in speciﬁc time al-

lowance. The ﬁnal role in crowdsourcing process is the crowd-

sourced platform. The crowd submits their works through the

crowdsourcing platform. In other word, when the task is done

by the crowd, crowdsourced platform intermediates between the

crowd and the requestor who evaluates the quality of the work.

According to the quality level of the work, crowdsourcing requestor

makes payment to the collaboratively worked members or to the

best solution ( Sari and Isiklar Alptekin, 2017 ).

The three well-known contributions via crowdsourcing are peer

production, competitions and micro-tasks ( LaToza and Hoek, 2016 ).

In peer production, large groups of people contribute software

projects e.g., Linux, Apache. In peer production approach, open

sourcing is the oldest model where people evolve works collabo-

ratively without any reward expectation. Competitions constitutes

the other approach of crowdsourcing, where workers compete with

each other for achieving projects’ goals in order to gain a mon-

etary reward. In competitive platforms, e.g., TopCoder, the client

identiﬁes project requirements in terms of its description, budget

and time. The requirements are submitted to the crowdsourced

platforms. The copilot/platform manager divides the projects into

several tasks, which are competition tasks with different rewards.

The large group of workers i.e. community propose diverse solu-

tions for these tasks. The best solution is voted or chosen among

other solutions and winning solution is rewarded ( LaToza and

Hoek, 2016 ). In micro-tasking, the crowd works on tasks that are

decomposed into a set of self-contained small tasks to be com-

pleted in a short time period via a platform such as Amazon’s Me-

chanical Turk (AMT) ( LaToza and Hoek, 2016 ).

In related literature, crowdfunding is deﬁned as one of the spe-

ciﬁc types of crowdsourcing. It is deﬁned as a development ap-

proach where ﬁnancial resources of projects are funded by large

number of individuals in terms of entrepreneurs, nonproﬁt organi-

zations and artists, in order to develop the projects on the inter-

net ( Bohliqa, 2015 ). The effect of crowdfunding has increased dra-

matically on design, ﬁlm making, music and photography projects

( Bohliqa, 2015 ). The difference between them is that in crowdfund-

ing, ﬁnancial resources of projects are collected from the crowd,

whereas in crowdsourcing, the crowd works collaboratively in or-

der to achieve common goals of the projects. Another speciﬁc type

of crowdsourcing is deﬁned as crowd wisdom ( Yin et al., 2015 ).

The main idea of crowd wisdom is that people ask their ques-

tions to the large group of people who are willing to answer. The

main concern is the independencies of individual opinions. Yahoo

Answers is one of the popular application area of crowd wisdom

on the internet. The last special type of crowdsourcing is given

as crowdsourced design ( Luther et al., 2015 ). The goal is designing

something by funding a large group of people. 99designs, which

performs designs such as logos, web design, etc. is a popular com-

mercial example of crowdsourced design approach.

202 A. Sarı, A. Tosun and G.I. Alptekin / The Journal of Systems and Software 153 (2019) 200–219

Table 1

Secondary studies on crowdsourcing in software engineering.

SLR # of

studies

Covered period

of time

Databases RQs Highlights

Hosseini et al.

(2015)

113 From 2006

until January

2014

ScienceDirect,

ACM, IEEE Xplore,

Google Scholar,

DBLP

RQ1. What are the main disciplines in

which crowdsourcing has been

researched?

RQ2. What types of research are used

to conduct the study?

RQ3.What forms of study are adopted

to conduct the researches of

empirical

nature?

RQ4. What communities have

conducted the study and what types of

publication are used to disseminate the

research?

RQ5. What are the research trends in

the domain?

RQ6. What features are commonly used

in crowdsourcing for characterizing?

1. Domains of studies

2. Types of studies

3. Forms of studies

4. Publication types of studies

5. Aﬃliation of studies

6. List of crowd features, crowdsoucer

features, crowdsourced task features,

crowdsourcing platform features

7. Frequencies

of each features

8. Distribution and emerging year of each

feature

9. Essentiality and optionality of each

feature

Ambreen and

Naveed (2016)

400 From 2005 till

the year 2015

ACM, IEEE Xplore,

ScienceDirect,

SpringerLink

RQ1.What is the state art of empirical

literature of crowdsourcing in

computing?

1. Yearly distribution of studies

2. Types of research methods

3. Frequency of research outputs

4. Distribution of studies along process

areas of SE

5. Platforms of CSE

6. Platforms used for empirical studies of

CSE

Thuan et al.

(2016)

38 Between

February and

March 2013

ACM, IEEE Xplore,

ScienceDirect,

SpringerLink,

Sage, Emerald

RQ1.What factors inﬂuence

organisation’s decision to crowdsource?

1. Factors inﬂuencing the decision to

crowdsource

2. Theoretical framework to support to

decision to crowdsource

Mao et al.

(2017)

210 Before April

2015

ACM, IEEE Xplore,

Springer Link,

Wiley, Science

Direct, ProQuest,

Google Scholar

No speciﬁc research question(s) 1. CSE deﬁnitions

2. Crowdsourcing Practice in SE as

commercial platforms and case studies

3. CSE applications

in different process

areas

Stol et al.

(2017)

18 No time zone No speciﬁed

database

No speciﬁc research question(s) 1. Ways how new developers join software

crowdsourcing projects

2. Ways how to engage the crowd in

requirements elicitation

3. Q&A sites of crowdsourcing platforms

4. Crowdsourced testing approaches

5. Ways of leveraging the crowd to

generate code summaries to motivate to

developers to contribute

6. Ways of leveraging

different types of

crowds beneﬁts

Morschheuser

et al. (2017)

110 Up to October

2016

Scopus Database No speciﬁc research question(s) 1. Bibliometric and descriptive information

of crowdsourcing and gamiﬁcation

2. Characteristics of gamiﬁed

crowdsourcing systems

3. Empirical studies on the psychological

and behavioral outcomes of gamiﬁed

crowdsourcing systems

4. Gamiﬁcation’s overall effectiveness in

crowdsourcing

2.3. Secondary studies on crowdsourcing in software engineering

We found six secondary studies in which various crowdsourc-

ing approaches have been summarized based on related literature

in the ﬁeld of software engineering. In Table 1 , we summarized

them by giving the considered databases, number of primary stud-

ies, research questions, covered period of time and main insights

in the paper.

In Mao et al. (2017) , researchers conduct a comprehensive sur-

vey to explore trends, open issues and opportunities on CSE. The

authors state that CSE has dramatically increased in both indus-

trial and academic communities, and hence, it is an emerging and

fast-expanding research area in software engineering. They iden-

tiﬁed in detail theory and model foundations, task decomposition,

planning and scheduling, motivation, remuneration, and quality as-

surance as issues in CSE. Another paper ( Hosseini et al., 2015 )

describes the features of crowdsourcing and its deﬁnitions from

various disciplines. Since it is one of the earlier works, the au-

thors concentrate on the basic characteristics and deﬁnitions of

crowdsourcing; including related application domains, crowdsourc-

ing types and forms and leading countries with the number of

published papers for crowdsourcing. The authors noticed a lack of

robust crowdsourcing deﬁnitions in both geographical areas and

certain scientiﬁc ﬁelds. Moreover, they stated that the platforms

are cornerstones for future studies and industry. In Ambreen and

Naveed (2016) , the authors aim at reviewing the state of the art of

crowdsourcing in computing ﬁeld by considering yearly distribu-

tions of studies, types of research methods, frequency of research

outputs, distribution of empirical studies through various areas of

software engineering, and platforms of crowdsourcing. This study

has a narrower scope when compared to the one in our paper.

Thuan et al. (2016) focuses directly on the factors inﬂuencing the

A. Sarı, A. Tosun and G.I. Alptekin / The Journal of Systems and Software 153 (2019) 200–219 203

Table 2

Research questions.

Research question Aim

RQ1: What are the business models used for crowdsourcing in software

engineering (CSE)?

Analyzing the business models used for CSE.

RQ2: What are the technological platforms used for management of CSE? Investigating technical infrastructure on which crowdsourcing process is

implemented

RQ3: How are crowdsourced software development processes modeled? Identifying crowdsourced software

development methodologies

RQ4: For which software process area(s) crowdsourcing is utilized? Identifying software process area(s) that crowdsourcing is utilized

RQ5: What kind of effort estimation approaches are employed in

crowdsourced software development?

Identifying effort estimation approaches in crowdsourced software

development

RQ6: What are the cost drivers used in effort estimation?

Identifying the factors that affect effort estimation

RQ7: How are task awards determined in CSE? Investigating task award strategies in CSE

RQ8: What kind of strategies exist for crowd selection or formation in

software engineering?

Analyzing strategies for crowd selection or formation in software engineering

RQ9: How are tasks decomposed into

micro-tasking process performed in CSE? Investigating micro-tasking process performed in CSE

RQ10: Which tools have been used to assist CSE? Identifying assisting tools for CSE

decision to crowdsource. The authors broadly classiﬁed main crite-

ria that affect the decision to crowdsource as: task, people, man-

agement and environmental issues. They proposed a theoretical

framework for deciding whether to crowdsource or not. They also

suggest mechanisms to increase commitment of internal employ-

ees in software companies, and pointed out the importance of

available platforms for crowdsourcing, and risks of achieving low

quality results and loss of intellectual property. In Stol et al. (2017) ,

the authors have only reviewed related articles instead of conduct-

ing a systematic review. The authors selected six articles out of 18

submissions to the crowdsourcing call to demonstrate how soft-

ware development can beneﬁt from crowdsourcing as a source of

knowledge, and as a source for new ideas and feedbacks on exist-

ing software. A survey in the context of gamiﬁcation and crowd-

sourcing is generated in Morschheuser et al. (2017) . They stated

that gamiﬁcation was used to promote a kind of competition be-

tween the participants rather than a collaborative experience. They

explored that in most of the analyzed cases, a supplementary ﬁ-

nancial incentive (monetary reward) was not involved. Moreover,

the research examined structured conceptual framework of gami-

ﬁed crowdsourcing systems and psychological and behavioral out-

comes associated with the use of gamiﬁcation affordance. Although

the paper examines gamiﬁcation with the crowdsourcing perspec-

tive and by using crowdsourcing terminology, its ﬁndings are all in

the gamiﬁcation ﬁeld.

As pointed out in each literature survey, the number of CSE-

related works has been expanding every day. Our SLR covers pub-

lications before January 2018, and synthesizes evidence from 67

primary studies that passed our quality assessment criteria. Except

Mao et al. (2017) , none of the earlier works conducted a SLR on

this topic. Hence, we have noticed that there is a need to conduct

a more comprehensive overview to identify and aggregate exist-

ing evidence in CSE, by putting more emphasis on unexplored is-

sues. In the most comprehensive SLR related to CSE ( Mao et al.,

2017 ), the authors state CSE theories and models, task decomposi-

tion and remuneration are among the open issues in crowdsourc-

ing applications in the ﬁeld of software engineering. Hence, we

introduced new research questions, namely RQ1, RQ2, RQ5, RQ6,

RQ7, RQ9. Although the tools that assist CSE are studied in detail

by Mao et al. (2017) , in our research we associate the assisted tools

with assisted task, award mechanism and supported crowd selec-

tion strategy. Our work can be considered as one of the pioneering

works that deeply examine the economic impact of CSE in terms of

effort estimation models, remuneration/t ask awarding models and

cost drivers, and the crowd formation strategies.

3. Research methodology

We used SLR guidelines proposed by Kitchenham and Char-

ters (2007) conducting this study. SLR steps were followed by

deﬁning research questions, search string and databases, exploring

relevant papers, running a two-phase ﬁltering method via quality

assessment checklists, generating the ﬁnal list of primary studies

and then synthesizing useful data through thematic analysis. This

section reports the details of our SLR steps.

3.1. Deﬁnition of research questions

We list our 10 research questions (RQs) to synthesize and ag-

gregate the existing evidence in CSE and to identify emerging re-

search and practical issues in Table 2 .

3.2. Conducting the research

A keyword search is a type of search that looks for match-

ing documents involving one or more words speciﬁed by the user.

These words need to distinguish the most relevant research among

scientiﬁc databases. We deﬁned several keywords and formulated

them via logical operators:

•

(crowdsourcing OR crowdsourced OR crowd) AND "software en-

gineering"

•

(crowdsourcing OR crowdsourced OR crowd) AND "software de-

velopment"

•

"competitive programming"

•

"crowdsourced development"

Although there are various other databases, we chose six pop-

ular ones, which have the richest publication content in our re-

search ﬁeld: IEEEXplore, ACM Digital Library, Web of Science, SCO-

PUS, SpringerLink and ScienceDirect. These databases were also

used in the earlier secondary studies ( Ambreen and Naveed, 2016;

Hosseini et al., 2015; Mao et al., 2017; Morschheuser et al., 2017;

Stol et al., 2017; Thuan et al., 2016 ). The searching was performed

by using our keywords and extracting all published work from

these libraries. Our searching process was carried out without any

date limit. We did not include Google Scholar in order to limit our

search domain to the publications only. In crowdsourcing, we are

aware that there are many blog posts and newspaper and mag-

azine articles that target software practitioners. However, in this

SLR we focus on academic publications only. Table 3 summarizes

the full-text search execution in these databases in terms of num-

ber of publications ﬁltered at each phase. Initial search revealed

5574 publications in total.

3.3. Screening of relevant papers

The screening process is comprised of establishing inclusion/

exclusion criteria in order to ﬁnd relevant studies regarding to our

RQs, and an additional quality assessment to select the ﬁnal list of

primary studies.

剩余19页未读，继续阅读

评论收藏

内容反馈

AE86Jag

粉丝: 43
资源: 18

软件工程众包系统文献综述

《空间众包》综述论文

asp下载系统文献综述撰写

软件工程专业毕业设计文献综述.docx

计算机 即时通讯系统 文献综述

指纹识别系统(文献综述).doc

移动众包系统IE控件设置自动化脚本及安装插件

众包数据库综述（2020年05月最新综述）

论文研究-精英蜂群算法及考虑利益相关者的众包定价模型.pdf

软件众包任务的定价模型与人员匹配方法研究及工具实现

机票查询系统文献综述.pdf

多智能体系统文献综述.docx

问卷调查系统文献综述.docx

思维导图-众包数据库综述.pdf

数字人文视域下文化遗产众包研究综述.docx

众包系统中基于双向拍卖的诚实多任务分配机制.pdf

基于PHP的NLP信息抽取准确性检测众包系统源码.zip

项目众包管理系统 论文.docx

多智能体系统文献综述.pdf

机器人聊天系统文献综述

ASP酒店系统文献综述

浇花系统文献综述

物业管理系统文献综述

高级软件工程课程作业习题及详解

百度外卖的众包软件安卓版

众包威客开源建站系统KPPW3.0正式版

知识图谱的众包构建

最新资源

计算机即时通讯系统文献综述

项目众包管理系统论文.docx