UnderstandingReal-WorldConcurrencyBugsinGo.pdf资源-CSDN文库

版权申诉

79 浏览量 2021-11-17 05:14:55 上传评论收藏 671KB PDF 举报

资源推荐

资源详情

资源评论

Understanding Real-World Concurrency Bugs in Go

Tengfei Tu

∗

BUPT, Pennsylvania State University

tutengfei.kevin@bupt.edu.cn

Xiaoyu Liu

Purdue University

liu1962@purdue.edu

Linhai Song

Pennsylvania State University

songlh@ist.psu.edu

Yiying Zhang

Purdue University

yiying@purdue.edu

Abstract

Go is a statically-typed programming language that aims

to provide a simple, ecient, and safe way to build multi-

threaded software. Since its creation in 2009, Go has ma-

tured and gained signicant adoption in production and

open-source software. Go advocates for the usage of mes-

sage passing as the means of inter-thread communication

and provides several new concurrency mechanisms and li-

braries to ease multi-threading programming. It is important

to understand the implication of these new proposals and the

comparison of message passing and shared memory synchro-

nization in terms of program errors, or bugs. Unfortunately,

as far as we know, there has been no study on Go’s concur-

rency bugs.

In this paper, we perform the rst systematic study on

concurrency bugs in real Go programs. We studied six pop-

ular Go software including Docker, Kubernetes, and gRPC.

We analyzed 171 concurrency bugs in total, with more than

half of them caused by non-traditional, Go-specic problems.

Apart from root causes of these bugs, we also studied their

xes, performed experiments to reproduce them, and eval-

uated them with two publicly-available Go bug detectors.

Overall, our study provides a better understanding on Go’s

concurrency models and can guide future researchers and

practitioners in writing better, more reliable Go software

and in developing debugging and diagnosis tools for Go.

CCS Concepts • Computing methodologies → Con-

current programming languages

;

• Software and its en-

gineering → Software testing and debugging.

∗

The work was done when Tengfei Tu was a visiting student at Pennsylvania

State University.

Permission to make digital or hard copies of all or part of this work for

personal or classroom use is granted without fee provided that copies are not

made or distributed for prot or commercial advantage and that copies bear

this notice and the full citation on the rst page. Copyrights for components

of this work owned by others than ACM must be honored. Abstracting with

credit is permitted. To copy otherwise, or republish, to post on servers or to

redistribute to lists, requires prior specic permission and/or a fee. Request

permissions from permissions@acm.org.

ASPLOS’19, April 13–17, 2019, Providence, RI, USA

ACM ISBN ISBN 978-1-4503-6240-5/19/04. .. $15.00

hps://doi.org/10.1145/3297858.3304069

Keywords Go; Concurrency Bug; Bug Study

ACM Reference Format:

Tengfei Tu, Xiaoyu Liu, Linhai Song, and Yiying Zhang. 2019. Un-

derstanding Real-World Concurrency Bugs in Go . In Proceedings

of 2019 Architectural Support for Programming Languages and Op-

erating Systems (ASPLOS’19). ACM, New York, NY, USA, 14 pages.

hps://doi.org/10.1145/3297858.3304069

1 Introduction

Go [

] is a statically typed language originally developed

by Google in 2009. Over the past few years, it has quickly

gained attraction and is now adopted by many types of soft-

ware in real production. These Go applications range from

libraries [

] and high-level software [

] to cloud infrastruc-

ture software like container systems [13, 36] and key-value

databases [10, 15].

A major design goal of Go is to improve traditional multi-

threaded programming languages and make concurrent pro-

gramming easier and less error-prone. For this purpose, Go

centers its multi-threading design around two principles:

1) making threads (called goroutines) lightweight and easy

to create and 2) using explicit messaging (called channel)

to communicate across threads. With these design princi-

ples, Go proposes not only a set of new primitives and new

libraries but also new implementation of existing semantics.

It is crucial to understand how Go’s new concurrency prim-

itives and mechanisms impact concurrency bugs, the type of

bugs that is the most dicult to debug and the most widely

studied [

] in traditional multi-threaded pro-

gramming languages. Unfortunately, there has been no prior

work in studying Go concurrency bugs. As a result, to date,

it is still unclear if these concurrency mechanisms actually

make Go easier to program and less error-prone to concur-

rency bugs than traditional languages.

In this paper, we conduct the rst empirical study on

Go concurrency bugs using six open-source, production-

grade Go applications: Docker [

] and Kubernetes [

two datacenter container systems, etcd [

], a distributed

key-value store system, gRPC [

], an RPC library, and Cock-

roachDB [10] and BoltDB [6], two database systems.

In total, we have studied 171 concurrency bugs in these ap-

plications. We analyzed the root causes of them, performed

experiments to reproduce them, and examined their xing

patches. Finally, we tested them with two existing Go con-

currency bug detectors (the only publicly available ones).

Our study focuses on a long-standing and fundamental

question in concurrent programming: between message pass-

ing [

] and shared memory, which of these inter-thread

communication mechanisms is less error-prone [

Go is a perfect language to study this question, since it pro-

vides frameworks for both shared memory and message

passing. However, it encourages the use of channels over

shared memory with the belief that explicit message passing

is less error-prone [1, 2, 21].

To understand Go concurrency bugs and the comparison

between message passing and shared memory, we propose

to categorize concurrency bugs along two orthogonal dimen-

sions: the cause of bugs and their behavior. Along the cause

dimension, we categorize bugs into those that are caused

by misuse of shared memory and those caused by misuse of

message passing. Along the second dimension, we separate

bugs into those that involve (any number of) goroutines that

cannot proceed (we call them blocking bugs) and those that

do not involve any blocking (non-blocking bugs).

Surprisingly, our study shows that it is as easy to make con-

currency bugs with message passing as with shared memory,

sometimes even more. For example, around 58% of blocking

bugs are caused by message passing. In addition to the viola-

tion of Go’s channel usage rules (e.g., waiting on a channel

that no one sends data to or close), many concurrency bugs

are caused by the mixed usage of message passing and other

new semantics and new libraries in Go, which can easily be

overlooked but hard to detect.

To demonstrate errors in message passing, we use a block-

ing bug from Kubernetes in Figure 1. The

finishReq

func-

tion creates a child goroutine using an anonymous func-

tion at line 4 to handle a request—a common practice in

Go server programs. The child goroutine executes

fn()

and

sends

result

back to the parent goroutine through channel

at line 6. The child will block at line 6 until the parent

pulls

result

from

at line 9. Meanwhile, the parent will

block at

select

until either when the child sends result to

(line 9) or when a timeout happens (line 11). If timeout hap-

pens earlier or if Go runtime (non-deterministically) chooses

the case at line 11 when both cases are valid, the parent will

return from

requestReq()

at line 12, and no one else can

pull

result

from

any more, resulting in the child being

blocked forever. The x is to change

from an unbuered

channel to a buered one, so that the child goroutine can

always send the result even when the parent has exit.

This bug demonstrates the complexity of using new fea-

tures in Go and the diculty in writing correct Go programs

like this. Programmers have to have a clear understanding

of goroutine creation with anonymous function, a feature

Go proposes to ease the creation of goroutines, the usage

of buered vs. unbuered channels, the non-determinism

of waiting for multiple channel operations using

select

1 func finishReq(timeout time.Duration) r ob {

2 - ch := make(chan ob)

3 + ch := make(chan ob, 1)

4 go func() {

5 result := fn()

6 ch <- result // block

7 } ()

8 select {

9 case result = <- ch:

10 return result

11 case <- time.After(timeout):

12 return nil

13 }

14 }

Figure 1. A blocking bug caused by channel.

and the special library

time

. Although each of these fea-

tures were designed to ease multi-threaded programming, in

reality, it is dicult to write correct Go programs with them.

Overall, our study reveals new practices and new issues of

Go concurrent programming, and it sheds light on an answer

to the debate of message passing vs. shared memory accesses.

Our ndings improve the understanding of Go concurrency

and can provide valuable guidance for future tool design.

This paper makes the following key contributions.

•

We performed the rst empirical study of Go concur-

rency bugs with six real-world, production-grade Go

applications.

•

We made nine high-level key observations of Go con-

currency bug causes, xes, and detection. They can

be useful for Go programmers’ references. We further

make eight insights into the implications of our study

results to guide future research in the development,

testing, and bug detection of Go.

•

We proposed new methods to categorize concurrency

bugs along two dimensions of bug causes and behav-

iors. This taxonomy methodology helped us to better

compare dierent concurrency mechanisms and corre-

lations of bug causes and xes. We believe other bug

studies can utilize similar taxonomy methods as well.

All our study results and studied commit logs can be found

at https://github.com/system-pclub/go-concurrency-bugs.

2 Background and Applications

Go is a statically-typed programming language that is de-

signed for concurrent programming from day one [

]. Al-

most all major Go revisions include improvements in its con-

currency packages [

]. This section gives a brief background

on Go’s concurrency mechanisms, including its thread model,

inter-thread communication methods, and thread synchro-

nization mechanisms. We also introduce the six Go applica-

tions we chose for this study.

2.1 Goroutine

Go uses a concept called goroutine as its concurrency unit.

Goroutines are lightweight user-level threads that the Go

runtime library manages and maps to kernel-level threads

in an

-to-

way. A goroutine can be created by simply

adding the keyword go before a function call.

To make goroutines easy to create, Go also supports creat-

ing a new goroutine using an anonymous function, a function

denition that has no identier, or “name”. All local variables

declared before an anonymous function are accessible to the

anonymous function, and are potentially shared between

a parent goroutine and a child goroutine created using the

anonymous function, causing data race (Section 6).

2.2 Synchronization with Shared Memory

Go supports traditional shared memory accesses across

goroutines. It supports various traditional synchroniza-

tion primitives like lock/unlock (

Mutex

), read/write lock

(

RWMutex

), condition variable (

Cond

), and atomic read/write

(

atomic

). Go’s implementation of

RWMutex

is dierent from

pthread_rwlock_t

in C. Write lock requests in Go have a

higher privilege than read lock requests.

As a new primitive introduced by Go,

Once

is designed

to guarantee a function is only executed once. It has a

method, with a function

as argument. When

Once.Do(f)

is invoked many times, only for the rst time,

is executed.

Once

is widely used to ensure a shared variable only be

initialized once by multiple goroutines.

Similar to

pthread_join

in C, Go uses

WaitGroup

to al-

low multiple goroutines to nish their shared variable ac-

cesses before a waiting goroutine. Goroutines are added to

WaitGroup

by calling

Add

. Goroutines in a

WaitGroup

use

Done

to notify their completion, and a goroutine calls

Wait

to wait for the completion notication of all goroutines in

WaitGroup

. Misusing

WaitGroup

can cause both blocking

bugs (Section 5) and non-blocking bugs (Section 6).

2.3 Synchronization with Message Passing

Channel (

chan

) is a new concurrency primitive introduced

by Go to send data and states across goroutines and to build

more complex functionalities [

]. Go supports two types

of channels: buered and unbuered. Sending data to (or

receiving data from) an unbuered channel will block a gor-

outine, until another goroutine receives data from (or sends

data to) the channel. Sending to a buered channel will only

block, when the buer is full. There are several underlying

rules in using channels and the violation of them can create

concurrency bugs. For example, channel can only be used

after initialization, and sending data to (or receiving data

from) a

nil

channel will block a goroutine forever. Sending

data to a closed channel or close an already closed channel

can trigger a runtime panic.

The

select

statement allows a goroutine to wait on mul-

tiple channel operations. A

select

will block until one of its

cases can make progress or when it can execute a

default

branch. When more than one cases in a

select

are valid, Go

will randomly choose one to execute. This randomness can

cause concurrency bugs as will be discussed in Section 6.

Go introduces several new semantics to ease the interac-

tion across multiple goroutines. For example, to assist the

programming model of serving a user request by spawn-

ing a set of goroutines that work together, Go introduces

context

to carry request-specic data or metadata across

goroutines. As another example,

Pipe

is designed to stream

data between a

Reader

and a

Writer

. Both

context

and

Pipe

are new forms of passing messages and misusing them

can create new types of concurrency bugs (Section 5).

Application Stars Commits Contributors LOC Dev History

Docker 48975 35149 1767 786K 4.2 Years

Kubernetes 36581 65684 1679 2297K 3.9 Years

etcd 18417 14101 436 441K 4.9 Years

CockroachDB 13461 29485 197 520k 4.2 Years

gRPC* 5594 2528 148 53K 3.3 Years

BoltDB 8530 816 98 9K 4.4 Years

Table 1. Information of selecte d applications.

The num-

ber of stars, commits, contributors on GitHub, total source lines of

code, and development history on GitHub. *: the gRPC version that is

written in Go.

2.4 Go Applications

Recent years have seen a quick increase in popularity and

adoption of the Go language. Go was the 9th most popular

language on GitHub in 2017 [

]. As of the time of writing,

there are 187K GitHub repositories written in Go.

In this study, we selected six representative, real-world

software written in Go, including two container systems

(Docker and Kubernetes), one key-value store system (etcd),

two databases (CockroachDB and BoltDB), and one RPC

library (gRPC-go

) (Table 1). These applications are open-

source projects that have gained wide usages in datacenter

environments. For example, Docker and Kubernetes are the

top 2 most popular applications written in Go on GitHub,

with 48.9K and 36.5K stars (etcd is the 10th, and the rest are

ranked in top 100). Our selected applications all have at least

three years of development history and are actively main-

tained by developers currently. All our selected applications

are of middle to large sizes, with lines of code ranging from 9

thousand to more than 2 million. Among the six applications,

Kubernetes and gRPC are projects originally developed by

Google.

3 Go Concurrency Usage Patterns

Before studying Go concurrency bugs, it is important to rst

understand how real-world Go concurrent programs are like.

This section presents our static and dynamic analysis results

of goroutine usages and Go concurrency primitive usages in

our selected six applications.

We will use gRPC to represent the gRPC version that is written Go in the

following paper, unless otherwise specied.

剩余13页未读，继续阅读

评论收藏

内容反馈

版权申诉

Roc-xb

粉丝: 12w+
资源: 8198

Understanding Real-World Concurrency Bugs in Go.pdf

最新资源

Understanding Real-World Concurrency Bugs in Go.pdf

Go_Design+Patterns+for+Real-World+Projects-Packt+Publishing(2017).pdf

Go Design Patterns for Real-World Projects 无水印pdf

Learning-Concurrency-in-Python-Build-highly-efficient-robust-and-concurrent-applications.pdf.pdf

python2-oslo-concurrency-3.25.1-1.el7.noarch.rpm

Java.Concurrency.in.Practice.pdf

Go-style concurrency in C++11.zip

Java-Concurrency-in-Practice.rar

Go-style concurrency in C.zip

Python库 | django-concurrency-0.4.tar.gz

开源项目-kevchn-go-concurrency-patterns.zip

Vert.xguide-for-java-devs.pdf

python-oslo-concurrency-doc-3.27.0-1.el7.noarch.rpm

Concurrency in C- Cookbook.2014.pdf

Go: Design Patterns for Real-World Projects

Seven.Concurrency.Models.in.Seven.Weeks.2014.7.pdf

python2-oslo-concurrency-3.25.0-1.el7.noarch.rpm

Concurrency.in.Csharp.Cookbook.2014.5.pdf

python2-oslo-concurrency-3.30.1-1.el7.noarch.rpm

python-oslo-concurrency-doc-3.30.1-1.el8.noarch.rpm

python-oslo-concurrency-doc-3.30.1-1.el7.noarch.rpm

Java 面经手册·小傅哥.pdf

解压后拖入浏览器扩展程序使用.zip

103套PPT模板.zip

Beyond Compare 免安装直接使用

notepad++.exe官网下载

Mars4_5.zip

QT自制精美Ui模板系列（一）桃子风格模板 - 二次开发专用

Postman9.12.2安装包

python爬虫数据可视化分析大作业.zip

最新资源