CrowdLearn: A Crowd-AI Hybrid System for Deep
Learning-based Damage Assessment Applications
Daniel (Yue) Zhang, Yang Zhang, Qi Li, Thomas Plummer, Dong Wang
Department of Computer Science and Engineering
University of Notre Dame, IN, USA
{yzhang40, yzhang42, qli8, tplumme2, dwang5}@nd.edu
Abstract—Artificial Intelligence (AI) has been widely adopted
in many important application domains such as speech recog-
nition, computer vision, autonomous driving, and AI for social
good. In this paper, we focus on the AI-based damage assessment
applications where deep neural network approaches are used to
automatically identify damage severity of impacted areas from
imagery reports in the aftermath of a disaster (e.g., earthquake,
hurricane, landslides). While AI algorithms often significantly
reduce the detection time and labor cost in such applications,
their performance sometimes falls short of the desired accuracy
and is considered to be less reliable than domain experts.
To exacerbate the problem, the black-box nature of the AI
algorithms also makes it difficult to troubleshoot the system
when their performance is unsatisfactory. The emergence of
crowdsourcing platforms (e.g., Amazon Mechanic Turk, Waze)
brings about the opportunity to incorporate human intelligence
into AI algorithms. However, the crowdsourcing platform is
also black-box in terms of the uncertain response delay and
crowd worker quality. In this work, we propose the CrowdLearn,
a crowd-AI hybrid system that leverages the crowdsourcing
platform to troubleshoot, tune, and eventually improve the black-
box AI algorithms by welding crowd intelligence with machine
intelligence. The system is specifically designed for deep learning-
based damage assessment (DDA) applications where the crowd
tend to be more accurate but less responsive than machines.
Our evaluation results on a real-world case study on Amazon
Mechanic Turk demonstrate that CrowdLearn can provide timely
and more accurate assessments to natural disaster events than
the state-of-the-art AI-only and human-AI integrated systems.
I.
INTRODUCTION
The recent advances in artificial intelligence (AI) has trans-
formed many important domains of modern life (e.g., trans-
portation, finance, education, healthcare, and entertainment)
and its encroachment is expected to intensify [1]–[4]. In
this paper, we focus on an important AI application - deep
learning-based disaster damage assessment (DDA) where
deep neural network approaches are used to automatically
identify damage severity of impacted areas from imagery re-
ports in the aftermath of a disaster (e.g., earthquake, hurricane,
landslides) [5], [6]. The assessments are then delegated to
emergency response agencies (e.g., FEMA and police depart-
ments) to adopt appropriate countermeasures. Traditionally,
the damage assessment was primarily done by domain experts,
which suffers from the apparent limitation of the heavy labor
cost of labeling and low efficiency in the presence of massive
amount of data [7]. A set of AI algorithms have recently
been developed to automatically label the damage severity
in images from social media posts without the presence of
domain experts [5], [6].
While AI algorithms can significantly reduce the labor cost
and improve the detection efficiency in DDA applications,
they are prone to various failure scenarios (see Figure 1). For
example, the AI algorithms mistakenly report a severe damage
for the images in Figure 1(a) and 1(b) and report no damage
for images in Figure 1(c) and 1(d) (please refer to the detailed
discussions under Figure 1). One main reason for the above
failure scenarios is that the AI-based DDA algorithms can only
capture the low level features of the images (e.g., color, layout,
shapes) but fail to “understand” the high level context of the
images (e.g., the story behind the image). Such failure may
lead to severe consequences (e.g., the rescue team may be
sent to the wrong places while places where people’s lives are
at stake are not responded). In contrast, human intelligence
(HI) is often more accurate in such failure scenarios [8]. For
example, humans can reliably assess the damage severity by
identifying fake or irrelevant images (e.g., Figure 1(a) and
1(b)) and observing the actual events happening in the images
(e.g., Figure 1(c) and 1(d)).
(a) Fake Image (b) Close Up
(c) Low Resolution (d) Implicit
Image (a) is a fake image showing a car falling from a huge
cleavage of a road. Image (b) is a close-up of a crack on a road.
The AI algorithms mistakenly return false detection result of
“severe damage” for both images. Image (c) shows a disaster scene
image with low resolution. Image (d) shows kids were injured and
taken away from a damaged area. The AI algorithms mistakenly
return false detection result of “no damage” for both images.
Figure 1: Examples failures of AI algorithm of DDA
Motivated by the unique and complementary advantages and
剩余11页未读，继续阅读
资源评论
百态老人
- 粉丝: 6779
- 资源: 2万+
下载权益
C知道特权
VIP文章
课程特权
VIP享7折，此内容立减5.97元 开通VIP
