Apache Hadoop Goes Realtime at Facebook
Dhruba Borthakur
Kannan Muthukkaruppan
Karthik Ranganathan
Samuel Rash
Joydeep Sen Sarma
Nicolas Spiegelberg
Dmytro Molkov
Rodrigo Schmidt
Facebook
{dhruba,jssarma,jgray,kannan,
nicolas,hairong,kranganathan,dms,
aravind.menon,rash,rodrigo,
amitanand.s}@fb.com
Jonathan Gray
Hairong Kuang
Aravind Menon
Amitanand Aiyer
ABSTRACT
!
"#$%&''(! )%$%*+,-! .%/,'-%.! !"#$%&&'( )$**"+$*0! 1+2! 31)2+! %4%)!
52%)63#$1*7! #//,1$#+1'*! &51,+! '*! +8%! 9/#$8%! :#.''/! /,#+3');<!
9/#$8%!:=#2%!12!#!.#+#%6,1(%!,#-%)!&51,+!'*!:#.''/!.%217*%.!
+'!25//')+!&1,,1'*2!'3!;%22#7%2!/%)!.#-<!>812!/#/%)!.%2$)1&%2!+8%!
)%#2'*2! ?8-! "#$%&''(! $8'2%! :#.''/! #*.! :=#2%! '4%)! '+8%)!
2-2+%;2!25$8!#2!9/#$8%!@#22#*.)#!#*.!A',.%;')+!#*.!.12$522%2!
+8%! #//,1$#+1'*B2! )%C51)%;%*+2! 3')! $'*212+%*$-0! #4#1,#&1,1+-0!
/#)+1+1'*! +',%)#*$%0! .#+#! ;'.%,! #*.! 2$#,#&1,1+-<! D%! %E/,')%! +8%!
%*8#*$%;%*+2! ;#.%! +'! :#.''/! +'! ;#(%! 1+! #! ;')%! %33%$+14%!
)%#,+1;%! 2-2+%;0! +8%! +)#.%'332! ?%! ;#.%! ?81,%! $'*3175)1*7! +8%!
2-2+%;0!#*.!8'?!+812!2',5+1'*!8#2!217*131$#*+!#.4#*+#7%2!'4%)!+8%!
28#).%.! F-GHI! .#+#%!2$8%;%!52%.!1*!'+8%)!#//,1$#+1'*2!#+!
"#$%&''(! #*.! ;#*-! '+8%)! ?%&62$#,%! $';/#*1%2<! D%! .12$522! +8%!
;'+14#+1'*2! &%81*.! '5)! .%217*! $8'1$%20! +8%! $8#,,%*7%2! +8#+! ?%!
3#$%! 1*! .#-6+'6.#-! '/%)#+1'*20! #*.! 35+5)%! $#/#&1,1+1%2! #*.!
1;/)'4%;%*+2! 2+1,,! 5*.%)! .%4%,'/;%*+<! D%! '33%)! +8%2%!
'&2%)4#+1'*2! '*! +8%! .%/,'-;%*+! #2! #! ;'.%,! 3')! '+8%)! $';/#*1%2!
?8'! #)%! $'*+%;/,#+1*7! #! :#.''/6%.! 2',5+1'*! '4%)! +)#.1+1'*#,!
28#).%.!JK=FG!.%/,'-;%*+2<!!
Categories and Subject Descriptors
!
:<;!LInformation SystemsMN!F12$%,,#*%'52<(
General Terms
!
F#*#7%;%*+0! F%#25)%;%*+0! O%)3');#*$%0! K12+)1&5+%.! G-2+%;20!
K%217*0!J%,1#&1,1+-0!I#*75#7%2<!
Keywords
!
K#+#0! 2$#,#&1,1+-0! )%2'5)$%! 28#)1*70! .12+)1&5+%.! 31,%! 2-2+%;0!
:#.''/0! :14%0! :=#2%0! "#$%&''(0!G$)1&%0!,'7!#77)%7#+1'*0!
.12+)1&5+%.!2-2+%;2<!
(
(
!
1. INTRODUCTION
9/#$8%! :#.''/! LPM! 12! #! +'/6,%4%,! 9/#$8%! /)'Q%$+! +8#+! 1*$,5.%2!
'/%*!2'5)$%! 1;/,%;%*+#+1'*2!'3!#!.12+)1&5+%.!31,%!2-2+%;!LRM!#*.!
F#/J%.5$%! +8#+! ?%)%! 1*2/1)%.! &-! S''7,%B2! S"G! LTM! #*.!
F#/J%.5$%! LUM! /)'Q%$+2<! >8%! :#.''/! %$'2-2+%;! #,2'! 1*$,5.%2!
/)'Q%$+2! ,1(%! 9/#$8%! :=#2%! LVM! ?81$8! 12! 1*2/1)%.! &-! S''7,%B2!
=17>#&,%0! 9/#$8%! :14%! LWM0! #! .#+#! ?#)%8'52%! &51,+! '*! +'/! '3!
:#.''/0! #*.! 9/#$8%! X''Y%%/%)! LZM0! #! $'').1*#+1'*! 2%)41$%! 3')!
.12+)1&5+%.!2-2+%;2<!
9+!"#$%&''(0! :#.''/! 8#2! +)#.1+1'*#,,-!&%%*!52%.!1*!$'*Q5*$+1'*!
?1+8!:14%!3')!2+')#7%!#*.!#*#,-212!'3!,#)7%!.#+#!2%+2<!F'2+!'3!+812!
#*#,-212!'$$5)2!1*!'33,1*%!&#+$8!Q'&2!#*.!+8%!%;/8#212!8#2!&%%*!'*!
;#E1;1[1*7!+8)'578/5+!#*.!%331$1%*$-<!>8%2%!?')(,'#.2!+-/1$#,,-!
)%#.! #*.! ?)1+%! ,#)7%! #;'5*+2! '3! .#+#! 3)';! .12(! 2%C5%*+1#,,-<!92!
25$80!+8%)%!8#2!&%%*!,%22!%;/8#212!'*!;#(1*7!:#.''/!/%)3');#*+!
3')!)#*.';!#$$%22!?')(,'#.2!&-!/)'41.1*7!,'?!,#+%*$-!#$$%22!+'!
:K"G<!\*2+%#.0! ?%! 8#4%! 52%.! #! $';&1*#+1'*! '3! ,#)7%! $,52+%)2! '3!
F-GHI!.#+#%2!#*.!$#$81*7!+1%)2!&51,+!521*7!;%;$#$8%.L]M<!\*!
;#*-! $#2%20! )%25,+2! 3)';! :#.''/! #)%! 5/,'#.%.! 1*+'! F-GHI! ')!
;%;$#$8%.!3')!$'*25;/+1'*!&-!+8%!?%&!+1%)<!
J%$%*+,-0!#!*%?!7%*%)#+1'*!'3!#//,1$#+1'*2!8#2!#)12%*!#+!"#$%&''(!
+8#+! )%C51)%! 4%)-! 8178! ?)1+%! +8)'578/5+! #*.! $8%#/! #*.! %,#2+1$!
2+')#7%0! ?81,%! 21;5,+#*%'52,-! )%C51)1*7! ,'?! ,#+%*$-! #*.! .12(!
%331$1%*+! 2%C5%*+1#,! #*.! )#*.';! )%#.! /%)3');#*$%<! F-GHI!
2+')#7%! %*71*%2! #)%! /)'4%*! #*.! 8#4%! 4%)-! 7''.! )#*.';! )%#.!
/%)3');#*$%0! &5+! +-/1$#,,-! 2533%)! 3)';! ,'?! )#*.';! ?)1+%!
+8)'578/5+<!\+!12!.1331$5,+!+'!2$#,%!5/!'5)!F-GHI!$,52+%)2!)#/1.,-!
?81,%! ;#1*+#1*1*7! 7''.! ,'#.! &#,#*$1*7! #*.! 8178! 5/+1;%<!
9.;1*12+)#+1'*! '3! F-GHI! $,52+%)2! )%C51)%2! #! )%,#+14%,-! 8178!
;#*#7%;%*+! '4%)8%#.! #*.! +8%-! +-/1$#,,-! 52%! ;')%! %E/%*214%!
8#).?#)%<! S14%*! '5)! 8178! $'*31.%*$%! 1*! +8%! )%,1#&1,1+-! #*.!
2$#,#&1,1+-!'3!:K"G0!?%!&%7#*!+'!%E/,')%!:#.''/!#*.!:=#2%!3')!
25$8!#//,1$#+1'*2<!
>8%! 31)2+! 2%+! '3! #//,1$#+1'*2! )%C51)%2! )%#,+1;%! $'*$5))%*+0! &5+!
2%C5%*+1#,0! )%#.! #$$%22! +'! #! 4%)-! ,#)7%! 2+)%#;! '3! )%#,+1;%! .#+#!
&%1*7!2+')%.!1*!:K"G<!9*!%E#;/,%!2-2+%;!7%*%)#+1*7!#*.!2+')1*7!
25$8! .#+#! 12! G$)1&%! LP^M0! #*! '/%*! 2'5)$%! .12+)1&5+%.! ,'7!
#77)%7#+1'*!2%)41$%!$)%#+%.!&-!#*.!52%.!%E+%*214%,-!#+!"#$%&''(<!
O)%41'52,-0!.#+#!7%*%)#+%.!&-!G$)1&%!?#2!2+')%.!1*!%E/%*214%!#*.!
8#).!+'!;#*#7%!_"G!2%)4%)2<!>?'!;#1*!#//,1$#+1'*2!+8#+!3#,,!1*+'!
+812! $#+%7')-! #)%! J%#,+1;%! 9*#,-+1$2! LPPM! #*.! F-GHI! &#$(5/2<!
D%! 8#4%! %*8#*$%.! :K"G! +'! &%$';%! #! 8178! /%)3');#*$%! ,'?!
,#+%*$-! 31,%! 2-2+%;! #*.! 8#4%! &%%*! #&,%! +'! )%.5$%! '5)! 52%! '3!
%E/%*214%!31,%!2%)4%)2<!
O%);1221'*!+'!;#(%!.171+#,!')!8#).!$'/1%2!'3!#,,!')!/#)+!'3!+812!?')(!3')!
/%)2'*#,!')!$,#22)'';!52%!12!7)#*+%.!?1+8'5+!3%%!/)'41.%.!+8#+!$'/1%2!#)%!
*'+! ;#.%! ')! .12+)1&5+%.! 3')! /)'31+! ')! $';;%)$1#,! #.4#*+#7%! #*.! +8#+!
$'/1%2! &%#)! +812! *'+1$%! #*.! +8%! 35,,! $1+#+1'*! '*! +8%! 31)2+! /#7%<! >'! $'/-!
'+8%)?12%0! ')! )%/5&,1280! +'! /'2+! '*! 2%)4%)2! ')! +'! )%.12+)1&5+%! +'! ,12+20!
)%C51)%2!/)1')!2/%$131$!/%);1221'*!#*.`')!#!3%%<!
,-.)/0(1220!a5*%!PRbPU0!R^PP0!9+8%*20!S)%%$%<!
@'/-)178+!R^PP!9@F!]cZ6P6VT^W6^UUP6V`PP`^U<<<dP^<^^<!
剩余9页未读，继续阅读
资源评论
mrking01052011-11-10好东西呀，好东西。 可惜是全英文，慢慢啃吧。
fan_dx02022013-05-05不错的资料，挺好的！
chaoping3152013-02-26hadoop的实时计算实现还是不够通啊。不过资料还是不错！
