没有合适的资源？快使用搜索试试~ 我知道了~

文库首页大数据sparkCloudera hadoop spark培训题

Cloudera hadoop spark培训题

Cloudera

hadoop

spark

scala

python

需积分: 16 26 下载量 109 浏览量 2017-08-24 22:10:54 上传评论收藏 3.71MB PDF 举报

温馨提示

试读

87页

Cloudera培训题目，主要关于spark，语言用的scala和python

资源推荐

资源详情

资源评论

Cloudera Custom Training:

Hands-On Exercises

General'Notes'............................................................................................................................'3!

Hands-On'Exercise:'Query'Hadoop'Data'with'Apache'Impala'.................................'6!

Hands-On'Exercise:'Access'HDFS'with'the'Command'Line'and'Hue'......................'8!

Hands-On'Exercise:'Run'a'YARN'Job'..............................................................................'14!

Hands-On'Exercise:'Explore'RDDs'Using'the'Spark'Shell'.......................................'20!

Hands-On'Exercise:'Process'Data'Files'with'Apache'Spark'...................................'30!

Hands-On'Exercise:'Use'Pair'RDDs'to'Join'Two'Datasets'.......................................'34!

Hands-On'Exercise:'Write'and'Run'an'Apache'Spark'Application'......................'39!

Hands-On'Exercise:'Configure'an'Apache'Spark'Application'................................'44!

Hands-On'Exercise:'View'Jobs'and'Stages'in'the'Spark'Application'UI'..............'49!

Hands-On'Exercise:'Persist'an'RDD'................................................................................'55!

Not to be reproduced or shared without prior written consent from Cloudera.

Hands-On'Exercise:'Implement'an'Iterative'Algorithm'with'Apache'Spark'....'58!

Hands-On'Exercise:'Use'Apache'Spark'SQL'for'ETL'..................................................'62!

Hands-On'Exercise:'Write'an'Apache'Spark'Streaming'Application'..................'68!

Hands-On'Exercise:'Process'Multiple'Batches'with'Apache'Spark'Streaming' 73!

Hands-On'Exercise:'Process'Apache'Kafka'Messages'with'Apache'Spark'

Streaming'................................................................ ................................................................'78!

Appendix'A:'Enabling'Jupyter'Notebook'for'PySpark'..............................................'82!

Appendix'B:'Managing'Services'on'the'Course'Virtual'Machine'..........................'85!

Not to be reproduced or shared without prior written consent from Cloudera.

General Notes

"#$%&'()*+!,()-.-./!0$%(+'+!%+'!)!1-(,%)#!2)03-.'!(%..-./!,3'!"'.,45!6-.%7!

&-+,(-8%,-$.9!:3-+!12!3)+!";<!-.+,)##'&!-.!=+'%&$>&-+,(-8%,'&!?$&'9!@+'%&$>

&-+,(-8%,'&!?$&'!-+!)!?',3$&!$A!(%..-./!<)&$$=!B3'('8C!)##!<)&$$=!&)'?$.+!(%.!

$.!,3'!+)?'!?)03-.'9!D,!-+E!'++'.,-)##CE!)!0#%+,'(!0$.+-+,-./!$A!)!+-./#'!?)03-.'9!D,!

B$(F+!G%+,!#-F'!)!#)(/'(!<)&$$=!0#%+,'(H!,3'!$.#C!F'C!&-AA'('.0'!-+!,3),!,3'!<;I5!

8#$0F!('=#-0),-$.!A)0,$(!-+!+',!,$!JE!+-.0'!,3'('!-+!$.#C!)!+-./#'!;),)K$&'!)L)-#)8#'9!

Points to Note while Working in the Virtual Machine

• :3'!1-(,%)#!2)03-.'!M12N!-+!+',!,$!#$/!-.!)+!,3'!%+'(!training!)%,$?),-0)##C9!DA!

C$%!#$/!$%,E!C$%!0).!#$/!8)0F!-.!)+!,3'!%+'(!training!B-,3!,3'!=)++B$(&!

training9!

• DA!C$%!.''&!-,E!,3'!($$,!=)++B$(&!-+!training9!O$%!?)C!8'!=($?=,'&!A$(!,3-+!-AE!

A$(!'7)?=#'E!C$%!B).,!,$!03)./'!,3'!F'C8$)(&!#)C$%,9!D.!/'.'()#E!C$%!+3$%#&!.$,!

.''&!,3-+!=)++B$(&!+-.0'!,3'!training!%+'(!3)+!%.#-?-,'&!+%&$!=(-L-#'/'+9!

• D.!+$?'!0$??).&>#-.'!+,'=+!-.!,3'!'7'(0-+'+E!C$%!B-##!+''!#-.'+!#-F'!,3-+P!

$ hdfs dfs -put united_states_census_data_2010 \

/user/training/example

:3'!&$##)(! +-/.!MQN!),!,3'!8'/-..-./!$A!')03!#-.'!-.&-0),'+!,3'!6-.%7! +3'##!=($?=,9!

:3'!)0,%)#!=($?=,!B-##!-.0#%&'!)&&-,-$.)#!-.A$(?),-$.!MA$(!'7)?=#'E!

[training@localhost training_materials]$N!8%,!,3-+!-+!$?-,,'&!

A($?!,3'+'!-.+,(%0,-$.+!A$(!8('L-,C9!

:3'!8)0F+#)+3!M\N!),!,3'!'.&!$A!,3'!A-(+,!#-.'!+-/.-A-'+!,3),!,3'!0$??).&!-+!.$,!

0$?=#','&E!).&!0$.,-.%'+!$.!,3'!.'7,!#-.'9!O$%!0).!'.,'(!,3'!0$&'!'7)0,#C!)+!

+3$B.!M$.!,B$!#-.'+NE!$(!C$%!0).!'.,'(!-,!$.!)!+-./#'!#-.'9!DA!C$%!&$!,3'!#),, '(E!

C$%!+3$%#&!not!,C='!-.!,3'!8)0F+#)+39!

• R#,3$%/3!?$+,!+,%&'.,+!)('!0$?A$(,)8#'!%+-./!SKDT!,'7,!'&-,$(+!#-F'!L-!$(!'?)0+E!

+$?'!?-/3,!=('A'(!)!/()=3-0)#!,'7,!'&-,$(9!:$!-.L$F'!,3'!/()=3-0)#!'&-,$(!A($?!

,3'!0$??).&!#-.'E!,C='!gedit!A$##$B'&!8C!,3'!=),3!$A!,3'!A-#'!C$%!B-+3! ,$!'&-,9!

Not to be reproduced or shared without prior written consent from Cloudera.

R=='.&-./!&!,$!,3'!0$??).&!)##$B+!C$%!,$!,C='!)&&-,-$.)#!0$??).&+!B3-#'!,3'!

'&-,$(!-+!+,-##!$='.9!<'('!-+!).!'7)?=#'!$A!3$B!,$!'&-,!)!A-#'!.)?'&!myfile.txtP!

$ gedit myfile.txt &

Points to Note during the Exercises

Directories

• :3'!?)-.!&-('0,$(C!A$(!,3'!'7'(0-+'+!-+!

~/training_materials/devsh/exercises9!U)03!&-('0,$(C!%.&'(!,3),!

$.'!0$(('+=$.&+!,$!).!'7'(0-+'!$(!+',!$A!'7'(0-+'+V,3-+!-+!('A'(('&!,$!-.!,3'!

-.+,(%0,-$.+!)+!W,3'!'7'(0-+'!&-('0,$(C9X!R.C!+0(-=,+!$(!A-#'+!('Y%-('&!A$(!,3'!

'7'(0-+'!M$,3'(!,3).!&),)N!)('!-.!,3'!'7'(0-+'!&-('0,$(C9!

• Z-,3-.!')03!'7'(0-+'!&-('0,$(C!C$%!?)C!A-.&!,3'!A$##$B-./!+%8&-('0,$(-'+P!

• solutionV:3-+!0$.,)-.+!+$#%,-$.!0$&'!A$(!')03!'7'(0-+'9!

• stubsVR!A'B!$A!,3'!'7'(0-+'+!&'='.&!$.!=($L-&'&!+,)(,'(!A-#'+!

0$.,)-.-./!+F'#',$.!0$&'9!!

• 2)L'.!=($G'0,!&-('0,$(-'+VI$(!'7'(0-+'+!A$(!B3-03!C$%!?%+,!B(-,'!50)#)!

0#)++'+E!C$%!3)L'!8''.!=($L-&'&!B-,3!=('0$.A-/%('&!2)L'.!=($G'0,!

&-('0,$(-'+9!Z-,3-.!,3'+'!=($G'0,+!)('!,B$!=)0F)/'+P!stubsE!B3'('!C$%!

B-##!&$!C$%(!B$(F!%+-./!+,)(,'(!+F'#',$.!0#)++'+H!).&!solutionE!

0$.,)-.-./!,3'!+$#%,-$.!0#)++9!

• ;),)!A-#'+!%+'&!-.!,3'!'7'(0-+'+!)('!-.!~/training_materials/data9!

S+%)##C!C$%!B-##!%=#$)&!,3'!A-#'+!,$!<;I5!8'A$('!B$(F-./!B-,3!,3'?9!

• :3'!12!&'A-.'+!)!A'B!'.L-($.?'.,!L)(-)8#'+!,3),!)('!%+'&!-.!=#)0'!$A!#$./'(!

=),3+!-.!,3'!-.+,(%0,-$.+9!5-.0'!')03!L)(-)8#'!-+!)%,$?),-0)##C!('=#)0'&!B-,3!-,+!

0$(('+=$.&-./!L)#%'+!B3'.!C$%!(%.!0$??).&+!-.!,3'!,'(?-.)#E!,3-+!?)F'+!-,!

')+-'(!).&!A)+,'(!A$(!C$%!,$!'.,'(!)!0$??).&9!!

• :3'!,B$!'.L-($.?'.,!L)(-)8#'+!A$(!,3-+!0$%(+'!)('!$DEVSH!).&!$DEVDATA9!

S.&'(!$DEVSH!C$%!0).!A-.&!exercisesE!examplesE!).&!scripts9!

• O$%!0).!)#B)C+!%+'!,3'!echo!0$??).&!-A!C$%!B$%#&!#-F'!,$!+''!,3'!L)#%'!$A!

).!'.L-($.?'.,!L)(-)8#'P!

Not to be reproduced or shared without prior written consent from Cloudera.

$ echo $DEVSH

Step-by-Step Instructions

R+!,3'!'7'(0-+'+!=($/('++E!).&!C$%!/)-.!?$('!A)?-#-)(-,C!B-,3!,3'!,$$#+!).&!

'.L-($.?'.,E!B'!=($L-&'!A'B'(!+,'=>8C>+,'=!-.+,(%0,-$.+H!)+!-.!,3'!(')#!B$(#&E!B'!

?'('#C!/-L'!C$%!)!('Y%-('?'.,!).&!-,*+!%=!,$!C$%!,$!+$#L'!,3'!=($8#'?[!O$%!+3$%#&!

A''#!A(''!,$!('A'(!,$!,3'!3-.,+!$(!+$#%,-$.+!=($L-&'&E!)+F!C$%(!-.+,(%0,$(!A$(!

)++-+,).0'E!$(!0$.+%#,!B-,3!C$%(!A'##$B!+,%&'.,+9!

Bonus Exercises

:3'('!)('!)&&-,-$.)#!03)##'./'+!A$(!+$?'!$A!,3'!3).&+>$.!'7'(0-+'+9!DA!C$%!A-.-+3!,3'!

?)-.!'7'(0-+'E!=#')+'!),,'?=,!,3'!)&&-,-$.)#!+,'=+9!

Catch-Up Script

DA!C$%!)('!%.)8#'!,$!0$?=#','!).!'7'(0-+'E!B'!3)L'!=($L-&'&!)!+0(-=,!,$!0),03!C$%!%=!

)%,$?),-0)##C9!U)03!'7'(0-+'!3)+!-.+,(%0,-$.+!A$(!(%..-./!,3'!0),03>%=!+0(-=,E!B3'('!

)==#-0)8#'9!

$ $DEVSH/scripts/catchup.sh

:3'!+0(-=,!B-##!=($?=,!A$(!B3-03!'7'(0-+'!C$%!)('!+,)(,-./H!-,!B-##!+',!%=!)##!,3'!

('Y%-('&!&),)!)+!-A!C$%!3)&!0$?=#','&!)##!,3'!=('L-$%+!'7'(0-+'+9!

Z)(.-./P!DA!C$%!(%.!,3'!0),03!%=!+0(-=,E!C$%!?)C!#$+'!C$%(!B$(F9!MI$(!'7)?=#'E!)##!

&),)!B-##!8'!&'#','&!A($?!<;I59N!

剩余86页未读，继续阅读

评论收藏

内容反馈

资源评论

资源反馈

评论星级较低，若资源使用遇到问题可联系上传者，3个工作日内问题未解决可申请退款~

VinLSJ2020

粉丝: 4
资源: 1

上传资源快速赚钱

我的内容管理展开

我的资源快来上传第一个资源

我的收益

登录查看自己的收益

我的积分登录查看自己的积分

我的C币登录后查看C币余额

我的收藏

我的下载

下载帮助

前往需求广场，查看用户热搜

Cloudera hadoop spark培训题

spark入门培训

scala培训课程

Spark培训PPT

hadoop-training:Hadoop培训的源代码-spark source code

Centos7 安装Cloudera.pdf

Hadoop从业者为什么需要Spark？

spark2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el6.zip

Big.Data.Analytics.with.Spark.and.Hadoop.17858

Cloudera Custom Training: Hands-On Exercises

Cloudera Enterprise 5.8.x 中文文档

Cloudera CDH大数据平台搭建参考手册

你需要Spark的10个理由

hadoop-2.4.zip

Sandy Ryza：Spark and Resource Management on Hadoop

Hadoop.The.Definitive.Guide.4th.Edition.1491901632

Hadoop综合分析报告.pdf

cloudera manager 5.6在centos7.1上的安装

使用kafka，spark，hbase开发日志分析系统

clouderaManager配置.xlsx

CDH搭建Hadoop集群

Hadoop技术最前沿资讯报告和Cloudera大数据产品介绍

妳那伊抹微笑_云计算之ClouderaManager以及CHD5.1.0集群部署安装文档V1.0.docx

基于Python+pytorch的图像处理+附完整代码图像处理，能够轻松实现图像的读取、显示、裁剪等还有机器学习等操作

python大作业 含爬虫、数据可视化、地图、报告、及源码（2016-2021全国各地区粮食产量）.rar

《点燃我温暖你》中李峋的同款爱心代码

Python金融量化的高级库：TA-Lib-0.4.24（包含python3.7、3.8、3.9、3.10的32位和64位版本）

第十五届蓝桥杯大赛软件赛省赛-PythonB组题目

大麦网抢票脚本【Python脚本】

Python数据分析项目实践，包括数据读取、评估、清洗、分析、可视化机器学习相关内容等

最新资源

python大作业含爬虫、数据可视化、地图、报告、及源码（2016-2021全国各地区粮食产量）.rar