Hadoop环境下的分布式协同过滤算法设计与实现

所需积分/C币:29 2018-07-03 17:14:34 732KB PDF
收藏 收藏 1
举报

以开源项目Hadoop为实验平台,论证传统协同过滤算法无法适应云平台;从相似度和预测偏好两方面,借鉴共词分析法,将传统协同过滤算法改进为适应Hadoop平台的分布式协同过滤算法;实现顺序组合式MapRe-duce协同过滤任务,并做进一步实验分析。
总第229期2013年第1期 Reduce 已duCe <K3 M HDFS pReduce Ha adop ser item preference Mapreduce de oop Item prefered Hadoop user Item prefer Lva ence Item preference? Item preference Namenode、 Datan ode, Secondary, Jobtracter and Tasktracker C BUG BLG Item n user item, preference item orefer 2.3 Hadoop ence,.". ltem preference. item preference Mapreduce preference Map <item, item, Item -User Item. Ite K Reduce item K er item preference item itel ltem2n2…tem;nk user preference Reduce Item item, n tem. n Item k K tem <KV> Item -User HDES It User Hade l1 D 21 Mapreduce Hado p nmI n Hadoop 图2ltem共现矩阵 3 Hade P,P…P nop XIANDAI TUSHU QINGBAO JISHU 85 21994-2018ChinaAcademicJournalElectronicPublishingHouse.Allrightsreservedhttp://www.cnki.net 2n Pu lte em U′;P1P Colly I L2 k 12 n P P 12 P coll,=rowl, T P 图3预测偏好向量U Mapreduce Item Hadoop Mapped uce 3 m n P"in=ΣmnP1 0 0 Mapped U",P"1,P",n T oken k 4 ∑nnP n1P1u+n1Px+…+ nImE 4 Hadoop ∑n2P P1u+n22+…+n2aP Hadoop nmi Pu+nm,Pu+.+nmmP Mapreduce M aI redu l n21 12P2 +p P t≤ < Off u,(,P 图4U,公式数学变形 Reduce: <u, (I, P)>-<u, list (I, P) k Coll ∑P.Coll Mapreduce k≤m 86 21994-2018ChinaAcademicJournalElectronicPublishingHouse.Allrightsreservedhttp://www.cnki.net 总第229期2013年第1期 Multiple outputs Maple duce Mapreduce Map: Offset, U>-+<i, (1.1)> Combiner: <i, (1,1)>-<i, list(l, n)> Map: c Offset, I'>-+<u, (I,P;> Reduce: <i, list(I, n)>-+<i, list(I, n>/<i, sum Combiner: <u, (L, P)>-<u,list(L, P)> U Reduee: <u, list(I, P)>-<u, list(I, P)> Coluu I col L, col, adop P Rowl Mapreduce 6 Mapreduce Hadoop Hadoop Mapreduce MapReduce Datajoir Reduce T Group- Hadoop Reduce Tagged MapOut Ubuntu12. 10 Tag Map Re Hadoop. 1.0 E DataJoin DataJoin Map Movielens erAse DataJoin ReducerBase tp//www.grouplens.org T Redu P Movielens 90571 R I Movielens Mapreduce Map: Offset, U/T <i, Tagged MapOut l084 4855 17748 Reduce: <i, TaggedMap Output)>-><u, Colu> 40318 U" I col I 800 167l 77533 90571 Col ColU Mapreduce 50、200、400、800943 Map: <Offset, ( u, ColU)>-+<u, ColU> Reduce: <u, ColU>+<u,U> U P 4 U Group key mapreduce 非分布式(p2 节点 Map:≤ Offset,lm/Uy>→≤i, TaggedMapOutput> 3节点 Reduce: <i, TaggedMapOutput)>-<i, (U, P)> 图5响应时间 XIANDAI TUSHU QINGBAO JISHU 21994-2018ChinaAcademicJournalElectronicPublishingHouse.Allrightsreservedhttp://www.cnki.net P top20 Hado MAE 2009 32 5 107-113. Li Shuqin. Review of Personalized In formation Retrieval Technology J. Information Studies Theory 4 prlic ation2009325107-113 ∑mpl-P = 2 L in 7.B Qu WY Ii HT el al. A Hyhrid Collaborative Filtering Recommendation Mechanism for p2p Networks. Future Genera tion Computer Systcms 2010 268 1409-1417 MAE 3 Pan R Scholz M. Mind the Caps Weighting the Unknown in ⅥAE Large-Scale One-Class Collaborative Filtering C. In Proceed ings of the 15th ACM SIGKDD International Conference on Knowi 1.2 edge Discouery and Data Mining Paris France. New York ACM2009667-676 08 4 Pank Zhou yh cao et al. one- class collaborative filte ring C. In Proccedings of the &th IEEE International Conferenc 02 on Data Mining Pisa. Washington DC USA IEEE Computer Society2008502-511 200 400 5 Salakhutdinoy r mnih a. probabilistic matrix Factorization c 非分布式tp20 分布式 In Proceedings of the 25th International Conference on Machine 图6MAE比较 Learning. New York ACM 2008 880-887 2013 39 1 112-118. Hou Jingchuan Fang Jingyi. Review on Data Citation in the Context of Big Data J. Journal of Library science in China 2013 39 1 112-118 7 item reference PI 2012 5 37-40. Ilan Cuifeng. The: Impact and Challenges of Map <Itcm. ltcm. the library Based on Big Data J. Library Information 2012 8 Sarwar B Karypis G Konstan J et al. Item- based Collaborative Filtering Recommendation Algorithms C. In Proceedings of the 10th International Conference on World Wide Web. New York NY Hado USA4CM2001285-295 p Hadoop 9 Resnick P lacowou n Suchak M et al. Group Lens An Open Archi tecture for Collaborative Filtering of Netnews C. In Proceeding Hado of the 1994 ACM Conference on Computer Supported Cooperate Work. New York NY USA ACM 1994 175-186 Hadoop 10 Sarwar B Karypis G Konstan J et al. Item- based Collaborative anReduce Filtering Recommendation Algorithms C. In Proceedings of the 21994-2018ChinaAcademicJournalElectronicPublishingHouse.Allrightsreservedhttp://www.cnki.net 总第229期2013年第1期 1Oth International Conference on World Wide Web. New York NY tribuicd Systems 2012 23 10 1831-1843 USA ACM2001285-295 11 White T. Hadoop The Definitive Guide M. The 3rd Edition 2006 32 2 88-92. Feng Lu Leng Fuhai. Development of USA0’ Reilly meria2012 Theorelical Studies of Cn -word Analysis J. Journal of Lin 12 Dean J Ghemawat S. MapReduce Simplified Data Processing on Science in china 2006 32 2 88-92 arge Clusters J. Communications of the ACM 2008 51 I 10 16 Sarwar b Karypis C Konstan J et al. Analysis of Recommenda tion Algorithms for E-commerce C. In Proceedings of the 2nd 13Hadoop.HdfsuSersGuideEb/oL.2012-12-02.http:// ACM Conference on Electronic Commerce. New York ACM 2000 hadoop. apache. org/docs/stable/hdfs_user_guide. htm 158-167. 14 Bahga A Madisetti V K. Analyzing Massive Machine Maintenance E-mailnjhnxq@163.com Data in a Computing Cloud J. IEEE Transactions on Parallel and Facebook 20112 Facebook Facebook book 2012 1 ” Facebook Graph Search FGS。 News Feed ” Timeline Facebook“ 410 240 Facebook Google Google Google Page rank Facebook Facebook Lars rasinusseri LarsRastuussenl Facebook Facebook Facebook 28 80% 20]12上 176 F acebo Google Google Facebook Facebook Google acebo Google oogle Google “ eople” http://shibeichen.com/post/40833850372 XIANDAI TUSHU QINGBAO JISHU 89 21994-2018ChinaAcademicJournalElectronicPublishingHouse.Allrightsreservedhttp://www.cnki.net

...展开详情
立即下载 低至0.43元/次 身份认证VIP会员低至7折
    抢沙发
    一个资源只可评论一次,评论内容不能少于5个字
    img
    qq_28339273

    关注 私信 TA的资源

    上传资源赚积分,得勋章
    最新推荐