MooseFS分布式文件系统调研分析

MooseFS 分布式文件系统调研分析报告

第一章

MooseFS

文件系统介绍

...................................................................................7

1.1 MooseFS 系统简介........................................................................................................7

1.1.1 MoosFS 特点............................................................................................................................7

1.1.2 MooseFS 的体系结构..............................................................................................................8

1.1.3 系统如何工作..........................................................................................................................8

1.1.4 容错性......................................................................................................................................9

1.1.5 平台..........................................................................................................................................9

2.2 master 整体运行框架..................................................................................................20

2.2.1 一切从 main 开始...................................................................................................................20

2.2.2 文件锁与进程启停控制........................................................................................................22

2.2.3 系统各模块的初始化............................................................................................................22

2.2.4 大循环 mainloop....................................................................................................................24

2.2.5 总结........................................................................................................................................27

2.3 master 的初始化..........................................................................................................28

2.3.1 日志模块初始化....................................................................................................................28

2.3.2 随机数产生器初始化............................................................................................................28

2.3.3 数据 cache 管理初始化..........................................................................................................29

2.4.3 对应 chunkserver 的消息结构...............................................................................................36

2.4.3 对应 client 的消息结构..........................................................................................................37

2.5 master 与 client 的消息交互........................................................................................39

2.5.1 消息 REGISTER....................................................................................................................45

2.5.2 消息 STATFS..........................................................................................................................46

2.5.3 消息 ACCESS........................................................................................................................47

2.5.4 消息 LOOKUP.......................................................................................................................47

2.5.5 消息 GETATTR......................................................................................................................48

2.5.6 消息 SETATTR.......................................................................................................................48

2.5.7 消息 READLINK...................................................................................................................49

2.5.14 消息 LINK............................................................................................................................53

2.5.15 消息 GETDIR.......................................................................................................................54

2.5.16 消息 OPEN...........................................................................................................................54

2.5.17 消息 READ CHUNK 及读数据过程...................................................................................55

2.5.18 消息 WRITE CHUNK 及写数据过程.................................................................................56

2.6 exports 机制.................................................................................................................58

2.6.1 mfsexports.mfs 文件结构.......................................................................................................59

2.6.2 exports 数据结构....................................................................................................................60

2.6.3 session 数据结构....................................................................................................................61

2.6.4 客户端 IP 访问控制...............................................................................................................62

2.7.1 数据结构................................................................................................................................67

2.7.2 保存加载与初始化................................................................................................................69

2.7.3 访问控制................................................................................................................................70

2.7.4 用户与组的相关命令............................................................................................................71

2.7.5 密码的维护............................................................................................................................74

2.7.6 root 用户的特殊性.................................................................................................................75

2.7.7 掉电保护和恢复....................................................................................................................75

2.8 session 机制..................................................................................................................75

2.8.1 session 数据结构....................................................................................................................76

2.8.2 sessions.mfs 文件结构...........................................................................................................76

2.9.2 块 ID.......................................................................................................................................81

2.9.2 块的版本号............................................................................................................................81

2.9.4 副本状态及转换....................................................................................................................82

2.9.5 块的副本数............................................................................................................................84

2.10 快照机制.....................................................................................................................97

2.10.1 快照的生成..........................................................................................................................97

2.10.2 快照的写时复制..................................................................................................................99

2.11 配额机制...................................................................................................................100

2.11.1 配额相关的数据结构和变量............................................................................................100

2.11.2 配额的维护........................................................................................................................102

2.12.4 trashpath 的获取和设置.....................................................................................................106

2.12.5 trashtime 的获取和设置.....................................................................................................106

2.12.6 文件的清除 PURGE..........................................................................................................107

2.12.7 文件的恢复 UNDEL..........................................................................................................108

2.12.8 回收站的维护....................................................................................................................108

2.13 reserved 机制...........................................................................................................108

2.13.1 记录打开文件....................................................................................................................109

2.13.2 reserved 机制数据结构......................................................................................................109

2.13.3 META 系统.........................................................................................................................110

2.13.4 文件信息的获取.................................................................................................................111

2.14.4 总结....................................................................................................................................115

2.15 优化机制...................................................................................................................117

2.15.1 freenode 与位管理节点编号..............................................................................................117

2.16 master 的容错恢复...................................................................................................119

2.16.1 changelog 机制...................................................................................................................120

2.16.2 mfsmetarestore 机制...........................................................................................................122

2.17 异地备份机制（自定义模块）...............................................................................125

2.17.1 工作模型............................................................................................................................125

2.18 master 端函数调用逻辑框架...................................................................................135

第三章

chunkserver

3.2 chunkserver 的初始化...............................................................................................142

3.2.1 块文件管理初始化..............................................................................................................142

3.2.2 与 client&cs 通信模块初始化.............................................................................................143

3.2.3 与 master 通信模块初始化..................................................................................................144

3.2.4 表格数据模块初始化..........................................................................................................145

3.3 周期性数据结构维护.................................................................................................145

3.3.1 周期性 chunk 维护...............................................................................................................145

3.3.2 周期性 folder 维护...............................................................................................................146

3.3.3 延时计数器维护..................................................................................................................147

3.3.4 块目录操作状态维护..........................................................................................................149

3.4.6 消息 REPLICATE................................................................................................................153

3.4.7 消息 TRUNCATE.................................................................................................................155

3.4.8 消息 DUPTRUNC................................................................................................................156

3.4.9 消息 REGISTER..................................................................................................................156

3.4.10 信息反馈............................................................................................................................157

3.5 线程池机制.................................................................................................................157

3.5.1 数据结构..............................................................................................................................158

3.5.2 基本操作函数......................................................................................................................161

3.5.3 工作原理..............................................................................................................................163

3.5.4 总结......................................................................................................................................166

3.6 与 client/cs 的消息交互..............................................................................................166