The log file contents are a sequence of 32KB blocks. The only
exception is that the tail of the file may contain a partial block.
Each block consists of a sequence of records:
block := record* trailer?
record :=
checksum: uint32 // crc32c of type and data[]
length: uint16
type: uint8 // One of FULL, FIRST, MIDDLE, LAST
data: uint8[length]
A record never starts within the last six bytes of a block (since it
won't fit). Any leftover bytes here form the trailer, which must
consist entirely of zero bytes and must be skipped by readers.
Aside: if exactly seven bytes are left in the current block, and a new
non-zero length record is added, the writer must emit a FIRST record
(which contains zero bytes of user data) to fill up the trailing seven
bytes of the block and then emit all of the user data in subsequent
blocks.
More types may be added in the future. Some Readers may skip record
types they do not understand, others may report that some data was
skipped.
FULL == 1
FIRST == 2
MIDDLE == 3
LAST == 4
The FULL record contains the contents of an entire user record.
FIRST, MIDDLE, LAST are types used for user records that have been
split into multiple fragments (typically because of block boundaries).
FIRST is the type of the first fragment of a user record, LAST is the
type of the last fragment of a user record, and MID is the type of all
interior fragments of a user record.
Example: consider a sequence of user records:
A: length 1000
B: length 97270
C: length 8000
A will be stored as a FULL record in the first block.
B will be split into three fragments: first fragment occupies the rest
of the first block, second fragment occupies the entirety of the
second block, and the third fragment occupies a prefix of the third
block. This will leave six bytes free in the third block, which will
be left empty as the trailer.
C will be stored as a FULL record in the fourth block.
===================
Some benefits over the recordio format:
(1) We do not need any heuristics for resyncing - just go to next
block boundary and scan. If there is a corruption, skip to the next
block. As a side-benefit, we do not get confused when part of the
contents of one log file are embedded as a record inside another log
file.
(2) Splitting at approximate boundaries (e.g., for mapreduce) is
simple: find the next block boundary and skip records until we
hit a FULL or FIRST record.
(3) We do not need extra buffering for large records.
Some downsides compared to recordio format:
(1) No packing of tiny records. This could be fixed by adding a new
record type, so it is a shortcoming of the current implementation,
not necessarily the format.
(2) No compression. Again, this could be fixed by adding new record types.
没有合适的资源?快使用搜索试试~ 我知道了~
温馨提示
Leveldb是一个google实现的非常高效的kv数据库,目前的版本1.2能够支持billion级别的数据量了。 在这个数量级别下还有着非常高的性能,主要归功于它的良好的设计。特别是LSM算法。 LevelDB 是单进程的服务,性能非常之高,在一台4个Q6600的CPU机器上,每秒钟写数据超过40w,而随机读的性能每秒钟超过10w。
资源推荐
资源详情
资源评论
收起资源包目录
leveldb google (373个子文件)
all-wcprops 3KB
all-wcprops 3KB
all-wcprops 1KB
all-wcprops 1KB
all-wcprops 1KB
all-wcprops 539B
all-wcprops 524B
all-wcprops 162B
all-wcprops 71B
AUTHORS 193B
db_impl.cc 35KB
version_set.cc 31KB
db_test.cc 29KB
table_test.cc 23KB
db_bench.cc 18KB
env_chromium.cc 17KB
env_posix.cc 15KB
crc32c.cc 15KB
repair.cc 11KB
skiplist_test.cc 10KB
sha1_portable.cc 10KB
corruption_test.cc 9KB
log_test.cc 9KB
db_iter.cc 8KB
block.cc 8KB
version_edit.cc 7KB
table_builder.cc 7KB
cache.cc 6KB
table.cc 5KB
log_reader.cc 5KB
coding.cc 5KB
merger.cc 5KB
coding_test.cc 5KB
two_level_iterator.cc 5KB
histogram.cc 4KB
cache_test.cc 4KB
format.cc 4KB
write_batch.cc 4KB
dbformat_test.cc 4KB
filename.cc 4KB
block_builder.cc 4KB
memtable.cc 3KB
filename_test.cc 3KB
dbformat.cc 3KB
log_writer.cc 3KB
table_cache.cc 3KB
write_batch_test.cc 2KB
builder.cc 2KB
env_test.cc 2KB
arena.cc 2KB
comparator.cc 2KB
logging.cc 2KB
crc32c_test.cc 2KB
iterator.cc 2KB
arena_test.cc 2KB
env.cc 2KB
port_android.cc 1KB
port_chromium.cc 1KB
status.cc 1KB
testutil.cc 1KB
testharness.cc 1KB
version_edit_test.cc 1KB
port_posix.cc 1KB
sha1_test.cc 1KB
hash.cc 958B
options.cc 715B
doc.css 1KB
entries 6KB
entries 5KB
entries 3KB
entries 2KB
entries 2KB
entries 1KB
entries 1KB
entries 408B
entries 254B
format 2B
format 2B
format 2B
format 2B
format 2B
format 2B
format 2B
format 2B
format 2B
leveldb.gyp 7KB
skiplist.h 11KB
env.h 10KB
version_set.h 10KB
options.h 7KB
db_impl.h 6KB
db.h 5KB
dbformat.h 5KB
testharness.h 4KB
coding.h 4KB
port_example.h 4KB
cache.h 3KB
port_android.h 3KB
iterator.h 3KB
table_builder.h 3KB
共 373 条
- 1
- 2
- 3
- 4
LyonYu
- 粉丝: 1
- 资源: 12
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功
- 1
- 2
- 3
前往页