lucene-2.3.2

Apache Lucene - Index File Formats

by Doug Cutting

Table of contents

1 Index File Formats............................................................................................................. 3

2 Definitions..........................................................................................................................3

2.1 Inverted Indexing...........................................................................................................4

2.2 Types of Fields.............................................................................................................. 4

2.3 Segments........................................................................................................................4

2.4 Document Numbers.......................................................................................................4

5.4 VInt................................................................................................................................6

5.5 Chars..............................................................................................................................7

5.6 String............................................................................................................................. 7

6 Per-Index Files...................................................................................................................8

6.1 Segments File................................................................................................................ 8

6.2 Lock File........................................................................................................................9

6.3 Deletable File...............................................................................................................10

6.4 Compound Files...........................................................................................................10

7 Per-Segment Files............................................................................................................ 10

7.1 Fields........................................................................................................................... 10

7.7 Deleted Documents..................................................................................................... 19

8 Limitations....................................................................................................................... 19

Apache Lucene - Index File Formats

Page 2

Copyright © 2006 The Apache Software Foundation. All rights reserved.

1. Index File Formats

This document defines the index file formats used in Lucene version 2.1. If you are using a

different version of Lucene, please consult the copy of docs/fileformats.html that

was distributed with the version you are using.

Apache Lucene is written in Java, but several efforts are underway to write versions of

of this document.

Compatibility notes are provided in this document, describing how file formats have changed

from prior versions.

In version 2.1, the file format was changed to allow lock-less commits (ie, no more commit

lock). The change is fully backwards compatible: you can open a pre-2.1 index for searching

or adding/deleting of docs. When the new segments file is saved (committed), it will be

written in the new file format (meaning no specific "upgrade" process is needed). But note

that once a commit has occurred, pre-2.1 Lucene will not be able to read the index.

In version 2.3, the file format was changed to allow segments to share a single set of doc

store (vectors & stored fields) files. This allows for faster indexing in certain cases. The

A document is a sequence of fields.

•

A field is a named sequence of terms.

•

A term is a string.

The same string in two different fields is considered a different term. Thus terms are

represented as a pair of strings, the first naming the field, and the second naming text within

Apache Lucene - Index File Formats

Page 3

Copyright © 2006 The Apache Software Foundation. All rights reserved.

the field.

The index stores statistics about terms in order to make term-based search more efficient.

and indexed.

The text of a field may be tokenized into terms to be indexed, or the text of a field may be

used literally as a term to be indexed. Most fields are tokenized, but sometimes it is useful for

certain identifier fields to be indexed literally.

See the Field java docs for more information on Fields.

Lucene indexes may be composed of multiple sub-indexes, or segments. Each segment is a

fully independent index, which could be searched separately. Indexes evolve by:

1.

Creating new segments for newly added documents.

added to an index is numbered zero, and each subsequent document added gets a number one

greater than the previous.

Note that a document's number may change, so caution should be taken when storing these

numbers outside of Lucene. In particular, numbers may change in the following situations:

•

The numbers stored in each segment are unique only within the segment, and must be

converted before they can be used in a larger context. The standard technique is to

Apache Lucene - Index File Formats

Page 4

Copyright © 2006 The Apache Software Foundation. All rights reserved.

allocate each segment a range of values, based on the range of numbers used in that

segment. To convert a document number from a segment to an external value, the

segment's base document number is added. To convert an external value back to a

removed as the index evolves through merging. Deleted documents are dropped when

segments are merged. A freshly-merged segment thus has no gaps in its numbering.

3. Overview

Each segment index maintains the following:

•

Field names. This contains the set of field names used in the index.

•

Stored Field values. This contains, for each document, a list of attribute-value pairs,

where the attributes are field names. These are used to store auxiliary information about

the document, such as its title, url, or an identifier to access a database. The set of stored

•

Term Frequency data. For each term in the dictionary, the numbers of all the documents

that contain that term, and the frequency of the term in that document.

•

Term Proximity data. For each term in the dictionary, the positions that the term occurs in

each document.

•

Normalization factors. For each field in each document, a value is stored that is

multiplied into the score for hits on that field.

•

4. File Naming

Apache Lucene - Index File Formats

Page 5

Copyright © 2006 The Apache Software Foundation. All rights reserved.