lucence-3.5.0

Lucene Change Log For more information on past and future Lucene versions, please see: http://s.apache.org/luceneversions ======================= Lucene 3.5.0 ======================= Changes in backwards compatibility policy * LUCENE-3390: The first approach in Lucene 3.4.0 for missing values support for sorting had a design problem that made the missing value be populated directly into the FieldCache arrays during sorting, leading to concurrency issues. To fix this behaviour, the method signatures had to be changed: - FieldCache.getUnValuedDocs() was renamed to FieldCache.getDocsWithField() returning a Bits interface (backported from Lucene 4.0). - FieldComparator.setMissingValue() was removed and added to constructor As this is expert API, most code will not be affected. (Uwe Schindler, Doron Cohen, Mike McCandless) * LUCENE-3464: IndexReader.reopen has been renamed to IndexReader.openIfChanged (a static method), and now returns null (instead of the old reader) if there are no changes in the index, to prevent the common pitfall of accidentally closing the old reader. * LUCENE-3541: Remove IndexInput's protected copyBuf. If you want to keep a buffer in your IndexInput, do this yourself in your implementation, and be sure to do the right thing on clone()! (Robert Muir) * LUCENE-2822: TimeLimitingCollector now expects a counter clock instead of relying on a private daemon thread. The global time limiting clock thread has been exposed and is now lazily loaded and fully optional. TimeLimitingCollector now supports setting clock baseline manually to include prelude of a search. Previous versions set the baseline on construction time, now baseline is set once the first IndexReader is passed to the collector unless set before. (Simon Willnauer) Changes in runtime behavior * LUCENE-3520: IndexReader.openIfChanged, when passed a near-real-time reader, will now return null if there are no changes. The API has always reserved the right to do this; it's just that in the past for near-real-time readers it never did. (Mike McCandless) Bug fixes * SOLR-2762: (backport form 4.x line): FSTLookup could return duplicate results or one results less than requested. (David Smiley, Dawid Weiss) * LUCENE-3412: SloppyPhraseScorer was returning non-deterministic results for queries with many repeats (Doron Cohen) * LUCENE-3421: PayloadTermQuery's explain was wrong when includeSpanScore=false. (Edward Drapkin via Robert Muir) * LUCENE-3432: IndexWriter.expungeDeletes with TieredMergePolicy should ignore the maxMergedSegmentMB setting (v.sevel via Mike McCandless) * LUCENE-3442: TermQuery.TermWeight.scorer() returns null for non-atomic IndexReaders (optimization bug, introcuced by LUCENE-2829), preventing QueryWrapperFilter and similar classes to get a top-level DocIdSet. (Dan C., Uwe Schindler) * LUCENE-3390: Corrected handling of missing values when two parallel searches using different missing values for sorting: the missing value was populated directly into the FieldCache arrays during sorting, leading to concurrency issues. (Uwe Schindler, Doron Cohen, Mike McCandless) * LUCENE-3439: Closing an NRT reader after the writer was closed was incorrectly invoking the DeletionPolicy and (then possibly deleting files) on the closed IndexWriter (Robert Muir, Mike McCandless) * LUCENE-3215: SloppyPhraseScorer sometimes computed Infinite freq (Robert Muir, Doron Cohen) * LUCENE-3465: IndexSearcher with ExecutorService was always passing 0 for docBase to Collector.setNextReader. (Robert Muir, Mike McCandless) * LUCENE-3503: DisjunctionSumScorer would give slightly different scores for a document depending if you used nextDoc() versus advance(). (Mike McCandless, Robert Muir) * LUCENE-3529: Properly support indexing an empty field with empty term text. Previously, if you had assertions enabled you would receive an error during flush, if you didn't, you would get an invalid index. (Mike McCandless, Robert Muir) * LUCENE-2633: PackedInts Packed32 and Packed64 did not support internal structures larger than 256MB (Toke Eskildsen via Mike McCandless) * LUCENE-3540: LUCENE-3255 dropped support for pre-1.9 indexes, but the error message in IndexFormatTooOldException was incorrect. (Uwe Schindler, Mike McCandless) * LUCENE-3541: IndexInput's default copyBytes() implementation was not safe across multiple threads, because all clones shared the same buffer. (Robert Muir) * LUCENE-3548: Fix CharsRef#append to extend length of the existing char[] and preserve existing chars. (Simon Willnauer) * LUCENE-3582: Normalize NaN values in NumericUtils.floatToSortableInt() / NumericUtils.doubleToSortableLong(), so this is consistent with stored fields. Also fix NumericRangeQuery to not falsely hit NaNs on half-open ranges (one bound is null). Because of normalization, NumericRangeQuery can now be used to hit NaN values by creating a query with upper == lower == NaN (inclusive). (Dawid Weiss, Uwe Schindler) API Changes * LUCENE-3454: Rename IndexWriter.optimize to forceMerge to discourage use of this method since it is horribly costly and rarely justified anymore. MergePolicy.findMergesForOptimize was renamed to findForcedMerges. IndexReader.isOptimized was deprecated. IndexCommit.isOptimized was replaced with getSegmentCount. (Robert Muir, Mike McCandless) * LUCENE-3205: Deprecated MultiTermQuery.getTotalNumerOfTerms() [and related methods], as the numbers returned are not useful for multi-segment indexes. They were only needed for tests of NumericRangeQuery. (Mike McCandless, Uwe Schindler) * LUCENE-3574: Deprecate outdated constants in org.apache.lucene.util.Constants and add new ones for Java 6 and Java 7. (Uwe Schindler) * LUCENE-3571: Deprecate IndexSearcher(Directory). Use the constructors that take IndexReader instead. (Robert Muir) * LUCENE-3577: Rename IndexWriter.expungeDeletes to forceMergeDeletes, and revamped the javadocs, to discourage use of this method since it is horribly costly and rarely justified. MergePolicy.findMergesToExpungeDeletes was renamed to findForcedDeletesMerges. (Robert Muir, Mike McCandless) New Features * LUCENE-3448: Added FixedBitSet.and(other/DISI), andNot(other/DISI). (Uwe Schindler) * LUCENE-2215: Added IndexSearcher.searchAfter which returns results after a specified ScoreDoc (e.g. last document on the previous page) to support deep paging use cases. (Aaron McCurry, Grant Ingersoll, Robert Muir) * LUCENE-1990: Adds internal packed ints implementation, to be used for more efficient storage of int arrays when the values are bounded, for example for storing the terms dict index (Toke Eskildsen via Mike McCandless) * LUCENE-3558: Moved SearcherManager, NRTManager & SearcherLifetimeManager into core. All classes are contained in o.a.l.search. (Simon Willnauer) Optimizations * LUCENE-3426: Add NGramPhraseQuery which extends PhraseQuery and tries to reduce the number of terms of the query when rewrite(), in order to improve performance. (Robert Muir, Koji Sekiguchi) * LUCENE-3494: Optimize FilteredQuery to remove a multiply in score() (Uwe Schindler, Robert Muir) * LUCENE-3534: Remove filter logic from IndexSearcher and delegate to FilteredQuery's Scorer. This is a partial backport of a cleanup in FilteredQuery/IndexSearcher added by LUCENE-1536 to Lucene 4.0. (Uwe Schindler) * LUCENE-2205: Very substantial (3-5X) RAM reduction required to hold the terms index on opening an IndexReader (Aaron McCurry via Mike McCandless) * LUCENE-3443: FieldCache can now set docsWithField, and create an array, in a single pass. This results in faster init time for apps that need both (such as sorting by a field with a missing value). (Mike McCandless) Test Cases * LUCENE-3420: Disable the finalness checks in TokenStream and Analyzer