lucene-1.4-final

Lucene Change Log $Id: CHANGES.txt,v 1.94 2004/07/01 17:40:41 cutting Exp $ 1.4 final 1. Added "an" to the list of stop words in StopAnalyzer, to complement the existing "a" there. Fix for bug 28960 (http://issues.apache.org/bugzilla/show_bug.cgi?id=28960). (Otis) 2. Added new class FieldCache to manage in-memory caches of field term values. (Tim Jones) 3. Added overloaded getFieldQuery method to QueryParser which accepts the slop factor specified for the phrase (or the default phrase slop for the QueryParser instance). This allows overriding methods to replace a PhraseQuery with a SpanNearQuery instead, keeping the proper slop factor. (Erik Hatcher) 4. Changed the encoding of GermanAnalyzer.java and GermanStemmer.java to UTF-8 and changed the build encoding to UTF-8, to make changed files compile. (Otis Gospodnetic) 5. Removed synchronization from term lookup under IndexReader methods termFreq(), termDocs() or termPositions() to improve multi-threaded performance. (cutting) 6. Fix a bug where obsolete segment files were not deleted on Win32. 1.4 RC3 1. Fixed several search bugs introduced by the skipTo() changes in release 1.4RC1. The index file format was changed a bit, so collections must be re-indexed to take advantage of the skipTo() optimizations. (Christoph Goller) 2. Added new Document methods, removeField() and removeFields(). (Christoph Goller) 3. Fixed inconsistencies with index closing. Indexes and directories are now only closed automatically by Lucene when Lucene opened them automatically. (Christoph Goller) 4. Added new class: FilteredQuery. (Tim Jones) 5. Added a new SortField type for custom comparators. (Tim Jones) 6. Lock obtain timed out message now displays the full path to the lock file. (Daniel Naber via Erik) 7. Fixed a bug in SpanNearQuery when ordered. (Paul Elschot via cutting) 8. Fixed so that FSDirectory's locks still work when the java.io.tmpdir system property is null. (cutting) 9. Changed FilteredTermEnum's constructor to take no parameters, as the parameters were ignored anyway (bug #28858) 1.4 RC2 1. GermanAnalyzer now throws an exception if the stopword file cannot be found (bug #27987). It now uses LowerCaseFilter (bug #18410) (Daniel Naber via Otis, Erik) 2. Fixed a few bugs in the file format documentation. (cutting) 1.4 RC1 1. Changed the format of the .tis file, so that: - it has a format version number, which makes it easier to back-compatibly change file formats in the future. - the term count is now stored as a long. This was the one aspect of the Lucene's file formats which limited index size. - a few internal index parameters are now stored in the index, so that they can (in theory) now be changed from index to index, although there is not yet an API to do so. These changes are back compatible. The new code can read old indexes. But old code will not be able read new indexes. (cutting) 2. Added an optimized implementation of TermDocs.skipTo(). A skip table is now stored for each term in the .frq file. This only adds a percent or two to overall index size, but can substantially speedup many searches. (cutting) 3. Restructured the Scorer API and all Scorer implementations to take advantage of an optimized TermDocs.skipTo() implementation. In particular, PhraseQuerys and conjunctive BooleanQuerys are faster when one clause has substantially fewer matches than the others. (A conjunctive BooleanQuery is a BooleanQuery where all clauses are required.) (cutting) 4. Added new class ParallelMultiSearcher. Combined with RemoteSearchable this makes it easy to implement distributed search systems. (Jean-Francois Halleux via cutting) 5. Added support for hit sorting. Results may now be sorted by any indexed field. For details see the javadoc for Searcher#search(Query, Sort). (Tim Jones via Cutting) 6. Changed FSDirectory to auto-create a full directory tree that it needs by using mkdirs() instead of mkdir(). (Mladen Turk via Otis) 7. Added a new span-based query API. This implements, among other things, nested phrases. See javadocs for details. (Doug Cutting) 8. Added new method Query.getSimilarity(Searcher), and changed scorers to use it. This permits one to subclass a Query class so that it can specify it's own Similarity implementation, perhaps one that delegates through that of the Searcher. (Julien Nioche via Cutting) 9. Added MultiReader, an IndexReader that combines multiple other IndexReaders. (Cutting) 10. Added support for term vectors. See Field#isTermVectorStored(). (Grant Ingersoll, Cutting & Dmitry) 11. Fixed the old bug with escaping of special characters in query strings: http://issues.apache.org/bugzilla/show_bug.cgi?id=24665 (Jean-Francois Halleux via Otis) 12. Added support for overriding default values for the following, using system properties: - default commit lock timeout - default maxFieldLength - default maxMergeDocs - default mergeFactor - default minMergeDocs - default write lock timeout (Otis) 13. Changed QueryParser.jj to allow '-' and '+' within tokens: http://issues.apache.org/bugzilla/show_bug.cgi?id=27491 (Morus Walter via Otis) 14. Changed so that the compound index format is used by default. This makes indexing a bit slower, but vastly reduces the chances of file handle problems. (Cutting) 1.3 final 1. Added catch of BooleanQuery$TooManyClauses in QueryParser to throw ParseException instead. (Erik Hatcher) 2. Fixed a NullPointerException in Query.explain(). (Doug Cutting) 3. Added a new method IndexReader.setNorm(), that permits one to alter the boosting of fields after an index is created. 4. Distinguish between the final position and length when indexing a field. The length is now defined as the total number of tokens, instead of the final position, as it was previously. Length is used for score normalization (Similarity.lengthNorm()) and for controlling memory usage (IndexWriter.maxFieldLength). In both of these cases, the total number of tokens is a better value to use than the final token position. Position is used in phrase searching (see PhraseQuery and Token.setPositionIncrement()). 5. Fix StandardTokenizer's handling of CJK characters (Chinese, Japanese and Korean ideograms). Previously contiguous sequences were combined in a single token, which is not very useful. Now each ideogram generates a separate token, which is more useful. 1.3 RC3 1. Added minMergeDocs in IndexWriter. This can be raised to speed indexing without altering the number of files, but only using more memory. (Julien Nioche via Otis) 2. Fix bug #24786, in query rewriting. (bschneeman via Cutting) 3. Fix bug #16952, in demo HTML parser, skip comments in javascript. (Christoph Goller) 4. Fix bug #19253, in demo HTML parser, add whitespace as needed to output (Daniel Naber via Christoph Goller) 5. Fix bug #24301, in demo HTML parser, long titles no longer hang things. (Christoph Goller) 6. Fix bug #23534, Replace use of file timestamp of segments file with an index version number stored in the segments file. This resolves problems when running on file systems with low-resolution timestamps, e.g., HFS under MacOS X. (Christoph Goller) 7. Fix QueryParser so that TokenMgrError is not thrown, only ParseException. (Erik Hatcher) 8. Fix some bugs introduced by change 11 of RC2. (Christoph Goller) 9. Fixed a problem compiling TestRussianStem. (Christoph Goller) 10. Cleaned up some build stuff. (Erik Hatcher) 1.3 RC2 1. Added getFieldNames(boolean) to IndexReader, SegmentReader, and SegmentsReader. (Julie