fscrawler-2.4

# FileSystem Crawler for Elasticsearch Welcome to the FS Crawler for [Elasticsearch](https://elastic.co/) This crawler helps to index documents from your local file system and over SSH. It crawls your file system and index new files, update existing ones and removes old ones. You need to install a version matching your Elasticsearch version: | Elasticsearch | FS Crawler | Released | Docs | |--------------------|-------------|----------|------------------------------------------------------------------------------| | 2.x, 5.x, 6.x | 2.4-SNAPSHOT| |See below | | 2.x, 5.x, 6.x | **2.3** |2017-07-10|[2.3](https://github.com/dadoonet/fscrawler/blob/fscrawler-2.3/README.md) | | 1.x, 2.x, 5.x | 2.2 |2017-02-03|[2.2](https://github.com/dadoonet/fscrawler/blob/fscrawler-2.2/README.md) | | 1.x, 2.x, 5.x | 2.1 |2016-07-26|[2.1](https://github.com/dadoonet/fscrawler/blob/fscrawler-2.1/README.md) | | es-2.0 | 2.0.0 |2015-10-30|[2.0.0](https://github.com/dadoonet/fscrawler/blob/fscrawler-2.0.0/README.md) | ## Build Status Thanks to Travis for the [build status](https://travis-ci.org/dadoonet/fscrawler): [](https://travis-ci.org/dadoonet/fscrawler) # Table of content * [Installation guide](#installation-guide) * [Download fscrawler](#download-fscrawler) * [Upgrade fscrawler](#upgrade-fscrawler) * [User guide](#user-guide) * [Getting Started](#getting-started) * [Searching for docs](#searching-for-docs) * [Crawler options](#crawler-options) * [Starting with a REST gateway](#starting-with-a-rest-gateway) * [Supported formats](#supported-formats) * [Administration guide](#administration-guide) * [CLI options](#cli-options) * [JVM Settings](#jvm-settings) * [Job file specification](#job-file-specification) * [Local FS settings](#local-fs-settings) * [SSH settings](#ssh-settings) * [Elasticsearch settings](#elasticsearch-settings) * [REST service](#rest-service) * [Tips and tricks](#tips-and-tricks) * [License](#license) # Installation Guide ## Download fscrawler FS Crawler binary is available on [Maven Central](https://repo1.maven.org/maven2/fr/pilato/elasticsearch/crawler/fscrawler/). Just download the latest release (or any other specific version you want to try). The filename ends with `.zip`. For example, if you wish to download [fscrawler-2.3](https://repo1.maven.org/maven2/fr/pilato/elasticsearch/crawler/fscrawler/2.3/fscrawler-2.3.zip): ```sh wget https://repo1.maven.org/maven2/fr/pilato/elasticsearch/crawler/fscrawler/2.3/fscrawler-2.3.zip unzip fscrawler-2.3.zip ``` The distribution contains: ``` $ tree . ├── LICENSE ├── NOTICE ├── README.md ├── bin │   ├── fscrawler │   └── fscrawler.bat └── lib ├── ... All needed jars ``` Note that you can also download a SNAPSHOT version [from sonatype](https://oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler/2.3-SNAPSHOT/) without needing to build it by yourself. ## Upgrade fscrawler It can happen that you need to [upgrade a mapping](#upgrading-an-existing-mapping) before starting fscrawler after a version upgrade. Read carefully the following update instructions. To update fscrawler, just download the new version, unzip it in another directory and launch it as usual. It will still pick up settings from the configuration directory. Of course, you need to stop first the existing running instances. ### Upgrade to 2.2 * fscrawler comes with new default mappings for files. They have better defaults as they consume less disk space and CPU at index time. You should remove existing files in `~/.fscrawler/_default/_mappings` before starting the new version so default mappings will be updated. If you modified manually mapping files, apply the modification you made on sample files. * `excludes` is now set by default for new jobs to `["~*"]`. In previous versions, any file or directory containing a `~` was excluded. Which means that if in your jobs, you are defining any exclusion rule, you need to add `*~*` if you want to get back the exact previous behavior. * If you were indexing `json` or `xml` documents with the `filename_as_id` option set, we were previously removing the suffix of the file name, like indexing `1.json` was indexed as `1`. With this new version, we don't remove anymore the suffix. So the `_id` for your document will be now `1.json`. ### Upgrade to 2.3 * fscrawler comes with new mapping for folders. The change is really tiny so you can skip this step if you wish. We basically removed `name` field in the folder mapping as it was unused. * The way FSCrawler computes now `path.virtual` for docs has changed. It now includes the filename. Instead of `/path/to` you will now get `/path/to/file.txt`. * The way FSCrawler computes now `virtual` for folders is now consistent with what you can see for folders. * `path.encoded` in documents and `encoded` in folders have been removed as not needed by FSCrawler after all. * [OCR](#ocr-integration) is now properly activated for PDF documents. This can be time, cpu and memory consuming though. You can disable explicitly it by setting `fs.pdf_ocr` to `false`. * All dates are now indexed in elasticsearch in UTC instead of without any time zone. For example, we were indexing previously a date like `2017-05-19T13:24:47.000`. Which was producing bad results when you were located in a time zone other than UTC. It's now indexed as `2017-05-19T13:24:47.000+0000`. * In order to be compatible with the coming 6.0 elasticsearch version, we need to get rid of types as only one type per index is still supported. Which means that we now create index named `job_name` and `job_name_folder` instead of one index `job_name` with two types `doc` and `folder`. If you are upgrading from FSCrawler 2.2, it requires that you reindex your existing data either by deleting the old index and running again FSCrawler or by using the [reindex API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html) as follows: ``` # Create folder index job_name_folder based on existing folder data POST _reindex { "source": { "index": "job_name", "type": "folder" }, "dest": { "index": "job_name_folder" } } # Remove old folder data from job_name index POST job_name/folder/_delete_by_query { "query": { "match_all": {} } } ``` Note that you will need first to create the right settings and mappings so you can then run the reindex job. You can do that by launching `bin/fscrawler job_name --loop 0`. Better, you can run `bin/fscrawler job_name --upgrade` and let FSCrawler do all that for you. Note that this can take a loooong time. Also please be aware that some APIs used by the upgrade action are only available from elasticsearch 2.3 (reindex) or elasticsearch 5.0 (delete by query). If you are running an older version than 5.0 you need first to upgrade elasticsearch. This procedure only applies if you did not set previously `elasticsearch.type` setting (default value was `doc`). If you did, then you also need to reindex the existing documents to the default `doc` type as per elasticsearch 6.0: ``` # Copy old type doc to the default doc type POST _reindex { "source": { "index": "job_name", "type": "your_type_here" }, "dest": { "index": "job_name", "type": "doc" } } # Remove old type data from job_name index POST job_name/your_type_here/_delete_by_query { "query": { "match_all": {} } } ``` But note that this last step can take a very loooong time and will generate a lot of IO on your disk. It might be easier in such case to restart fscrawler from scratch. *