elasticdump迁移数据到新es

elasticdump ================== Tools for moving and saving indices.  --- [](https://npmjs.org/package/elasticdump) <br /> [](https://hub.docker.com/r/elasticdump/elasticsearch-dump/) [](https://hub.docker.com/r/taskrabbit/elasticsearch-dump/) [](http://travis-ci.org/elasticsearch-dump/elasticsearch-dump) [](https://npmjs.com/elasticdump) ## Version Warnings! - Version `1.0.0` of Elasticdump changes the format of the files created by the dump. Files created with version `0.x.x` of this tool are likely not to work with versions going forward. To learn more about the breaking changes, vist the release notes for version [`1.0.0`](https://github.com/elasticsearch-dump/elasticsearch-dump/releases/tag/v1.0.0). If you recive an "out of memory" error, this is probably or most likely the cause. - Version `2.0.0` of Elasticdump removes the `bulk` options. These options were buggy, and differ between versions of Elasticsearch. If you need to export multiple indexes, look for the `multielasticdump` section of the tool. - Version `2.1.0` of Elasticdump moves from using `scan/scroll` (ES 1.x) to just `scan` (ES 2.x). This is a backwards-compatible change within Elasticsearch, but performance may suffer on Elasticsearch versions prior to 2.x. - Version `3.0.0` of Elasticdump has the default queries updated to only work for ElasticSearch version 5+. The tool *may* be compatible with earlier versions of Elasticsearch, but our version detection method may not work for all ES cluster topologies - Version `5.0.0` of Elasticdump contains a breaking change for the s3 transport. _s3Bucket_ and _s3RecordKey_ params are no longer supported please use s3urls instead - Version `6.1.0` and higher of Elasticdump contains a change to the upload/dump process. This change allows for overlapping promise processing. The benefit of which is improved performance due increased parallel processing, but a side-effect exists where-by records (data-set) aren't processed in a sequential order (the ordering is no longer guaranteed) - Version `6.67.0` and higher of Elasticdump will quit if the node version does not match the minimum requirement needed (v10.0.0) ## Installing (local) ```bash npm install elasticdump ./bin/elasticdump ``` (global) ```bash npm install elasticdump -g elasticdump ``` ## Use ### Standard Install Elasticdump works by sending an `input` to an `output`. Both can be either an elasticsearch URL or a File. Elasticsearch: - format: `{protocol}://{host}:{port}/{index}` - example: `http://127.0.0.1:9200/my_index` File: - format: `{FilePath}` - example: `/Users/evantahler/Desktop/dump.json` Stdio: - format: stdin / stdout - format: `$` You can then do things like: ```bash # Copy an index from production to staging with analyzer and mapping: elasticdump \ --input=http://production.es.com:9200/my_index \ --output=http://staging.es.com:9200/my_index \ --type=analyzer elasticdump \ --input=http://production.es.com:9200/my_index \ --output=http://staging.es.com:9200/my_index \ --type=mapping elasticdump \ --input=http://production.es.com:9200/my_index \ --output=http://staging.es.com:9200/my_index \ --type=data # Backup index data to a file: elasticdump \ --input=http://production.es.com:9200/my_index \ --output=/data/my_index_mapping.json \ --type=mapping elasticdump \ --input=http://production.es.com:9200/my_index \ --output=/data/my_index.json \ --type=data # Backup and index to a gzip using stdout: elasticdump \ --input=http://production.es.com:9200/my_index \ --output=$ \ | gzip > /data/my_index.json.gz # Backup the results of a query to a file elasticdump \ --input=http://production.es.com:9200/my_index \ --output=query.json \ --searchBody="{\"query\":{\"term\":{\"username\": \"admin\"}}}" # Specify searchBody from a file elasticdump \ --input=http://production.es.com:9200/my_index \ --output=query.json \ --searchBody=@/data/searchbody.json # Copy a single shard data: elasticdump \ --input=http://es.com:9200/api \ --output=http://es.com:9200/api2 \ --input-params="{\"preference\":\"_shards:0\"}" # Backup aliases to a file elasticdump \ --input=http://es.com:9200/index-name/alias-filter \ --output=alias.json \ --type=alias # Import aliases into ES elasticdump \ --input=./alias.json \ --output=http://es.com:9200 \ --type=alias # Backup templates to a file elasticdump \ --input=http://es.com:9200/template-filter \ --output=templates.json \ --type=template # Import templates into ES elasticdump \ --input=./templates.json \ --output=http://es.com:9200 \ --type=template # Split files into multiple parts elasticdump \ --input=http://production.es.com:9200/my_index \ --output=/data/my_index.json \ --fileSize=10mb # Import data from S3 into ES (using s3urls) elasticdump \ --s3AccessKeyId "${access_key_id}" \ --s3SecretAccessKey "${access_key_secret}" \ --input "s3://${bucket_name}/${file_name}.json" \ --output=http://production.es.com:9200/my_index # Export ES data to S3 (using s3urls) elasticdump \ --s3AccessKeyId "${access_key_id}" \ --s3SecretAccessKey "${access_key_secret}" \ --input=http://production.es.com:9200/my_index \ --output "s3://${bucket_name}/${file_name}.json" # Import data from MINIO (s3 compatible) into ES (using s3urls) elasticdump \ --s3AccessKeyId "${access_key_id}" \ --s3SecretAccessKey "${access_key_secret}" \ --input "s3://${bucket_name}/${file_name}.json" \ --output=http://production.es.com:9200/my_index --s3ForcePathStyle true --s3Endpoint https://production.minio.co # Export ES data to MINIO (s3 compatible) (using s3urls) elasticdump \ --s3AccessKeyId "${access_key_id}" \ --s3SecretAccessKey "${access_key_secret}" \ --input=http://production.es.com:9200/my_index \ --output "s3://${bucket_name}/${file_name}.json" --s3ForcePathStyle true --s3Endpoint https://production.minio.co # Import data from CSV file into ES (using csvurls) elasticdump \ # csv:// prefix must be included to allow parsing of csv files # --input "csv://${file_path}.csv" \ --input "csv:///data/cars.csv" --output=http://production.es.com:9200/my_index \ --csvSkipRows 1 # used to skip parsed rows (this does not include the headers row) --csvDelimiter ";" # default csvDelimiter is ',' ``` ### Non-Standard Install If Elasticsearch is not being served from the root directory the `--input-index` and `--output-index` are required. If they are not provided, the additional sub-directories will be parsed for index and type. Elasticsearch: - format: `{protocol}://{host}:{port}/{sub}/{directory...}` - example: `http://127.0.0.1:9200/api/search` ```bash # Copy a single index from a elasticsearch: elasticdump \ --input=http://es.com:9200/api/search \ --input-index=my_index \ --output=http://es.com:9200/api/search \ --output-index=my_index \ --type=mapping # Copy a single type: elasticdump \ --input=http://es.com:9200/api/search \ --input-index=my_index/my_type \ --output=http://es.com:9200/api/search \ --output-index=my_index \ --type=mapping ``` ### Docker install If you prefer using docker to use elasticdump, you can download this project from docker hub: ```bash docker pull elasticdump/elasticsearch-dump ``` Then you can use it just by : - using `docker run --rm -ti elasticdump/elasticsearch-dump` - you'll need to mount your file storage dir `-v <your dumps dir>:<your mount point>` to your docker container Example: ```bash # Copy