# Logstash
Logstash is part of the [Elastic Stack](https://www.elastic.co/products) along with Beats, Elasticsearch and Kibana. Logstash is a server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash." (Ours is Elasticsearch, naturally.). Logstash has over 200 plugins, and you can write your own very easily as well.
For more info, see <https://www.elastic.co/products/logstash>
## Documentation and Getting Started
You can find the documentation and getting started guides for Logstash
on the [elastic.co site](https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html)
For information about building the documentation, see the README in https://github.com/elastic/docs
## Downloads
You can download officially released Logstash binaries, as well as debian/rpm packages for the
supported platforms, from [downloads page](https://www.elastic.co/downloads/logstash).
## Need Help?
- [Logstash Forum](https://discuss.elastic.co/c/logstash)
- [Logstash Documentation](https://www.elastic.co/guide/en/logstash/current/index.html)
- [#logstash on freenode IRC](https://webchat.freenode.net/?channels=logstash)
- [Logstash Product Information](https://www.elastic.co/products/logstash)
- [Elastic Support](https://www.elastic.co/subscriptions)
## Logstash Plugins
Logstash plugins are hosted in separate repositories under the [logstash-plugins](https://github.com/logstash-plugins) github organization. Each plugin is a self-contained Ruby gem which gets published to RubyGems.org.
### Writing your own Plugin
Logstash is known for its extensibility. There are hundreds of plugins for Logstash and you can write your own very easily! For more info on developing and testing these plugins, please see the [working with plugins section](https://www.elastic.co/guide/en/logstash/current/contributing-to-logstash.html)
### Plugin Issues and Pull Requests
**Please open new issues and pull requests for plugins under its own repository**
For example, if you have to report an issue/enhancement for the Elasticsearch output, please do so [here](https://github.com/logstash-plugins/logstash-output-elasticsearch/issues).
Logstash core will continue to exist under this repository and all related issues and pull requests can be submitted here.
## Developing Logstash Core
### Prerequisites
* Install JDK version 8 or 11. Make sure to set the `JAVA_HOME` environment variable to the path to your JDK installation directory. For example `set JAVA_HOME=<JDK_PATH>`
* Install JRuby 9.2.x It is recommended to use a Ruby version manager such as [RVM](https://rvm.io/) or [rbenv](https://github.com/sstephenson/rbenv).
* Install `rake` and `bundler` tool using `gem install rake` and `gem install bundler` respectively.
### RVM install (optional)
If you prefer to use rvm (ruby version manager) to manage Ruby versions on your machine, follow these directions. In the Logstash folder:
```sh
gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
\curl -sSL https://get.rvm.io | bash -s stable --ruby=$(cat .ruby-version)
```
### Check Ruby version
Before you proceed, please check your ruby version by:
```sh
$ ruby -v
```
The printed version should be the same as in the `.ruby-version` file.
### Building Logstash
The Logstash project includes the source code for all of Logstash, including the Elastic-Licensed X-Pack features and functions; to run Logstash from source using only the OSS-licensed code, export the `OSS` environment variable with a value of `true`:
``` sh
export OSS=true
```
* Set up the location of the source code to build
``` sh
export LOGSTASH_SOURCE=1
export LOGSTASH_PATH=/YOUR/LOGSTASH/DIRECTORY
```
* To run Logstash from the repo you must first bootstrap the environment:
```sh
rake bootstrap
```
* You can then use `bin/logstash` to start Logstash, but there are no plugins installed. To install default plugins, you can run:
```sh
rake plugin:install-default
```
This will install the 80+ default plugins which makes Logstash ready to connect to multiple data sources, perform transformations and send the results to Elasticsearch and other destinations.
To verify your environment, run the following to send your first event:
```sh
bin/logstash -e 'input { stdin { } } output { stdout {} }'
```
This should start Logstash with stdin input waiting for you to enter an event
```sh
hello world
2016-11-11T01:22:14.405+0000 0.0.0.0 hello world
```
**Advanced: Drip Launcher**
[Drip](https://github.com/ninjudd/drip) is a tool that solves the slow JVM startup problem while developing Logstash. The drip script is intended to be a drop-in replacement for the java command. We recommend using drip during development, in particular for running tests. Using drip, the first invocation of a command will not be faster but the subsequent commands will be swift.
To tell logstash to use drip, set the environment variable `` JAVACMD=`which drip` ``.
Example (but see the *Testing* section below before running rspec for the first time):
JAVACMD=`which drip` bin/rspec
**Caveats**
Drip does not work with STDIN. You cannot use drip for running configs which use the stdin plugin.
## Building Logstash Documentation
To build the Logstash Reference (open source content only) on your local
machine, clone the following repos:
[logstash](https://github.com/elastic/logstash) - contains main docs about core features
[logstash-docs](https://github.com/elastic/logstash-docs) - contains generated plugin docs
[docs](https://github.com/elastic/docs) - contains doc build files
Make sure you have the same branch checked out in `logstash` and `logstash-docs`.
Check out `master` in the `docs` repo.
Run the doc build script from within the `docs` repo. For example:
```
./build_docs.pl --doc ../logstash/docs/index.asciidoc --chunk=1 -open
```
## Testing
Most of the unit tests in Logstash are written using [rspec](http://rspec.info/) for the Ruby parts. For the Java parts, we use junit. For testing you can use the *test* `rake` tasks and the `bin/rspec` command, see instructions below:
### Core tests
1- To run the core tests you can use the Gradle task:
./gradlew test
or use the `rspec` tool to run all tests or run a specific test:
bin/rspec
bin/rspec spec/foo/bar_spec.rb
Note that before running the `rspec` command for the first time you need to set up the RSpec test dependencies by running:
./gradlew bootstrap
2- To run the subset of tests covering the Java codebase only run:
./gradlew javaTests
3- To execute the complete test-suite including the integration tests run:
./gradlew check
4- To execute a single Ruby test run:
SPEC_OPTS="-fd -P logstash-core/spec/logstash/api/commands/default_metadata_spec.rb" ./gradlew :logstash-core:rubyTests --tests org.logstash.RSpecTests
5- To execute single spec for integration test, run:
./gradlew integrationTests -PrubyIntegrationSpecs=specs/slowlog_spec.rb
Sometimes you might find a change to a piece of Logstash code causes a test to hang. These can be hard to debug.
If you set `LS_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"` you can connect to a running Logstash with your IDEs debugger which can be a great way of finding the issue.
### Plugins tests
To run the tests of all currently installed plugins:
rake test:plugins
You can install the default set of plugins included in the logstash package:
rake test:install-default
---
Note that if a plugin is installed using the plugin manager `bin/logstash-plugin install ...` do not forget to also install the plugins development dependencies using the following command after the plugin installation:
bin/logstash-plugin install --development
## Building Artifacts
Built artifacts will be placed in the `LS_HOME/build` directory, and will create the d
没有合适的资源?快使用搜索试试~ 我知道了~
ElasticSearch + Logstash 自动同步mysql数据
共2000个文件
json:1123个
java:471个
txt:171个
需积分: 5 0 下载量 129 浏览量
2024-01-04
11:59:38
上传
评论
收藏 21.73MB ZIP 举报
温馨提示
ElasticSearch + Logstash 自动同步mysql数据
资源推荐
资源详情
资源评论
收起资源包目录
ElasticSearch + Logstash 自动同步mysql数据 (2000个子文件)
index.asciidoc 3KB
offline.c 3KB
plugin-doc.css 14KB
arcsight.conf.erb 2KB
template.cfg.erb 2KB
example.rb.erb 2KB
azure.conf.erb 2KB
logstash-input-example.gemspec.erb 1KB
CONTRIBUTORS.erb 432B
example_spec.rb.erb 277B
DEVELOPER.md.erb 129B
Gemfile 39B
env2yaml.go 5KB
build.gradle 2KB
settings.gradle 28B
sample.gz 1KB
gradle-wrapper.jar 58KB
QueueTest.java 40KB
Queue.java 33KB
CompiledPipelineTest.java 32KB
JavaKeyStoreTest.java 31KB
EventCondition.java 30KB
RubyUtil.java 30KB
CompiledPipeline.java 23KB
DeadLetterQueueReaderTest.java 22KB
DatasetCompiler.java 22KB
JavaKeyStore.java 19KB
EventTest.java 18KB
AbstractPipelineExt.java 18KB
Graph.java 16KB
Event.java 16KB
ObjectMappers.java 15KB
MmapPageIOV2.java 14KB
PluginFactoryExt.java 14KB
JrubyEventExtLibrary.java 14KB
LineTest.java 14KB
RecordIOReader.java 14KB
DeadLetterQueueWriter.java 14KB
CommonActionsTest.java 13KB
DSL.java 12KB
LogstashInstallation.java 12KB
JvmOptionsParser.java 12KB
Page.java 12KB
JrubyTimestampExtLibrary.java 12KB
PipelineConfigTest.java 12KB
PluginUtilValidateConfigTest.java 12KB
FieldReference.java 12KB
RubyJavaIntegration.java 11KB
OutputStrategyExt.java 11KB
ReportGenerator.java 11KB
Main.java 11KB
MmapPageIOV1.java 11KB
PipelineReporterExt.java 11KB
SecretStoreCli.java 10KB
ConfigurationImplTest.java 10KB
CloudSettingIdTest.java 10KB
ComputeStepSyntaxElement.java 10KB
RubyfierTest.java 10KB
PipelineBus.java 10KB
RecordIOReaderTest.java 9KB
MetricExt.java 9KB
ConvergeResultExt.java 9KB
PqRepair.java 9KB
JavaCodecDelegatorTest.java 9KB
SyntaxFactory.java 9KB
Logstash.java 9KB
IngestConverter.java 9KB
IRHelpers.java 9KB
ReportGeneratorTest.java 8KB
JsUtil.java 8KB
PluginConfigSpec.java 8KB
LoggerExt.java 8KB
SecretStoreCliTest.java 8KB
QueueReadClientBase.java 8KB
ConfigCompilerTest.java 8KB
OutputDelegatorTest.java 8KB
ImperativeToGraphtest.java 8KB
PlainTest.java 8KB
Valuefier.java 8KB
SettingsImpl.java 8KB
CommonActions.java 8KB
PipelineBusTest.java 8KB
HotThreadsMonitor.java 8KB
DeadLetterQueueWriterTest.java 8KB
SecretStoreFactoryTest.java 8KB
Generator.java 8KB
ShutdownWatcherExt.java 8KB
AccessorsTest.java 8KB
SecretStoreFactory.java 8KB
EventConditionTest.java 7KB
PipelineConfig.java 7KB
Vertex.java 7KB
AbstractDeadLetterQueueWriterExt.java 7KB
LsMetricsMonitor.java 7KB
RecordIOWriter.java 7KB
CloudSettingId.java 7KB
Accessors.java 7KB
DeadLetterQueueReader.java 7KB
JavaInputDelegatorExt.java 7KB
FieldReferenceTest.java 7KB
共 2000 条
- 1
- 2
- 3
- 4
- 5
- 6
- 20
资源评论
中年程序员一枚
- 粉丝: 1099
- 资源: 32
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功