logstash-filter-plugin example 테스트

해당 문서는 https://www.elastic.co/guide/en/logstash/7.3/java-filter-plugin.html 를 기반으로 순서대로 따라해 보았습니다.

logstash 7.3 빌드

샘플 java-filter-plugin을 빌드하려면 logstash-core가 필요하여, 해당 버전에 맞는 소스 코드 다운로드 후 빌드를 해 줍니다.

naruui-Mac-Pro:IdeaProjects naru$ git clone --branch 7.3 --single-branch https://github.com/elastic/logstash.git logstash
Cloning into 'logstash'...
remote: Enumerating objects: 66335, done.
remote: Counting objects: 100% (103/103), done.
remote: Compressing objects: 100% (44/44), done.
remote: Total 66335 (delta 87), reused 62 (delta 59), pack-reused 66232
Receiving objects: 100% (66335/66335), 102.24 MiB | 13.83 MiB/s, done.
Resolving deltas: 100% (41882/41882), done.
Checking out files: 100% (2789/2789), done.

naruui-Mac-Pro:logstash naru$ ./gradlew assemble
Downloading https://services.gradle.org/distributions/gradle-4.10-bin.zip
..........................................................................

Welcome to Gradle 4.10!

Here are the highlights of this release:
 - Incremental Java compilation by default
 - Periodic Gradle caches cleanup
 - Gradle Kotlin DSL 1.0-RC3
 - Nested included builds
 - SNAPSHOT plugin versions in the `plugins {}` block

For more details see https://docs.gradle.org/4.10/release-notes.html


Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1m 57s
26 actionable tasks: 26 executed

logstash filter plugin example 빌드

https://github.com/logstash-plugins/logstash-filter-java_filter_example 의 소스코드를 받아서 아래와 같이 빌드합니다.

빌드하기전에 gradle.properties파일을 생성하여 아래와 같이 logstash-core가 빌드된 경로를 지정합니다.

LOGSTASH_CORE_PATH=/Users/naru/IdeaProjects/logstash/logstash-core

naruui-Mac-Pro:logstash-filter-java\_filter\_example naru$ ./gradlew gem

> Task :downloadJRuby  
> Unable to get progress logger. Download progress will not be displayed.

> Task :downloadAndInstallJRuby  
> Fetching: rake-12.3.1.gem (100%)  
> Successfully installed rake-12.3.1  
> 1 gem installed  
> Fetching: json-1.8.6-java.gem (100%)  
> Successfully installed json-1.8.6-java  
> 1 gem installed

> Task :gem  
> WARNING: open-ended dependency on jar-dependencies (>= 0) is not recommended  
> if jar-dependencies is semantically versioned, use:  
> add\_runtime\_dependency 'jar-dependencies', '
> 
> ~\> 0'  
> WARNING: open-ended dependency on logstash-devutils (>= 0, development) is not recommended  
> if logstash-devutils is semantically versioned, use:  
> add\_development\_dependency 'logstash-devutils', '~
> 
> \> 0'  
> WARNING: See [http://guides.rubygems.org/specification-reference/](http://guides.rubygems.org/specification-reference/) for help  
> Successfully built RubyGem  
> Name: logstash-filter-java\_filter\_example  
> Version: 1.0.2  
> File: logstash-filter-java\_filter\_example-1.0.2.gem

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.  
Use '--warning-mode all' to show the individual deprecation warnings.  
See [https://docs.gradle.org/6.5.1/userguide/command\_line\_interface.html#sec:command\_line\_warnings](https://docs.gradle.org/6.5.1/userguide/command_line_interface.html#sec:command_line_warnings)

BUILD SUCCESSFUL in 31s  
9 actionable tasks: 9 executed

저의 경우에는 빌드시 아래와 같은 2가지 문제가 나타났습니다. 여러가지 블로그를 참고하여 해결한 내용입니다.

빌드 에러 처리

naruui-Mac-Pro:logstash-filter-java\_filter\_example naru$ ./gradlew gem

FAILURE: Build failed with an exception.

-   Where:  
    Build file '/Users/naru/IdeaProjects/logstash-filter-java\_filter\_example/build.gradle' line: 101
-   What went wrong:  
    A problem occurred configuring root project 'logstash-filter-java\_filter\_example'.
-   Could not create task ':gem'.  
    No signature of method: org.gradle.api.internal.tasks.DefaultTaskDependency$TaskDependencySet.getAt() is applicable for argument types: (ArrayList) values: \[\[task ':downloadAndInstallJRuby', task ':removeObsoleteJars', ...\]\]  
    Possible solutions: getAt(java.lang.String), getAt(java.lang.String), getAt(int), head(), putAt(java.lang.String, java.lang.Object), wait()
-   Try:  
    Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
-   Get more help at [https://help.gradle.org](https://help.gradle.org)

아래와 같이 build.gradle 파일 수정

dependsOn \[downloadAndInstallJRuby, removeObsoleteJars, vendor, generateRubySupportFiles\]

\=> dependsOn (\[downloadAndInstallJRuby, removeObsoleteJars, vendor, generateRubySupportFiles\])

naruui-Mac-Pro:logstash-filter-java\_filter\_example naru$ ./gradlew gem

> Task :downloadJRuby FAILED

FAILURE: Build failed with an exception.

-   What went wrong:  
    A problem was found with the configuration of task ':downloadJRuby' (type 'Download').
-   File '/Users/naru/IdeaProjects/logstash-filter-java\_filter\_example/versions.yml' specified for property '$1' does not exist.
-   Try:  
    Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
-   Get more help at [https://help.gradle.org](https://help.gradle.org)

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.  
Use '--warning-mode all' to show the individual deprecation warnings.  
See [https://docs.gradle.org/6.5.1/userguide/command\_line\_interface.html#sec:command\_line\_warnings](https://docs.gradle.org/6.5.1/userguide/command_line_interface.html#sec:command_line_warnings)

BUILD FAILED in 2s  
1 actionable task: 1 executed

내려받은 logstash 최상위 디렉터리에서 versions.yml을 logstash-filter-java_filter_example의 최상위 디렉터리로 복사

Plguin 설치

bin/logstash-plugin install --no-verify --local /path/to/logstash-filter-java\_filter\_example-1.0.2.gem  
Installing logstash-filter-java\_filter\_example  
Installation successful

테스트

# java\_filter.conf 내용

input {  
generator { message => "Hello world!" count => 1 }  
}  
filter {  
java\_filter\_example {}  
}  
output {  
stdout { codec => rubydebug }  
}

# 실행

bin/logstash -f java\_filter.conf

# 결과 - filter의 예제는 메시지를 reverse하는 것이다. java\_filter.conf의 message가 아래와 같이 reverse되어 출력된다.

{  
"host" => "naruui-Mac-Pro.local",  
"@version" => "1",  
"sequence" => 0,  
"message" => "!dlrow olleH",  
"@timestamp" => 2021-10-01T06:12:02.621Z  
}