Redhat ppc64le 서버에서 logstash를 설치하고 구성하기

IBM POWER9 서버, 즉 ppc64le 아키텍처 기반의 Redhat 서버에 logstash를 설치하고 구성하는 것을 간단히 해보겠습니다. 

먼저, logstash는 java 기반이기 때문에, 그냥 x86_64용으로 elastic.co에서 제공되는 RPM 파일을 그냥 그대로 받아다 설치해서 쓰시면 됩니다.

[user607@p607-kvm1 ~]$ wget https://artifacts.elastic.co/downloads/logstash/logstash-7.4.2.rpm

[user607@p607-kvm1 ~]$ sudo rpm -Uvh logstash-7.4.2.rpm
warning: logstash-7.4.2.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:7.4.2-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: TieredCompilation is disabled in this release.
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash


다만 이걸 그대로 수행해보시면 아래와 같이 "load error: ffi/ffi" error가 나는 것을 보실 수 있습니다.

[root@p607-kvm1 user607]# /usr/share/logstash/bin/logstash
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2019-11-12 23:56:52.605 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2019-11-12 23:56:52.633 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[ERROR] 2019-11-12 23:56:53.198 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) load error: ffi/ffi -- java.lang.NullPointerException: null


이는 https://github.com/elastic/logstash/issues/10755 에 기술된 bug 때문인데, 요약하면 jruby-complete-*.jar 파일 안의 powerpc64le-linux platform을 위한 directory에 실수로 platform.conf가 빠져 있기 때문에 발생하는 것입니다.  이는 아래와 같은 command로 간단히 확인하실 수 있습니다.   일단 아래와 같이 error가 나는지 확인하십시요.

[root@p607-kvm1 user607]# java -jar /usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar -rffi -e 1
LoadError: load error: ffi/ffi -- java.lang.NullPointerException: null
  require at org/jruby/RubyKernel.java:987
  require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
   <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/ffi.rb:1
  require at org/jruby/RubyKernel.java:987
   (root) at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1

확인되면 아래와 같이 powerpc64le-linux/types.conf 파일을 추출해내십시요.

[root@p607-kvm1 user607]# jar -xvf /usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar META-INF/jruby.home/lib/ruby/stdlib/ffi/platform/powerpc64le-linux/types.conf
 inflated: META-INF/jruby.home/lib/ruby/stdlib/ffi/platform/powerpc64le-linux/types.conf

이 파일을 다음과 같이 powerpc64le-linux/platform.conf로 copy하십시요.

[root@p607-kvm1 user607]# cp META-INF/jruby.home/lib/ruby/stdlib/ffi/platform/powerpc64le-linux/types.conf META-INF/jruby.home/lib/ruby/stdlib/ffi/platform/powerpc64le-linux/platform.conf

이렇게 만들어진 powerpc64le-linux/platform.conf을 원래의 jruby-complete-*.jar 파일에 update하십시요.

[root@p607-kvm1 user607]# jar -uvf /usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar META-INF/jruby.home/lib/ruby/stdlib/ffi/platform/powerpc64le-linux/platform.conf
adding: META-INF/jruby.home/lib/ruby/stdlib/ffi/platform/powerpc64le-linux/platform.conf(in = 3796) (out= 483)(deflated 87%)

이제 다시 테스트해보면 "load error: ffi/ffi"가 더 발생하지 않는 것을 보실 수 있습니다.

[root@p607-kvm1 user607]# java -jar /usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar -rffi -e 1

[root@p607-kvm1 user607]# echo $?
0

이제 간단한 logstash config 파일을 작성합니다.  여기서는 /tmp/in.log에 뭔가 내용이 쌓이면 그걸 grok으로 filtering한 뒤 rubydebug codec으로 format하여 /tmp/out.log에 집어넣는 것입니다.

[root@p607-kvm1 user607]# vi logstash-simple.conf
input {
  file {
    path => "/tmp/in.log"
  }
}

filter {
  grok {
    match => { "message" => "%{NUMBER:timestamp} %{NUMBER:channel} %{MAC:client_mac} %{MAC:mac} %{INT:rssi}" }
 }
}

output {
  file {
    path => "/tmp/out.log"
}
  stdout { codec => rubydebug }
}

아래와 같이 logstash를 start 합니다.

[root@p607-kvm1 user607]# /usr/share/logstash/bin/logstash -f logstash-simple.conf
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-11-13 00:23:47.244 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-13 00:23:47.267 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.4.2"}
[INFO ] 2019-11-13 00:23:50.305 [Converge PipelineAction::Create<main>] Reflections - Reflections took 75 ms to scan 1 urls, producing 20 keys and 40 values
[WARN ] 2019-11-13 00:23:52.059 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2019-11-13 00:23:52.065 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, :thread=>"#<Thread:0x370844ca run>"}
[INFO ] 2019-11-13 00:23:52.938 [[main]-pipeline-manager] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_b3374bae0eefd65d5f3cdbcfd41ca004", :path=>["/tmp/in.log"]}
[INFO ] 2019-11-13 00:23:52.992 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2019-11-13 00:23:53.081 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2019-11-13 00:23:53.089 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-11-13 00:23:53.488 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}


여기서 다음과 같이 일정 format의 string을 /tmp/in.log에 집어넣습니다.

[root@p607-kvm1 user607]#  echo "1524708231.794252221  8  c0:4a:00:40:e6:0e  c0:4a:00:40:AA:AA   -68" >> /tmp/in.log


그러면 logstash의 standard output log에 아래와 같이 그에 대한 처리가 되는 메시지가 나오는 것을 보실 수 있습니다.


[INFO ] 2019-11-13 00:24:54.175 [[main]>worker9] file - Opening file {:path=>"/tmp/out.log"}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
    "@timestamp" => 2019-11-13T05:24:53.785Z,
          "path" => "/tmp/in.log",
      "@version" => "1",
          "host" => "p607-kvm1",
          "tags" => [
        [0] "_grokparsefailure"
    ],
       "message" => "1524708231.794252221  8  c0:4a:00:40:e6:0e  c0:4a:00:40:AA:AA   -68"
}
[INFO ] 2019-11-13 00:25:11.708 [[main]>worker9] file - Closing file /tmp/out.log


그리고 그 결과로 아래와 같이 format된 string이 지정된 file로 저장된 것을 보실 수 있습니다. 

[root@p607-kvm1 user607]# cat /tmp/out.log
{"@timestamp":"2019-11-13T05:24:53.785Z","path":"/tmp/in.log","@version":"1","host":"p607-kvm1","tags":["_grokparsefailure"],"message":"1524708231.794252221  8  c0:4a:00:40:e6:0e  c0:4a:00:40:AA:AA   -68"}