Logstash + Elasticsearch + Kibana

Logstash  + Elasticsearch + Kibana
類似 Splunk 的功能 OpenSource

Logstash 的使用結合  Elasticsearch  與  Kibana

Logstash + Elasticsearch + Kibana

角色說明:

(0) rsyslog [Remote linux server ]
(1) syslog collector [ Logstash ]
(2) search server [Elasticsearch ]
(3) web ui [ Kibana ]

官方實用說明
http://www.logstash.net/docs/1.4.2/tutorials/10-minute-walkthrough/
http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash


分別使用 rpm & yum 安裝必要的軟體

(Logstash)

官網上下載該 rpm 安裝即可
 (  http://www.elasticsearch.org/overview/elkdownloads/  )
[root@Test-Logstash syslog]# rpm -qa | grep logstash
logstash-contrib-1.4.2-1_efd53ef.noarch
logstash-1.4.2-1_2c0f5a1.noarch

(Java)

yum 安裝即可
[root@Test-Logstash syslog]# rpm -qa | grep java
tzdata-java-2014e-1.el6.noarch
java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el6_5.x86_64

(Elasticsearch)

官網上下載該 rpm 安裝即可
 ( http://www.elasticsearch.org/overview/elkdownloads/ )
[root@Test-Logstash src]# rpm -qa | grep elasticsearch
elasticsearch-1.3.2-1.noarch

(Kinaba)

kibana 官網上下載該檔案即可
 ( http://www.elasticsearch.org/overview/elkdownloads/ )
kibana-3.1.0.tar.gz 並解壓縮後放置於 /var/www/html 內即可,目錄名稱可自行命名之

(Remote Syslog Client)

修改 rsyslog.conf 並重啟該服務即可
最簡單的設定方式

在最下方加入
*.* @xxx.xxx.xxx
or
*.* @@xxx.xxx.xxx




設定 logstash 的 conf

[root@Test-Logstash conf.d]# pwd
/etc/logstash/conf.d
[root@Test-Logstash conf.d]# cat syslog.conf 
input {
  tcp {
    type => "syslog"
    port => 514
  }
  udp {
    type => "syslog"
    port => 514
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}


output {
  elasticsearch {
    embedded => true
  }
}

[root@Test-Logstash conf.d]#


將 logstash 服務改由 root 身份啟動

[root@Test-Logstash init.d]# pwd
/etc/init.d
[root@Test-Logstash init.d]# cat logstash
#!/bin/sh
# Init script for logstash
# Maintained by Elasticsearch
# Generated by pleaserun.
# Implemented based on LSB Core 3.1:
#   * Sections: 20.2, 20.3
#
### BEGIN INIT INFO
# Provides:          logstash
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description:
# Description:        Starts Logstash as a daemon.
### END INIT INFO

PATH=/sbin:/usr/sbin:/bin:/usr/bin
export PATH

if [ `id -u` -ne 0 ]; then
   echo "You need root privileges to run this script"
   exit 1
fi

name=root
pidfile="/var/run/$name.pid"

LS_USER=root
LS_GROUP=root


設定 Kibana config  內的 elasticsearch 的 url

[root@Test-Logstash syslog]# pwd
/var/www/html/syslog
[root@Test-Logstash syslog]# cat config.js
/** @scratch /configuration/config.js/1
 *
 * == Configuration
 * config.js is where you will find the core Kibana configuration. This file contains parameter that
 * must be set before kibana is run for the first time.
 */
define(['settings'],
function (Settings) {


  /** @scratch /configuration/config.js/2
   *
   * === Parameters
   */
  return new Settings({

    /** @scratch /configuration/config.js/5
     *
     * ==== elasticsearch
     *
     * The URL to your elasticsearch server. You almost certainly don't
     * want +http://localhost:9200+ here. Even if Kibana and Elasticsearch are on
     * the same host. By default this will attempt to reach ES at the same host you have
     * kibana installed on. You probably want to set it to the FQDN of your
     * elasticsearch host
     *
     * Note: this can also be an object if you want to pass options to the http client. For example:
     *
     *  +elasticsearch: {server: "http://localhost:9200", withCredentials: true}+
     *
     */
    elasticsearch: "http://192.168.111.145:9200",


Demo

elasticsearch test 

Kibana