Rsyslog + Elasticsearch + Kibana

類似 Splunk 的功能 OpenSource

rsyslog 的進階使用結合  Elasticsearch  與  Kibana

Rsyslog + Elasticsearch + Kibana

角色說明:
(0) rsyslog [Remote linux server ]
(1) rsyslog collector [ Rsyslog ]
(2) search server [Elasticsearch ]
(3) web ui [ Kibana ]


(1)升級並安裝 rsyslog 8.x等必要套件

[root@Rsyslog yum.repos.d]# pwd
/etc/yum.repos.d
[root@Rsyslog yum.repos.d]# cat rsyslog.repo
[rsyslog_v8]
name=Adiscon CentOS-$releasever - local packages for $basearch
baseurl=http://rpms.adiscon.com/v8-stable/epel-$releasever/$basearch
enabled=1
gpgcheck=0
gpgkey=http://rpms.adiscon.com/RPM-GPG-KEY-Adiscon
protect=1
[root@Rsyslog yum.repos.d]#

yum update rsyslog
yum install rsyslog-debuginfo rsyslog-libdbi rsyslog-mysql rsyslog-relp rsyslog-snmp rsyslog-elasticsearch rsyslog-mmanon rsyslog-mmfields rsyslog-mmjsonparse rsyslog-mmnormalize rsyslog-mmutf8fix rsyslog-ommail

[root@Rsyslog rsyslog.d]# rpm -qa | grep rsyslog
rsyslog-8.2.2-1.el6.x86_64
rsyslog-mmfields-8.2.2-1.el6.x86_64
rsyslog-relp-8.2.2-1.el6.x86_64
rsyslog-snmp-8.2.2-1.el6.x86_64
rsyslog-ommail-8.2.2-1.el6.x86_64
rsyslog-mmjsonparse-8.2.2-1.el6.x86_64
rsyslog-mysql-8.2.2-1.el6.x86_64
rsyslog-debuginfo-8.2.2-1.el6.x86_64
rsyslog-mmutf8fix-8.2.2-1.el6.x86_64
rsyslog-mmnormalize-8.2.2-1.el6.x86_64
rsyslog-mmanon-8.2.2-1.el6.x86_64
rsyslog-elasticsearch-8.2.2-1.el6.x86_64
rsyslog-libdbi-8.2.2-1.el6.x86_64
[root@Rsyslog rsyslog.d]#

(2)安裝 java & Elasticsearch

http://www.elasticsearch.org/overview/elkdownloads/

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.noarch.rpm

rpm 安裝即可

(3)安裝 Kibana ( search web api )
wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz
放在 web -root 即可

其它重要設定

vi /etc/rsyslog.conf

新增 rsyslog  listen port

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514


新增與 Elasticsearch 介接設定 syslog.conf

[root@Rsyslog rsyslog.d]# pwd
/etc/rsyslog.d
[root@Rsyslog rsyslog.d]# cat syslog.conf
module(load="omelasticsearch") # for outputting to Elasticsearch

# this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index" type="list") {
 constant(value="logstash-")
 property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
 constant(value=".")
 property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
 constant(value=".")
 property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

# this is for formatting our syslog in JSON with @timestamp
 template(name="plain-syslog" type="list") {
 constant(value="{")
 constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
 constant(value="\",\"@host\":\"")       property(name="hostname")
 constant(value="\",\"@severity\":\"")   property(name="syslogseverity-text")
 constant(value="\",\"@facility\":\"")   property(name="syslogfacility-text")
 constant(value="\",\"@syslogtag\":\"")  property(name="syslogtag" format="json")
 constant(value="\",\"@message\":\"")    property(name="msg" format="json")
 constant(value="\"}")
 }
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch" template="plain-syslog" searchIndex="logstash-index" dynSearchIndex="on")

參考 url http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/

demo