⼩⽩都会超详细--ELK⽇志管理平台搭建教程
⼀、介绍
⼆、安装JDK
三、安装Elasticsearch
四、安装Logstash
五、安装Kibana
六、Kibana简单使⽤
系统环境：CentOS Linux release 7.4.1708 (Core)
当前问题状况
1. 开发⼈员不能登录线上服务器查看详细⽇志。
2. 各个系统都有⽇志，⽇志数据分散难以查。
3. ⽇志数据量⼤，查询速度慢，或者数据不够实时。
⼀、介绍
1、组成
ELK由Elasticsearch、Logstash和Kibana三部分组件组成；
Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，⾃动发现，索引⾃动分⽚，索引副本机制，restful风格接⼝，多数据源，⾃动搜索负载等。
Logstash是⼀个完全开源的⼯具，它可以对你的⽇志进⾏收集、分析，并将其存储供以后使⽤
kibana 是⼀个开源和免费的⼯具，它可以为 Logstash 和 ElasticSearch 提供的⽇志分析友好的 Web 界⾯，可以帮助您汇总、分析和搜索重要数据⽇志。
2、四⼤组件
Logstash: logstash server端⽤来搜集⽇志；
Elasticsearch: 存储各类⽇志；
Kibana: web化接⼝⽤作查寻和可视化⽇志；
Logstash Forwarder: logstash client端⽤来通过lumberjack ⽹络协议发送⽇志到logstash server；
3、⼯作流程
在需要收集⽇志的所有服务上部署logstash，作为logstash agent（logstash shipper）⽤于监控并过滤收集⽇志，将过滤后的内容发送到Redis，然后logstash indexer将⽇志收集在⼀起交给全⽂搜索服务ElasticSearch，可以⽤ElasticSearch进⾏⾃定义搜索通过Kibana 来结合⾃定义搜索进⾏页⾯展⽰。
下⾯是在两台节点上都安装⼀下环境。⼆、安装JDK
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15配置阿⾥源：wget -O /pos.po mirrors.aliyun/po yum clean all
yum makecache
Logstash的运⾏依赖于Java运⾏环境，Elasticsearch 要求⾄少 Java 7。
[root@controller ~]# yum install java-1.8.0-openjdk -y
[root@controller ~]# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
1、关闭防⽕墙
systemctl stop firewalld.service #停⽌firewall
systemctl disable firewalld.service #禁⽌firewall开机启动
2、关闭selinux
exited
sed-i 's/SELINUX=enforcing/SELINUX=disabled/g'/etc/selinux/config
setenforce 0
三、安装Elasticsearch
基础环境安装（elk-node1和elk-node2同时操作）
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 321）下载并安装GPG Key
[root@elk-node1 ~]# rpm --import /GPG-KEY-elasticsearch
2）添加yum仓库
[root@elk-node1 ~]# vim /pos.po
[elasticsearch-2.x]
name=Elasticsearch repository for2.x packages
baseurl=/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=/GPG-KEY-elasticsearch
enabled=1
3）安装elasticsearch
[root@elk-node1 ~]# yum install www.taohuayuan178-y elasticsearch
4）添加⾃启动
chkconfig --add elasticsearch
5）启动命令
systemctl daemon-reload
systemctl enable elasticsearch.service
6）修改配置
[root@elk-node1 ~]# cd /etc/elasticsearch/
[root@elk-node1 elasticsearch]# ls
[root@elk-node1 elasticsearch]# l{,.bak}
[root@elk-node1 elasticsearch]# mkdir -p /data/es-data
[root@elk-node1 elasticsearch]# l
[root@elk-node1 elasticsearch]# grep www.leyou2'^[a-z]' l cluster.name: hejianlai //集名称
node.name: elk-node1 //节点名称
path.data: /data/es-data//数据存放⽬录
path.logs: /var/log/elasticsearch///⽇志存放⽬录
<_lock: true//打开内存
33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100network.host: 0.0.0.0 //监听⽹络
http.port: 9200 //端⼝
[root@elk-node1 elasticsearch]# systemctl start elasticsearch
You have new mail in/var/spool/mail/root
[root@elk-node1 elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since www.dongfan178/ Thu 2018-07-12 22:00:47 CST; 9s ago
Docs:
Process: 22333 ExecStart=/usr/share/elasticsearch/bin/elasticsearch-Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -
Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.f=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 22331 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec(code=exited, status=0/SUCCESS)
Main PID: 22333 (code=exited, status=1/FAILURE)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]: at sun.nio.hrowAsIOException(UnixException.java:102)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]:www.mhylpt at sun.nio.hrowAsIOException(UnixException.java:107)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]: at sun.nio.ateDirectory(UnixFileSystemProvider.java:384)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]:www.feifanyule/ at java.nio.ateDirectory(Files.java:674)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]:www.078881 at java.nio.ateAndCheckIsDirectory(Files.java:781)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]: at java.nio.ateDirectories(Files.java:767)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]: at org.elasticsearch.sureDirectoryExists(Security.java:337)
Jul 12 22:00:47 elk-node1 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 12 22:00:47 elk-node1 systemd[1]: Unit elasticsearch.service entered failed state.
Jul 12 22:00:47 elk-node1 systemd[1]: elasticsearch.service failed.
[root@elk-node1 elasticsearch]# cd /var/log/elasticsearch/
[root@elk-node1 elasticsearch]# ll
total 4
-rw-r--r-- 1 elasticsearch elasticsearch 0 Jul 12 22:00 hejianlai_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 Jul 12 22:00 hejianlai_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 Jul 12 22:00 hejianlai_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 2232 Jul 12 22:00 hejianlai.log
[root@elk-node1 elasticsearch]# tail hejianlai.log
at sun.nio.anslateToIOException(UnixException.java:84)
at sun.nio.hrowAsIOException(UnixException.java:102)
at sun.nio.hrowAsIOException(UnixException.java:107)
at sun.nio.ateDirectory(UnixFileSystemProvider.java:384)
at java.nio.ateDirectory(Files.java:674)
at java.nio.ateAndCheckIsDirectory(Files.java:781)
at java.nio.ateDirectories(Files.java:767)
at org.elasticsearch.sureDirectoryExists(Security.java:337)
at org.elasticsearch.bootstrap.Security.addPath(Security.java:314)
... 7 more
[root@elk-node1 elasticsearch]# less hejianlai.log
You have new mail in/var/spool/mail/root
[root@elk-node1 elasticsearch]# grep elas /etc/passwd
elasticsearch:x:991:988:elasticsearch user:/home/elasticsearch:/sbin/nologin
#报错/data/es-data没权限，赋权限即可
[root@elk-node1 elasticsearch]# chown -R elasticsearch:elasticsearch /data/es-data/
[root@elk-node1 elasticsearch]# systemctl start elasticsearch
[root@elk-node1 elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2018-07-12 22:03:28 CST; 4s ago
Docs:
Process: 22398 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec(code=exited, status=0/SUCCESS)
Main PID: 22400 (java)
CGroup: /system.slice/elasticsearch.service
└─22400 /bin/java-Xms256m -Xmx1g -Djava.awt.headless=true-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+
Jul 12 22:03:29 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:29,739][WARN ][bootstrap ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
Jul 12 22:03:29 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:29,899][INFO ][node ] [elk-node1] version[2.4.6], pid[22400], build[5376dca/2017-07-18T12 :17:44Z]
Jul 12 22:03:29 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:29,899][INFO ][node ] [elk-node1] initializing ...
Jul 12 22:03:30 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:30,644][INFO ][plugins ] [elk-node1] modules [reindex, lang-expression, lang-groovy], plugins
101 102 103[], sites []
Jul 12 22:03:30 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:30,845][INFO ][env] [elk-node1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [1.7gb], n...types [rootfs]
Jul 12 22:03:30 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:30,845][INFO ][env] [elk-node1] heap size [1007.3mb], compressed ordinary object pointers [true]
Jul 12 22:03:33 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:33,149][INFO ][node ] [elk-node1] initialized
Jul 12 22:03:33 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:33,149][INFO ][node ] [elk-node1] starting ...
Jul 12 22:03:33 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:33,333][INFO ][transport ] [elk-node1] publish_address {192.168.247.135:9300}, bound_addresses {[::]:9300}
Jul 12 22:03:33 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:33,345][INFO ][discovery ] [elk-node1] hejianlai/iUUTEKhyTxyL78aGtrrBOw
Hint: Some lines were ellipsized, use -l to show in full.
安装ES插件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27#统计索引数
[root@elk-node1 ~]# curl -i -XGET '192.168.247.135:9200/_count?pretty' -d '
> "query": {
> "match_all":{}
> }'
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 95
{
"count": 0,
"_shards": {
"total": 0,
"successful": 0,
"failed": 0
}
}
#es插件，收费的不建议使⽤(这个不安装)
[root@elk-node1 bin]# /usr/share/elasticsearch/bin/plugin install marvel-agent
#安装开源的elasticsearch-head插件
[root@elk-node1 bin]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
-> Installing
Trying github/mobz/elasticsearch-head/archive/master.zip ...
Downloading ...........................................................................................................................................
DONE Verifying github/mobz/elasticsearch-head/archive/master.zip checksums if available ... NOTE: Unable to verify checksum for downloaded plugin (unable to find.sha1 or .md5 file to verify)
使⽤POST⽅法创建查询
使⽤GET⽅法查询数据
版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。
发表评论