• 对于注定会优秀的人来说,他所需要的,只是时间----博主
  • 手懒得,必受贫穷,手勤的,必得富足----《圣经》
  • 帮助别人,成就自己。愿君在本站能真正有所收获!
  • 如果你在本站中发现任何问题,欢迎留言指正!
  • 本站开启了防爆破关小黑屋机制,如果您是正常登录但被关进小黑屋,请联系站长解除!

<五>ELK-6.5.3学习笔记–elasticsearch高可用的探索

ELK eryajf 2周前 (01-05) 92℃ 已收录 1个评论
本文预计阅读时间 31 分钟

es部署集群,以保障服务高可用。

主机规划如下:

主机IP 组件
192.168.111.3 elasticsearch-node1
192.168.111.4 elasticsearch-node2
192.168.111.5 elasticsearch-node3
192.168.111.6 elasticsearch-node0,kibana,logstash
192.168.111.16 nginx,filebeat

简述:

  • 其中node1,node2,node3作为数据节点,而node0则作为集群当中与kibana连接的节点。
  • nginx主机通过filebeat将日志转给logstash,logstash发给es集群,然后再由kibana展示。

1,安装配置。

1, 安装依赖。

yum -y install lrzsz vim curl wget java

2, 配置yum源。

cat > /etc/yum.repos.d/elk.repo << EOF
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

3, 安装。

在四台主机安装elasticsearch。

yum -y install elasticsearch

接下来进行配置:

elk-node1

[root@localhost ~]$egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: ebei-elk
node.name: elk-node1
path.data: /logs/elasticsearch6
path.logs: /logs/elasticsearch6/log
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.111.3:9300","192.168.111.4:9300","192.168.111.5:9300","192.168.111.6:9300"]
discovery.zen.minimum_master_nodes: 2
xpack.security.enabled: false

说明:

  • cluster.name:自定义集群名称
  • node.name:自定义节点名,不同主机名称不能一致。
  • path.data:data存储路径,这里更改成自定义以应对日志的big。
  • path.logs:log存储路径,是为es自己的日志。
  • 注意创建上边两项定义的两个文件目录。否则会启动失败。
mkdir -p /logs/elasticsearch6/log
cd /logs
chown -R elasticsearch.elasticsearch elasticsearch6/
  • network.host:es监听地址,采用"0.0.0.0",表示允许所有设备访问。
  • http.port:es监听端口,可不取消注释,默认即此端口。
  • discovery.zen.ping.unicast.hosts:集群节点发现列表
  • discovery.zen.minimum_master_nodes:定义可成为master的节点数量。为避免脑裂的情况,建议设置成2。参考
  • xpack.security.enabled:添加这条,这条是配置kibana的安全机制,暂时关闭。

4, 分发配置。

接着将配置分发给其他几台主机。

[root@localhost ~]$scp /etc/elasticsearch/elasticsearch.yml 192.168.111.4:/etc/elasticsearch/elasticsearch.yml
[root@localhost ~]$scp /etc/elasticsearch/elasticsearch.yml 192.168.111.5:/etc/elasticsearch/elasticsearch.yml
[root@localhost ~]$scp /etc/elasticsearch/elasticsearch.yml 192.168.111.6:/etc/elasticsearch/elasticsearch.yml

然后更改对应三台的配置,只需更改node-name即可。

列出另外三台的配置:

elk-node2

[root@localhost ~]$egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: ebei-elk
node.name: elk-node2
path.data: /logs/elasticsearch6
path.logs: /logs/elasticsearch6/log
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.111.3:9300","192.168.111.4:9300","192.168.111.5:9300","192.168.111.6:9300"]
discovery.zen.minimum_master_nodes: 2
xpack.security.enabled: false

elk-node3

[root@localhost ~]$egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: ebei-elk
node.name: elk-node3
path.data: /logs/elasticsearch6
path.logs: /logs/elasticsearch6/log
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.111.3:9300","192.168.111.4:9300","192.168.111.5:9300","192.168.111.6:9300"]
discovery.zen.minimum_master_nodes: 2
xpack.security.enabled: false

elk-node0

[root@localhost ~]$egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: ebei-elk
node.name: elk-node0
path.data: /logs/elasticsearch6
path.logs: /logs/elasticsearch6/log
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.111.3:9300","192.168.111.4:9300","192.168.111.5:9300","192.168.111.6:9300"]
discovery.zen.minimum_master_nodes: 2
xpack.security.enabled: false

node.master: false
node.data: false
node.ingest: false

注意最后多了三条配置。

一开始我也并没有想到这一点,而且也并没有node0这一节点的,但当我开始配置kibana的配置时发现,kibana只能写一条es的连接信息,于是乎,才发现有这样一个问题,后来了解到,官方给到了一种解决方式

方式就是添加如上三条配置,然后将这一节点变成与kibana对接的桥梁,但是并不参与数据方面的处理。

最后就是启动几台主机上的es服务。

systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
systemctl status elasticsearch.service

2,查看集群信息。

  • 查看es是否正常启动。
[root@localhost ~]$curl -X GET "127.0.0.1:9200/"
{
  "name" : "elk-node1",
  "cluster_name" : "ebei-elk",
  "cluster_uuid" : "53LLexx8RSW16nE4lsJMQQ",
  "version" : {
    "number" : "6.5.3",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "159a78a",
    "build_date" : "2018-12-06T20:11:28.826501Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
  • 另一种方式查看。
[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty'
{
  "cluster_name" : "ebei-elk",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 11,
  "active_shards" : 22,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
  • 查看集群状态。
[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/nodes'
192.168.111.3 48 39 0 0.22 0.07 0.06 mdi * elk-node1  #带*号的表示master
192.168.111.6 41 98 0 0.03 0.06 0.05 -   - elk-node0  #注意这个节点并不参与数据处理。
192.168.111.4 44 57 0 0.00 0.01 0.05 mdi - elk-node2
192.168.111.5 27 62 0 0.00 0.01 0.05 mdi - elk-node3
  • 另一种方式查看。
[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/nodes?v'
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.111.3           34          39   0    0.13    0.10     0.07 mdi       *      elk-node1
192.168.111.6           20          98   0    0.01    0.05     0.05 -         -      elk-node0
192.168.111.4           32          57   0    0.00    0.01     0.05 mdi       -      elk-node2
192.168.111.5           38          62   0    0.00    0.01     0.05 mdi       -      elk-node3
  • 查看集群详细信息。
[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cluster/state/nodes?pretty'
{
  "cluster_name" : "ebei-elk",  #集群名称
  "compressed_size_in_bytes" : 14291,
  "cluster_uuid" : "53LLexx8RSW16nE4lsJMQQ",  #集群id
  "nodes" : {
    "oCNARqdUT5KXLZTlcS22GA" : {  #node的ID值
      "name" : "elk-node1",   #node名称
      "ephemeral_id" : "y-NCFJULTEmWdWjNjCPS2A",
      "transport_address" : "192.168.111.3:9300", #集群通讯地址
      "attributes" : {
        "ml.machine_memory" : "8202727424",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20",
        "ml.enabled" : "true"
      }
    },
    "8F_rZuR1TByEb6bXz0EgzA" : {
      "name" : "elk-node0",
      "ephemeral_id" : "b3CtPKpyRUahT4njpRqjlQ",
      "transport_address" : "192.168.111.6:9300",
      "attributes" : {
        "ml.machine_memory" : "8202039296",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "ml.enabled" : "true"
      }
    },
    "ptEOHzaPTgmlqW3NRhd7SQ" : {
      "name" : "elk-node2",
      "ephemeral_id" : "YgypZZNcTfWcIYDhOlUAzw",
      "transport_address" : "192.168.111.4:9300",
      "attributes" : {
        "ml.machine_memory" : "8202039296",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "ml.enabled" : "true"
      }
    },
    "PJYNTw5nTeS2kjjft4UcrA" : {
      "name" : "elk-node3",
      "ephemeral_id" : "C2nnREt0TOGKeDw9OQRJUA",
      "transport_address" : "192.168.111.5:9300",
      "attributes" : {
        "ml.machine_memory" : "8202031104",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "ml.enabled" : "true"
      }
    }
  }
}
  • 查询master。
[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cluster/state/master_node?pretty'
{
  "cluster_name" : "ebei-elk",
  "compressed_size_in_bytes" : 14291,
  "cluster_uuid" : "53LLexx8RSW16nE4lsJMQQ",
  "master_node" : "oCNARqdUT5KXLZTlcS22GA"
}
[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/master?v'
id                     host          ip            node
oCNARqdUT5KXLZTlcS22GA 192.168.111.3 192.168.111.3 elk-node1
  • 查询集群健康状态。
[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/health?v'
epoch      timestamp cluster  status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1545386516 10:01:56  ebei-elk green           4         3     22  11    0    0        0             0                  -                100.0%

3,高可用验证。

通过刚刚的测试,已经看出,master在node1,现在将node1的es停掉,看看是否会自动漂移。

[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/nodes?v'
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.111.3           36          39   0    0.03    0.04     0.05 mdi       *      elk-node1
192.168.111.6           22          98   0    0.00    0.02     0.05 -         -      elk-node0
192.168.111.4           38          57   0    0.00    0.01     0.05 mdi       -      elk-node2
192.168.111.5           44          62   0    0.00    0.01     0.05 mdi       -      elk-node3
[root@localhost ~]$systemctl stop elasticsearch

然后到另外一个节点查看一下:

[root@localhost ~]$curl -XGET 'http://127.0.0.1:9200/_cat/nodes?v'
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.111.4           45          58   0    0.00    0.01     0.05 mdi       -      elk-node2
192.168.111.5           30          62   0    0.00    0.01     0.05 mdi       *      elk-node3
192.168.111.6           24          98   0    0.00    0.02     0.05 -         -      elk-node0

可以看到master已经跑到了node3。

4,对接配置。

filebeat往logstash转发日志的配置还是原来的样子,只不过logstash的配置文件要更改一下output了,如下:

input {
    beats {
        host => 192.168.111.16
        port => 5044
    }
}
filter {
    if [fields][logtype] == "nginx-access" {
        json {
        source => "message"
        target => "data"
        }
    }
    if [fields][logtype] == "nginx-error" {
        json {
        source => "message"
        target => "data"
        }
    }
}
output {
    if [fields][logtype] == "nginx-access" {
        elasticsearch {
        hosts => ["192.168.111.3:9200","192.168.111.4:9200","192.168.111.5:9200"]
        index => "logstash-nginx-access-%{+YYYY.MM.dd}"
        }
    }
    if [fields][logtype] == "nginx-error" {
        elasticsearch {
        hosts => ["192.168.111.3:9200","192.168.111.4:9200","192.168.111.5:9200"]
        index => "logstash-nginx-error-%{+YYYY.MM.dd}"
        }
    }
}

再看kibana的配置。

[root@localhost logs]$egrep -v "^$|^#" /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://127.0.0.1:9200"
kibana.index: ".kibana"
xpack.security.enabled: false

然后访问即可,可以看到日志能够源源不断的打进来,如果es当中某个节点挂了,并不会影响kibana当中日志的展示,但是,此时可以看到与kibana对接的es还是单个,所以这种还不能算是绝对高可用。

5,高可用。

所以正确的做法应该是,撤掉刚刚的node0,然后将集群当中的三台es通过前端nginx做一个代理,然后让kibana连接nginx配置的地址即可实现高可用。

现在就在192.168.111.16的nginx添加配置:

upstream elasticsearch {
    zone elasticsearch 64K;
    server 192.168.111.3:9200;
    server 192.168.111.4:9200;
    server 192.168.111.5:9200;
}
server {
    listen 9200;
    server_name 192.168.111.16;
    location / {
        proxy_pass     http://elasticsearch;
        proxy_redirect    off;
        proxy_set_header  Host             $host;
        proxy_set_header  X-Real-IP        $remote_addr;
        proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
    }
    access_log logs/es_access.log;
}

然后加载配置。

nginx -t
nginx -s reload

接着更改kibana当中的连接地址:

[root@localhost logs]$!egr
egrep -v "^$|^#" /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.111.16:9200"
kibana.index: ".kibana"
xpack.security.enabled: false

这样以来,数据还都正常流通,即便nginx后端的某个es挂掉,还不会影响整个链路的正常。当然目前nginx是单节点,其实问题不大,如果量实在非常大,可以在nginx前端再添加高可用组件即可。


weinxin
扫码订阅,第一时间获得更新
微信扫码二维码,订阅我们网站的动态,另外不定时发送WordPress小技巧,你可以随时退订,欢迎订阅哦~

二丫讲梵 , 版权所有丨如未注明 , 均为原创丨本网站采用BY-NC-SA协议进行授权 , 转载请注明<五>ELK-6.5.3学习笔记–elasticsearch高可用的探索
喜欢 (0)
[如果想支持本站,可支付宝赞助]
分享 (0)
eryajf
关于作者:
学无止境,我愿意无止境学。书山有路,我愿意举身投火,淬炼成金!

您必须 登录 才能发表评论!

(1)个小伙伴在吐槽
  1. eryajf
    各位有更好的经验的,可以在评论区一起讨论分享。
    eryajf2019-01-06 10:24 Windows 7 | Chrome 70.0.3538.9