ELK 安裝 (CentOS 7)

Reference:

[ELK教學]最新CentOS ElasticSearch、Logstash、Kibana、Filebeat ，快速安裝(照著貼上就對了)

Day 11 ELK 收集系統

【Day01】ELK環境建置與介紹

步驟：

安裝CentOS 7

1. software selection: server with GUI
2. security policy: off
3. set fix IP, auto connect
4. disable firewall
• systemctl stop firewalld
• systemctl disable firewalld
• vi /etc/selinux/config; SELINUX=disabled
• setenforce 0

安裝Java

• yum install java

安裝 ElasticSearch

• vi /etc/yum.repos.d/elasticsearch.repo

[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

• yum install --enablerepo=elasticsearch elasticsearch

如果有啟用selinux，需做以下設定。

vi /etc/sysconfig/elasticsearch

# Additional Java OPTS
ES_JAVA_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp"
ES_TMPDIR=/var/lib/elasticsearch/tmp

mkdir /var/lib/elasticsearch/tmp
chown elasticsearch:elasticsearch /var/lib/elasticsearch/tmp

• systemctl start elasticsearch
• systemctl enable elasticsearch
• 設定 Elasticsearch 記憶體使用上限及下限
• vi /etc/elasticsearch/jvm.options
• -Xms1g # Xms 記憶體使用下限
• -Xmx1g # Xmx 記憶體使用上限
• (2022/12/8 update: 設定記憶體限制反而造成使用問題，最後不啟用)
  
• 測試
• curl "http://127.0.0.1:9200/_cat/nodes"
• 127.0.0.1 42 97 3 0.03 0.12 0.09 cdhilmrstw * ELK01 <=有回應代表正常

安裝 Kibana

• vi /etc/yum.repos.d/kibana.repo

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

• yum install kibana
• systemctl start kibana
• systemctl enable kibana

安裝 NGINX

• vi /etc/yum.repos.d/nginx.repo

[nginx]
name=nginx repo
baseurl=https://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

• yum -y install nginx httpd-tools
• htpasswd -c /etc/nginx/htpasswd.users root
• cd /etc/nginx/conf.d
• mv default.conf default.conf.bk
• vi nginx.conf

server {
    listen 80;
    server_name _;
    #auth_basic "Restricted Access";
    #auth_basic_user_file /etc/nginx/htpasswd.users;
    location / {
        proxy_pass http://localhost:5601; 
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}

• systemctl start nginx
• systemctl enable nginx

如果有啟用selinux，需要執行 setsebool httpd_can_network_connect 1 -P

測試

• http://elk01.domain.com

安裝 LogStash

• vi /etc/yum.repos.d/logstash.repo

[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

• yum install logstash
• 配置 Logstash
• vi /etc/logstash/conf.d/logstash.conf

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

• 如果 fstab 的 /tmp 有設定 noexec，必須另外指定 tmpdir 
• vi /etc/logstash/jvm.options
• -Djava.io.tmpdir=/var/lib/logstash/tmp
• mkdir /var/lib/logstash/tmp
• chown logstash:logstash /var/lib/logstash/tmp
• systemctl restart logstash
• systemctl enable logstash
• 驗證

• systemctl status logstash
• netstat -ano|grep 5044    (要有listen 5044 port)
• 如果沒有 listen 5044，需檢查 /var/log/message 的錯誤訊息。可能是 /etc/logstash 的檔案權限，須設定給 logstash
  

安裝 Winlogbeat for Windows

• download & install Winlogbeat (WINDOWS MSI 32/64-BIT)
• Modify C:\ProgramData\Elastic\Beats\winlogbeat\winlogbeat.yml
• output.elasticsearch:
  • hosts: ["localhost:9200"]
• #output.logstash:
• #hosts: ["elk01.domain.com:5044"]
• #index: "Win01"
• C:\Program Files\Elastic\Beats\7.11.0\winlogbeat\winlogbeat.exe setup  (建立index template)
• Modify C:\ProgramData\Elastic\Beats\winlogbeat\winlogbeat.yml
• #output.elasticsearch:
  • #hosts: ["localhost:9200"]
• output.logstash:
• hosts: ["elk01.domain.com:5044"]
• index: "Win01"
• Start-Service winlogbeat

安裝 Filebeat for Linux syslog by yum

• cd /etc/yum.repos.d
• vi elastic.repo

[elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

• rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
• yum install filebeat
• systemctl enable filebeat
• vi /etc/filebeat/filebeat.yml

filebeat.inputs:

- type: log

  enabled: true

  paths:

    - /var/log/hosts/*.log

output.elasticsearch:

  hosts: ["localhost:9200"]

#output.logstash:

  #hosts: ["elk01.domain.com:5044"]

  #index: "linux01"

• filebeat setup   (建立index template)

filebeat.inputs:

- type: log

  enabled: true

  paths:

    - /var/log/hosts/*.log

#output.elasticsearch:

  #hosts: ["localhost:9200"]

output.logstash:

  hosts: ["elk01.domain.com:5044"]

  index: "linux01"

• systemctl start filebeat

到這邊就完成可以收工了。

如果要使用加密連線，必須安裝憑證。

如果需要帳號密碼驗證登入網頁，需啟用帳號。

Easy way to configure Filebeat-Logstash SSL/TLS Connection

SSL client fails to connect to Logstash

Using elasticsearch-certutil in Silent Modeedit

在ELK server建立憑證

mkdir $HOME/cert

vi $HOME/cert/instances.yml

instances:

  - name: 'elk01'

    dns: [ 'elk01.domain.com' ]

    ip: ['127.0.0.1', '<private_IP or public_IP>']

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --keep-ca-key --pem --in $HOME/cert/instances.yml --out $HOME/cert/elk-cert.zip --days 36500

unzip -d $HOME/cert $HOME/cert/elk-cert.zip

openssl pkcs8 -in $HOME/cert/elk01/elk01.key -topk8 -nocrypt -out $HOME/cert/elk01/elk01.pkcs8.key

mkdir /etc/cert

cp $HOME/cert/elk01/* /etc/cert/

cp $HOME/cert/ca/* /etc/cert/

ln -s /etc/cert /etc/elasticsearch/cert

修改 hosts

vi /etc/hosts

增加 <your IP> elk01.domain.com

ex: 192.168.11.1 elk01.domain.com

修改 elasticsearch 設定

vi /etc/elasticsearch/elasticsearch.yml

node.name: elk01
network.host: elk01.domain.com
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: cert/elk01.key
xpack.security.http.ssl.certificate: cert/elk01.crt
xpack.security.http.ssl.certificate_authorities: cert/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: cert/elk01.key
xpack.security.transport.ssl.certificate: cert/elk01.crt
xpack.security.transport.ssl.certificate_authorities: cert/ca.crt

discovery.type: single-node

systemctl restart elasticsearch

在ELK server建立帳號

/usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

請記下密碼，之後的設定檔需要用到。Ex: your_password

 

修改 logstash 設定

vi /etc/logstash/conf.d/logstash.conf

input {
  beats {
    port => 5044
    ssl => true
    ssl_key => '/etc/cert/elk01.pkcs8.key'
    ssl_certificate => '/etc/cert/elk01.crt'
  }
}

output {
  elasticsearch {
    hosts => ["https://elk01.domain.com:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    cacert => '/etc/cert/ca.crt'
    user => "elastic"
    password => "your_password"
  }
}

systemctl restart logstash

驗證 logstash.conf

/usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

若設定正確會回覆 Configuration OK

修改 kibana 設定

vi /etc/kibana/kibana.yml

server.name: "elk01"
server.host: "elk01.domain.com"
server.ssl.enabled: true
server.ssl.certificate: /etc/cert/elk01.crt
server.ssl.key: /etc/cert/elk01.key
elasticsearch.hosts: ["https://elk01.domain.com:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "your_password"
elasticsearch.ssl.certificateAuthorities: [ "/etc/cert/ca.crt" ]

修改 nginx 設定

vi /etc/nginx/conf.d/nginx.conf

server {
    listen 443 ssl;
    server_name _;
    ssl_certificate /etc/cert/elk01.crt;
    ssl_certificate_key /etc/cert/elk01.key;
    location / {
        proxy_pass https://elk01.domain.com:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

設定 filebeat 使用加密連線

登入filebeat linux

scp username@elk01.domain.com:$HOME/ca/ca.crt /etc/cert

vi /etc/filebeat/filebeat.yml

output.logstash:

  hosts: ["elk01.domain.com:5044"]

  ssl.certificate_authorities: ["/etc/cert/ca.crt"]

 

systemctl restart filebeat

驗證 filebeat.conf

filebeat -e

設定 winlogbeat 使用加密連線

登入winlogbeat Windows

將 elk01.domain.com的 $HOME/ca/ca.crt 複製到 c:\tmp

double click ca.crt，將憑證加入 本機電腦 的 受信任的根憑證

編輯 C:\ProgramData\Elastic\Beats\winlogbeat\winlogbeat.yml

output.logstash:

  # The Logstash hosts

  hosts: ["elk01.domain.com:5044"]

  # 如果沒有設定DNS可以改用 hosts: ["<private_IP>:5044"]

  ssl.certificate_authorities: ["c:\\ca\\ca.crt"]

 

restart winlogbeat service

驗證 winlogbeat.conf

開啟命令提示字元

cd C:\ProgramData\Elastic\Beats\winlogbeat

"C:\Program Files\Elastic\Beats\7.11.0\winlogbeat\winlogbeat.exe" -e

2021/12/15 update:

新增 metricbeat，在 Metrics / Metrics Explorer 出現 500 Internal Server Error。

Reference: Kibana Metrics – 500 Internal Server Error

因為使用 logstash 無法產生 Index Templates，解法是修改 metricbeat.yml。

先用 output.elasticsearch 產生 index templates，再改回 output.logstash。

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elk01.domain.com:9200"]
  ssl.certificate_authorities: ["/etc/cert/ca.crt"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "your_passwd"