技术博客
首页 所有文章 标签 分类 关于
技术博客

目录

ELKFilebeatKafkaZookeeper安装部署

 收录于 It
   约 959 字   预计阅读 2 分钟 
https://bing.ee123.net/img/rand?artid=146287078

ELK+Filebeat+Kafka+Zookeeper安装部署

1.安装zookeeper

zookpeer下载地址: [apache-zookeeper-3.7.1-bin.tar.gz

https://csdnimg.cn/release/blog_editor_html/release2.3.8/ckeditor/plugins/CsdnLink/icons/icon-default.png?t=P1C7 “apache-zookeeper-3.7.1-bin.tar.gz”)

1.1解压安装zookeeper软件包（三个节点做）

cd /root

tar zxf apache-zookeeper-3.7.1-bin.tar.gz

mv apache-zookeeper-3.7.1-bin /usr/local/zookeeper-3.7.1

cd /usr/local/zookeeper-3.7.1/conf/

cp zoo_sample.cfg zoo.cfg

https://i-blog.csdnimg.cn/direct/e76b20bc3e884a028d6b6a281478c1e2.png

https://i-blog.csdnimg.cn/direct/0ead6071e1c7496490d7655848c09403.png

https://i-blog.csdnimg.cn/direct/5e4c4058fc024297acb4b06af708c4c2.png

1.2 修改配置文件（第一个节点）

vim zoo.cfg

https://i-blog.csdnimg.cn/direct/42e9ef8ffe7842699a60eafcb3af2e73.png

https://i-blog.csdnimg.cn/direct/a4d7622d60924a8299042569e63e7dd1.png

将zoo.cfg复制到其他两个节点

scp zoo.cfg 172.25.23.8:/usr/local/zookeeper-3.7.1/conf/

scp zoo.cfg 172.25.23.9:/usr/local/zookeeper-3.7.1/conf/

https://i-blog.csdnimg.cn/direct/3f5d52fb68c0461990a755bcd1973ea4.png

1.3 给每个机器指定对应的节点号

mkdir data logs

echo 1 > data/myid

https://i-blog.csdnimg.cn/direct/529c34f37de24cf287424181cc794bef.png

mkdir data logs

echo 2 > data/myid

https://i-blog.csdnimg.cn/direct/51f5897c0841425882d4ed78488fb161.png

mkdir data logs

echo 3 > data/myid
cd /usr/local/zookeeper-3.7.1/bin

./zkServer.sh start

1.4 启动zookeeper

cd /usr/local/zookeeper-3.7.1/bin

./zkServer.sh start

1.5 开启之后，查看三个节点zookeeper状态

./zkServer.sh status

https://i-blog.csdnimg.cn/direct/709aa540deeb484faf400fff261ec669.png

2、安装kafka

2.1 安装 kafka（3个节点都要操作）

cd /root

tar zxf kafka_2.13-2.7.1.tgz

mv kafka_2.13-2.7.1 /usr/local/kafka

https://i-blog.csdnimg.cn/direct/fe42d5582ad5400a9fc606d222420ec5.png

2.2 修改配置文件 （3个节点都要操作）

cd /usr/local/kafka/config/

vim server.properties

https://i-blog.csdnimg.cn/direct/cd6e945c853144359422d8317f4d4d90.png

https://i-blog.csdnimg.cn/direct/83564810ddc44f04b5bfa1858efeae37.png

2.3 将相关命令加入到系统环境当中 （3个节点都要操作）

vim /etc/profile

export KAFKA_HOME=/usr/local/kafka

export PATH=$PATH:$KAFKA_HOME/bin

https://i-blog.csdnimg.cn/direct/a4bf70529efb42bd97585a2fd85dbd09.png

source /etc/profile

2.4 启动kafka

cd /usr/local/kafka/config/

kafka-server-start.sh -daemon server.properties

netstat -antp | grep 9092

三个节点都启动

https://i-blog.csdnimg.cn/direct/da81567132274a51a068b313b4f164fa.png

Kafka 命令行操作

创建topic

kafka-topics.sh --create --zookeeper 172.25.23.7:2181,172.25.23.8:2181,172.25.23.9:2181 --replication-factor 2 --partitions 3 --topic test

https://i-blog.csdnimg.cn/direct/8a610d0573c84b46a95d6143c3c22cf8.png

–zookeeper：定义 zookeeper 集群服务器地址，如果有多个 IP 地址使用逗号分割，一般使用一个 IP 即可

–replication-factor：定义分区副本数，1 代表单副本，建议为 2

–partitions：定义分区数

–topic：定义 topic 名称

查看当前服务器中的所有 topic

kafka-topics.sh --list --zookeeper 172.25.23.7:2181,172.25.23.8:2181,172.25.23.9:2181

查看某个 topic 的详情

kafka-topics.sh  --describe --zookeeper 172.25.23.7:2181,172.25.23.8:2181,172.25.23.9:2181

https://i-blog.csdnimg.cn/direct/137226a6cc414971b3b253dd3334fba0.png

发布消息

kafka-console-producer.sh --broker-list 172.25.23.7:9092,172.25.23.8:9092,172.25.23.9:9092  --topic test

https://i-blog.csdnimg.cn/direct/2f922cad05c046b1aad290cfe2bf94a9.png

消费消息

kafka-console-consumer.sh --bootstrap-server 172.25.23.7:9092,172.25.23.8:9092,172.25.23.9:9092 --topic test --from-beginning

https://i-blog.csdnimg.cn/direct/4a4673f87e084dfc9f9195e263b0bf04.png

–from-beginning：会把主题中以往所有的数据都读取出来

修改分区数

kafka-topics.sh --zookeeper 172.25.23.7:2181,172.25.23.8:2181,172.25.23.9:2181 --alter --topic test --partitions 6

https://i-blog.csdnimg.cn/direct/58623ff069c24c2180ef7b2cbd2eae1f.png

删除 topic

kafka-topics.sh --delete --zookeeper 172.25.23.7:2181,172.25.23.8:2181,172.25.23.9:2181 --topic test

https://i-blog.csdnimg.cn/direct/754010dc4dbf49fbb3c117c41a672d5d.png

2.5 创建topic

cd /usr/local/kafka/bin

kafka-topics.sh --create --zookeeper 172.25.23.7:2181,172.25.23.8:2181,172.25.23.9:2181 --partitions 3 --replication-factor 2 --topic wyn

https://i-blog.csdnimg.cn/direct/b2fc485242ae4e41b3964bda7a6adb04.png

kafka-topics.sh --describe --zookeeper 172.25.23.7:2181

https://i-blog.csdnimg.cn/direct/20594f62f4c0425f98a1584e3f8d8d28.png

2.6 测试topic

发布消息

kafka-console-producer.sh --broker-list 172.25.23.7:9092,172.25.23.8:9092,172.25.23.9:9092 --topic wyn

https://i-blog.csdnimg.cn/direct/b95e64a470d74f739b551c6ee90d5a16.png

kafka-console-producer.sh --broker-list 172.25.23.7:9092,172.25.23.8:9092,172.25.23.9:9092 --topic wyn

https://i-blog.csdnimg.cn/direct/4f966d00c63443e08729efc1dac0fff7.png

消费消息

kafka-console-consumer.sh --bootstrap-server 172.25.23.7:9092,172.25.23.8:9092,172.25.23.9:9092 --topic nginx-es --from-beginning

https://i-blog.csdnimg.cn/direct/f32cccf3a57e4e7c9ae41995b6da0c2c.png

3、配置数据采集层filebeat（172.25.23.8）

安装nginx

cd /root

rpm -ivh nginx-1.20.0-1.el7.ngx.x86_64.rpm

https://i-blog.csdnimg.cn/direct/8b20293ec449494d82f6dae712a10989.png

3.1 定制日志格式

vim /etc/nginx/nginx.conf
log_format  json '{"@timestamp":"$time_iso8601",'

                           '"@version":"1",'

                           '"client":"$remote_addr",'

                           '"url":"$uri",'

                           '"status":"$status",'

                           '"domain":"$host",'

                           '"host":"$server_addr",'

                           '"size":$body_bytes_sent,'

                           '"responsetime":$request_time,'

                           '"referer": "$http_referer",'

                           '"ua": "$http_user_agent"'

               '}';



access_log  /var/log/nginx/access.log  json;

https://i-blog.csdnimg.cn/direct/44f96d4fa9b343d3996206c48e890706.png

3.2 上传、解压安装包

cd /opt

rpm -ivh filebeat-5.5.0-x86_64.rpm

https://i-blog.csdnimg.cn/direct/82b1b0517cba48e8ac610c11f0fd732a.png

3.3 修改配置文件filebeat.yml

vim /etc/filebeat/filebeat.yml

https://i-blog.csdnimg.cn/direct/908ca5f5a16e40f793c447d82477b7cd.png

https://i-blog.csdnimg.cn/direct/3c4a32e8e82343e297fdab5ecfa8678e.png

3.4 启动filebeat

systemctl restart filebeat

systemctl status filebeat

https://i-blog.csdnimg.cn/direct/bb8f8ab073864bf5977888dcd59462a5.png

查看端口是否打开

netstat -antp|grep filebeat

https://i-blog.csdnimg.cn/direct/1f7b7fb9f5dc4cacbab74d636a862590.png

4、所有组件部署完成之后，开始配置部署

4.1 在kafka上创建一个话题nginx-es

cd /usr/local/kafka/bin

./kafka-topics.sh --create --zookeeper 172.25.23.7:2181,172.25.23.8:2181,172.25.23.9:2181 --replication-factor 1 --partitions 1 --topic nginx-es

https://i-blog.csdnimg.cn/direct/f1e725bf103a4f06a8c8e580de9fa41c.png

./kafka-topics.sh --list --zookeeper 172.16.39.66:2181,172.16.39.70:2181,172.16.39.71:2181

https://i-blog.csdnimg.cn/direct/125197c1e2064c6aa87943590b5cc790.png

4.2 修改logstash的配置文件

cd /etc/logstash/conf.d/

vim nginxlog.conf
input {

kafka {

    topics => "nginx-es"

    #codec => "json"

    decorate_events => true

    bootstrap_servers => ["172.25.23.7:9092,172.25.23.8:9092,172.25.23.9:9092"]

  }

}

output {

      elasticsearch {

      hosts => ["172.25.23.7:9200 "]

      index => "nginx-%{+YYYY-MM-dd}"

      }

  }

https://i-blog.csdnimg.cn/direct/6cb67a3603da423a889e2454efd2403c.png

​​​​​​​systemctl restart logstash

systemctl status logstash

https://i-blog.csdnimg.cn/direct/cd528fdd7b394cd5ba51d8ad03e6f767.png

4.3 验证网页

cd /usr/local/src/node_modules/grunt/bin

./grunt server

https://i-blog.csdnimg.cn/direct/f31713fb3af64aa980d9265c99f41cb8.png

https://i-blog.csdnimg.cn/direct/ec22786b65ea457b96c6a44c737f8d59.png

4.4 Kibana上创建索引

https://i-blog.csdnimg.cn/direct/7943353b8c154e6f8b882278fe5fedb9.png

更新于 2025-03-15
 集群部署, 分布式系统, Zookeeper, Nginx, Kafka, Filebeat, Elk
返回 | 主页