概述
为了对网内各终端的外网流量进行可视化的管理，经过各方面的查找，决定使用elasticsearch & elastiflow &Kibana进行实现。
Elasticsearch
- 功能：分布式搜索与分析引擎。提供近实时（NRT）全文搜索、结构化查询、数据分析能力，支持海量数据存储与高性能检索。
- 用途：日志/指标分析、安全监控、业务搜索、数据可视化底层存储。
- 官网：elastic.co/elasticsearch
Elastiflow
- 功能：网络流量分析工具。解析 NetFlow/IPFIX/sFlow 数据，将网络流量信息（如源/目的IP、端口、协议）导入 Elasticsearch。
- 用途：网络监控、流量可视化、安全威胁检测（如异常连接）。
- 官网：elastiflow.com
- GITHUB: elastiflow_for_elasticsearch
ℹ️ 注：Elastiflow 是商业产品，提供免费社区版与付费企业版。
Logstash
- 功能：数据采集与处理管道。支持从多种来源（日志、数据库、消息队列）采集数据，进行过滤/转换（如解析JSON、删除字段），并输出到目标存储（如 Elasticsearch）。
- 用途：日志聚合、数据清洗、ETL（提取-转换-加载）。
- 官网：elastic.co/logstash
Kibana
- 功能：Elasticsearch 数据可视化与分析平台。通过图表、仪表盘、地图等交互式界面展示数据，支持日志探索、机器学习分析、APM 监控等。
- 用途：日志分析、业务指标监控、安全事件调查、用户行为分析。
- 官网：elastic.co/kibana
软件架构说明
如何上图所示，整套系统的工作步骤如下：
- 在网络设备中配置sflow
- 使用Elastiflow进行流量收集
- Elastiflow将收集到的日志写入ES数据库中
- 通过Kibana进行流量数据可视化
方案设计
现网络架构如上图所示，核心交换机接入了一台zabbix服务器做网内设备的监控，下面接入了两个接入交换机，上行通过链路聚合接入防火墙。为了节省资源，将本方案搭建在zabbix的服务器上。
步骤
修改主机参数
vi /etc/sysctl.conf
vm.max_map_count=262144
net.core.netdev_max_backlog=4096
net.core.rmem_default=262144
net.core.rmem_max=67108864
net.ipv4.udp_rmem_min=131072
net.ipv4.udp_mem=2097152 4194304 8388608
sysctl -p
使用docker-compose安装
- 配置环境变量
#################### GENERAL SETTINGS ####################
# Unattended installation
FULL_AUTO=1
# Project namespace (defaults to the current folder name if not set)
#COMPOSE_PROJECT_NAME=myproject
# Components to install
INSTALL_FLOWCOLL=1
INSTALL_SNMPCOLLTRAP=0
#################### ELASTIC SETTINGS ###################
# Version of Elasticsearch. Note: 8.16.4 is the last version to support TSDS for free.
ELASTIC_VERSION=8.14.3
# Set the cluster name
CLUSTER_NAME=docker-cluster
# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=elastic
# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200
# JVM Heap Size
# Set heap size to about one-third of the system memory, but do not exceed 31g. Assuming 16GB of system memory, we'll set this to 5GB
JVM_HEAP_SIZE=5
# Set the memory limit to 2x the heap size (currently set to 10GB)
MEM_LIMIT_ELASTIC=10737418240
# Set the memory limit to 2GB for small to medium workloads (currently set to 2GB)
MEM_LIMIT_KIBANA=2147483648
#################### KIBANA SETTINGS ###################
# Version of Kibana
KIBANA_VERSION=8.14.3
# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=elastic
# Port to expose Kibana to the host
KIBANA_PORT=5601
################## ELASTIFLOW SETTINGS #################
# Not required to install and use but does unlock additional features
ELASTIFLOW_ACCOUNT_ID=''
ELASTIFLOW_LICENSE_KEY=''
################ ELASTIFLOW FLOW SETTINGS ##############
# Version of ElastiFlow
ELASTIFLOW_FLOW_VERSION=7.7.2
ELASTIFLOW_LICENSE_FLOW_RECORDS_PER_SECOND=16000
#If using ECS, set ECS_ENABLE to "true" and FLOW_DASHBOARDS_SCHEMA to "ecs"
ECS_ENABLE=false
FLOW_DASHBOARDS_SCHEMA=codex
FLOW_DASHBOARDS_VERSION=8.14.x
################ ELASTIFLOW SNMP COLLECTOR / SNMP TRAP COLLECTOR SETTINGS ###############
# ElastiFlow SNMP Collector and SNMP Traps Version
ELASTIFLOW_SNMP_VERSION=7.7.2
ELASTIFLOW_LICENSE_TELEMETRY_HOSTS=20
SNMP_DASHBOARDS_VERSION=8.2.x
SNMP_DASHBOARDS_SCHEMA=codex
SNMP_TRAPS_DASHBOARDS_VERSION=8.14.x
SNMP_TRAPS_DASHBOARDS_SCHEMA=codex
- 配置docker-compose.yml
services:
setup:
image: elasticsearch:${ELASTIC_VERSION}
volumes:
- ./data/certs:/usr/share/elasticsearch/config/certs
user: "0"
networks:
- elastiflow_net # 设置连接的网络
command: >
bash -c '
if [ x${ELASTIC_PASSWORD} == x ]; then
echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
exit 1;
elif [ x${KIBANA_PASSWORD} == x ]; then
echo "Set the KIBANA_PASSWORD environment variable in the .env file";
exit 1;
fi;
if [ ! -f config/certs/ca.zip ]; then
echo "Creating CA";
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f config/certs/certs.zip ]; then
echo "Creating certs";
echo -ne \
"instances:\n"\
" - name: es01\n"\
" dns:\n"\
" - es01\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
" - name: kibana\n"\
" dns:\n"\
" - kibana\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
> config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem \
-out config/certs/certs.zip \
--in config/certs/instances.yml \
--ca-cert config/certs/ca/ca.crt \
--ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
echo "Setting file permissions"
chown -R root:root config/certs;
find . -type d -exec chmod 750 \{\} \;;
find . -type f -exec chmod 640 \{\} \;;
echo "Waiting for Elasticsearch availability";
until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
echo "Setting kibana_system password";
until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
echo "All done!";
'
healthcheck:
test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
interval: 1s
timeout: 5s
retries: 120
es01:
depends_on:
setup:
condition: service_healthy
image: elasticsearch:${ELASTIC_VERSION}
networks:
- elastiflow_net # 设置连接的网络
restart: 'unless-stopped'
volumes:
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
- ./data/certs:/usr/share/elasticsearch/config/certs
- ./data/esdata:/usr/share/elasticsearch/data
ports:
- ${ES_PORT}:9200
environment:
- node.name=es01
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01
#- discovery.seed_hosts=es01 # Use if adding more ES nodes
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/es01/es01.key
- xpack.security.http.ssl.certificate=certs/es01/es01.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/es01/es01.key
- xpack.security.transport.ssl.certificate=certs/es01/es01.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
#- xpack.license.self_generated.type=${LICENSE}
- xpack.monitoring.collection.enabled=true
- ES_JAVA_OPTS=-Xms${JVM_HEAP_SIZE}g -Xmx${JVM_HEAP_SIZE}g
mem_limit: ${MEM_LIMIT_ELASTIC}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
kibana:
depends_on:
es01:
condition: service_healthy
image: kibana:${KIBANA_VERSION}
networks:
- elastiflow_net # 设置连接的网络
restart: 'unless-stopped'
volumes:
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
- ./data/certs:/usr/share/kibana/config/certs
- ./data/kibanadata:/usr/share/kibana/data
ports:
- ${KIBANA_PORT}:5601
environment:
# The environment variable "SERVER_NAME" maps to Kibana's "server.name" setting
- SERVER_NAME=kibana
- I18N_LOCALE="zh-CN"
# Force Kibana to serve HTTPS
- SERVER_SSL_ENABLED=true
- SERVER_SSL_KEY=config/certs/kibana/kibana.key
- SERVER_SSL_CERTIFICATE=config/certs/kibana/kibana.crt
# By default, it will redirect from http to https, so point "server.publicBaseUrl" to https
- SERVER_PUBLICBASEURL=https://kibana.example.com:5601
- ELASTICSEARCH_HOSTS=https://es01:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
- ELASTICSEARCH_SSL_VERIFICATIONMODE=certificate
# Example encryption key for encrypted saved objects
- XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=ElastiFlow_0123456789_0123456789_0123456789
# Prevent autocomplete dropdowns from timing out
- UNIFIEDSEARCH_AUTOCOMPLETE_VALUESUGGESTIONS_TIMEOUT=4000
- UNIFIEDSEARCH_AUTOCOMPLETE_VALUESUGGESTIONS_TERMINATEAFTER=100000
mem_limit: ${MEM_LIMIT_KIBANA}
healthcheck:
# Use HTTPS for the healthcheck; curl -k to skip certificate validation if using self-signed
test:
[
"CMD-SHELL",
"curl -k -s -I https://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
]
interval: 10s
timeout: 10s
retries: 120
# 通过elastiflow做netflow流量收集
flow-collector:
image: elastiflow/flow-collector:${ELASTIFLOW_FLOW_VERSION}
container_name: flow-collector
restart: 'unless-stopped'
networks:
- elastiflow_net # 设置连接的网络
ports:
- "2055:2055/udp"
- "6343:6343/udp"
- "4739:4739/udp"
volumes:
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
- ./data/elastiflow:/etc/elastiflow
- ./data/elastiflow/log/elastiflow:/var/log/elastiflow
- ./data/elastiflow/log/flowcoll:/var/lib/elastiflow/flowcoll
environment:
EF_LICENSE_ACCEPTED: 'true'
EF_ACCOUNT_ID: '${ELASTIFLOW_ACCOUNT_ID}'
EF_LICENSE_KEY: '${ELASTIFLOW_LICENSE_KEY}'
EF_LICENSE_FLOW_RECORDS_PER_SECOND: '${ELASTIFLOW_LICENSE_FLOW_RECORDS_PER_SECOND}'
#EF_FLOW_LICENSED_UNITS:
#EF_INSTANCE_NAME: default
#EF_API_PORT: 8080
#EF_API_TLS_ENABLE: ''
#EF_API_TLS_CERT_FILEPATH: ''
#EF_API_TLS_KEY_FILEPATH: ''
#EF_API_BASIC_AUTH_ENABLE: 'false'
#EF_API_BASIC_AUTH_USERNAME: ''
#EF_API_BASIC_AUTH_PASSWORD: ''
#EF_LOGGER_LEVEL: 'info'
#EF_LOGGER_ENCODING: 'json'
EF_LOGGER_FILE_LOG_ENABLE: 'true'
EF_LOGGER_FILE_LOG_FILENAME: '/var/log/elastiflow/flowcoll/flowcoll.log'
#EF_LOGGER_FILE_LOG_MAX_SIZE: 100
#EF_LOGGER_FILE_LOG_MAX_AGE: ''
#EF_LOGGER_FILE_LOG_MAX_BACKUPS: 4
#EF_LOGGER_FILE_LOG_COMPRESS: 'false'
# Fixes an issue where download does not complete in time
EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_TIMEOUT: 60
EF_FLOW_SERVER_UDP_IP: '0.0.0.0'
EF_FLOW_SERVER_UDP_PORT: 2055,4739,6343,9995
#EF_FLOW_SERVER_UDP_READ_BUFFER_MAX_SIZE: 134217728
#EF_FLOW_PACKET_STREAM_MAX_SIZE:
EF_AWS_VPC_FLOW_LOG_S3_ENABLE: 'false'
#EF_AWS_VPC_FLOW_LOG_S3_BUCKET: ''
#EF_AWS_VPC_FLOW_LOG_S3_PREFIX: 'AWSLogs'
#AWS_REGION: ''
#AWS_ACCESS_KEY_ID: ''
#AWS_SECRET_ACCESS_KEY: ''
#EF_AWS_VPC_FLOW_LOG_S3_TLS_ENABLE: 'false'
#EF_AWS_VPC_FLOW_LOG_S3_TLS_SKIP_VERIFICATION: 'false'
#EF_AWS_VPC_FLOW_LOG_S3_TLS_CA_CERT_FILEPATH: ''
#EF_AWS_VPC_FLOW_LOG_S3_TLS_MIN_VERSION: '1.2'
#EF_INPUT_FLOW_BENCHMARK_ENABLE: 'false'
#EF_INPUT_FLOW_BENCHMARK_PACKET_FILEPATH: '/etc/elastiflow/benchmark/flow/packets.txt'
#EF_PROCESSOR_POOL_SIZE:
#EF_PROCESSOR_DECODE_IPFIX_ENABLE: 'true'
#EF_PROCESSOR_DECODE_NETFLOW1_ENABLE: 'true'
#EF_PROCESSOR_DECODE_NETFLOW5_ENABLE: 'true'
#EF_PROCESSOR_DECODE_NETFLOW6_ENABLE: 'true'
#EF_PROCESSOR_DECODE_NETFLOW7_ENABLE: 'true'
#EF_PROCESSOR_DECODE_NETFLOW9_ENABLE: 'true'
#EF_PROCESSOR_DECODE_SFLOW5_ENABLE: 'true'
#EF_PROCESSOR_DECODE_SFLOW_FLOWS_ENABLE: 'true'
#EF_PROCESSOR_DECODE_SFLOW_FLOWS_KEEP_SAMPLES: 'false'
#EF_PROCESSOR_DECODE_SFLOW_COUNTERS_ENABLE: 'true'
#EF_PROCESSOR_DECODE_MAX_RECORDS_PER_PACKET: 64
#EF_PROCESSOR_TRANSLATE_KEEP_IDS: 'default'
EF_PROCESSOR_ENRICH_APP_ID_ENABLE: 'false'
#EF_PROCESSOR_ENRICH_APP_ID_PATH: '/etc/elastiflow/app/appid.yml'
#EF_PROCESSOR_ENRICH_APP_ID_TTL: 7200
EF_PROCESSOR_ENRICH_APP_IPPORT_ENABLE: 'true'
EF_PROCESSOR_ENRICH_APP_IPPORT_PATH: '/etc/elastiflow/app/ipport.yml'
#EF_PROCESSOR_ENRICH_APP_IPPORT_TTL: 7200
#EF_PROCESSOR_ENRICH_APP_IPPORT_PRIVATE: 'true'
#EF_PROCESSOR_ENRICH_APP_IPPORT_PUBLIC: 'false'
#EF_PROCESSOR_ENRICH_APP_REFRESH_RATE: 15
#EF_PROCESSOR_ENRICH_IPADDR_TTL: 7200
EF_PROCESSOR_ENRICH_IPADDR_METADATA_ENABLE: 'true'
EF_PROCESSOR_ENRICH_IPADDR_METADATA_USERDEF_PATH: '/etc/elastiflow/metadata/ipaddrs.yml'
#EF_PROCESSOR_ENRICH_IPADDR_METADATA_REFRESH_RATE: 15
EF_PROCESSOR_ENRICH_IPADDR_DNS_ENABLE: 'true'
EF_PROCESSOR_ENRICH_IPADDR_DNS_NAMESERVER_IP: ''
EF_PROCESSOR_ENRICH_IPADDR_DNS_NAMESERVER_TIMEOUT: 3000
#EF_PROCESSOR_ENRICH_IPADDR_DNS_RESOLVE_PRIVATE: 'true'
#EF_PROCESSOR_ENRICH_IPADDR_DNS_RESOLVE_PUBLIC: 'true'
#EF_PROCESSOR_ENRICH_IPADDR_DNS_USERDEF_PATH: '/etc/elastiflow/hostname/user_defined.yml'
#EF_PROCESSOR_ENRICH_IPADDR_DNS_USERDEF_REFRESH_RATE: 15
#EF_PROCESSOR_ENRICH_IPADDR_DNS_INCLEXCL_PATH: '/etc/elastiflow/hostname/incl_excl.yml'
#EF_PROCESSOR_ENRICH_IPADDR_DNS_INCLEXCL_REFRESH_RATE: 15
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_ASN_ENABLE: 'true'
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_ASN_PATH: '/etc/elastiflow/maxmind/GeoLite2-ASN.mmdb'
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_ENABLE: 'true'
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_PATH: '/etc/elastiflow/maxmind/GeoLite2-City.mmdb'
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_VALUES: 'city,country,country_code,location,timezone'
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_LANG: 'zh-CN'
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_INCLEXCL_PATH: '/etc/elastiflow/maxmind/incl_excl.yml'
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_INCLEXCL_REFRESH_RATE: 15
EF_PROCESSOR_ENRICH_ASN_PREF: 'lookup'
#EF_PROCESSOR_ENRICH_NETIF_TTL: 7200
EF_PROCESSOR_ENRICH_NETIF_METADATA_ENABLE: 'true'
EF_PROCESSOR_ENRICH_NETIF_METADATA_USERDEF_PATH: '/etc/elastiflow/metadata/netifs.yml'
#EF_PROCESSOR_ENRICH_NETIF_METADATA_REFRESH_RATE: 15
EF_PROCESSOR_ENRICH_NETIF_FLOW_OPTIONS_ENABLE: 'true'
EF_PROCESSOR_ENRICH_NETIF_SNMP_ENABLE: 'true'
#EF_PROCESSOR_ENRICH_NETIF_SNMP_PORT: 161
#EF_PROCESSOR_ENRICH_NETIF_SNMP_VERSION: 2
EF_PROCESSOR_ENRICH_NETIF_SNMP_COMMUNITIES: 'public'
#EF_PROCESSOR_ENRICH_NETIF_SNMP_V3_USERNAME: ''
#EF_PROCESSOR_ENRICH_NETIF_SNMP_V3_AUTHENTICATION_PROTOCOL: 'noauth'
#EF_PROCESSOR_ENRICH_NETIF_SNMP_V3_AUTHENTICATION_PASSPHRASE: ''
#EF_PROCESSOR_ENRICH_NETIF_SNMP_V3_PRIVACY_PROTOCOL: 'nopriv'
#EF_PROCESSOR_ENRICH_NETIF_SNMP_V3_PRIVACY_PASSPHRASE: ''
#EF_PROCESSOR_ENRICH_NETIF_SNMP_TIMEOUT: 2
#EF_PROCESSOR_ENRICH_NETIF_SNMP_RETRIES: 1
#EF_PROCESSOR_ENRICH_TOTALS_IF_NO_DELTAS: 'false'
#EF_PROCESSOR_ENRICH_SAMPLERATE_CACHE_SIZE: 32768
#EF_PROCESSOR_ENRICH_SAMPLERATE_USERDEF_ENABLE: 'false'
#EF_PROCESSOR_ENRICH_SAMPLERATE_USERDEF_PATH: '/etc/elastiflow/settings/sample_rate.yml'
#EF_PROCESSOR_ENRICH_SAMPLERATE_USERDEF_OVERRIDE: 'false'
#EF_PROCESSOR_ENRICH_COMMUNITYID_ENABLE: 'true'
#EF_PROCESSOR_ENRICH_COMMUNITYID_SEED: 0
#EF_PROCESSOR_ENRICH_CONVERSATIONID_ENABLE: 'true'
#EF_PROCESSOR_ENRICH_CONVERSATIONID_SEED: 0
#EF_PROCESSOR_ENRICH_JOIN_ASN: 'true'
#EF_PROCESSOR_ENRICH_JOIN_GEOIP: 'true'
#EF_PROCESSOR_ENRICH_JOIN_SEC: 'true'
#EF_PROCESSOR_ENRICH_JOIN_NETATTR: 'true'
#EF_PROCESSOR_ENRICH_JOIN_SUBNETATTR: 'true'
#EF_PROCESSOR_DURATION_PRECISION: 'ms'
#EF_PROCESSOR_TIMESTAMP_PRECISION: 'ms'
#EF_PROCESSOR_PERCENT_NORM: 100
#EF_PROCESSOR_EXPAND_CLISRV: 'true'
#EF_PROCESSOR_EXPAND_CLISRV_NO_L4_PORTS: 'true'
#EF_PROCESSOR_KEEP_CPU_TICKS: 'false'
#EF_PROCESSOR_DROP_FIELDS: ''
#EF_PROCESSOR_IFA_ENABLE: 'false'
#EF_PROCESSOR_IFA_WORKER_SIZE: 0
# stdout
#EF_OUTPUT_STDOUT_ENABLE: 'false'
#EF_OUTPUT_STDOUT_FORMAT: 'json_pretty'
# monitor
#EF_OUTPUT_MONITOR_ENABLE: 'false'
#EF_OUTPUT_MONITOR_INTERVAL: 300
# Elasticsearch
EF_OUTPUT_ELASTICSEARCH_ENABLE: 'true'
EF_OUTPUT_ELASTICSEARCH_ECS_ENABLE: '${ECS_ENABLE}'
#EF_OUTPUT_ELASTICSEARCH_BATCH_DEADLINE: 2000
#EF_OUTPUT_ELASTICSEARCH_BATCH_MAX_BYTES: 8388608
#EF_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE: 'collect'
#EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD: 'rollover'
EF_OUTPUT_ELASTICSEARCH_TSDS_ENABLE: 'false'
#EF_OUTPUT_ELASTICSEARCH_INDEX_SUFFIX: ''
#EF_OUTPUT_ELASTICSEARCH_DROP_FIELDS: ''
#EF_OUTPUT_ELASTICSEARCH_ALLOWED_RECORD_TYPES: 'as_path_hop,flow_option,flow,telemetry'
#EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ENABLE: 'true'
#EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_OVERWRITE: 'true'
EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SHARDS: 1
EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REPLICAS: 0
#EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REFRESH_INTERVAL: '10s'
#EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_CODEC: 'best_compression'
#EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ILM_LIFECYCLE: 'elastiflow'
#EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_PIPELINE_DEFAULT: '_none'
#EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_PIPELINE_FINAL: '_none'
# A comma separated list of Elasticsearch nodes to use. DO NOT include "http://" or "https://"
EF_OUTPUT_ELASTICSEARCH_ADDRESSES: 'es01:9200'
EF_OUTPUT_ELASTICSEARCH_USERNAME: 'elastic'
EF_OUTPUT_ELASTICSEARCH_PASSWORD: '${ELASTIC_PASSWORD}'
#EF_OUTPUT_ELASTICSEARCH_CLOUD_ID: ''
#EF_OUTPUT_ELASTICSEARCH_API_KEY: ''
#EF_OUTPUT_ELASTICSEARCH_CLIENT_CA_CERT_FILEPATH: ''
#EF_OUTPUT_ELASTICSEARCH_CLIENT_CERT_FILEPATH: ''
#EF_OUTPUT_ELASTICSEARCH_CLIENT_KEY_FILEPATH: ''
EF_OUTPUT_ELASTICSEARCH_TLS_ENABLE: 'true'
EF_OUTPUT_ELASTICSEARCH_TLS_SKIP_VERIFICATION: 'true'
EF_OUTPUT_ELASTICSEARCH_TLS_CA_CERT_FILEPATH: ''
#EF_OUTPUT_ELASTICSEARCH_RETRY_ENABLE: 'true'
#EF_OUTPUT_ELASTICSEARCH_RETRY_ON_TIMEOUT_ENABLE: 'true'
#EF_OUTPUT_ELASTICSEARCH_MAX_RETRIES: 3
#EF_OUTPUT_ELASTICSEARCH_RETRY_BACKOFF: 1000
# OpenSearch
EF_OUTPUT_OPENSEARCH_ENABLE: 'false'
EF_OUTPUT_OPENSEARCH_ECS_ENABLE: 'false'
#EF_OUTPUT_OPENSEARCH_BATCH_DEADLINE: 2000
#EF_OUTPUT_OPENSEARCH_BATCH_MAX_BYTES: 8388608
#EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE: 'collect'
#EF_OUTPUT_OPENSEARCH_INDEX_PERIOD: 'daily'
#EF_OUTPUT_OPENSEARCH_INDEX_SUFFIX: ''
#EF_OUTPUT_OPENSEARCH_DROP_FIELDS: ''
#EF_OUTPUT_OPENSEARCH_ALLOWED_RECORD_TYPES: 'as_path_hop,flow_option,flow,telemetry'
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ENABLE: 'true'
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_OVERWRITE: 'true'
EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_SHARDS: 1
EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REPLICAS: 0
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REFRESH_INTERVAL: '10s'
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_CODEC: 'best_compression'
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ISM_POLICY: 'elastiflow'
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_DEFAULT: '_none'
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_FINAL: '_none'
# A comma separated list of OpenSearch nodes to use. DO NOT include "http://" or "https://"
EF_OUTPUT_OPENSEARCH_ADDRESSES: 'es01:9200'
EF_OUTPUT_OPENSEARCH_USERNAME: 'admin'
EF_OUTPUT_OPENSEARCH_PASSWORD: 'admin'
#EF_OUTPUT_OPENSEARCH_CLIENT_CA_CERT_FILEPATH: ''
#EF_OUTPUT_OPENSEARCH_CLIENT_CERT_FILEPATH: ''
#EF_OUTPUT_OPENSEARCH_CLIENT_KEY_FILEPATH: ''
EF_OUTPUT_OPENSEARCH_TLS_ENABLE: 'false'
EF_OUTPUT_OPENSEARCH_TLS_SKIP_VERIFICATION: 'false'
EF_OUTPUT_OPENSEARCH_TLS_CA_CERT_FILEPATH: ''
#EF_OUTPUT_OPENSEARCH_RETRY_ENABLE: 'true'
#EF_OUTPUT_OPENSEARCH_RETRY_ON_TIMEOUT_ENABLE: 'true'
#EF_OUTPUT_OPENSEARCH_MAX_RETRIES: 3
#EF_OUTPUT_OPENSEARCH_RETRY_BACKOFF: 1000
# Splunk
EF_OUTPUT_SPLUNK_HEC_ENABLE: 'false'
#EF_OUTPUT_SPLUNK_HEC_CIM_ENABLE: 'false'
EF_OUTPUT_SPLUNK_HEC_ADDRESSES: '127.0.0.1:8088'
EF_OUTPUT_SPLUNK_HEC_TOKEN: ''
#EF_OUTPUT_SPLUNK_HEC_BATCH_MAX_BYTES: 8388608
#EF_OUTPUT_SPLUNK_HEC_BATCH_DEADLINE: 2000
#EF_OUTPUT_SPLUNK_HEC_TLS_ENABLE: 'true'
#EF_OUTPUT_SPLUNK_HEC_TLS_SKIP_VERIFICATION: 'false'
#EF_OUTPUT_SPLUNK_HEC_TLS_CA_CERT_FILEPATH: ''
#EF_OUTPUT_SPLUNK_HEC_DROP_FIELDS: ''
# Kafka
EF_OUTPUT_KAFKA_ENABLE: 'false'
EF_OUTPUT_KAFKA_BROKERS: ''
#EF_OUTPUT_KAFKA_VERSION: '1.0.0'
#EF_OUTPUT_KAFKA_TOPIC: 'elastiflow-flow-codex'
#EF_OUTPUT_KAFKA_PARTITION_KEY: 'flow.export.ip.addr'
#EF_OUTPUT_KAFKA_CLIENT_ID: 'elastiflow-flowcoll'
#EF_OUTPUT_KAFKA_RACK_ID: ''
#EF_OUTPUT_KAFKA_TIMEOUT: 30
#EF_OUTPUT_KAFKA_DROP_FIELDS: ''
#EF_OUTPUT_KAFKA_ALLOWED_RECORD_TYPES: 'as_path_hop,flow_option,flow,telemetry'
#EF_OUTPUT_KAFKA_FLAT_RECORD_ENABLE: 'true'
EF_OUTPUT_KAFKA_SASL_ENABLE: 'false'
#EF_OUTPUT_KAFKA_SASL_USERNAME: ''
#EF_OUTPUT_KAFKA_SASL_PASSWORD: ''
#EF_OUTPUT_KAFKA_TLS_ENABLE: 'false'
#EF_OUTPUT_KAFKA_TLS_CA_CERT_FILEPATH: ''
#EF_OUTPUT_KAFKA_TLS_CERT_FILEPATH: ''
#EF_OUTPUT_KAFKA_TLS_KEY_FILEPATH: ''
#EF_OUTPUT_KAFKA_TLS_SKIP_VERIFICATION: 'false'
#EF_OUTPUT_KAFKA_PRODUCER_MAX_MESSAGE_BYTES: 1000000
#EF_OUTPUT_KAFKA_PRODUCER_REQUIRED_ACKS: 1
#EF_OUTPUT_KAFKA_PRODUCER_TIMEOUT: 10
#EF_OUTPUT_KAFKA_PRODUCER_COMPRESSION: 3
#EF_OUTPUT_KAFKA_PRODUCER_COMPRESSION_LEVEL: -1000
#EF_OUTPUT_KAFKA_PRODUCER_FLUSH_BYTES: 1000000
#EF_OUTPUT_KAFKA_PRODUCER_FLUSH_MESSAGES: 1024
#EF_OUTPUT_KAFKA_PRODUCER_FLUSH_FREQUENCY: 1000
#EF_OUTPUT_KAFKA_PRODUCER_FLUSH_MAX_MESSAGES: 0
#EF_OUTPUT_KAFKA_PRODUCER_RETRY_MAX: 3
#EF_OUTPUT_KAFKA_PRODUCER_RETRY_BACKOFF: 100
# Cribl
EF_OUTPUT_CRIBL_ENABLE: 'false'
EF_OUTPUT_CRIBL_ADDRESSES: '127.0.0.1:10080'
EF_OUTPUT_CRIBL_TOKEN: ''
#EF_OUTPUT_CRIBL_BATCH_DEADLINE: 2000
#EF_OUTPUT_CRIBL_BATCH_MAX_BYTES: 8388608
#EF_OUTPUT_CRIBL_TLS_ENABLE: 'false'
#EF_OUTPUT_CRIBL_TLS_SKIP_VERIFICATION: 'false'
#EF_OUTPUT_CRIBL_TLS_CA_CERT_FILEPATH: ''
#EF_OUTPUT_CRIBL_DROP_FIELDS: ''
# Generic HTTP
EF_OUTPUT_GENERIC_HTTP_ENABLE: 'false'
EF_OUTPUT_GENERIC_HTTP_ECS_ENABLE: 'false'
#EF_OUTPUT_GENERIC_HTTP_BATCH_DEADLINE: 2000
#EF_OUTPUT_GENERIC_HTTP_BATCH_MAX_BYTES: 8388608
EF_OUTPUT_GENERIC_HTTP_ADDRESSES: ''
#EF_OUTPUT_GENERIC_HTTP_USERNAME: ''
#EF_OUTPUT_GENERIC_HTTP_PASSWORD: ''
#EF_OUTPUT_GENERIC_HTTP_TLS_ENABLE: 'false'
#EF_OUTPUT_GENERIC_HTTP_TLS_SKIP_VERIFICATION: 'false'
#EF_OUTPUT_GENERIC_HTTP_TLS_CA_CERT_FILEPATH: ''
#EF_OUTPUT_GENERIC_HTTP_DROP_FIELDS: ''
#EF_OUTPUT_GENERIC_HTTP_TIMESTAMP_SOURCE: 'collect'
networks:
elastiflow_net:
driver: bridge
- 创建存储目录
mkdir data
mkdir certs elastiflow esdata kibanadata
mkdir -p elastiflow/log
chown -R 1000:1000 *
- 下载所需的 ElastiFlow NetObserv Flow 支持文件
cd elastiflow
wget -O flow-collector_7.2.2_linux_amd64.deb https://elastiflow-releases.s3.us-east-2.amazonaws.com/flow-collector/flow-collector_7.2.2_linux_amd64.deb && sudo mkdir -p elastiflow_extracted
- 使用容器解压下载的文件
docker pull ubuntu
docker run -it --rm -v /docker/elastiflow/data/elastiflow:/etc/elastiflow/ ubuntu sh -c "dpkg-deb -x /etc/elastiflow/flow-collector_7.7.2_linux_amd64.deb /etc/elastiflow/elastiflow_extracted &&cp -r /etc/elastiflow/elastiflow_extracted/etc/elastiflow/. /etc/elastiflow"
- Geo and ASN 扩充
mkdir -p elastiflow/maxmind
wget wget https://raw.githubusercontent.com/P3TERX/GeoLite.mmdb/download/GeoLite2-ASN.mmdb
wget https://raw.githubusercontent.com/P3TERX/GeoLite.mmdb/download/GeoLite2-City.mmdb
wget https://raw.githubusercontent.com/P3TERX/GeoLite.mmdb/download/GeoLite2-Country.mmdb
-
访问https://ip:5601，进行登录，帐号和密码均为elastic。
-
导入仪表盘模板
https://github.com/elastiflow/elastiflow_for_elasticsearch/tree/master/kibana/flow
选择适合版本的仪表盘进行下载。
- 交换机配置
sflow collector 2 ip 192.168.100.250 description zabbix
sflow agent ip 192.168.100.254
interface GigabitEthernet0/0/47
eth-trunk 0
sflow counter-sampling collector 2
sflow counter-sampling interval 2
sflow flow-sampling collector 2
sflow flow-sampling rate 256
- 数据展示
借鉴资料
https://github.com/elastiflow/elastiflow_for_elasticsearch/tree/master/kibana/flow
https://github.com/elastiflow/ElastiFlow-Tools/blob/main/docker_install/readme.md
评论区