gohangout是golang编写的类似logstash的采集转换工具.
本文记录一下,我在从logstash转向gohangout时趟过一些坑,方便后面查询
调试
gohangout的设置目标就是kafak读取数据,经过转换写入es,因此当前最新版本1.4.2竟然还没有实现File读取.
inputs:
- Stdin:
codec: json
outputs:
- Stdout:{}这个上节有提过了,在调试filters时是最常用的.
./gohangout -logtostderr -v 5 --config config.yml
对了logtostderr是输出调试信息 -v 是日志级别
Input
读取大同小异
logstash
kafka {
bootstrap_servers => "192.168.1.7:9092,192.168.1.8:9092"
topics => ["nginx"]
group_id => "nginx"
consumer_threads => 2
codec => json
add_field => {"topic" => "nginx"}
}
gohangout
- Kafka:
codec: json
topic:
nginx: 3
consumer_settings:
group.id: nginx
bootstrap.servers: '192.168.1.7:9092,192.168.1.8:9092'
auto.commit.interval.ms: '10000'outputs
输出到es,gohangout很容易解决东八区的问题.
logstash
elasticsearch {
hosts=>["192.168.1.5:9200","192.168.1.6:9200"]
index => "nginx-%{index_day}"
document_type => "nginx"
}
gohangout
- Elasticsearch:
hosts:
- 'http://192.168.1.5:9200'
- 'http://192.168.1.6:9200'
index: 'nginx-%{+2006.01.02}'
index_type: 'nginx' # default logs
index_time_location: 'Asia/Shanghai' # defaut UTC
bulk_actions: 30000 #default 20000
bulk_size: 20 # default 15 MB
flush_interval: 5 # default 10 seconds
concurrent_requests: 10filters
Json解析
将message段进行json解析,方便在es查询
logstash
json {
source => "message"
}
gohangout
- Json:
field: message移除字段
原始无用,又不方便移除写入,在输出时移除是个好办法
logstash
mutate {
remove_field => ["message","tags","@version","source"]
}
gohangout
- Remove:
fields: ['message','tags','@version','source']类型转换
把数字转换成int或float,方便在kinaba排序
logstash
mutate {
convert =>[
"request_time","float",
"status","integer"
]
}
gohangout
- Convert:
fields:
request_time:
remove_if_fail: false
setto_if_fail: 0.0
to: float
status:
remove_if_fail: false
setto_if_fail: 0
to: int小写转换
各种原因,sdk实现出来,竟然大小写混合的,此处统一处理成小写
logstash
mutate {
lowercase =>["product_line","app_name"]
}
gohangout
- Lowercase:
fields: ['product_line','app_name']日期转换
logstash
date {
match => ["timestamp", "yyyy-MM-dd HH:mm:ss.SSS"]
target => "@timestamp"
}
gohangout
- Date:
src: 'timestamp'
target: '@timestamp'
location: Asia/Shanghai
overwrite: true
formats:
- '2006-01-02 15:04:05.999'一般在kinaba都是以@timestamp作为时序索引,此处将日志的生成时间写入到@timestamp,方便,查询.
我用gohantout也就一个月不到,暂时先总结这么多.
转载请注明: 转自Rainbird的个人博客 本文链接: ELK|gohangout替换logstash时的一些记录
