ELK|gohangout替换logstash时的一些记录


gohangout是golang编写的类似logstash的采集转换工具.
本文记录一下,我在从logstash转向gohangout时趟过一些坑,方便后面查询

调试
gohangout的设置目标就是kafak读取数据,经过转换写入es,因此当前最新版本1.4.2竟然还没有实现File读取.

inputs:
    - Stdin:
        codec: json
outputs:
    - Stdout:{}

这个上节有提过了,在调试filters时是最常用的.
./gohangout -logtostderr -v 5 --config config.yml
对了logtostderr是输出调试信息 -v 是日志级别

Input
读取大同小异

logstash
    kafka {
        bootstrap_servers => "192.168.1.7:9092,192.168.1.8:9092"
        topics => ["nginx"]
        group_id => "nginx"
        consumer_threads => 2
        codec => json
        add_field => {"topic" => "nginx"}
    }

gohangout
    - Kafka:
        codec: json
        topic:
            nginx: 3
        consumer_settings:
            group.id: nginx
            bootstrap.servers: '192.168.1.7:9092,192.168.1.8:9092'
            auto.commit.interval.ms: '10000'

outputs
输出到es,gohangout很容易解决东八区的问题.

logstash
 elasticsearch {
    hosts=>["192.168.1.5:9200","192.168.1.6:9200"]
    index => "nginx-%{index_day}"
    document_type => "nginx"
}

gohangout
    - Elasticsearch:
        hosts:
            - 'http://192.168.1.5:9200'
            - 'http://192.168.1.6:9200'
        index: 'nginx-%{+2006.01.02}'
        index_type: 'nginx' # default logs
        index_time_location: 'Asia/Shanghai' # defaut UTC
        bulk_actions: 30000 #default 20000
        bulk_size: 20 # default 15 MB
        flush_interval: 5 # default 10 seconds
        concurrent_requests: 10

filters

Json解析
将message段进行json解析,方便在es查询

logstash
    json {
        source => "message"
    }

gohangout
    - Json:
        field: message

移除字段
原始无用,又不方便移除写入,在输出时移除是个好办法

logstash
    mutate {
        remove_field => ["message","tags","@version","source"]
    }

gohangout
    - Remove:
        fields: ['message','tags','@version','source']

类型转换
把数字转换成int或float,方便在kinaba排序

logstash
    mutate {
        convert =>[
            "request_time","float",
            "status","integer"
        ]
    }
gohangout
    - Convert:
        fields:
            request_time:
                remove_if_fail: false
                setto_if_fail: 0.0
                to: float
            status:
                remove_if_fail: false
                setto_if_fail: 0
                to: int

小写转换
各种原因,sdk实现出来,竟然大小写混合的,此处统一处理成小写

logstash
    mutate {
        lowercase =>["product_line","app_name"] 
    }
gohangout
    - Lowercase:
        fields: ['product_line','app_name']

日期转换

logstash
    date { 
        match => ["timestamp", "yyyy-MM-dd HH:mm:ss.SSS"] 
        target => "@timestamp" 
    }

gohangout
    - Date:
        src: 'timestamp'
        target: '@timestamp'
        location: Asia/Shanghai
        overwrite: true
        formats:
            - '2006-01-02 15:04:05.999'

一般在kinaba都是以@timestamp作为时序索引,此处将日志的生成时间写入到@timestamp,方便,查询.
我用gohantout也就一个月不到,暂时先总结这么多.


相关博文

About rainbird

IOS攻城狮
This entry was posted in ELK and tagged , , , , , , , , , , . Bookmark the permalink.

发表评论