[hadoop@HadoopMaster filebeat-1.3.1-x86_64]$ pwd
/home/hadoop/app/filebeat-1.3.1-x86_64
[hadoop@HadoopMaster filebeat-1.3.1-x86_64]$ ll
total 11116
-rwxr-xr-x. 1 hadoop hadoop 11354200 Sep 15  2016 filebeat
-rw-r--r--. 1 hadoop hadoop      814 Sep 15  plate.json
-rw-r--r--. 1 hadoop hadoop    17212 Sep 15  l
[hadoop@HadoopMaster filebeat-1.3.1-x86_64]$ pwd正则匹配多行
/home/hadoop/app/filebeat-1.3.1-x86_64
[hadoop@HadoopMaster filebeat-1.3.1-x86_64]$ l
>>>#### Filebeat Configuration Example >>>>>
>>>>>#### Filebeat >>>>>>>###
filebeat:
# List of prospectors to fetch data.
prospectors:
# Each - is a prospector. Below are the prospector specific configurations
-
# Paths that should be crawled and fetched. Glob based paths.
# To fetch all ".log" files from a specific level of subdirectories
# /var/log/*/*.log can be used.
# For each file found under this path, a harvester is started.
# Make sure not file is defined twice as this can lead to unexpected behaviour.
# 指定要监控的⽇志，可以指定具体得⽂件或者⽬录
paths:
- /var/log/*.log  （这是默认的）（⾃⾏可以修改）(⽐如我放在/home/hadoop/app.log⾥）
#- c:\programdata\elasticsearch\logs\*
# Configure the file encoding for reading files with international characters
# following the W3C recommendation for HTML5 (/TR/encoding).
# Some sample encodings:
#  plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
#    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
# 指定被监控的⽂件的编码类型，使⽤plain和utf-8都是可以处理中⽂⽇志的
#encoding: plain
# Type of the files. Based on this the way the file is read is decided.
# The different types cannot be mixed in one prospector
#
# Possible options are:
# * log: Reads every line of the log file (default)
# * stdin: Reads the standard in
# 指定⽂件的输⼊类型log(默认)或者stdin
input_type: log
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list. The include_lines is called before
# 在输⼊中排除符合正则表达式列表的那些⾏。
# exclude_lines. By default, no lines are dropped.
# exclude_lines: ["^DBG"]
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list. The include_lines is called before
# exclude_lines. By default, all the lines are exported.
# 包含输⼊中符合正则表达式列表的那些⾏（默认包含所有⾏），include_lines执⾏完毕之后会执⾏exclude_lines      # include_lines: ["^ERR", "^WARN"]
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
# 忽略掉符合正则表达式列表的⽂件
# exclude_files: [".gz$"]
# Optional additional fields. These field can be freely picked
# to add additional information to the crawled log files for filtering
# 向输出的每⼀条⽇志添加额外的信息，⽐如“level:debug”，⽅便后续对⽇志进⾏分组统计。
# 默认情况下，会在输出信息的fields⼦⽬录下以指定的新增fields建⽴⼦⽬录，例如fields.level
# 这个得意思就是会在es中多添加⼀个字段，格式为 "filelds":{"level":"debug"}
#fields:
#  level: debug
#  review: 1
# Set to true to store the additional fields as top level fields instead
# of under the "fields" sub-dictionary. In case of name conflicts with the
# fields added by Filebeat itself, the custom fields overwrite the default
# fields.
# 如果该选项设置为true，则新增fields成为顶级⽬录，⽽不是将其放在fields⽬录下。
# ⾃定义的field会覆盖filebeat默认的field
# 如果设置为true，则在es中新增的字段格式为："level":"debug"
#fields_under_root: false
# Ignore files which were modified more then the defined timespan in the past.
# In case all files on your system must be read you can set this value very large.
# Time strings like 2h (2 hours), 5m (5 minutes) can be used.
# 可以指定Filebeat忽略指定时间段以外修改的⽇志内容，⽐如2h（两个⼩时）或者5m(5分钟)。
#ignore_older: 0
# Close older closes the file handler for which were not modified
# for longer then close_older
# Time strings like 2h (2 hours), 5m (5 minutes) can be used.
# 如果⼀个⽂件在某个时间段内没有发⽣过更新，则关闭监控的⽂件handle。默认1h
#close_older: 1h
# Type to be published in the 'type' field. For Elasticsearch output,
# the type defines the document type these entries should be stored
# in. Default: log
# 设定Elasticsearch输出时的document的type字段可以⽤来给⽇志进⾏分类。Default: log
#document_type: log
# Scan frequency in seconds.
# How often these files should be checked for changes. In case it is set
# to 0s, it is done as often as possible. Default: 10s
# Filebeat以多快的频率去prospector指定的⽬录下⾯检测⽂件更新（⽐如是否有新增⽂件）
# 如果设置为0s，则Filebeat会尽可能快地感知更新（占⽤的CPU会变⾼）。默认是10s
#scan_frequency: 10s
# Defines the buffer size every harvester uses when fetching the file
# 每个harvester监控⽂件时，使⽤的buffer的⼤⼩
#harvester_buffer_size: 16384
# Maximum number of bytes a single log event can have
# All bytes after max_bytes are discarded and not sent. The default is 10MB.
# This is especially useful for multiline log messages which can get large.
# ⽇志⽂件中增加⼀⾏算⼀个⽇志事件，max_bytes限制在⼀次⽇志事件中最多上传的字节数，多出的字节会被丢弃      #max_bytes: 10485760
# Mutiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# 适⽤于⽇志中每⼀条⽇志占据多⾏的情况，⽐如各种语⾔的报错信息调⽤栈
#multiline:
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
# 多⾏⽇志开始的那⼀⾏匹配的pattern
#pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
# 是否需要对pattern条件转置使⽤，不翻转设为true，反转设置为false。【建议设置为true】
#negate: false
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
# 匹配pattern后，与前⾯（before）还是后⾯（after）的内容合并为⼀条⽇志
#match: after
# The maximum number of lines that are combined to one event.
# In case there are more the max_lines the additional lines are discarded.
# Default is 500
# 合并的最多⾏数（包含匹配pattern的那⼀⾏）
#max_lines: 500
# After the defined timeout, an multiline event is sent even if no new pattern was found to start a new event
# Default is 5s.
# 到了timeout之后，即使没有匹配⼀个新的pattern（发⽣⼀个新的事件），也把已经匹配的⽇志事件发送出去
#timeout: 5s
# Setting tail_files to true means filebeat starts readding new files at the end
# instead of the beginning. If this is used in combination with log rotation
# this can mean that the first entries of a new file are skipped.
# 如果设置为true，Filebeat从⽂件尾开始监控⽂件新增内容，把新增的每⼀⾏⽂件作为⼀个事件依次发送，
# ⽽不是从⽂件开始处重新发送所有内容
#tail_files: false
# Backoff values define how agressively filebeat crawls new files for updates
# The default values can be used in most cases. Backoff defines how long it is waited
# to check a file again after EOF is reached. Default is 1s which means the file
# is checked every second if new lines were added. This leads to a near real time crawling.
# Every time a new line appears, backoff is reset to the initial value.
# Filebeat检测到某个⽂件到了EOF（⽂件结尾）之后，每次等待多久再去检测⽂件是否有更新，默认为1s
#backoff: 1s
# Max backoff defines what the maximum backoff time is. After having backed off multiple times
# from checking the files, the waiting time will never exceed max_backoff idenependent of the
# backoff factor. Having it set to 10s means in the worst case a new line can be added to a log
# file after having backed off multiple times, it takes a maximum of 10s to read the new line
# Filebeat检测到某个⽂件到了EOF之后，等待检测⽂件更新的最⼤时间，默认是10秒
#max_backoff: 10s
# The backoff factor defines how fast the algorithm backs off. The bigger the backoff factor,
# the faster the max_backoff value is reached. If this value is set to 1, no backoff will happen.
# The backoff value will be multiplied each time with the backoff_factor until max_backoff is reached
# 定义到达max_backoff的速度，默认因⼦是2，到达max_backoff后，变成每次等待max_backoff那么长的时间才backoff⼀次，
# 直到⽂件有更新才会重置为backoff
# 根据现在的默认配置是这样的，每隔1s检测⼀下⽂件变化，如果连续检测两次之后⽂件还没有变化，下⼀次检测间隔时间变为10s
#backoff_factor: 2
# This option closes a file, as soon as the file name changes.
# This config option is recommended on windows only. Filebeat keeps the files it's reading open. This can cause
# issues when the file is removed, as the file will not be fully removed until also Filebeat closes
# the reading. Filebeat closes the file handler after ignore_older. During this time no new file with the
# same name can be created. Turning this feature on the other hand can lead to loss of data
# on rotate files. It can happen that after file rotation the beginning of the new
# file is skipped, as the reading starts at the end. We recommend to leave this option on false
# but lower the ignore_older value to release files faster.
# 这个选项关闭⼀个⽂件，当⽂件名称的变化。#该配置选项建议只在windows
#force_close_files: false
# Additional prospector
#-
# Configuration to use stdin input
#input_type: stdin
# General filebeat configuration options
#
# Event count spool threshold - forces network flush if exceeded
# spooler的⼤⼩，spooler中的事件数量超过这个阈值的时候会清空发送出去（不论是否到达超时时间）
#spool_size: 2048
# Enable async publisher pipeline in filebeat (Experimental!)
# 是否采⽤异步发送模式（实验功能）
#publish_async: false
# Defines how often the spooler is flushed. After idle_timeout the spooler is
# Flush even though spool_size is not reached.
# spooler的超时时间，如果到了超时时间，spooler也会清空发送出去（不论是否到达容量的阈值）
#idle_timeout: 5s
# Name of the registry file. Per default it is put in the current working
# directory. In case the working directory is changed after when running
# filebeat again, indexing starts from the beginning again.
# 记录filebeat处理⽇志⽂件的位置的⽂件，默认是在启动的根⽬录下
#registry_file: .filebeat
# Full Path to directory with additional prospector configuration files. Each file must end with .yml
# These config files must have the full filebeat config part inside, but only
# the prospector part is processed. All global options like spool_size are ignored.
# The config_dir MUST point to a different directory then where the main filebeat config file is in.
# 如果要在本配置⽂件中引⼊其他位置的配置⽂件，可以写在这⾥（需要写完整路径），但是只处理prospector的部分
#config_dir:
>>>>>>>>>>>>>>>####
>>>>>#### Libbeat Config >>>>>>####
# Base config file used by all other beats for using libbeat features
>>>>>#### Output >>>>>>>>##
# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:
### Elasticsearch as output
elasticsearch:            （这是默认的，filebeat收集后放到es⾥）（⾃⾏可以修改，⽐如我有时候想filebeat收集后，然后到redis，再到es，就可以注销这⾏）
# Array of hosts to connect to.
# Scheme and port can be left out and will be set to the default (http and 9200)
# In case you specify and additional path, the scheme is required: localhost:9200/path
# IPv6 addresses should always be defined as: [2001:db8::1]:9200
hosts: ["localhost:9200"]        （这是默认的，filebeat收集后放到es⾥）（⾃⾏可以修改，⽐如我有时候想filebeat收集后，然后到redis，再到es，就可以注销这⾏）
# Optional protocol and basic auth credentials. #protocol: "https" #username: "admin" #password: "s3cr3t" # Number of workers per Elasticsearch host. #worker: 1 # Optional index name. The default is "filebeat" and generates # [filebeat-]YYYY.M

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。

gawk 语法
« 上一篇
linux 根据多个关键字 匹配行写入新文件的方法
下一篇 »

发表评论

Copyright © 2024 baidu.com
Powered By 688IT编程网 豫ICP备2022007602号 豫公网安备41160202000602