Getting a File Beat to Read in Data Again

Equally mentioned earlier:Super dry! Collect nginx logs through filebeat, logstash and rsyslog。 The filebeat used in this article is version 7.7.0. This article volition explain it from the following aspects:

Introduction to filebeat

Relationship between filebeat and beats

First, filebeat is a member of beats.

Beats is a lightweight log collector in. In fact, the beats family has six members. In the early on elk architecture, logstash was used to collect and parse logs, but logstash consumes high resources such as retentiveness, CPU and Io. Compared with logstash, beats occupies almost negligible CPU and memory of the system.

Beats currently contains vi tools:

• Packetbeat: network data (collect network traffic data)
• Metricbeat: Metrics (collect data such as CPU and memory usage at the system, process, and file system levels)
• Filebeat: log file (collect file data)
• Winlogbeat: Windows consequence log (collect windows result log information)
• Auditbeat: audit data (collect audit logs)
• Heartbeat: Runtime Monitoring (collect data during system functioning)

What is filebeat

Filebeat is a lightweight delivery tool for forwarding and centralizing log data. Filebeat monitors the log files or locations you specify, collects log events, and forwards them to elasticsearch or logstash for indexing.

Filebeat works as follows: when you offset filebeat, it will starting time i or more inputs that will be constitute in the location specified for log data. For each log found by filebeat, filebeat starts the collector. Each collector reads a unmarried log to get new content and sends the new log data to libbeat, which aggregates events and sends the aggregated data to the output configured for filebeat.

Catamenia chart of filebeat work

Relationship between filebeat and logstash

Because logstash is run past the JVM and consumes a lot of resource, the writer later on wrote a lightweight logstash forwarder with less functions merely less resource consumption. Just the writer is just a person, joinhttp://rubberband.co After the company, because es visitor has also acquired another open source project packetbeat, which is dedicated to golang and has a whole squad, ES company simply merged the development of logstash forwarder into the same golang team, then the new project is called filebeat.

Introduction to filebeat principle

Composition of filebeat

Filebeat structure: it is equanimous of 2 components, namely inputs and collectors. These components piece of work together to rail files and send event data to your specified output. Harvester is responsible for reading the contents of a single file. Harvester reads each file line by line and sends the contents to the output. Start a harvester for each file. Harvester is responsible for opening and endmost files, which means that the file descriptor remains open up when harvester is running. If you delete or rename a file while collecting it, filebeat volition proceed to read the file. The side outcome of this is that the infinite on the disk is reserved until the harvester is closed. By default, filebeat keeps the file open up until information technology reaches close_ inactive。

Closing harvester can produce the following results:

• The file handler closes. If the harvester is still deleted while reading the file, the underlying resource will exist released.
• Only in scan_ The drove of files will non be started again until the frequency is over.
• If the file is moved or deleted when harvester is closed, the collection of the file will not go along.

An input is responsible for managing harvesters and finding all source reads. If the input blazon is log, input volition find all files on the drive that match the divers path and start a harvester for each file. Each input runs in its own go process, and filebeat currently supports multiple input types. Each input type can be defined more than once. Log input checks each file to see if harvester needs to be started, if harvester is already running, or if the file tin exist ignored.

How does filebeat save the country of a file

Filebeat keeps the state of each file and oft flushes the land to the registry file on disk. This state is used to remember the last showtime read past the harvester and ensure that all log lines are sent. If the output cannot exist accessed (such as elasticsearch or logstash), filebeat tracks the last line sent and continues to read the file when the output is available again. When filebeat runs, the status data of each input is also saved in retentiveness. When filebeat restarts, the information from the registry file is used to rebuild the country, and filebeat continues each harvester at the last known location. For each input, filebeat retains the country of each file it finds. Because the file tin be renamed or moved, the file name and path are non sufficient to identify the file. For each file, filebeat stores a unique identifier to find whether the file was previously captured.

How can filebeat guarantee at to the lowest degree one data consumption

Filebeat guarantees that the event will be delivered to the configured output at least in one case without losing data. Considering it stores the delivery condition of each upshot in a registry file. When the defined output is blocked and all events are not acknowledged, filebeat will proceed to try to send events until the output acknowledges that the event has been received. If filebeat closes while sending events, information technology does not await for the output to acknowledge all events earlier endmost. When filebeat restarts, all events non best-selling before filebeat closes will be sent to the output again. This ensures that each event is sent at least once, but there may eventually be duplicate events sent to the output. Past setting shutdown_ With the timeout option, you can configure filebeat to expect a specific time before shutting down.

Filebeat installation

Compressed parcel installation

This article is installed in a compressed packet, Linux version, filebeat-7.7.0-linux-x86_ 64.tar.gz。

            curl-50-Ohttps://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.0-linux-x86_64.tar.gz tar -xzvf filebeat-7.vii.0-linux-x86_64.tar.gz                      

Configuration sample file: filebeat.reference.yml (including all configuration items that are not outdated)

Configuration file: filebeat.yml

Basic command

See the official website for details:https://world wide web.rubberband.co/guide/…

            consign    # export run       # Execute (default) test      # Examination configuration keystore  # Secret key storage modules   # Module configuration direction setup     # Set initial environment For example:. / filebeat examination   config  # Used to test whether the configuration file is correct          

Input and output

Supported input components:

            Multilinemessages，Azureeventhub，CloudFoundry，Container，Docker，GooglePub/Sub，HTTPJSON，Kafka，Log，MQTT，NetFlow，Office   three hundred and 60-five   Management   Activity   API, redis, S3, stdin, syslog, TCP, UDP (log is the most commonly used)          

Supported output components:

            Elasticsearch, logstash, Kafka, redis, file, panel, elasticcloud, changetheoutputcodec (the most unremarkably used is elasticsearch, logstash)          

Use of keystore

Keystore is mainly used to prevent sensitive information from being leaked, such equally passwords, such as ES passwords. Hither, a central can be generated as es_ PWD, a corresponding relationship of the countersign with the value of ES. When using the countersign of ES, ${es can exist used_ PWD} use.

            Create a keystore that stores passwords: filebeat keystore create Then add a fundamental value pair to it, for example: filebeatk eystore add es_ PWD Use to overwrite the value of the original cardinal: filebeat cardinal shop add es_ PWD–force Delete key value pairs: filebeat central store remove es_ PWD View existing key value pairs: filebeat key store listing          

For example, you lot can use ${es later_ PWD} uses its value, for example:

            output.elasticsearch.password:"${ES_PWD}"                      

Filebeat.yml configuration (log input type as an example)

Come across the official website for details:https://www.elastic.co/guide/…

            type:   log  # The input blazon is log enable:   true  # Indicates that the log blazon configuration is effective paths：      # Specify the log to be monitored, which is currently processed according to the glob function of go language. The configuration directory is not recursively processed. For example, if the configuration directory is: -  / var/log/*  /*. log   # It volition only look for files ending in ". Log" in all subdirectories of / var / log directory, non files ending in ". Log" in / var / log directory. recursive_ glob.enabled:  # Enable global recursive mode, such as / foo / * * including / foo,  / foo/*,  / foo/*/* Encoding: # specify the encoding type of the monitored file. Both patently and UTF-8 tin handle Chinese logs exclude_ lines:  ['^ DBG']  # Does not contain rows that match regular include_ lines:  ['^ ERR',  '^ WARN']   # Contains rows that match regular harvester_ buffer_ size:   sixteen 1000 three hundred and fourscore-4  # The byte size of the buffer used past each harvester to get the file max_ bytes:   ten million four hundred and eighty-five one thousand seven hundred and lx  # The maximum number of bytes a single log message can have. max_ All bytes after bytes are discarded and not sent. The default value is 10MB (10485760) exclude_ files:  ['. gz$']   # List of regular expressions used to lucifer files that you want filebeat to ignore ingore_ older:   0  # The default value is 0, which means information technology is disabled. 2h, 2m, etc. can be configured. Note ignore_ Older must be greater than close_ The value of inactive. It means to ignore those that are non updated beyond the set value Files or files take never been nerveless by harvester close_*  # close_  * The configuration option is used to turn off the harvester after a specific standard or time.   Closing harvester ways closing the file handler.   If harvester is closed Later the file is updated, it is in scan_ After frequency, the file will be picked up again.   However, if y'all move or delete files while harvester is closed, filebeat will not be able to receive files again , and whatsoever data not read by the harvester will be lost. close_ inactive   # When the selection is started, if it is non read at the specified fourth dimension, the file handle will be closed The last log read is defined as the starting point of the side by side read, non based on the modification fourth dimension of the file If the closed file changes, a new harverster will be in scan_ Frequency is started after running It is recommended to set at least one value greater than the log reading frequency, and configure multiple prospectors to realize log files with unlike update speeds The internal timestamp mechanism is used to reverberate the reading of the tape log. Each fourth dimension the last line of log is read, the countdown starts, and information technology is used for 2h   5m   To represent close_ rename  # When the option is enabled, if the file is renamed and moved, filebeat turns off the processing and reading of the file close_ removed  # When the option is enabled and the file is deleted, filebeat closes the file processing. Subsequently reading this selection, clean must exist started_ removed close_ eof  # It is suitable for files that write logs only in one case, and so filebeat turns off the processing and reading of files close_ timeout  # When the pick is enabled, filebeat will fix a predefined fourth dimension for each harvester. No matter whether the file is read or not, it volition be airtight afterwards reaching the set time close_ timeout   Cannot be equal to ignore_ Older will cause the file to not be read when information technology is updated. If the output does non output log events, this timeout will non exist started, At least one event must be sent, and and so the haverter will be closed Set 0   Indicates not to start clean_ inactived  # Removes the status of previously harvested files from the registry file The setting must be greater than ignore_ older+scan_ Frequency to ensure that no condition is deleted while the file is still existence collected Configuration options help reduce the size of registry files, especially if a large number of new files are generated every day This configuration pick can too be used to preclude the filebeat problem of reusing inodes on Linux clean_ removed  # After the startup option, if the file cannot be found on deejay, filebeat volition exist cleared from the registry If shut is closed   removed   Clean must be closed   removed scan_ frequency  # The frequency at which the prospector checks new files in the path specified for harvesting. The default is 10s tail_ Files: # if it is set to true, filebeat monitors the new contents of the file from the cease of the file, and sends each new line of file equally an event in turn, Instead of resending everything from the beginning of the file. Symlinks: # symbolic links option allows filebeat to collect symbolic links in improver to regular files. When collecting symbolic links, even if the path of the symbolic link is reported, Filebeat also opens and reads the original file. backoff：  # The backoff option specifies how filebeat actively grabs new files for updates. By default 1s, the backoff selection defines filebeat after reaching EOF Bank check the waiting time between files over again. max_ backoff：  # The maximum time filebeat waits before checking the file again later on reaching EOF backoff_ cistron：  # Specify the number of backoff attempts to wait. The default is 2 harvester_ limit：#harvester_ The limit selection limits the number of harvesters started in parallel by a prospector, which directly affects the number of files opened tags  # Add together tags to the list and filter them, for case: Tags:  [" json"] fields  # Optional fields. Select additional fields for output, which can be nested types such as scalar value, tuple, dictionary, etc The default is in the sub dictionary position filebeat.inputs: fields: app_id: query_engine_12 fields_ under_ root  # If the value is true, fields are stored at the top of the output document multiline.pattern  # Must lucifer regexp pattern multiline.negate  # The action to define the above pattern matching criteria is   No, the default is false If the pattern matching status '^ B', the default is false pattern, which means that matching is carried out according to design matching   Merge log lines that exercise not outset with B If true, the log lines starting with B volition not be merged multiline.match  #  Specifies how filebeat combines matching rows into events, earlier or after, depending on the negate specified above multiline.max_ lines  # The maximum number of rows that can exist combined into one event. If information technology exceeds, it will be discarded. The default is 500 multiline.timeout  # Define the timeout. If a new consequence is started and no match is establish within the timeout, the log will likewise be sent. The default is 5S max_ procs  # Sets the maximum number of CPUs that can be executed simultaneously. The default value is the number of logical CPUs available in the arrangement. name  # Specify a name for the filebeat. The default is the hostname of the host          

Example one: logstash equally output

Filebeat.yml configuration:

            #=========================== Filebeat inputs ============================= filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can employ different inputs for various configurations. # Beneath are the input specific configurations. - blazon: log   # Alter to truthful to enable this input configuration.   enabled: truthful   # Paths that should be crawled and fetched. Glob based paths.   paths:   # Configure multiple log paths     -/var/logs/es_aaa_index_search_slowlog.log     -/var/logs/es_bbb_index_search_slowlog.log     -/var/logs/es_ccc_index_search_slowlog.log     -/var/logs/es_ddd_index_search_slowlog.log     #- c:programdataelasticsearchlogs*   # Exclude lines. A list of regular expressions to match. It drops the lines that are   # matching whatsoever regular expression from the list.   #exclude_lines: ['^DBG']   # Include lines. A list of regular expressions to match. It exports the lines that are   # matching whatsoever regular expression from the listing.   #include_lines: ['^ERR', '^WARN']   # Exclude files. A list of regular expressions to match. Filebeat drops the files that   # are matching any regular expression from the list. By default, no files are dropped.   #exclude_files: ['.gz$']   # Optional additional fields. These fields tin be freely picked   # to add additional information to the crawled log files for filtering   #fields:   #  level: debug   #  review: one   ### Multiline options   # Multiline can be used for log messages spanning multiple lines. This is common   # for Java Stack Traces or C-Line Continuation   # The regexp Design that has to exist matched. The example pattern matches all lines starting with [   #multiline.design: ^[   # Defines if the pattern ready nether design should exist negated or not. Default is simulated.   #multiline.negate: false   # Match tin be set to "after" or "before". It is used to define if lines should exist append to a pattern   # that was (non) matched before or afterward or as long as a pattern is non matched based on negate.   # Notation: Subsequently is the equivalent to previous and before is the equivalent to to next in Logstash   #multiline.match: after #================================ Outputs ===================================== #----------------------------- Logstash output -------------------------------- output.logstash:   #   The   Logstash   hosts  # Use load balancing mechanism with multiple logstash   hosts: ["192.168.110.130:5044","192.168.110.131:5044","192.168.110.132:5044","192.168.110.133:5044"]     loadbalance:   true   # Load balancing is used   # Optional SSL. By default is off.   # List of root certificates for HTTPS server verifications   #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]   # Certificate for SSL client authentication   #ssl.document: "/etc/pki/client/cert.pem"   # Client Certificate Key   #ssl.cardinal: "/etc/pki/client/cert.key"          

            ./filebeat  - due east  # Start filebeat          

Configuration of logstash

            input {   beats {     port => 5044      } } output {   elasticsearch {     hosts  => [" http://192.168.110.130:9200 "]  # Multiple can be configured hither     index => "query-%{yyyyMMdd}"    } }          

Example 2: elasticsearch equally output

Configuration of filebeat.yml:

            ###################### Filebeat Configuration Example ######################### # This file is an example configuration file highlighting merely the nigh mutual # options. The filebeat.reference.yml file from the same directory contains all the # supported options with more comments. You tin can apply it as a reference. # # You can find the full configuration reference here: # https://www.elastic.co/guide/en/beats/filebeat/alphabetize.html # For more available modules and options, delight see the filebeat.reference.yml sample # configuration file. #=========================== Filebeat inputs ============================= filebeat.inputs: # Each - is an input. About options can be set at the input level, so # y'all can use different inputs for various configurations. # Beneath are the input specific configurations. - type: log   # Change to true to enable this input configuration.   enabled: true   # Paths that should be crawled and fetched. Glob based paths.   paths:     -/var/logs/es_aaa_index_search_slowlog.log     -/var/logs/es_bbb_index_search_slowlog.log     -/var/logs/es_ccc_index_search_slowlog.log     -/var/logs/es_dddd_index_search_slowlog.log     #- c:programdataelasticsearchlogs*   # Exclude lines. A list of regular expressions to friction match. It drops the lines that are   # matching any regular expression from the list.   #exclude_lines: ['^DBG']   # Include lines. A list of regular expressions to friction match. Information technology exports the lines that are   # matching any regular expression from the list.   #include_lines: ['^ERR', '^WARN']   # Exclude files. A list of regular expressions to match. Filebeat drops the files that   # are matching whatever regular expression from the list. Past default, no files are dropped.   #exclude_files: ['.gz$']   # Optional additional fields. These fields can be freely picked   # to add additional information to the crawled log files for filtering   #fields:   #  level: debug   #  review: 1   ### Multiline options   # Multiline can exist used for log messages spanning multiple lines. This is common   # for Java Stack Traces or C-Line Continuation   # The regexp Pattern that has to be matched. The example blueprint matches all lines starting with [   #multiline.blueprint: ^[   # Defines if the pattern gear up nether pattern should be negated or non. Default is false.   #multiline.negate: false   # Match can be fix to "after" or "earlier". It is used to define if lines should be append to a pattern   # that was (not) matched before or after or as long every bit a pattern is not matched based on negate.   # Notation: After is the equivalent to previous and before is the equivalent to to side by side in Logstash   #multiline.lucifer: after #============================= Filebeat modules =============================== filebeat.config.modules:   # Glob blueprint for configuration loading   path: ${path.config}/modules.d/*.yml   # Set to truthful to enable config reloading   reload.enabled: false   # Period on which files nether path should be checked for changes   #reload.period: 10s #==================== Elasticsearch template setting ========================== #================================ General ===================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. name: filebeat222 # The tags of the shipper are included in their own field with each # transaction published. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add together additional information to the # output. #fields: #  env: staging #cloud.auth: #================================ Outputs ===================================== #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch:   # Array of hosts to connect to.   hosts: ["192.168.110.130:9200","92.168.110.131:9200"]   # Protocol - either `http` (default) or `https`.   #protocol: "https"   # Authentication credentials - either API key or username/password.   #api_key: "id:api_key"   username: "elastic"   countersign:  "${ ES_ PWD}"    # Set up password through keystore          

            ./filebeat  - e  # Starting time filebeat          

Check the elasticsearch cluster. There is a default index name filebeat -% {[beat. Version]} -% {+ yyyy. Mm. DD}

Filebeat module

Official website:https://www.elastic.co/guide/…

Hither, I apply elasticsearch fashion to parse the slow log query of ES. The operation steps are every bit follows, and the operations of other modules are the same:

Premise: install elasticsearch and kibana software, so use filebeat.

Specific operation official websites include:https://www.elastic.co/guide/…

Stride 1: configure the filebeat.yml file:

            #============================== Kibana ===================================== # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. # This requires a Kibana endpoint configuration. setup.kibana:   # Kibana Host   # Scheme and port can be left out and will be set to the default (http and 5601)   # In case y'all specify and additional path, the scheme is required: http://localhost:5601/path   # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601   host:  " 192.168.110.130:5601"   # Specify kibana   username:  " elastic"    # user   countersign:  "${ ES_ PWD}"   # Password. Keystore is used here to prevent plaintext password   # Kibana Space ID   # ID of the Kibana Space into which the dashboards should exist loaded. By default,   # the Default Space volition be used.   #space.id: #================================ Outputs ===================================== # Configure what output to apply when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch:   # Assortment of hosts to connect to.   hosts: ["192.168.110.130:9200","192.168.110.131:9200"]   # Protocol - either `http` (default) or `https`.   #protocol: "https"   # Hallmark credentials - either API key or username/password.   #api_key: "id:api_key"   username:  " rubberband"   # Users of ES   password:  "${ ES_ PWD}"  #  Password for ES   #Index cannot exist specified here because I accept non configured the template. An index named filebeat -% {[vanquish. Version]} -% {+ yyyy. Mm. DD} volition exist generated automatically          

Footstep ii: configure the slow log path of elasticsearch:

            cd filebeat-vii.7.0-linux-x86_64/modules.d vim elasticsearch.yml：                      

Step 3: effective es module:

            ./filebeat modules elasticsearch View effective modules: ./filebeat modules listing          

Step four: initialize the environs:

            ./filebeat setup -e                      

Step 5: start filebeat:

            ./filebeat -e                      

Check the elasticsearch cluster, every bit shown in the following figure, and automatically resolve the logs of deadening log query:

At this betoken, the experiment of elasticsearch module is successful.

Author: Yicun Hui
Original text:https://www.cnblogs.com/zsql/…

glasstakis1955.blogspot.com

Source: https://developpaper.com/its-not-difficult-to-understand-filebeat-a-sharp-tool-for-log-collection/

Share this post