tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Cloud Log Service
    Documentation
    Download PDF

    LogListener Limits

    Last updated: 2022-05-07 11:20:50
    Download PDF

      This document describes LogListener's capabilities and limits of file collection, configuration, resources, performance, and error handling related to data collection.

      File Collection Capabilities and Limits

      Item Capability and Limit
      File encoding Supports UTF8 and GBK codec.
      Note: only LogListener v2.6.2 and above support GBK codec.
      Soft link Supported.
      Size of a single log The size of a single-line log is limited to 512 KB. If the size of a single-line log exceeds 512 KB, the log will be truncated and only the first 512 KB is retained. For a multi-line log, after it is divided according to the first-line regular expression, the maximum limit for a single log is 1 MB.
      Regular expression Perl-compatible regular expressions are supported.
      First log collection behavior LogListener supports full/incremental collection policies:
      • Full collection: after LogListener is started for the first time after installation, all logs that meet requirements are collected, including files that have not been written.
      • Incremental collection: after LogListener is started for the first time after installation, existing files will be collected from the latest position.
      Note: Full/Incremental collection policies are available starting from LogListener v2.6.2.
      Log file rotation Supported. (It is recommended that the file name after rotation not be matched by the wildcard collection path.)
      Collection behavior when log parsing fails We recommend you enable the feature of uploading parsing-failure logs. If the feature is enabled, parsing-failure logs will be uploaded to the preset index in the format of full text in a single line. Otherwise, the logs will be discarded.
      File opening LogListener opens files when reading files for log collection and closes the files when the reading is completed.

      Checkpoint Management Capabilities and Limits

      Item Capability and Limit
      Checkpoint storage location The default save path is the data directory in the LogListener installation directory. You can customize a save path in the loglistener.conf file.
      Checkpoint storage policy LogListener stores two copies of checkpoint metadata:
      • One copy records the information only of checkpoints where upload is completed, and the information is persisted to a disk in real time.
      • The other copy records the information of checkpoints where upload is completed and checkpoints where upload is not completed, and the information is periodically persisted to a disk. Persistence takes precedence when the program exits.

      Resource and Performance Capabilities and Limits

      Item Capability and Limit
      Default CPU resource limits LogListener does not limit CPU resources by default. You can configure CPU resource limits in the configuration file.
      Without CPU resource limits, the current implementation architecture of LogListener can achieve a maximum CPU usage of 110% (up to 100% for a single business thread and about 10% for a control thread).
      Default memory resource limits LogListener's default memory threshold is 2 GB. You can change the threshold to 300 MB or higher as needed.
      Default bandwidth resource limits LogListener does not limit bandwidth resources by default. You can configure bandwidth resource limits in the loglistener.conf file.
      Resource overrun handling policy If LogListener overruns CPU and memory resources for more than 5 minutes, LogListener will be forced to restart automatically.
      Log compression Collected logs are shipped as compact logs by default. You can modify log compression configuration in the loglistener.conf file.
      Number of monitored directories The recommended maximum number of monitored directories is 5,000. If this limit is exceeded, log collection may fail.
      Number of monitored files The recommended maximum number of monitored files is 10,000. If this limit is exceeded, log collection may fail.

      Error Handling

      Item Capability and Limit
      Network error handling Except exceptions requiring special handling (such as log topic deletion), all other errors (such as network exceptions, timeout, frequency control, and overdue payment) will be retried.
      Maximum retry timeout duration If data fails to be sent after retrying for more than 1 hour, the system discards the data.
      The default behavior is to retry at an interval, and the retry interval becomes longer and longer until the maximum retry timeout duration is exceeded.
      Maximum number of retries The maximum number of retries can be set in the loglistener.conf configuration file and is not set by default. By default, the system retries until the maximum retry duration is exceeded, and then discards the data.
      If the maximum number of retries is set, the system retries until the maximum number of retries is exceeded, and then discards the data.

      File Collection Rules

      Item Capability and Limit
      Log upload policy LogListener automatically aggregates and uploads logs of the same file when any of the following aggregation conditions is met: 10,000 logs, the total logset size reaching 1 MB, or the log collection time exceeding 3 seconds.
      File collection handling policy A single target file (a file that can be matched by the collection path) can be uploaded to only one log topic. The same file cannot be matched by the collection paths of multiple topics.
      If you need to upload a file to multiple topics, use soft links. You can create different soft links for the same target file so that it can be collected to different topics via different soft links.
      Log collection delay In the case of real-time collection, data collection, transmission, and storage to disks will be completed within 1 minute, and the logs are retrievable in the console.
      If the log volume is large, or LogListener can use only a limited amount of resources, there will be some collection delay.
      Item Capability and Limit
      Machine group logic At present, machine groups are divided into the following two categories, and their usage methods are independent of and incompatible with each other. If the two usage methods are used together, LogListener will not be able to pull the correct collection configuration, resulting in no collection.
      1. IP machine group: a machine IP must be manually added to the machine group in the console, and the group_label field in loglistener.conf on the corresponding machine must be empty.
      2. Label machine group: the machine group label is set in the console, and the group_label field in loglistener.conf on the corresponding machine must be set to the same label.
      Relationships between machine groups and log topics
      • A single log topic can be bound to multiple machine groups.
      • A single machine group can be bound to multiple log topics.
      Relationships between machine groups and machines
      • A single machine can be added to multiple machine groups.
      • For an IP machine group, there is no limit to the number of machine groups that a machine can join.
      • For a label machine group, the maximum number of machine groups that a machine can join is 20.
      Limits on the labels of label machine groups
      • Currently, the label of a label machine group can contain up to 32 characters.
      • Up to 20 labels can be configured for a single label machine group.

      Collection Path/Collection Blocklist Usage

      Item Capability and Limit
      Collection blocklist This feature is used to specify the content to ignore in the collection path. Currently, collection blocklists support two modes:
      • FILE mode: specify the files to ignore in the collection path.
      • PATH mode: specify the subdirectories to ignore in the collection path. Wildcard filtering is supported.
      Notes:
      • The FILE and PATH modes can be used together.
      • A collection blocklist is to exclude content from the collection path, and for both the FILE and PATH modes, the specified path to exclude must be a subset of the collection path.
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      tencent
      Copyright © 2013-2022 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy