PrevNext
search by keyword

Recent Pages

Documentation
Cloud Log Service
    Documentation
    Download PDF

    LogListener Service Logs

    Last updated: 2021-07-20 11:57:20
    Download PDF

      Overview

      LogListener service logs are used to record the operation, collection, and monitoring activities of LogListener, and you can configure visualized graphs to display such log data.

      Default configuration

      Default Configuration Item Description
      Log Topic When you enable LogListener service logs, the logset cls_service_logging will be created for you automatically, and all log data generated by associated machine groups will be categorized and stored in corresponding log topics. The following three log topics are created by default:
    • loglistener_status: the heartbeat status logs for the corresponding LogListener.
    • loglistener_alarm: logs corresponding to LogListener's monitoring by collection metric/error type .
    • loglistener_business: logs corresponding to LogListener's collection operations, with each log corresponds to one request.
      • Region After the LogListener log services are enabled, logsets and log topics will be created under the machine groups in the same region of LogListener.
      Log Storage Duration The default storage duration is 7 days, and the value cannot be modified.
      Index Full-text index and key-value index are enabled for all collected log data by default. You can modify the index configuration. For details, please see Configuring Index.
      Dashboard Dashboard service_log_dashboard will be created in the same region of LogListener by default.
      Note：

      • LogListener service logs are only about collection and monitoring activities of LogListener, and do not support writing into other kinds of data.
      • Log data generated by this feature does not incur costs.
      • cls_service_logging is a unified logset for LogListener service logs.

      Operations

      • Viewing LogListener status
        When the LogListener service logs are enabled, you can view LogListener running status and collection statistics. You can view the number of active LogListeners, LogListener status distribution and other statistical metrics on the service_log_dashboard dashboard.

      • Log collection monitoring configuration
        You can configure the collection and monitoring service logs by metric/error type. For example:

        • MEM, CPU, collection speed, collection latency, or other metrics
        • The number of parsing errors of LogListener

      • File-level monitoring
        With the LogListener service logs are enabled, you can view the monitoring logs of files and directories. For example:

        • All collection statistics files of one IP
        • The amount of logs collected under a certain path on a certain IP, such as app1 application logs located in /var/log/app1/. You can get the statistics of logs collected under this path.
        • The collection statistics of a topic.

      Prerequisites

      Only LogListener v2.5.4 and above support collection and monitoring service logs by machine/machine group. You’re advised to upgrade LogListener to the latest version.

      Directions

      Enabling the service logs

      1. Log in to the CLS console.
      2. In the left sidebar, click Machine Group to go to the machine group management page.
      3. On the machine group list page, select the target machine group and click to enable the LogListener service logs.

      Disabling LogListener service logs

      1. Log in to the CLS console.
      2. In the left sidebar, click Machine Group to view the machine group list.
      3. Select the target machine group, and click to disable the LogListener service logs.
      Note：

      After this feature is disabled, the log data saved in the logset cls_service_logging will not be deleted automatically. You can manually delete the logset where the service logs are saved.

      Dashboard of Service Logs

      When the LogListener service logs are enabled, CLS will create a dashboard search_log_dashboard by the type of recorded logs to display LogListener’s collection and monitoring statistics.

      Collection statistics dashboard

      You can go to the Dashboard page of the CLS console, click the ID of the target dashboard to view LogListener collection statistics, including its status, parsing failure rate, sending success rate, and other metrics.

      Log Types

      LogListener status logs

      The parameters of the log topic loglistener_status are detailed as follows:

      ParameterDescription
      InstanceIdLogListener unique identifier
      IPMachine group IP
      LabelAn array of machine IDs
      VersionVersion number
      MemoryUsedMemory utilization of LogListener
      MemMaxMemory utilization threshold on this machine set by the Agent
      CpuUsageLogListener CPU utilization
      StatusLogListener running status
      TotalSendLogSizeSize of logs sent
      SendSuccessLogSizeSize of successfully sent logs
      SendFailureLogSizeSize of sending-failed logs
      SendTimeoutLogSizeSize of logs with sending timed out
      TotalParseLogCountTotal number of logs parsed
      ParseFailureLogCountNumber of parsing-failed logs
      TotalSendLogCountNumber of logs sent
      SendSuccessLogCountNumber of successful sent logs
      SendFailureLogCountNumber of sending-failed logs
      SendTimeoutLogCountNumber of logs with sending timed out
      TotalSendReqsTotal number of requests sent
      SendSuccessReqsNumber of successful sent requests
      SendFailureReqsThe number of sending-failed requests
      SendTimeoutReqsNumber of requests with sending timed out
      TotalFinishRspsTotal number of RSP files received
      TotalSuccessFromStartTotal number of successfully sent requests since LogListener was enabled
      AvgReqSizeAverage request packet size
      SendAvgCostAverage sending time
      AvailConnNumNumber of available connections
      QueueSizeThe size of queued requests

      LogListener alarm logs

      The parameters of the log topic loglistener_alarm are detailed as follows:

      Monitoring Metric Description
      InstanceId LogListener unique identifier
      Label An array of machine IDs
      IP Machine group IP
      Version LogListener version
      AlarmType.count Statistics of alarm types
      AlarmType.example Sample alarm type

      AlarmType:

      alarm type type ID Description
      UnknownError 0 Initializing the alarm type.
      UnknownError 1 Failed to parse.
      CredInvalid 2 Failed to verify.
      SendFailure 3 Failed to send.
      RunException 4 Abnormal LogListener running.
      MemLimited 5 Reached the memory utilization threshold.
      FileProcException 6 Exceptions occurred in file processing.
      FilePosGetError 7 Failed to get the file publishing info.
      HostIpException 8 Exceptions occurred in the server IP thread.
      StatException 9 Failed to get the process info.
      UpdateException 10 Exceptions occurred in the CLS modification feature.
      DoSendError 11 Failed to confirm sending.
      FileAddError 12 Failed to create the file.
      FileMetaError 13 Failed to create the metadata file.
      FileOpenError 14 Failed to open the file.
      FileOpenError 15 Failed to read the file.
      FileStatError 16 Failed to get the file status.
      getTimeError 17 Failed to get the time from the log content.
      HandleEventError 18 Exceptions occurred in processing the file.
      handleFileCreateError 19 Exceptions occurred in handleFileCreateEvent().
      LineParseError 20 Failed to parse the log directory.
      Lz4CompressError 21 Failed to compress.
      readEventException 22 Failed to read.
      ReadFileBugOn 23 A bug exists.
      ReadFileException 24 Exceptions occurred in the read file.
      ReadFileInodeChange 25 File node changed.
      ReadFileTruncate 26 The read file is truncated.
      WildCardPathException 27 Exceptions occurred in addWildcardPathInotify().

      LogListener collection logs

      The parameters of the log topic loglistener_business are detailed as follows:

      ParameterDescription
      InstanceIdLogListener unique identifier
      LabelAn array of machine IDs
      IPMachine group IP
      VersionLogListener version
      TopicIdThe target topic of the collected file
      FileNameFile path name
      FileNameActual file path
      FileInodeFile node
      FileSizeFile size
      LastReadTimeThe most recent read time of the file
      ParseFailLinesNumber of parsing-failed logs within a time window
      ParseFailSizeSize of parsing-failed logs within a time window
      ParseSuccessLinesNumber of logs successful parsed within a time window
      ParseSuccessSizeSize of logs successful parsed within a time window
      ReadOffsetOffset of file reading in bytes
      TruncateSizeSize of truncated log files within a time window
      ReadAvgDelayAverage time delay for reads within a time window
      TimeFormatFailuresLinesNumber of timestamp matching errors within a time window
      SendSuccessSizeSize of logs successful sent within a time window
      SendSuccessCountNumber of logs successful sent within a time window
      SendFailureSizeSize of sending-failed logs within a time window
      SendFailureCountNumber of sending-failed logs within a time window
      SendTimeoutSizeSize of logs with sending timed out within a time window
      SendTimeoutCountNumber of logs with sending timed out within a time window
      DroppedLogSizeSize of dropped logs within a time window
      DroppedLogCountNumber of dropped logs in a time window
      ProcessBlockWhether the current file has triggered collection blocking in a statistical period (collection blocking will be triggered if the sliding window of a file has not moved for 10 minutes)
      tencent
      Copyright © 2013-2021 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy