Storing Access Logs in CLS

Last updated: 2020-08-03 11:13:05

    CLB supports configuring layer-7 (HTTP/HTTPS) access logs that can help you better understand client requests, troubleshoot issues, and analyze user behaviors. Currently, access logs can be stored in CLS, reported at a minute granularity, and searched online by multiple rules.

    Access logs of CLB are mainly used to quickly locate and troubleshoot issues. The access logging feature includes log reporting, storage, and search:

    • Log reporting provides best-effort service, that is, it prioritizes service forwarding over log reporting.
    • Log storage and search provide SLA based on the storage service currently in use.

    Note:

    • As the feature of storing access logs in COS will be officially disused after 00:00:00, June 30, 2020, you are recommended to use CLS for CLB access log storage.
    • Currently, access logs can be stored in CLS only for layer-7 protocols (HTTP/HTTPS) but not layer-4 protocols (TCP/UDP/TCP SSL).
    • The feature of storing CLB access logs in CLS is free of charge, and you only need to pay for CLS usage.
    • Currently, access logs can be stored in CLS in the Guangzhou, Shanghai, Nanjing, Beijing, Chongqing, Chengdu, Hong Kong (China), Singapore, Mumbai, Silicon Valley, Toronto, and Frankfurt regions through the console or APIs.
    • Currently, CLS is in beta test. To try it out, please submit a ticket for application.

    Enabling Access Log Storage in CLS

    1. Log in to the CLB Console.
    2. Click the ID of the CLB instance to be configured to enter the "Basic Information" page.
    3. In the "Log Access" module, edit "Cloud Log Service".
    4. In the pop-up box, enable access logging and select the destination logset and log topic for access log storage. If you haven't created a logset or log topic yet, please create relevant resources and then select them as the storage location.
    5. Click Submit and access logs will be collected into the corresponding topic.
    6. Then, click the logset or log topic to redirect to the log search page in CLS.
    7. (Optional) If you want to disable access logging, you can edit "Cloud Log Service" again to disable it and submit in the pop-up window.

    Note:

    With the CLS scheme, you can also store access logs in COS according to your actual business needs such as costs and usage. For more information, please see Shipping Overview.

    Searching for Access Log

    Step 1. Configure log topic indexes

    Note:

    The log topics must be configured with indexes; otherwise, no logs can be searched for.

    The recommended indexes are as follows:

    Key-Value Index Field Type Delimiter
    server_addr text No delimiter required
    server_name text No delimiter required
    http_host text No delimiter required
    status long -
    vip_vpcid long -

    The steps are as follows:

    1. Log in to the CLS Console.
    2. On the left sidebar, click Logset to enter the "Logset Management" page.
    3. Click a logset ID to enter the logset details page.
    4. On the logset details page, click a log topic ID to enter the log topic details page.
    5. On the log topic details page, select the Index Configuration tab. You can select some variables from the log variables and configure the index fields as needed. For more information on how to configure, please see Enabling Index.
    6. The result of index configuration is as shown below:

    Step 2. Search for access logs

    1. Log in to the CLS Console.
    2. On the left sidebar, click Search and Analysis to enter the "Search Analysis" page.
    3. On the "Search Analysis" page, select a logset, log topic, and time range, and click Search Analysis to search for the access logs reported by CLB to CLS. For more information on the search syntax, please see Syntax and Rules.

    Log Format and Variable Description

    Log format

    [$stgw_request_id] [$time_local] [$protocol_type] [$server_addr:$server_port] [$server_name] [$remote_addr:$remote_port] [$status] [$upstream_addr] [$upstream_status] [$proxy_host] [$request] [$request_length] [$bytes_sent] [$http_host] [$http_user_agent] [$http_referer] [$request_time] [$upstream_response_time] [$upstream_connect_time] [$upstream_header_time] [$tcpinfo_rtt] [$connection] [$connection_requests] [$ssl_handshake_time] [$ssl_cipher] [$ssl_protocol] [$vip_vpcid]

    Field type

    Currently, CLS supports the following three field types:

    Name Type Description
    text Text type
    long Integer type (Int 64)
    double Floating point type (64-bit)

    Log variable description

    VariableDescriptionField Type
    stgw_request_idRequest ID.text
    time_local Access time and time zone, such as "01/Jul/2019:11:11:00 +0800" where "+0800" represents UTC+8, i.e., Beijing time.text
    protocol_typeProtocol type (HTTP/HTTPS/SPDY/HTTP2/WS/WSS).text
    server_addrDestination IP of request.text
    server_portDestination port of request.long
    server_nameRule's `server_name`, i.e., server name.text
    remote_addrClient IP.text
    remote_portClient port.long
    statusStatus code returned to client.long
    upstream_addrRS address.text
    upstream_statusStatus code returned by RS to CLB.text
    proxy_hostStream ID.text
    requestRequest line.text
    request_lengthNumber of bytes of request received from client.long
    bytes_sentNumber of bytes sent to client.long
    http_hostRequest domain name.text
    http_user_agent`user_agent` field of the HTTP header.text
    http_refererHTTP request source.text
    request_timeRequest processing time. The timing begins when the first byte is received from the client and stops when the last byte is sent to the client, i.e., the total time the whole process takes, where the client request reaches a CLB instance, the CLB instance forwards the request to an RS, the RS responds and sends data to the CLB instance, and finally the CLB instance forwards the data to the client.double
    upstream_response_timeThe time that an entire backend request process takes. The timing begins when a CLB instance connects with an RS and stops when the RS receives the request and responds.double
    upstream_connect_timeThe time it takes to establish a TCP connection with an RS. The timing begins when a CLB instance connects with an RS and stops when it sends the HTTP request.double
    upstream_header_timeThe time it takes to receive an HTTP header from the RS. The timing begins when a CLB instance connects with an RS and stops when the HTTP response header is received from the RS.double
    tcpinfo_rttTCP connection RTT.long
    connectionConnection ID.long
    connection_requestsNumber of requests on connection.long
    ssl_handshake_timeThe time that an SSL handshake takes.double
    ssl_cipherSSL cipher suite.text
    ssl_protocolSSL protocol version.text
    vip_vpcidVPC ID of CLB instance VIP.long

    Was this page helpful?

    Was this page helpful?

    • Not at all
    • Not very helpful
    • Somewhat helpful
    • Very helpful
    • Extremely helpful
    Send Feedback
    Help