tencent cloud

Feedback

Bot Traffic Details

Last updated: 2022-06-24 11:05:54

    Overview

    With data provided by the bot behavior management module, bot traffic analysis quickly analyzes the bot impacts on domain names in terms of bot feature metrics, including the types of bots, proportion of actions, bot score distribution, top data by request count and URLs that may be affected. You can click View details to view the bot details of an access source, and its access characteristics and exceptions detected.

    The bot traffic details section displays the bot traffic details of top 10 access sources. You can also view their session access information and logs if session settings are configured, and the information of these access sources/IPs by retrieval.

    Prerequisites

    You have subscribed to the bot behavior management service and enabled bot traffic analysis.

    Directions

    1. Log in to the WAF console and select Bot Traffic Analysis on the left sidebar. Open the Bot traffic details tab.
    2. On the page displayed, click the drop-down list in the upper-left corner and select a domain name.
    3. Specify a date or use the filter to search top 10 access sources of all domain names or a specific domain name. Then click View logs of the access source you want to view.
    4. To view traffic details of an access source, click View details on the right.

    Viewing access overview

    The bot traffic details page shows you the estimated risk value of the access source and the information of the hit policy, and allows you to take measures such as adding the access source to the allowlist/blocklist and creating custom rules targeting the access source.

    Field description:

    • IP address and tag: It displays the IP address of an access source and the hit tag that identifies the bot category.
    • Last request: It displays the score of the last access request from the access source and the risk level.
    • Session count: It displays the number of continuous sessions on the website accessed occurred in the last access request.
    • Access address: It displays the domain name of the access source.
    • Exception feature: It displays modules that detect exceptional features of the access source.
    • Hit modules: It displays modules that take actions to combat bots.
    • Policy ID: It is the ID of a hit policy.
    • View access logs: It redirects you to the access logs page where you can view the access details of the access source.
    • Add to allowlist: It allows you to add the access source to the allowlist.
    • Add to blocklist: It allows you to add the access source to the blocklist.
    • Add custom rules: It allows you to add custom rules targeting the access source.

    Viewing bot scores

    In the bot score distribution and bot action distribution sections, you can view the distribution of bot scores and bot actions within the selected period, helping you determine the risk level of the access source.

    Viewing bot information

    The bot traffic details page also displays information of the automated access source including the features of the bot and access request, threat intelligence and AI evaluation information, bot flow statistics and sessions. Using this data, you can quickly identify exceptions in the access request and take measures against the bot.

    Basic session information

    On the basic session information tab, you can view the information of the access source IP and session separately.

    Field description:

    • IP information
    • Access source IP: IP address of an access source.
    • City: City where the access source is located.
    • Region: Country where the access source is located.
    • IP type: IP type of the access source.
    • IP owner: IP owner of the access source.
    • Session information
    • Session average speed: The average session speed of the access source in the latest session, which is calculated by the total number of session requests/session duration. Unit: times/minute.
    • Total sessions: The total number of sessions of the access source in the latest session.
    • Whether Robots.txt exists: Whether the access source has accessed the Robots.txt file, which is often accessed by bot sessions.
    • Session duration: Amount of time the latest session initiated by the access source.

    Request feature information

    On the request feature information tab, you can view the information of the request features, Cookie, User-Agent, Referer, and Query in the session request.

    Field description:

    Type Metric Description
    Request feature information Percentage of repeated URLs Percentage of the repeated URLs in a session request. The value range is 0-1. Set this parameter based on your actual business needs. A value that is too high or too low suggests an exception (which must be determined based on the actual business conditions).
    Total URL types Number of deduplicated URLs in a session request.
    Minimum URL depth Minimum directory levels of URLs in a session request.
    Maximum URL depth Maximum directory levels of URLs in a session request.
    Average URL depth Average directory levels of URLs in a session request.
    Total URLs Total number of URLs visited in a session request (including duplicates).
    Cookie information Whether Cookie is abused Different types of UAs use the same Cookie.
    Cookie exist Whether the Cookie exists in all session requests.
    Percentage of repeated Cookies Percentage of the repeated Cookies in a session request. The value ranges from 0-1.
    Cookie validity Percentage of the Cookies that can be parsed in a session request.
    Most used Cookie Most used Cookies in a session request.
    Percentage of the most used Cookies Percentage of the most used Cookies in a session request.
    User-Agent information User-Agent type User-Agent type of the request user in a session request.
    User-Agent exist Whether User-Agent exists in a session request.
    User-Agent randomness index Random distribution of User-Agents in a session request. When the reference value threshold exceeds 0.6, an exception is suspected; when it exceeds 0.92, an exception is basically confirmed.
    User-Agent type Number of deduplicated User-Agents in a session request, which is valid only for non-proxy IPs. A value that is too high suggests an exception (which must be determined based on the actual business conditions).
    User-Agent existence rate Existence rate of UAs in a session request, which ranges from 0 to 1. A value too small may be suspected as an exception (which must be determined based on the actual business conditions).
    Most used User-Agent Most used value of the HTTP User-Agent in a session request.
    Percentage of the most used User-Agents Percentage of the most used HTTP User-Agent values ​in a session request.
    User-Agent similarity rate Similarity between the most used value and the rest in a session request.
    Referer information Percentage of repeated Referers Percentage of repeated Referers in a session request, which is valid only for access through browsers and ranges from 0 to 1. A value that is too high suggests an exception (which must be determined based on the actual business conditions).
    Referer exist Existence rate of Referers in a session request, which ranges from 0 to 1. A value too small may be suspected as an exception. It's available for browser access.
    Referer existence rate Existence rate of Referers in a session request, which ranges from 0 to 1. A value too small may be suspected as an exception. It's available for browser access.
    Whether Referer is abused Different types of UAs use the same Referer.
    Most used Referer Most used value of the HTTP Referer in a session request.
    Percentage of the most used Referers Percentage of the most used HTTP Referer values in a session request.
    Query information Percentage of repeated Query parameters Percentage of repeated GET request parameters (`Query` content) or POST request parameters (`Body` content) in a session request, which ranges from 0 to 1. Set this parameter based on your actual business needs. A value that is too high or too low suggests an exception (which must be determined based on the actual business conditions).
    Total query parameter types Most used parameters in a session request, which may be GET request parameters (`Query` content) or POST request parameters (`Body` content).

    Threat intelligence

    The threat intelligence tab displays the known information that matches the current access source IP/session ID, such as the IDC details of the access source.

    • IDC details: If the information of the access source includes the IDC, the IDC information will be displayed.
    • Threat Intelligence: If the IP information of the access source is matched, the matched tag and its definition will be displayed.

    AI evaluation

    The AI evaluation tab displays the following feature metrics detected exceptionally and the corresponding probability values, including information of Cookie, User-Agent, Referer, and Query. If a metric's value larger than 0, this metric is considered exceptional.

    Bot flow statistics

    The bot flow statistics tab displays the feature metrics detected exceptionally, and the corresponding values estimated, as well as the reference values.


    Field description:

    Metric Description
    Session average speed This metric indicates that the session average speed is considered exceptional, and gives a probability threshold to confirm the exception.
    User-Agent type This metric indicates that the User-Agent type is considered exceptional, and gives a probability threshold to confirm the exception.
    URL type This metric indicates that the URL type is considered exceptional, and gives a probability threshold to confirm the exception.
    Session duration This metric indicates that the session duration is considered exceptional, and gives a probability threshold to confirm the exception.
    Total session count This metric indicates that the total session count is considered exceptional, and gives a probability threshold to confirm the exception.

    Session management

    The session management tab displays the IP addresses accessed by the current session, the number of accesses by each IP address, and the access logs of the current session ID.

    Note:

    When session settings are configured, the session management tab will be displayed.

    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support