Bot Protection Settings

Last updated: 2020-11-25 19:15:20

    Overview

    Bot protection configuration allows you to set bot protection policies based on the characteristics of bot session behaviors and take corresponding actions. You can also observe and analyze bot details and fine-tune the policies based on the provided session status details to ensure the security of your website's core APIs and businesses. This feature supports two types of protection policies, public policies and custom session policies.

    Public categories

    WAF provides protection against 12 public categories of bots with over 1,000 subcategories, such as search engine, speed tester, content aggregator, scanner, and website crawler. You can set protection actions (passing, monitoring, and blocking) for public bots as needed, and WAF will process bot requests that hit a public category accordingly.

    Custom session policies

    WAF custom session policies allow you to set features of protocols, IP intelligence, and custom sessions, each of which can be determined in multiple dimensions. You can set custom session policies and the processing actions (passing, monitoring, CAPTCHA, redirect, and blocking) based on your actual business needs and bot details, and WAF will process bot requests that hit a custom protection policy accordingly.

    Note:

    Only WAF Enterprise and Ultimate Editions support bot behavior management. We recommended WAF Advanced users upgrade to the Enterprises or Ultimate Edition.

    Instructions

    Enabling or disabling bot protection

    Log in to the WAF Console and choose Bot Behavior Management -> Bot Protection Settings on the left sidebar to go to the bot policy settings page.

    Field description:

    • Domain Name: protected domain name that is added to WAF in Web Application Firewall -> Defense settings. Sorting is supported.
    • BOT Switch: disabled by default. You can enable it as needed.

      Note:

      BOT Switch takes effect only when WAF Switch is enabled.

    • WAF Switch: the WAF switch status, which is displayed in the protected domain name list in Web Application Firewall -> Defense settings.
    • Operation: click Defense settings to set a bot protection policy.

    Setting public categories

    1. Log in to the WAF Console and choose Bot Behavior Management > Bot Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the Operation column on the right.
    2. On the Defense settings page, click Public Type to enter the corresponding list page.

    Field description:
    - Categories: WAF can identify 12 public bot categories, including search engine, speed tester, content aggregator, scanner, and website crawler.
    - Number of BOT Classes: number of BOT subcategories in each category.
    - Action: the action supported by the corresponding public bot category, which is "Monitor" by default. You can set it to "Pass" or "Block" in the "Operation" column on the right. You can view the blocking results in Attack Log and view the IP addresses blocked in real time in IP Blocking Status.
    - Operation: the action taken for the corresponding public bot category. For more information, see Action Category Description.
    3. In the upper-left corner, click Copy to copy the public bot category settings of the current domain name to another one whose original settings will be overridden.

    Custom session policies

    Protocol features

    1. Log in to the WAF Console and choose Bot Behavior Management > Bot Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the Operation column on the right.
    2. On the Defense settings page, choose Custom Session Policy > Protocol Features to go to the corresponding list page.
      • Field description:
        • Rule Name: protocol feature.
        • Action: the default action of the protocol feature policy, which is "Monitor" by default. You can set its value in the Operation column on the right.
        • Rule Switch: disabled by default.
        • Modification Time: the last time the policy was modified.
        • Operation: click Edit to set the action, which can be "Pass", "Monitor", or "Block". For more information, see Action Category Description. You can view the blocking results in Attack Log and view the IP addresses blocked in real time in IP Blocking Status.
      • The following table lists the names of protocol feature policies.
        Protocol Feature CategoryPolicy Name
        User-AgentUser-Agent is empty or does not exist.
        User-Agent is BOT.
        User-Agent is Unknown BOT.
        User-Agent is HTTP Library.
        User-Agent is Tools.
        User-Agent is Framework.
        User-Agent is Scanner.
        HTTP headerReferer is empty or does not exist.
        Referer abuse (multiple UAs use the same Referer).
        Cookie abuse (multiple UAs use the same Cookie).
        Cookie is empty or does not exist.
        Connection is empty or does not exist.
        Accept is empty or does not exist.
        Accept-Language is empty or does not exist.
        Accept-Encoding is empty or does not exist.
        HTTP featuresThe HTTP HEAD method is used.
        The HTTP version is 1.0 or earlier.

    IP intelligence features

    1. Log in to the WAF Console and choose Bot Behavior Management -> Bot Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the "Operation" column on the right.

    2. On the Defense settings page, select Custom Session Policy > IP Intelligence Characteristics to enter the corresponding list page.

      • Field description:

        • Rule Name: IP intelligence policy.
        • Action: the default action of the intelligence feature policy, which is "Monitor" by default. You can set its value in the "Operation" column on the right.
        • Rule Switch: disabled by default. We recommend that you enable the switch to automatically check the health of your domain names in Tencent Cloud WAF.
        • Modification Time: the last time the policy was modified.
        • Operation: click Edit to set the action, which can be "Pass", "Monitor", "CAPTCHA", or "Block". For more information, see Action Category Description.

          Note:

          This is used to automatically check the health of your domain names. Do not set it to "Block".

      • The following table lists the names of IP intelligence feature policies.

        IP Intelligence CategoryPolicy Name
        Automated testingAutomated testing provided by Tencent Cloud WAF.
        IDC-IP libraryIDC-IP library - Tencent Cloud.
        IDC-IP library - Alibaba Cloud.
        IDC-IP library - Huawei Cloud.
        IDC-IP library - Kingsoft Cloud.
        IDC-IP library - Ucloud.
        IDC-IP library - Baidu Cloud.
        IDC-IP library - JD Cloud.
        IDC-IP library - QingCloud.
        IDC-IP library - AWS.
        IDC-IP library - Azure.
        IDC-IP library - Google Cloud.

    Custom session features

    1. Log in to the WAF Console and choose Bot Behavior Management -> Bot Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the "Operation" column on the right.

    2. On the Defense settings page, choose Custom Session Policy > Custom Session Features to go to the corresponding list page.

      • Field description:
        • SN: the policy number, which automatically increases.
        • Rule name/Description: the policy name and description.
        • Condition: policy match conditions. Up to 10 match conditions can be added for one policy, which are connected by the "AND" relationship.
        • Action: the action that is set when the policy is added. You can set its value in the "Operation" column on the right.
        • Rule Switch: policy switch status, which is set when the policy is added.
        • Modification Time: the last time the policy was added or modified.
        • Operation: modification or deletion of the policy. Click Edit to set the action, which can be "Pass", "Monitor", "CAPTCHA", "Redirect", or "Block". For more information, see Action Category Description.
    3. Add custom session features. Click Add in the upper-right corner of the list to go to the Add custom session features page.

      Note:

      Policy priority is determined by the action category in the order: Pass > Monitor > Redirect > CAPTCHA > Block. For policies of the same action category, the more recently a policy was added, the higher its priority.

      • Field description:
        • Rule Name: the policy name.
        • Rule Description: the policy description.
        • Rule Switch: enabled by default.
        • Condition: conditions for matching BOT policies. Up to 10 match conditions can be set, which are connected by the “AND” relationship. When you hover the curse over a match condition, you can view its description.
        • Action: the action to be taken.
      • Action Category Description:
        CategoryDescription
        PassSession requests that match the set conditions will be allowed and will not be logged.
        MonitorSession requests that match the set conditions will be monitored. You can view more information on the monitored sessions in the custom bot details.
        CAPTCHAThis is applicable only to access through browsers. Session requests that match the set conditions will be verified through a CAPTCHA. If they fail verification, they will be blocked. Otherwise, they can normally access the domain name within the penalty period.
        RedirectSession requests that match the set conditions will be redirected to a specified URL of the current domain name within the specified penalty period.
        BlockSession requests that match the set conditions will be blocked. You can set the penalty period to 5 to 10,080 minutes (7 days). You can view the blocking results in the [Attack Log], and view the IP addresses blocked in real time in [IP Blocking Status].
      • The following table lists the match conditions for custom session features:
        CategoryFilter ConditionDescription
        Session featuresAverage SpeedTotal Session Requests/Session Duration in requests/minute.
        Window SpeedSession access frequency for every 2-minute interval (window) in requests/minute.
        Total SessionsTotal number of access requests in a bot session.
        Session DurationDuration of the bot session.
        Robots.txtThe session request wants to access the `Robots.txt` file.
        Session in early morningThe session request is initiated between 2:00 AM and 5:00 AM.
        IP featuresAccess real IPAccess real IP.
        IP TypeIP type, which can be IDC or base station (ISP base station). When the IP type is IDC, exceptions may occur.
        IP OwnerIP owner information such as `tencent.com`, which can be queried in bot details and is valid only if the IP type is IDC.
        Request featuresMost Requested URLThe most frequently requested URL.
        URL Duplicate RateProportion of repeated URLs in session requests, which ranges from 0 to 1. Set this parameter based on your actual business needs. A value that is too high or too low suggests an exception (which must be determined based on the actual business conditions).
        URL ClassNumber of URLs in session requests after deduplication.
        Most Requested ParameterThe most frequently requested parameter, which may be a GET request parameter (`Query` content) or POST request parameter (`Body` content).
        Parameter Duplicate RateProportion of repeated GET request parameters (`Query` content) or POST request parameters (`Body` content) in session requests, which ranges from 0 to 1. Set this parameter based on your actual business needs. A value that is too high or too low suggests an exception (which must be determined based on the actual business conditions).
        COOKIECOOKIE ExistenceUsed to check whether the HTTP headers fo session requests have cookies.
        Most Requested COOKIEThe most frequently requested cookie in session requests.
        COOKIE Duplicate RateProportion of repeated cookies in session requests, which ranges from 0 to 1.
        COOKIE Existence RateProportion of requests with a cookie among all session requests, which ranges from 0 to 1.
        COOKIE AbuseMultiple different UAs use the same cookie.
        COOKIE ClassNumber of cookies in session requests after deduplication.
        RefererReferer ExistenceUsed to check whether the HTTP header has a referer in session requests.
        Most Requested RefererThe most frequent HTTP referer in session requests.
        Referer Duplicate RateProportion of repeated referers in session requests, which is valid only for access through browsers and ranges from 0 to 1. A value that is too high suggests an exception (which must be determined based on the actual business conditions).
        Referer Existence RateProportion of requests with a referer in session requests, which is valid only for access through browsers and ranges from 0 to 1. A value that is too low suggests an exception (which must be determined based on the actual business conditions).
        Referer AbuseMultiple different UAs use the same referer.
        Referer ClassNumber of referers in session requests after deduplication.
        UAUA ExistenceUsed to check whether the HTTP headers of session requests have UAs.
        Most Requested UAThe most frequent HTTP `User-Agent` value in session requests.
        UA Existence RateProportion of requests with a UA among session requests, which ranges from 0 to 1. A value that is too low suggests an exception (which must be determined based on the actual business conditions).
        UA ClassNumber of UAs in session requests after deduplication, which is valid only for non-proxy IP addresses. A value that is too high suggests an exception (which must be determined based on the actual business conditions).
        UA Type
        • Browser.
        • Mobile device.
        • Game console or TV.
        • Public bot.
        • Private bot.
        • Automated tool.
        • Unknown.
        • Public scanner.
        • Framework.
        • Programming language HTTP library.
          UA Randomness IndexThe random distribution status, which ranges from 0 to 1. The higher the value, the more likely the request is exceptional.
          Reference thresholds: a value above 0.6 suggests an exception; a value above 0.92 is almost certainly an exception.
          HTTP HeaderAccept ExistenceThe HTTP headers in session requests will be checked for the `Accept` field.
          Accept-Language ExistenceThe HTTP headers in session requests will be checked for the `Accept-Language` field.
          Accept-Encoding ExistenceThe HTTP headers in session requests will be checked for the `Accept-Encoding` field.
          Connection ExistenceThe HTTP headers in session requests will be checked for the `Connectiton` field.
          Request Method RateProportion of session requests with request methods.
          HTTP Version RateProportion of session requests that use HTTP versions.
          Returned Status Code RateProportion of session requests with status codes returned by WAF.
          Advanced FeaturesPrediction TagSuspicious behavior predicted by the system. Not all sessions have a prediction tag. A prediction tag can indicate the following:
          • Suspected illegal crawler.
          • Suspected regular crawler.
          • Suspected login without user value.
          • Suspected login without user parameter.
          • Suspected login without username and password.
          • Suspected login without user login action.
          • Suspected brute force attack.
          • Suspected credential stuffing attack.
          • Suspected unauthorized access to SMS APIs.
          • Suspected unauthorized access to CAPTCHA APIs.
          • Suspected malicious registration.
          • Suspected repeated API access
          ScoreBot score for the session given by the intelligent bot analysis engine. The higher the score, the more likely the session is initiated by a bot. The reference value is 5.
          AI Model CheckoutResult of AI behavior analysis model detection. If the AI model flags a request, this suggests an exception.
          Public BOT ForgingSession requests pretend to be in public bot categories. For example, a request uses the UA of a Baidu crawler, but its IP address does not belong to Baidu.
          Access to Sensitive APIUsed to determine whether sensitive APIs (such as SMS APIs, registration APIs, and login APIs) are accessed.
          Index Exception for Time-series BehaviorAn algorithm used to detect abnormal time-series behaviors. A smaller index suggestions a higher probability of an exception.
          Reference value thresholds: a value below 0.5 suggests an exception; a value below 0.24 is almost certainly an exception.
          Index Exception of Sequential EntropyAn algorithm used to detect the time-series behavior entropy. A smaller index suggests a higher probability of exception. The reference value threshold is 0.5. A value below 0.5 suggests an exception.

      Was this page helpful?

      Was this page helpful?

      • Not at all
      • Not very helpful
      • Somewhat helpful
      • Very helpful
      • Extremely helpful
      Send Feedback
      Help