Overview

Bot protection configuration allows you to set bot protection policies based on the characteristics of bot session behaviors and take corresponding actions. You can also observe and analyze bot details and fine-tune the policies based on the provided session status details to ensure the security of your website's core APIs and businesses. This feature supports two types of protection policies, public policies and custom session policies.

Public categories

WAF provides protection against 12 public categories of bots with over 1,000 subcategories, such as search engine, speed tester, content aggregator, scanner, and website crawler. You can set protection actions (passing, monitoring, and blocking) for public bots as needed, and WAF will process bot requests that hit a public category accordingly.

Custom session policies

WAF custom session policies allow you to set features of protocols, IP intelligence, and custom sessions, each of which can be determined in multiple dimensions. You can set custom session policies and the processing actions (passing, monitoring, CAPTCHA, redirect, and blocking) based on your actual business needs and bot details, and WAF will process bot requests that hit a custom protection policy accordingly.

Note：

Only WAF Enterprise and Ultimate Editions support bot behavior management. We recommended WAF Advanced users upgrade to the Enterprises or Ultimate Edition.

Instructions

Enabling or disabling bot protection

Log in to the WAF Console and choose Bot Behavior Management -> Bot Protection Settings on the left sidebar to go to the bot policy settings page.

Field description:

• Domain Name: protected domain name that is added to WAF in Web Application Firewall -> Defense settings. Sorting is supported.
• BOT Switch: disabled by default. You can enable it as needed.

  Note：

  BOT Switch takes effect only when WAF Switch is enabled.

• WAF Switch: the WAF switch status, which is displayed in the protected domain name list in Web Application Firewall -> Defense settings.
• Operation: click Defense settings to set a bot protection policy.

Setting public categories

1. Log in to the WAF Console and choose Bot Behavior Management > Bot Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the Operation column on the right.
2. On the Defense settings page, click Public Type to enter the corresponding list page.
   

Field description:
- Categories: WAF can identify 12 public bot categories, including search engine, speed tester, content aggregator, scanner, and website crawler.
- Number of BOT Classes: number of BOT subcategories in each category.
- Action: the action supported by the corresponding public bot category, which is "Monitor" by default. You can set it to "Pass" or "Block" in the "Operation" column on the right. You can view the blocking results in Attack Log and view the IP addresses blocked in real time in IP Blocking Status.
- Operation: the action taken for the corresponding public bot category. For more information, see Action Category Description.
3. In the upper-left corner, click Copy to copy the public bot category settings of the current domain name to another one whose original settings will be overridden.

Custom session policies

Protocol features

1. Log in to the WAF Console and choose Bot Behavior Management > Bot Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the Operation column on the right.
2. On the Defense settings page, choose Custom Session Policy > Protocol Features to go to the corresponding list page.
   
   • Field description:
     • Rule Name: protocol feature.
     • Action: the default action of the protocol feature policy, which is "Monitor" by default. You can set its value in the Operation column on the right.
     • Rule Switch: disabled by default.
     • Modification Time: the last time the policy was modified.
     • Operation: click Edit to set the action, which can be "Pass", "Monitor", or "Block". For more information, see Action Category Description. You can view the blocking results in Attack Log and view the IP addresses blocked in real time in IP Blocking Status.
   • The following table lists the names of protocol feature policies.

     Protocol Feature Category	Policy Name
     User-Agent	User-Agent is empty or does not exist.
     	User-Agent is BOT.
     	User-Agent is Unknown BOT.
     	User-Agent is HTTP Library.
     	User-Agent is Tools.
     	User-Agent is Framework.
     	User-Agent is Scanner.
     HTTP header	Referer is empty or does not exist.
     	Referer abuse (multiple UAs use the same Referer).
     	Cookie abuse (multiple UAs use the same Cookie).
     	Cookie is empty or does not exist.
     	Connection is empty or does not exist.
     	Accept is empty or does not exist.
     	Accept-Language is empty or does not exist.
     	Accept-Encoding is empty or does not exist.
     HTTP features	The HTTP HEAD method is used.
     	The HTTP version is 1.0 or earlier.

IP intelligence features

1. Log in to the WAF Console and choose Bot Behavior Management -> Bot Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the "Operation" column on the right.

2. On the Defense settings page, select Custom Session Policy > IP Intelligence Characteristics to enter the corresponding list page.
   

   • Field description:

     • Rule Name: IP intelligence policy.
     • Action: the default action of the intelligence feature policy, which is "Monitor" by default. You can set its value in the "Operation" column on the right.
     • Rule Switch: disabled by default. We recommend that you enable the switch to automatically check the health of your domain names in Tencent Cloud WAF.
     • Modification Time: the last time the policy was modified.
     • Operation: click Edit to set the action, which can be "Pass", "Monitor", "CAPTCHA", or "Block". For more information, see Action Category Description.

       Note：

       This is used to automatically check the health of your domain names. Do not set it to "Block".

   • The following table lists the names of IP intelligence feature policies.

     IP Intelligence Category	Policy Name
     Automated testing	Automated testing provided by Tencent Cloud WAF.
     IDC-IP library	IDC-IP library - Tencent Cloud.
     	IDC-IP library - Alibaba Cloud.
     	IDC-IP library - Huawei Cloud.
     	IDC-IP library - Kingsoft Cloud.
     	IDC-IP library - Ucloud.
     	IDC-IP library - Baidu Cloud.
     	IDC-IP library - JD Cloud.
     	IDC-IP library - QingCloud.
     	IDC-IP library - AWS.
     	IDC-IP library - Azure.
     	IDC-IP library - Google Cloud.

Custom session features

1. Log in to the WAF Console and choose Bot Behavior Management -> Bot Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the "Operation" column on the right.

2. On the Defense settings page, choose Custom Session Policy > Custom Session Features to go to the corresponding list page.
   

   • Field description:
     • SN: the policy number, which automatically increases.
     • Rule name/Description: the policy name and description.
     • Condition: policy match conditions. Up to 10 match conditions can be added for one policy, which are connected by the "AND" relationship.
     • Action: the action that is set when the policy is added. You can set its value in the "Operation" column on the right.
     • Rule Switch: policy switch status, which is set when the policy is added.
     • Modification Time: the last time the policy was added or modified.
     • Operation: modification or deletion of the policy. Click Edit to set the action, which can be "Pass", "Monitor", "CAPTCHA", "Redirect", or "Block". For more information, see Action Category Description.

3. Add custom session features. Click Add in the upper-right corner of the list to go to the Add custom session features page.
   

   Note：

   Policy priority is determined by the action category in the order: Pass > Monitor > Redirect > CAPTCHA > Block. For policies of the same action category, the more recently a policy was added, the higher its priority.

   • Field description:
     • Rule Name: the policy name.
     • Rule Description: the policy description.
     • Rule Switch: enabled by default.
     • Condition: conditions for matching BOT policies. Up to 10 match conditions can be set, which are connected by the “AND” relationship. When you hover the curse over a match condition, you can view its description.
     • Action: the action to be taken.
       
   • Action Category Description:

     Category	Description
     Pass	Session requests that match the set conditions will be allowed and will not be logged.
     Monitor	Session requests that match the set conditions will be monitored. You can view more information on the monitored sessions in the custom bot details.
     CAPTCHA	This is applicable only to access through browsers. Session requests that match the set conditions will be verified through a CAPTCHA. If they fail verification, they will be blocked. Otherwise, they can normally access the domain name within the penalty period.
     Redirect	Session requests that match the set conditions will be redirected to a specified URL of the current domain name within the specified penalty period.
     Block	Session requests that match the set conditions will be blocked. You can set the penalty period to 5 to 10,080 minutes (7 days). You can view the blocking results in the [Attack Log], and view the IP addresses blocked in real time in [IP Blocking Status].

   • The following table lists the match conditions for custom session features:

     Category	Filter Condition	Description
     Session features	Average Speed	Total Session Requests/Session Duration in requests/minute.
     	Window Speed	Session access frequency for every 2-minute interval (window) in requests/minute.
     	Total Sessions	Total number of access requests in a bot session.
     	Session Duration	Duration of the bot session.
     	Robots.txt	The session request wants to access the `Robots.txt` file.
     	Session in early morning	The session request is initiated between 2:00 AM and 5:00 AM.
     IP features	Access real IP	Access real IP.
     	IP Type	IP type, which can be IDC or base station (ISP base station). When the IP type is IDC, exceptions may occur.
     	IP Owner	IP owner information such as `tencent.com`, which can be queried in bot details and is valid only if the IP type is IDC.
     Request features	Most Requested URL	The most frequently requested URL.
     	URL Duplicate Rate	Proportion of repeated URLs in session requests, which ranges from 0 to 1. Set this parameter based on your actual business needs. A value that is too high or too low suggests an exception (which must be determined based on the actual business conditions).
     	URL Class	Number of URLs in session requests after deduplication.
     	Most Requested Parameter	The most frequently requested parameter, which may be a GET request parameter (`Query` content) or POST request parameter (`Body` content).
     	Parameter Duplicate Rate	Proportion of repeated GET request parameters (`Query` content) or POST request parameters (`Body` content) in session requests, which ranges from 0 to 1. Set this parameter based on your actual business needs. A value that is too high or too low suggests an exception (which must be determined based on the actual business conditions).
     COOKIE	COOKIE Existence	Used to check whether the HTTP headers fo session requests have cookies.
     	Most Requested COOKIE	The most frequently requested cookie in session requests.
     	COOKIE Duplicate Rate	Proportion of repeated cookies in session requests, which ranges from 0 to 1.
     	COOKIE Existence Rate	Proportion of requests with a cookie among all session requests, which ranges from 0 to 1.
     	COOKIE Abuse	Multiple different UAs use the same cookie.
     	COOKIE Class	Number of cookies in session requests after deduplication.
     Referer	Referer Existence	Used to check whether the HTTP header has a referer in session requests.
     	Most Requested Referer	The most frequent HTTP referer in session requests.
     	Referer Duplicate Rate	Proportion of repeated referers in session requests, which is valid only for access through browsers and ranges from 0 to 1. A value that is too high suggests an exception (which must be determined based on the actual business conditions).
     	Referer Existence Rate	Proportion of requests with a referer in session requests, which is valid only for access through browsers and ranges from 0 to 1. A value that is too low suggests an exception (which must be determined based on the actual business conditions).
     	Referer Abuse	Multiple different UAs use the same referer.
     	Referer Class	Number of referers in session requests after deduplication.
     UA	UA Existence	Used to check whether the HTTP headers of session requests have UAs.
     	Most Requested UA	The most frequent HTTP `User-Agent` value in session requests.
     	UA Existence Rate	Proportion of requests with a UA among session requests, which ranges from 0 to 1. A value that is too low suggests an exception (which must be determined based on the actual business conditions).
     	UA Class	Number of UAs in session requests after deduplication, which is valid only for non-proxy IP addresses. A value that is too high suggests an exception (which must be determined based on the actual business conditions).
     	UA Type	Browser. Mobile device. Game console or TV. Public bot. Private bot. Automated tool. Unknown. Public scanner. Framework. Programming language HTTP library.
     	UA Randomness Index	The random distribution status, which ranges from 0 to 1. The higher the value, the more likely the request is exceptional. Reference thresholds: a value above 0.6 suggests an exception; a value above 0.92 is almost certainly an exception.
     HTTP Header	Accept Existence	The HTTP headers in session requests will be checked for the `Accept` field.
     	Accept-Language Existence	The HTTP headers in session requests will be checked for the `Accept-Language` field.
     	Accept-Encoding Existence	The HTTP headers in session requests will be checked for the `Accept-Encoding` field.
     	Connection Existence	The HTTP headers in session requests will be checked for the `Connectiton` field.
     	Request Method Rate	Proportion of session requests with request methods.
     	HTTP Version Rate	Proportion of session requests that use HTTP versions.
     	Returned Status Code Rate	Proportion of session requests with status codes returned by WAF.
     Advanced Features	Prediction Tag	Suspicious behavior predicted by the system. Not all sessions have a prediction tag. A prediction tag can indicate the following: Suspected illegal crawler. Suspected regular crawler. Suspected login without user value. Suspected login without user parameter. Suspected login without username and password. Suspected login without user login action. Suspected brute force attack. Suspected credential stuffing attack. Suspected unauthorized access to SMS APIs. Suspected unauthorized access to CAPTCHA APIs. Suspected malicious registration. Suspected repeated API access
     	Score	Bot score for the session given by the intelligent bot analysis engine. The higher the score, the more likely the session is initiated by a bot. The reference value is 5.
     	AI Model Checkout	Result of AI behavior analysis model detection. If the AI model flags a request, this suggests an exception.
     	Public BOT Forging	Session requests pretend to be in public bot categories. For example, a request uses the UA of a Baidu crawler, but its IP address does not belong to Baidu.
     	Access to Sensitive API	Used to determine whether sensitive APIs (such as SMS APIs, registration APIs, and login APIs) are accessed.
     	Index Exception for Time-series Behavior	An algorithm used to detect abnormal time-series behaviors. A smaller index suggestions a higher probability of an exception. Reference value thresholds: a value below 0.5 suggests an exception; a value below 0.24 is almost certainly an exception.
     	Index Exception of Sequential Entropy	An algorithm used to detect the time-series behavior entropy. A smaller index suggests a higher probability of exception. The reference value threshold is 0.5. A value below 0.5 suggests an exception.