Overview

Bot protection configuration allows you to set bot protection policies based on the characteristics of bot session behaviors and take corresponding actions. You can also observe and analyze bot details and fine-tune the policies based on the provided session status details to ensure the security of your website's core APIs and businesses. This feature supports two types of protection policies, public policies and custom session policies.

Public categories

WAF provides protection against 12 public categories of bots with over 1000 subcategories, such as search engine, speed tester, content aggregator, scanner, and website crawler. You can set protection actions (allowing, monitoring, and blocking) for public bots as needed, and WAF will process bot requests that hit a public category accordingly.

Custom session policies

WAF custom session policies allow you to set features of protocols, IP intelligence, and custom sessions, each of which can be determined in multiple dimensions. You can set custom session policies and the processing actions (allowing, monitoring, CAPTCHA, redirection, and blocking) based on your actual business needs and bot details, and WAF will process bot requests that hit a custom protection policy accordingly.

Note：

Only WAF Enterprise and Ultimate Editions support bot behavior management. We recommend WAF Advanced users upgrade to the Enterprise or Ultimate Edition.

Directions

Enabling or disabling bot protection

Log in to the WAF console, and click BOT Behavior Management -> BOT Protection Settings on the left sidebar to enter the bot policy settings page.

Field description:

• Domain Name: protected domain name that is added to WAF in Web Application Firewall -> Defense settings. Sorting is supported.
• BOT Switch: disabled by default. You can enable it as needed.

  Note：

  The bot switch takes effect only when the WAF switch is enabled.

• WAF Switch: the WAF switch status, which is displayed in the protected domain name list in Web Application Firewall -> Defense settings.
• Operation: click Defense settings to set a bot protection policy.

Setting public categories

1. Log in to the WAF console, and click BOT Behavior Management -> BOT Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the operation column on the right.
2. On the defense settings page, click Public Type to enter the corresponding list page.
   

Parameter description:
- Categories: WAF can identify 12 public bot categories, including search engine, speed tester, content aggregator, scanner, and website crawler.
- Number of BOT Classes: number of BOT subcategories in each category.
- Action: the action supported by the corresponding public bot category, which is "Monitor" by default. You can set it to "Allow" or "Block" in the operation column on the right. You can view the blocking results in Attack Log and view the blocked IPs in real time in IP Blocking On/Off.
- Operation: the action taken for the corresponding public bot category. For more information, see Action Category Description.
3. At the upper-left corner, click Copy to copy the public bot category settings of the current domain name to another one whose original settings will be overwritten.

Custom session policies

Protocol features

1. Log in to the WAF console, and click BOT Behavior Management -> BOT Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the operation column on the right.
2. On the defense settings page, click Custom Session Policy -> Protocol Features to enter the corresponding list page.
   
   • Field description:
     • Rule Name: protocol feature.
     • Action: the default action of the protocol feature policy, which is "Monitor" by default. You can set it in the operation column on the right.
     • Rule Switch: disabled by default.
     • Modification Time: the last time the rule was modified.
     • Operation: click Edit to set the action, which can be "Allow", "Monitor", "CAPTCHA", or "Block". For more information, see Action Category Description. You can view the blocking results in Attack Log and view the blocked IPs in real time in IP Blocking On/Off.
   • The following table lists the names of protocol feature policies:

     Protocol Feature Category	Policy Name
     User-Agent	User-Agent is empty or does not exist.
     	User-Agent is BOT.
     	User-Agent is Unknown BOT.
     	User-Agent is HTTP Library.
     	User-Agent is Tools.
     	User-Agent is Framework.
     	User-Agent is Scanner.
     HTTP header	Referer is empty or does not exist.
     	Referer abuse (multiple UAs use the same referer).
     	Cookie abuse (multiple UAs use the same cookie).
     	Cookie is empty or does not exist.
     	Connection is empty or does not exist.
     	Accept is empty or does not exist.
     	Accept-Language is empty or does not exist.
     	Accept-Encoding is empty or does not exist.
     HTTP features	The HTTP HEAD method is used.
     	The HTTP version is 1.0 or earlier.

IP intelligence features

1. Log in to the WAF console, and click BOT Behavior Management -> BOT Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the operation column on the right.
2. On the defense settings page, click Custom Session Policy -> IP Intelligence Features to enter the corresponding list page.
   
   • Field description:
     • Rule Name: IP intelligence policy.
     • Action: the default action of the intelligence feature policy, which is "Monitor" by default. You can set it in the operation column on the right.
     • Rule Switch: disabled by default. We recommend enabling the switch to automatically check the health of your domain names in Tencent Cloud WAF.
     • Modification Time: the last time the rule was modified.
     • Operation: click Edit to set the action, which can be "Allow", "Monitor", "CAPTCHA", or "Block". For more information, see Action Category Description.

       Note：

       This is used to automatically check the health of your domain names. Please do not set it to "Block".

   • The following table lists the names of IP intelligence feature policies:

     IP Intelligence Category	Rule Name
     Automated testing	Automated testing provided by Tencent Cloud WAF.
     IDC-IP library	IDC-IP library - Tencent Cloud.
     	IDC-IP library - Alibaba Cloud.
     	IDC-IP library - Huawei Cloud.
     	IDC-IP library - Kingsoft Cloud.
     	IDC-IP library - UCloud.
     	IDC-IP library - Baidu Cloud.
     	IDC-IP library - JD Cloud.
     	IDC-IP library - QingCloud.
     	IDC-IP library - AWS.
     	IDC-IP library - Azure.
     	IDC-IP library - Google.

Custom session features

1. Log in to the WAF console, and click BOT Behavior Management -> BOT Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the operation column on the right.
2. On the defense settings page, click Custom Session Policy -> Custom Session Features to enter the corresponding list page.
   
   • Field description:
     • No. auto-incrementing rule number.
     • Rule Name/Description: the rule name and description.
     • Condition: rule match conditions. Up to 10 match conditions can be added for one rule, which are connected by the "AND" relationship.
     • Action: the action that is set when the rule is added. You can set it in the operation column on the right.
     • Rule Switch: rule switch status, which is set when the rule is added.
     • Modification Time: the last time the rule was added or modified.
     • Operation: modification or deletion of the rule. Click Edit to set the action, which can be "Allow", "Monitor", "CAPTCHA", "Redirection", or "Block". For more information, see Action Category Description.
3. Click Add at the upper-left corner of the list to add a custom session feature rule.
   

   Note：

   Rule priority is determined by the priority number. The smaller the number, the higher the priority. The default number is 100. For rules with the same number, the later a rule was added, the higher its priority.

   • Field description:
     • Rule Name: the rule name.
     • Rule Description: the rule description.
     • Rule Switch: enabled by default.
     • Condition: conditions for matching bot policies. Up to 10 match conditions can be set, which are connected by the "AND" relationship. When you hover the cursor over a match condition, you can view its description.
     • Action: the action to be taken.
   • Action Category Description:

     Category	Description
     Allow	Session requests that match the set conditions will be allowed and will not be logged.
     Monitor	Session requests that match the set conditions will be monitored and logged. You can view more information on the monitored sessions in the custom bot details.
     CAPTCHA	This is applicable only to the access through browsers. Session requests that match the set conditions will be verified through a CAPTCHA. If they fail, they will be blocked. Otherwise, they can normally access within the penalty period.
     Redirection	Session requests that match the set conditions will be redirected to a specified URL of the current domain name within the specified penalty period.
     Block	Session requests that match the set conditions will be blocked. You can set the penalty period to 5 to 10,080 minutes (7 days). You can view the blocking results in [Attack Log](https://console.cloud.tencent.com/guanjia/log/attack) and view the blocked IPs in real time in [IP Blocking On/Off](https://console.cloud.tencent.com/guanjia/ip/record).

   • The following table lists the match conditions for custom session features:

     Category	Filter Condition	Description
     Session features	Average speed	Total session requests/Session duration in requests/minute.
     	Window speed	Session access frequency for every 2-minute interval (window) in requests/minute.
     	Total sessions	Total number of access requests in a bot session.
     	Session duration	Duration of the bot session.
     	Robots.txt	The session requests the file `Robots.txt`.
     	Session in early morning	The session request is initiated between 2 AM and 5 AM.
     IP features	Access source IP	Access source IP.
     	IP type	IP type, which can be IDC or base station (ISP base station). When the IP type is IDC, exceptions may occur.
     	IP owner	IP owner information such as `tencent.com`, which can be queried in bot details and is valid only if the IP type is IDC.
     Request features	Most requested URL	The most frequently requested URL.
     	URL duplicate rate	Proportion of repeated URLs in session requests, which ranges from 0 to 1. Set this parameter based on your actual business needs. A value that is too high or too low suggests an exception (which must be determined based on the actual business conditions).
     	URL class	Number of URLs in session requests after deduplication.
     	Most requested parameter	The most frequently requested parameter, which may be a GET request parameter (`Query` content) or POST request parameter (`Body` content).
     	Parameter duplicate rate	Proportion of repeated GET request parameters (`Query` content) or POST request parameters (`Body` content) in session requests, which ranges from 0 to 1. Set this parameter based on your actual business needs. A value that is too high or too low suggests an exception (which must be determined based on the actual business conditions).
     COOKIE	COOKIE existence	Used to check whether the HTTP headers of session requests have cookies.
     	Most requested COOKIE	The most frequently requested cookie in session requests.
     	COOKIE duplicate rate	Proportion of repeated cookies in session requests, which ranges from 0 to 1.
     	COOKIE existence rate	Proportion of requests with a cookie among all session requests, which ranges from 0 to 1.
     	COOKIE abuse	Multiple different UAs use the same cookie.
     	COOKIE class	Number of cookies in session requests after deduplication.
     Referer	Referer existence	Used to check whether the HTTP header has a referer in session requests.
     	Most requested referer	The most frequent HTTP referer in session requests.
     	Referer duplicate rate	Proportion of repeated referers in session requests, which is valid only for access through browsers and ranges from 0 to 1. A value that is too high suggests an exception (which must be determined based on the actual business conditions).
     	Referer existence rate	Proportion of requests with a referer in session requests, which is valid only for access through browsers and ranges from 0 to 1. A value that is too low suggests an exception (which must be determined based on the actual business conditions).
     	Referer abuse	Multiple different UAs use the same referer.
     	Referer class	Number of referers in session requests after deduplication.
     UA	UA existence	Used to check whether the HTTP headers of session requests have UAs.
     	Most requested UA	The most frequent HTTP `User-Agent` value in session requests.
     	UA existence rate	Proportion of requests with a UA among session requests, which ranges from 0 to 1. A value that is too low suggests an exception (which must be determined based on the actual business conditions).
     	UA class	Number of UAs in session requests after deduplication, which is valid only for non-proxy IPs. A value that is too high suggests an exception (which must be determined based on the actual business conditions).
     	UA type	Browser. Mobile device. Game console or TV. Public bot. Private bot. Automated tool. Unknown. Public scanner. Framework. Programming language HTTP library.
     	UA randomness index	The UA random distribution status, which ranges from 0 to 1. The higher the value, the more likely the request is exceptional. Reference thresholds: a value above 0.6 suggests an exception; a value above 0.92 is almost certainly an exception.
     Other HTTP headers	Accept existence	The HTTP headers in session requests will be checked for the `Accept` field.
     	Accept-Language existence	The HTTP headers in session requests will be checked for the `Accept-Language` field.
     	Accept-Encoding existence	The HTTP headers in session requests will be checked for the `Accept-Encoding` field.
     	Connection existence	The HTTP headers in session requests will be checked for the `Connection` field.
     	Request method rate	Proportion of the request methods used in session requests.
     	HTTP version rate	Proportion of the HTTP versions used in session requests.
     	Status code returning rate	Proportion of session requests with status codes returned by WAF.
     Advanced features	Prediction tag	Suspicious behavior predicted by the system. Not all sessions have a prediction tag. A prediction tag can indicate the following: Suspected irregular crawler. Suspected regular crawler. Suspected login without user value. Suspected login without user parameter. Suspected login without username and password. Suspected login without user login action. Suspected brute force attack. Suspected credential stuffing attack. Suspected unauthorized access to SMS APIs. Suspected unauthorized access to CAPTCHA APIs. Suspected malicious registration. Suspected repeated API access.
     	Bot score	Bot score for the session given by the intelligent bot analysis engine. The higher the score, the more likely the session is initiated by a bot. The reference value is 5.
     	AI model checkout	Result of AI behavior analysis model detection. If the AI model flags a request, this suggests an exception.
     	Public bot forging	Session requests pretend to be in public bot categories. For example, a request uses the UA of a Baidu crawler, but its IP address does not belong to Baidu.
     	Access to sensitive APIs	Used to determine whether sensitive APIs (such as SMS APIs, registration APIs, and login APIs) are accessed.
     	Index exception for time-series behavior	An algorithm used to detect abnormal time-series behaviors. A smaller index suggests a higher probability of an exception. Reference value thresholds: a value below 0.5 suggests an exception; a value below 0.24 is almost certainly an exception.
     	Index exception of sequential entropy	An algorithm used to detect the time-series behavior entropy. A smaller index suggests a higher probability of exception. The reference value threshold is 0.5. A value below 0.5 suggests an exception.