Bot protection configuration allows you to set bot protection policies based on the characteristics of bot session behaviors and take corresponding actions. You can also observe and analyze bot details and fine-tune the policies based on the provided session status details to ensure the security of your website's core APIs and businesses. This feature supports two types of protection policies, public policies and custom session policies.
WAF provides protection against 12 public categories of bots with over 1,000 subcategories, such as search engine, speed tester, content aggregator, scanner, and website crawler. You can set protection actions (passing, monitoring, and blocking) for public bots as needed, and WAF will process bot requests that hit a public category accordingly.
WAF custom session policies allow you to set features of protocols, IP intelligence, and custom sessions, each of which can be determined in multiple dimensions. You can set custom session policies and the processing actions (passing, monitoring, CAPTCHA, redirect, and blocking) based on your actual business needs and bot details, and WAF will process bot requests that hit a custom protection policy accordingly.
Note:
Only WAF Enterprise and Ultimate Editions support bot behavior management. We recommended WAF Advanced users upgrade to the Enterprises or Ultimate Edition.
Log in to the WAF Console and choose Bot Behavior Management -> Bot Protection Settings on the left sidebar to go to the bot policy settings page.
Field description:
Note:
BOT Switch takes effect only when WAF Switch is enabled.
Field description:
- Categories: WAF can identify 12 public bot categories, including search engine, speed tester, content aggregator, scanner, and website crawler.
- Number of BOT Classes: number of BOT subcategories in each category.
- Action: the action supported by the corresponding public bot category, which is "Monitor" by default. You can set it to "Pass" or "Block" in the "Operation" column on the right. You can view the blocking results in Attack Log and view the IP addresses blocked in real time in IP Blocking Status.
- Operation: the action taken for the corresponding public bot category. For more information, see Action Category Description.
3. In the upper-left corner, click Copy to copy the public bot category settings of the current domain name to another one whose original settings will be overridden.
Protocol Feature Category | Policy Name |
---|---|
User-Agent | User-Agent is empty or does not exist. |
User-Agent is BOT. | |
User-Agent is Unknown BOT. | |
User-Agent is HTTP Library. | |
User-Agent is Tools. | |
User-Agent is Framework. | |
User-Agent is Scanner. | |
HTTP header | Referer is empty or does not exist. |
Referer abuse (multiple UAs use the same Referer). | |
Cookie abuse (multiple UAs use the same Cookie). | |
Cookie is empty or does not exist. | |
Connection is empty or does not exist. | |
Accept is empty or does not exist. | |
Accept-Language is empty or does not exist. | |
Accept-Encoding is empty or does not exist. | |
HTTP features | The HTTP HEAD method is used. |
The HTTP version is 1.0 or earlier. |
Log in to the WAF Console and choose Bot Behavior Management -> Bot Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the "Operation" column on the right.
On the Defense settings page, select Custom Session Policy > IP Intelligence Characteristics to enter the corresponding list page.
Field description:
Note:
This is used to automatically check the health of your domain names. Do not set it to "Block".
The following table lists the names of IP intelligence feature policies.
IP Intelligence Category | Policy Name |
---|---|
Automated testing | Automated testing provided by Tencent Cloud WAF. |
IDC-IP library | IDC-IP library - Tencent Cloud. |
IDC-IP library - Alibaba Cloud. | |
IDC-IP library - Huawei Cloud. | |
IDC-IP library - Kingsoft Cloud. | |
IDC-IP library - Ucloud. | |
IDC-IP library - Baidu Cloud. | |
IDC-IP library - JD Cloud. | |
IDC-IP library - QingCloud. | |
IDC-IP library - AWS. | |
IDC-IP library - Azure. | |
IDC-IP library - Google Cloud. |
Log in to the WAF Console and choose Bot Behavior Management -> Bot Protection Settings on the left sidebar. Find the target domain name and click Defense settings in the "Operation" column on the right.
On the Defense settings page, choose Custom Session Policy > Custom Session Features to go to the corresponding list page.
Add custom session features. Click Add in the upper-right corner of the list to go to the Add custom session features page.
Note:
Policy priority is determined by the action category in the order: Pass > Monitor > Redirect > CAPTCHA > Block. For policies of the same action category, the more recently a policy was added, the higher its priority.
Category | Description |
---|---|
Pass | Session requests that match the set conditions will be allowed and will not be logged. |
Monitor | Session requests that match the set conditions will be monitored. You can view more information on the monitored sessions in the custom bot details. |
CAPTCHA | This is applicable only to access through browsers. Session requests that match the set conditions will be verified through a CAPTCHA. If they fail verification, they will be blocked. Otherwise, they can normally access the domain name within the penalty period. |
Redirect | Session requests that match the set conditions will be redirected to a specified URL of the current domain name within the specified penalty period. |
Block | Session requests that match the set conditions will be blocked. You can set the penalty period to 5 to 10,080 minutes (7 days). You can view the blocking results in the [Attack Log], and view the IP addresses blocked in real time in [IP Blocking Status]. |
Category | Filter Condition | Description |
---|---|---|
Session features | Average Speed | Total Session Requests/Session Duration in requests/minute. |
Window Speed | Session access frequency for every 2-minute interval (window) in requests/minute. | |
Total Sessions | Total number of access requests in a bot session. | |
Session Duration | Duration of the bot session. | |
Robots.txt | The session request wants to access the `Robots.txt` file. | |
Session in early morning | The session request is initiated between 2:00 AM and 5:00 AM. | |
IP features | Access real IP | Access real IP. |
IP Type | IP type, which can be IDC or base station (ISP base station). When the IP type is IDC, exceptions may occur. | |
IP Owner | IP owner information such as `tencent.com`, which can be queried in bot details and is valid only if the IP type is IDC. | |
Request features | Most Requested URL | The most frequently requested URL. |
URL Duplicate Rate | Proportion of repeated URLs in session requests, which ranges from 0 to 1. Set this parameter based on your actual business needs. A value that is too high or too low suggests an exception (which must be determined based on the actual business conditions). | |
URL Class | Number of URLs in session requests after deduplication. | |
Most Requested Parameter | The most frequently requested parameter, which may be a GET request parameter (`Query` content) or POST request parameter (`Body` content). | |
Parameter Duplicate Rate | Proportion of repeated GET request parameters (`Query` content) or POST request parameters (`Body` content) in session requests, which ranges from 0 to 1. Set this parameter based on your actual business needs. A value that is too high or too low suggests an exception (which must be determined based on the actual business conditions). | |
COOKIE | COOKIE Existence | Used to check whether the HTTP headers fo session requests have cookies. |
Most Requested COOKIE | The most frequently requested cookie in session requests. | |
COOKIE Duplicate Rate | Proportion of repeated cookies in session requests, which ranges from 0 to 1. | |
COOKIE Existence Rate | Proportion of requests with a cookie among all session requests, which ranges from 0 to 1. | |
COOKIE Abuse | Multiple different UAs use the same cookie. | |
COOKIE Class | Number of cookies in session requests after deduplication. | |
Referer | Referer Existence | Used to check whether the HTTP header has a referer in session requests. |
Most Requested Referer | The most frequent HTTP referer in session requests. | |
Referer Duplicate Rate | Proportion of repeated referers in session requests, which is valid only for access through browsers and ranges from 0 to 1. A value that is too high suggests an exception (which must be determined based on the actual business conditions). | |
Referer Existence Rate | Proportion of requests with a referer in session requests, which is valid only for access through browsers and ranges from 0 to 1. A value that is too low suggests an exception (which must be determined based on the actual business conditions). | |
Referer Abuse | Multiple different UAs use the same referer. | |
Referer Class | Number of referers in session requests after deduplication. | |
UA | UA Existence | Used to check whether the HTTP headers of session requests have UAs. |
Most Requested UA | The most frequent HTTP `User-Agent` value in session requests. | |
UA Existence Rate | Proportion of requests with a UA among session requests, which ranges from 0 to 1. A value that is too low suggests an exception (which must be determined based on the actual business conditions). | |
UA Class | Number of UAs in session requests after deduplication, which is valid only for non-proxy IP addresses. A value that is too high suggests an exception (which must be determined based on the actual business conditions). | |
UA Type |
| |
UA Randomness Index | The random distribution status, which ranges from 0 to 1. The higher the value, the more likely the request is exceptional. Reference thresholds: a value above 0.6 suggests an exception; a value above 0.92 is almost certainly an exception. | |
HTTP Header | Accept Existence | The HTTP headers in session requests will be checked for the `Accept` field. |
Accept-Language Existence | The HTTP headers in session requests will be checked for the `Accept-Language` field. | |
Accept-Encoding Existence | The HTTP headers in session requests will be checked for the `Accept-Encoding` field. | |
Connection Existence | The HTTP headers in session requests will be checked for the `Connectiton` field. | |
Request Method Rate | Proportion of session requests with request methods. | |
HTTP Version Rate | Proportion of session requests that use HTTP versions. | |
Returned Status Code Rate | Proportion of session requests with status codes returned by WAF. | |
Advanced Features | Prediction Tag | Suspicious behavior predicted by the system. Not all sessions have a prediction tag. A prediction tag can indicate the following:
|
Score | Bot score for the session given by the intelligent bot analysis engine. The higher the score, the more likely the session is initiated by a bot. The reference value is 5. | |
AI Model Checkout | Result of AI behavior analysis model detection. If the AI model flags a request, this suggests an exception. | |
Public BOT Forging | Session requests pretend to be in public bot categories. For example, a request uses the UA of a Baidu crawler, but its IP address does not belong to Baidu. | |
Access to Sensitive API | Used to determine whether sensitive APIs (such as SMS APIs, registration APIs, and login APIs) are accessed. | |
Index Exception for Time-series Behavior | An algorithm used to detect abnormal time-series behaviors. A smaller index suggestions a higher probability of an exception. Reference value thresholds: a value below 0.5 suggests an exception; a value below 0.24 is almost certainly an exception. | |
Index Exception of Sequential Entropy | An algorithm used to detect the time-series behavior entropy. A smaller index suggests a higher probability of exception. The reference value threshold is 0.5. A value below 0.5 suggests an exception. |
Was this page helpful?