Enforcing a Web Scraping Policy

To enforce a web scraping policy for a web application, perform the following steps:

1. Add a Bot to the Allowed Bots List

2. Create a Web Scraping Policy

3. Associate the Web Scraping Policy with a Service

    

1. Go to the BOT MITIGATION > Libraries page, and click Add Bot in the Allowed Bots section.

2. In the Add Bot page, specify values for the parameters (Parent Name, User Agent and Host).

3. Click Save.

1. Go to the BOT MITIGATION > Bot Mitigation page, and click Add Policy in the Web Scraping Policies section.

2. In the Add Web Scraping Policy page:

   1. Specify values for the parameters under Honey Traps and Bot Detection.

   2. Select the allow-listed bot created in Step 1 - Add a Bot to the Allowed Bots List.

   3. Select the categories that you want to block.

   4. Click Save.

1. Go to the BOT MITIGATION > Bot Mitigation page.

2. In the Bot Mitigation Policy section:

   1. Identify the service that you want to associate the web scraping policy to.

   2. Click Edit next to the URL policy associated with the service.

3. In the Edit URL Policy page:

   1. Set Status to On.

   2. From the Bot Detection section, select the policy created in Step 2 - Create a Web Scraping Policy from the Web Scraping Policy list.

   3. Specify values for other parameters as required and click Save.