Features of Bright Data Crawl API

The Crawl API provides several key features:

• Customizable crawling depth

  Control how deep the crawler goes into your target websites

• Filter URLs with regular expressions

  Use regex patterns to target specific URLs and content

• Option to ignore sitemaps

  Streamlined crawling by bypassing sitemap restrictions