Web Crawlers

We have a bunch of sophisticated crawlers designed to scrape websites. Crawlers are configured via source records.

The heavy-lifting is done right out of hardcore testing applications: playwright. We have some tricks involving running headless (but actually not) in order to mitigate the arms race against bots and crawlers that our target sources array against us.