Overview

Locust is a distributed web data discovery and collection framework intended to enable non-power users to quickly create web crawlers and web scrapers that benefit from the cost effciences of serverless technologies and can process modern dynamic web pages.

The design goals of Locust are:

• Lower the technical barrier of entry to building a web crawler/scraper
• Shorten the build stage of the build-measure-learn cycle for web data collection
• Simplify the use of serverless platforms for purposes of web data collection