Crawler-Lib Engine

The Crawler-Lib Engine is a fully generalized task processor. Although it is way beyond a crawler it can be used to develop high performance bots (robots) for web crawling and web scraping. But also data extraction, publishing and data distribution applications can use it in C#, Visual Basic and any other .NET programming language. It is optimized to perform thousands of requests per second. It has a great support for the async pattern (IAsync implementations) and the Task Parallel Library (TPL) to perform async and parallel requests of any kind. Due to its workflow capabilities it can be used to perform complex request sequences with dependencies. For example a web crawling workflow: Establish a cookie, fill a form, post it and process the results.

Get Started

Download and Install

Please download and install the Crawler-Lib Engine NuGet Package in your project with the NuGet Package Manager. A license file for the free “Anonymous Edition” is included in the packages so the Crawler-Lib Engine can be used and tested without registering as customer. All packages are licensed under the Crawler-Lib Framework End User License Agreement. You accept the terms and conditions of this agreement by downloading and installing the package. To generate a license file for your free or purchased edition go to the Downloads Page and click the “Generate License” link. Follow the installation instructions there.

Code Samples

Take a look at the code samples section to see the available samples and to get an overview over the capabilities. Please download also the Code Samples (Visual Studio Solution with all C# Samples) and watch the Tutorials and C# Sample Videos on YouTube.

Compare Editions