You can download the C# code samples for the Crawler-Lib Engine at once. It is available as a zipped Visual Studio Solution and contains all the listed code samples. Keep in mind that the included TestWebApp project must run on a development webserver on port 10080 to run the samples. Please use the Suppport Forum to discuss issues concerning the samples.
Each sample contains general information about the design and the usage of the Crawler-Lib Engine. The samples are intend to be an introduction for programmers.
StartWork()method – the entry point of the task workflow. The workflow contains simply a HTTP request and extracts the links with the Html Agility Pack. The request itself uses the async/await pattern to provide the success handler.
CrawlerEnginecan be controlled from outside, tasks are scheduled and results are processed. It includes also a
SendMailTaskas a sample how different tasks are started and processed on the same engine and how non HTTP tasks can be designed.
The articles and blog posts can have their own source code associated. Download the ZIP file linked in the article or blog post.