Crawler-Lib Engine Code Samples

You can download the C# code samples for the Crawler-Lib Engine at once. It is available as a zipped Visual Studio Solution and contains all the listed code samples. Keep in mind that the included TestWebApp project must run on a development webserver on port 10080 to run the samples. Please use the Suppport Forum to discuss issues concerning the samples.

Crawler-Lib Engine Samples Solution Download

Crawler-Lib Engine Support Forum

Samples in the Visual Studio Solution

Each sample contains general information about the design and the usage of the Crawler-Lib Engine. The samples are intend to be an introduction for programmers.

• Simple Task Sample: Crawl a Website and Extract all Links
  It shows the general building block of tasks that can be processed by the Crawler-Lib Engine. It shows the building of a task, with a task request and a task result and introduces the StartWork() method – the entry point of the task workflow. The workflow contains simply a HTTP request and extracts the links with the Html Agility Pack. The request itself uses the async/await pattern to provide the success handler.
• Website Crawler Sample
  A basic implementation of a crawler that crawls a website starting from a certain URL. It follows the links to a certain depth and crawls a certain amount of pages. It also sends status Email messages to monitor the crawling progress. This sample shows how the CrawlerEngine can be controlled from outside, tasks are scheduled and results are processed. It includes also a SendMailTask as a sample how different tasks are started and processed on the same engine and how non HTTP tasks can be designed.   
• Workflow Control Elements Sample
  This sample shows the concept of a crawler workflow and the various workflow control elements. It shows ho sequences of operations can be coded and how to execute operations in parallel. It is also a basic introduction of the Delay, Group, Retry, Expense and the Limit workflow elements. 

Articles / Blog Posts

The articles and blog posts can have their own source code associated. Download the ZIP file linked in the article or blog post.

• Social Media APIs and Cloud Service Operation Costs Measurement: Expense Workflow Element
• Twitter Crawler and Status Analyzer with the Crawler-Lib Engine

Video Tutorials