Is it possible to set/change User Agent string when I do the HttpRequest?
I found no property about this. Is there something, I'm not seeing?

Thanks.

This can be done by adding it to the headers collection of the HttpRequestConfig. But it is not very nice, I will introduce properties for this.

Pending a new version with Headers properties headers, I do this

Before :

HttpRequest request = await new HttpRequest(new Uri(new Uri("http://localhost:10080/), 
                                                                   new HttpRequestQuota { MaxDownloadSize = 100000, OperationTimeoutMilliseconds = 10000, ResponseTimeoutMilliseconds = 5000 },
                                                                   AwaitProcessingEnum.All);

I replace by :

HttpRequest request = await new HttpRequest( new HttpRequestConfig
                                                    {
                                                        Url = new Uri("http://localhost:10080/"),
                                                        Quota = new HttpRequestQuota { MaxDownloadSize = 100000, OperationTimeoutMilliseconds = 10000, ResponseTimeoutMilliseconds = 5000 },
                                                        Headers = new List<KeyValuePair<string, string>>() { new KeyValuePair<string, string> ("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0") },
                                                        AwaitProcessing = AwaitProcessingEnum.All
                                                    });

It's OK?

Yes that is Ok. I have taken the header issue on priority.

I've been assigned to that issue. We are currently implementing the common header properties. Have you additional suggestions for this? I will merge it with main and trigger the build process. When everything is ok, (unit tests, test deploy, …) i will trigger the release process. I think we can do this this eveneing.
  

Good new 🙂
Tell me when release version will be available on Nuget. I test it with my current devellopement.

We have finished the releasing of version 2.3.5529.650. nuget packages and zip files. you can update your project.

Can you explain what has been added in this version?



I design my crawler with your example Website Crawler. With this new version, I encounter an error on this line

this.engine.AddLimiter("PageRequestLimiter",  new QuotaLimiter( new QuotaLimiterConfig { MaxRunningLimited = 20 }));

It seems that AddLimiter method no longer accepts 2 parameters but one of type ILimiter.
But I found no example, to add limiter to the engine with type of ILimiter.

Documentation about this is no up to date : CrawlerEngine.AddLimiter Method

EDIT : I found !

this.engine.AddLimiter(new QuotaLimiter(new QuotaLimiterConfig { Name = "PageRequestLimiter", MaxRunningLimited = 20 }));

[b][/b]

the change was in version 2.3.5513.24819 not in the latest version.
Breaking: AddLimiter() takes name form LimiterConfig now.

https://www.nuget.org/packages/CrawlerLib.Engine

it makes no sens that the AddLimiter() method and the configuration has a Name property / parameter.
set name in LimiterConfig and all is ok.

OK.

an example on how to use header properties in HttpRequest?