Back End Hosting Infrastructure

There are two questions about the infrastructure when a back-end for a web server or a crawler is developed. First, how should the back-end process communicate with the front ends? From the perspective of the framework can be considered as answered because the .NET Framework itself has plenty of communication components (like WCF) to provide the architectural possibilities out of the box.

The second is harder to answer and quite annoying to implement: How should the back-end be hosted? There are several possiblilities too. For example: Hosting in the Web Server (so called IIS hosting), hosing in a Windows service, hosting in a Linux deamon, hosting in the cloud,  integrate in the front-end, and so on. These solutions have some things in common: They are propertiary, monolythic, unflexible not very robust and therefore crap. These are the problems:

Hanging Worker Process

Sometimes it happens that a back-end or worker process hangs. If this is a Windows service, Linux daemon or a process in the cloud the game is over. The Microsoft Internet Information Server have faced this and isolated his worker processes. A master process monitors them and terminates and restarts them when they are not working. So the process comes at least back and the operation can continue. But this is not very controllable and sometimes all worker processes disappear because the web site was idle for a longer time. It’s no solution for back-ends with a lot of background tasks.

…