Storage Operation Processing

The database or in general storage or persistence service is one of the big bottlenecks of any application or service. These days everything is massive parallel and event driven which overwhelm most storage systems with a vast amount of small storage operations. We have this in the back-end,  especially with our Crawler-Lib Engine because it can deliver ten thousands of task results per second. A web application is not much better, it starts at leas for every page request a storage operation (often many).

The Crawler-Lib Storage Operation Processor is the remedy for this. It reduces the degree of parallelism to the optimum for the underlying storage system. It groups and reorders the pending storage operations to get more cache hits and less materializations for certain data. And last but not least it prioritizes interactive foreground storage operations before background task storage operations, so that the web site or user interface keeps responsive even under heavy load.