Social media service APIs, cloud services or even internal datacenter resources have operation costs that must sometimes be measured. The Expense workflow element is an upcoming feature of the Crawler-Lib Engine. It introduces a generalized measuring mechanism for operation costs. It can be used when a task accesses APIs like the Facebook Graph API, Facebook FQL , Twitter REST API, Google GData API, Microsoft Azure, Amazon EC2 and so on. It tracks how many costs this API calls have used and delivers them aggregated in the task result.
It works hand in hand with another new workflow element called Limit, which allows limiting the parallelism and the throughput. In conjunction with the Expense workflow element workflows can be limited based on the operation costs they will produce.
With these upcoming features it is easy to access those APIs respecting the quotas and without exceeding any limits. It is also possible to collect internal operation costs and bill them to the customers as a cost based service fee.