SPADE: The System S Declarative Stream Processing Engine

Buğra Gedik, Henrique Andrade, Kun-Lung Wu, Philip S. Yu, and MyungCheol Doo.SPADE: The System S Declarative Stream Processing Engine. International Conference on Management of Data, ACM SIGMOD, 2008.

Summary

The paper presents SPADE, a declarative stream processing engine for IBM System S, a real-time stream computing middleware. SPADE is an operator-level declarative programming language supported by a compilation framework consisting of libraries and tools such as parsers, optimizers, and code generators. It provides high-level abstractions for composing large-scale, distributed, and scalable stream processing applications as dataflow graphs. SPADE leverages the underlying System S runtime for performance, scalability, fault tolerance, and distributed job management, while offering an effective compilation and optimization framework for rapid application development. Its intermediate representation includes well-defined, type-generic, and configurable operators and stream adapters that enable flexible construction of distributed streaming applications.

Strong Points

1) SPADE supports flexible windowing schemes, including tumbling and sliding windows with grouping mechanisms, making it suitable for domains such as data mining and signal processing. The SPADE compiler generates application-specific execution plans that are tailored to the underlying System S runtime, taking into account factors such as operator placement, network topology, and execution environment to produce highly optimized distributed streaming applications.

2) SPADE uses learning based statistics collection framework to decide best operator grouping and applies operator fusion to yield PE's optimized for CPU and network load and therefore reducing inter PE transmission delays. Also SPADE compiler supports incremental builds of dynamically changing applications, which reduces the build costs.

3) SPADE provides support for external edge adapters and incremental addition of new operators forming potentially sharable toolkits. These user defined operators can make use of external libraries and implement operations that are customized to a particular application domain.

Weak Points

1) While SPADE and System S can place multiple Processing Elements on the same physical node, the paper does not explicitly address fine-grained intra-operator parallelism, such as mapping multiple threads to a single operator to better exploit multi-core architectures.

2) The model can be more simplified and extended to various domains, and domain specific PE's and operator mapping to PE's is not discussed in the paper.

3) Failure of Processing Elements and their recovery from crashes is not discussed in this paper, instead relying on the underlying System S runtime for fault tolerance without elaborating on PE-level recovery semantics.