Hybrid CEP Engine

Description

A Hybrid Solution proposed for near-real-time Complex Event Processing in a resource-constrained environment. This is achieved through the combination of two existing state-of-the-art engines, namely the SASE system, which was originally introduced in SASE, and a CET (Complete Event Trend) detection solution, proposed by Mei H. The latter departs from the NFA rationale, and it is capable of trading latency for significantly lower resource usage in such a beneficial manner that, in practice, it can run much more queries than previously.

---

Architecture

---

Engine Decision Mechanism

Decides the appropriate engine depending on the input query, based on predefined rules. These rules were generated using insights extracted from the experiments.

The criteria considered before deciding on the execution engine are:

• (a) the presence or absence of Kleene events in the sought pattern,
• (b) the strategy by which the relevant events will be selected from the incoming input stream,
• (c) the size of the sliding window imposed by the user.

If the pattern contains Kleene, the selection strategy is Skip-Till-Any-Match, and the window length >= 100 then CET engine is employed. Otherwise the selected engine is that of SASE.

---

Compilation

For Linux users, first install maven, if necessary, by running sudo apt install maven.

Run mvn package under /implementation folder. Then, a JAR named hybridEngineCEP-v01.jar is created under /target folder

---

Input Parameters

1. Required parameters:

-q, --query          Path of query file

-i, --inputStream    Path of Input Stream file

-t, --eventtype      Type of events: stock or check

2. Optional parameters:

-e, --engine         The engine to run: sase or cet

-p, --parallelism    Degree of parallelism for cet

-w, --write          Whether to write or not the output

-o, --output         The output file for results

-c, --conf           The stream configuration file

3. Guide:

-h, --help           Prints the helper, with required and optional parameters

---

Run

java -jar target/hybridEngineCEP-v01.jar [options]

---

Output

Profiling Numbers

• Engine used: SASE OR CET
• Total Running Time: SECONDS seconds
• Number Of Events Processed: EVENTS
• Number Of Runs Created: RUNS
• Number Of Matches Found: MATCHES
• Used memory is bytes: BYTES
• Used memory is megabytes: MB
• Number of cets per slide: (only for CET engine)
• Maximum Latency per slide in nano: (only for CET engine)
• Minimum Latency per slide in nano: (only for CET engine)
• Average Latency per slide in nano: (only for CET engine)
• Maximum Latency in nano: MAX_L
• Minimum Latency in nano: MIN_L
• Average Latency in nano: AVG_L
• Throughput: X events/second

Examples

1. Circular Kiting Fraud Detection (using CET engine)

java -jar target/hybridEngineCEP-v01.jar -q queries/qkite.query -i datasets/kite-big-transf.stream -t check -e cet -w -o output/results-kite.res -p 8

2. Sub-sequences with same symbol (using SASE engine)

java -jar target/hybridEngineCEP-v01.jar -q queries/qstock1.query -i datasets/stock-random.stream -t stock -e sase -w -o output/results-symbol.res 

3. Sub-sequences with same price (using SASE engine)

java -jar target/hybridEngineCEP-v01.jar -q queries/qstock2.query -i datasets/dataset-test.stream -t stock -e sase -w -o output/results-price.res 

Stream Generators

There are two stream generators under the folder datasets/generators/.

Run python checkGen.py -h or python stockGen.py -h to find out the required and optional parameters for each generator.