This repository provides XML code examples that use SAS Event Stream Processing to process real-time streaming data. Project code is written to run in SAS Event Stream Processing Studio. Use these examples with SAS Event Stream Processing 2020.1 and later, unless stated otherwise.
To obtain XML code examples that run with earlier releases of SAS Event Stream Processing, visit the SAS Event Stream Processing Code Examples download page.
Beginning with SAS Event Stream Processing 2023.09, you can install some examples directly from SAS Event Stream Processing Studio:
- In SAS Event Stream Processing Studio, click
on any page. - Select Examples.
- In the SAS Event Stream Processing Examples window, select an example with the Install example button in the right pane.
- Click Install example. A project package ZIP file is imported from the relevant example's directory in this GitHub. The project XML file in the project package has connectors that point to input files in the product package. As a result, you do not need to configure connectors.
- Run the imported example in test mode. For more information, see Running a Test.
For more information about how the example works, return to the SAS Event Stream Processing Examples window, select the example and click Go to GitHub to read the example README. When you are viewing an example project in SAS Event Stream Processing Studio, you can also select each window in turn in the workspace to read brief descriptions of those windows in the right pane.
When the Install example button is not available but the example's directory contains a project package ZIP file, you can manually import the project package to SAS Event Stream Processing Studio, but only when you are using SAS Event Stream Processing 2023.09 or later. For more information, see Import a Project.
Note: The remainder of the files in each example's directory might differ slightly from the files in the project package. For example, the project XML file might not include connectors.
For more information about project packages in general, see Project Package.
Some examples in this repository do not include a project package. To use these examples, complete the following steps:
- Save the contents of the directory for a specific example to a convenient location on your computer.
- Open SAS Event Stream Processing Studio and sign in with your credentials. For more information, see Accessing SAS Event Stream Processing Studio.
- When you are using an edge server, you must manually start an ESP server. For more information, see Add or Edit an ESP Server to run on an Edge Server. If you are using a Kubernetes cluster, an ESP server is created automatically on the cluster and you are not required to manually start it.
- Import the project XML file to SAS Event Stream Processing Studio. For more information, see Import a Project.
- Open the project in SAS Event Stream Processing Studio and examine its contents. You can select each window in turn in the workspace to read brief descriptions of those windows in the right pane.
- Use test mode to verify that the example operates as intended. For more information, see Running a Test.
Note: When you first enter the test mode you do not see data streaming through the model. This is expected behavior unless publisher connectors have been configured. In step 7, you publish events from the project's CSV file and data streams through the project. - When you run a project in test mode, in most cases you can publish events from the project's corresponding CSV file by clicking the Publish button. This enables you to send events to a Source window without configuring a publisher connector. The CSV file can reside in a convenient location on your computer without having to upload it to a Kubernetes persistent volume. For more information, see Publish Events from a CSV file.
Note: For projects that contain multiple Source windows, you must publish events for each Source window separately. You cannot publish events to multiple Source windows simultaneously.
Alternatively, you can configure publisher connectors prior to running the project in test mode. For more information, see Configure a File and Socket Connector.
Important: When connector groups are used (for example, in the geofence and sailing examples), it is not appropriate to publish events from the project's CSV file when in test mode. Instead, it is recommended that you configure publisher connectors to the relevant CSV file and connector groups in SAS Event Processing Modeler. Follow the instructions for each specific example.
The GettingStarted directory contains a set of projects to help you get started quickly. Window code contains a description that describes its purpose.
| Example | Description |
|---|---|
| aggregate | Apply aggregation functions to a group of stock market trades. When new events arrive in the Aggregate window, they are placed into aggregate groups. After a new event enters the event stream, the aggregation functions are called. The functions then modify the aggregation groups accordingly. Aggregate field calculation functions or expressions that are registered to the Aggregate window must appear in non-key fields, which in this project are totalQuant and maxPrice. |
| compute | Use a Compute window to match people from the city of Apex. You can use Compute windows to project input fields from one event to a new event and to augment this event with fields that result from a calculation. |
| fitstat | Use the FitStat (fit statistics) algorithm. The goodness-of-fit of a statistical model describes how well a model fits a set of data. The measures summarize the difference between observed values and predicted values of the model under consideration. |
| join | Join stock market trades with the corresponding traders. A Join window receives events from a left input window and a right input window. It produces a single output stream of joined events. Joined events are created according to a join type and join conditions. |
| splitter | Use a splitter. You can use expressions to define window-output splitter-slot calculations (for example, you can use an expression to evaluate where to send a generated event). In this project, the splitter works with user-defined functions to calculate the slot number to decide which Copy window the event goes to. |
The Basic directory contains projects that demonstrate how to use a few windows. See also GettingStarted.
| Example | Description |
|---|---|
| lua_compute | Use Lua code to modify user data. The Lua window in this example is used as an alternative to a Compute window. Note: Use this example with SAS Event Stream Processing 2021.2.1 and later. |
| lua_connector | Use a Lua connector. This example contains a Source window with a Lua connector that reads RSS feeds from scientific sites and publishes them into the project. Note: Use this example with SAS Event Stream Processing 2022.10 and later. |
| lua_parse | Use Lua code to parse JSON data and generate multiple events. This example contains two projects. The Lua window in these projects is used as an alternative to a Functional window. The Basic Example Project contains step-by-step instructions about how to run the project. The Advanced Example Project is for demonstration purposes. Note: Use these examples with SAS Event Stream Processing 2022.10 and later. |
| lua_state | Use Lua code to maintain and use event state to generate events. The Lua window in this project generates events every time a new stock symbol price exceeds the current maximum price for that stock symbol. Note: Use these examples with SAS Event Stream Processing 2022.10 and later. |
| pattern_empty_index | Identify increases in a stock's price within a specific time interval. The Pattern window uses Lua code to define events of interest (EOI) to be matched. Note: Use this example with SAS Event Stream Processing 2022.1.2 and later. |
| removestate | Transition a model from stateful to stateless. This example demonstrates how to facilitate the transition of a stateful part of a model to a stateless part of a model. |
| union | Merge three event streams of stock market trades together. A Union window unites two or more event streams using a strict policy or a loose policy. |
The Advanced directory contains projects that demonstrate the use of a larger number of windows together, or that are otherwise more complex than examples in the Basic or GettingStarted directories.
| Example | Description |
|---|---|
| activitytracker | Use a job template to deploy a project to an edge server. This example processes GPS data from activity tracker devices worn by players in a football match. The example demonstrates how you can publish a project in SAS Event Stream Processing Studio to make it available in SAS Event Stream Manager, and then use a job template to deploy the project to an edge server. |
| geofence | Display a list of wanted vehicles found near critical infrastructure sites. This example uses Join, Geofence, and Filter windows to match wanted vehicles that are in close proximity to critical infrastructure sites. |
| lua_snippet | Use a Lua snippet. This example simulates health-care data streaming into a model and detects measurements that fall outside a specified range. A Lua snippet in the project subscribes to windows, stores values, and reads a JSON file. Lua connectors are also used to generate and inject data. Note: Use this example with SAS Event Stream Processing 2023.04 and later. |
| onnx_object_detection | Use an ONNX model to detect objects. This example demonstrates how to use a project to reference an open-source ONNX model, in order to detect objects in a video. Note: Use this example with SAS Event Stream Processing 2023.09 and later. |
| sailing | Visualize data obtained from a set of boats. This example identifies two geographical areas of interest. One area is an exclusion zone that the boats identified are not permitted to enter. The other is an area where a speed restriction has been applied. |
| slot_exp_xml | Split events across output slots. The Compute window in this example determines which output slot or slots should be used for each stock market event. Three Copy windows connect to the Compute window using different output slots. |
| trades | Identify large trades in stock market transactions. This project identifies large securities transactions and the traders who were involved in those trades. |
| transpose_long | Transpose data from an aircraft, in long mode. This project conceptualizes an event as a row that consists of multiple columns. You can use a Transpose window to interchange an event's rows as columns, and columns as rows. You will process information about the pitch, yaw, roll, and velocity of an aircraft in flight. |
| transpose_wide | Transpose data from an aircraft, in wide mode. This project conceptualizes an event as a row that consists of multiple columns. You can use a Transpose window to interchange an event's rows as columns, and columns as rows. You will process information about the pitch, yaw, roll, and velocity of an aircraft in flight. |
The Analytics directory contains projects that show how to use various algorithms that are packaged with SAS Event Stream Processing.
| Example | Description |
|---|---|
| analytics_cepstrum | Use the Cepstrum Transform (Cepstrum) algorithm. A cepstrum results from taking the inverse Fourier transform of the logarithm of the estimated spectrum of a signal. This application is often used in speech analysis, specifically with voice detection. |
| analytics_changedetection | Use the Change Detection (ChangeDetection) algorithm. With change detection, a stream of measures is monitored and an alert is raised when values deviate from what is expected. |
| analytics_correlation | Use the streaming Pearson's correlation algorithm. Pearson's correlation coefficient is the most common measure of how data correlate with one another. It shows the linear relationship between two sets of data. |
| analytics_dbscan | Use DBSCAN clustering. It applies an insert-only input stream that consists of the following: 1. An ID that acts as the streams key. 2. An x coordinate of data. 3. A y coordinate of data. |
| analytics_distributionfitting | Use the Distribution Fitting algorithm. The algorithm supports fitting a Weibull, Gamma, or Normal distribution to a variable in the incoming data stream. |
| analytics_fitstatistics | Use the FitStat (fit statistics) algorithm. The goodness-of-fit of a statistical model describes how well a model fits a set of data. The measure summarizes the difference between observed values and predicted values of the model under consideration. |
| analytics_kmeans | Use k-means clustering. K-means clustering is often used in data mining. The algorithm assigns data points to their nearest cluster centroid. Each cluster centroid is then recomputed based on the average of the cluster's data points. In k-means clustering, an input event is augmented with a cluster number. This indicates the cluster that the observation falls into. |
| analytics_lagmonitoring | Use the Lag Monitoring (LagMonitor) algorithm. The Lag Monitoring algorithm computes the cross-correlation between a target time series and one or more additional time series. Results contain the selected lags and computed cross-correlation values that correspond to minimum, maximum, and maximum absolute value cross-correlations for each of the variables. |
| analytics_linearregression | Use online linear regression on streaming data. This project has an insert-only input stream that consists of the following: 1. An ID that acts as the streams key. 2. A y coordinate of data. 3. 784 x coordinates. |
| analytics_logisticregression | Use online logistic regression on streaming data. This project has an insert-only input data stream that consists of the following: 1. An ID that acts as the stream's key. 2. A y coordinate of data. 3. 784 x coordinates. |
| analytics_modelReader_digitalFilter | Score data by using SODFIL.sasast. This project shows how to use a Model Reader window to facilitate the scoring of an input data set using a pre-trained analytic store model. The analytic store model is SODFIL.sasast, which is a Digital Filter analytic store of Butterworth filter type. The analytic store was created using CAS. |
| analytics_modelReader_RPCA | Score data by using RPCA.sasast. This project shows how to use a Model Reader window to facilitate the scoring of an input data set using a pre-trained analytic store model. The analytic store model is RPCA.sasast, which is a Robust Principal Components Analysis (RPCA) analytic store. The analytic store was created using CAS. |
| analytics_movingrelativerange | Use the moving relative range (MRR) algorithm. The moving relative range provides a measure of volatility for a nonstationary time series, where the mean and variance of the series changes over time. |
| analytics_ROC | Use the receiver operating curve characteristic (ROC) information algorithm. ROC information shows the diagnostic ability of a classifier system as you vary its discrimination threshold. |
| analytics_segmentedcorrelation | Use the segmented correlation algorithm. Segmented correlation is similar to autocorrelation. It specifies the correlation between the elements of a series and others from the same series that are separated from them by a specified interval. |
| analytics_STFT | Use the Short-Time Fourier Transform (STFT) algorithm. A Fourier transform decomposes a function of time into its underlying frequencies. The amplitude, offset, and rotation speed of every underlying cycle is returned by the function. STFT is commonly used to monitor the time-varying frequency content of a signal. It can be used in digital filters to detect anomalies in continuous streams of data. |
| analytics_streaminghistogram | Use the streaming histogram algorithm. The streaming histogram algorithm processes a stream of numerical data and puts it in bins to generate boundaries in order to create a histogram that fits the data. |
| analytics_subspacetracking | Use the Subspace Tracking (SST) algorithm. When a data set contains a sequence of nx1 vectors: x(t), subspace tracking estimates the covariance matrix for each vector x(t) and then computes the first p principal eigenvectors of the covariance matrix. For each iteration at time t, the covariance matrix C(t) is obtained. SST can be applied to industrial data to detect outliers. Results can be used to identify potential errors before they occur. |
| analytics_summary | Use the streaming summary statistics algorithm. The algorithm is used in a Calculate window, which calculates univariate summary statistics for input variable data that it receives from a Source window. |
| analytics_supportvectormachine | Use a support vector machine. Support vector machines are supervised learning models with associated algorithms. They apply classification and regression analysis on incoming data. You supply training examples and mark them as belonging to a category. A support vector machine builds a model that assigns new examples to that category. |
| analytics_texttokenization | Use a tokenization algorithm. This project shows how to use the tokenization algorithm to perform text tokenization on streaming input. |
| analytics_textvectorization | Use the TextVectorization algorithm. Vectorizing text creates maps from words or n-grams to vector space. A vector space is an algebraic model to represent text documents as vectors of identifiers (for example, index terms). |
| analytics_TFIDF | Use the Term Frequency - Inverse Document Frequency (TFIDF) algorithm. TFIDF can be used as a weighing factor in text mining or general information searches. |
| analytics_tSNE | Use the t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm. The t-SNE algorithm is a machine learning algorithm for dimensionality reduction that is used to visualize high-dimensional data sets. |
The following examples in the Advanced directory contain practical use cases.
- activitytracker
- geofence
- onnx_object_detection
- sailing
- trades
The following examples demonstrate how you can use Lua code in projects.
- lua_compute
- lua_connector
- lua_parse
- lua_snippet
- lua_state
- geofence
- pattern_empty_index
- trades
We welcome your contributions! Please read CONTRIBUTING.md for details on how to submit contributions to this project.
This project is licensed under the Apache 2.0 License.