Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions docs/docs/internals/runtime.md
Original file line number Diff line number Diff line change
@@ -1,58 +1,58 @@
The Mantis Runtime is consists of two components:

1. A single **[Mantis Master]** which coordinates the execution of [Mantis Jobs].
1. Independent **Mantis Jobs** which receive streams of events as input, [transform] events one at a
1. A single **Mantis Master** which coordinates the execution of Mantis Jobs.
1. Independent **Mantis Jobs** which receive streams of events as input, transform events one at a
time, and produce streams of events as output.

This page assumes familiarity with Mantis Job high-level concepts. An introduction can be found in
[Writing Mantis Jobs](../../developing/writingjobs). This page presents internal details for Mantis Jobs.

## Mantis Job Components
A Mantis Job consists of three [components]. Each one is based on a [cold Observable] that emits
A Mantis Job consists of three components. Each one is based on a cold Observable that emits
events to the next Observer in the Observable chain:

1. **Source**

The [Source] component is an `RxFunction` that consumes data in a streaming, non-blocking,
[backpressure]-aware manner from an external service.
The Source component is an `RxFunction` that consumes data in a streaming, non-blocking,
backpressure-aware manner from an external service.

1. **Processing Stage**

A [Processing Stage] component is based on an `RxFunction`. This is where
A Processing Stage component is based on an `RxFunction`. This is where
event transformations take place. There can be many Processing Stages in a Mantis Job.

1. **Sink**

The [Sink] component is based on an `RxAction`. It asynchronously emits results of the
The Sink component is based on an `RxAction`. It asynchronously emits results of the
final Processing Stage to an external service.

!!! note
Mantis Jobs can consume events from typical external services such as APIs, databases, and
[Kafka] topics. Mantis Jobs can also consume events emitted by other Mantis Jobs. This is
Kafka topics. Mantis Jobs can also consume events emitted by other Mantis Jobs. This is
referred to in Mantis as **job chaining**.

## Runtime Lifecycle
The entry point for a Mantis Job is the Mantis [Worker]. The Mantis Master starts three primary
The entry point for a Mantis Job is the Mantis Worker. The Mantis Master starts three primary
services on a Mantis Worker when the Master boots the Worker up:

1. The **virtual machine worker** service interacts with the underlying substrate, currently
[Mesos]. This service subscribes to task updates and registers the Mantis Worker with Mesos
Mesos. This service subscribes to task updates and registers the Mantis Worker with Mesos
executor callbacks to launch Mantis Jobs.

1. The **heartbeat service** sends HTTP heartbeat requests to notify the Mantis Master that the
worker is alive and available to process events.

1. The **stage executor** dynamically loads bytecode for a Mantis Job, creates an in-memory
representation of all the [metadata] required to execute events for that Mantis Job, and
representation of all the metadata required to execute events for that Mantis Job, and
processes events for the current Processing Stage.

### Job Master Stage
The [Job Master] [autoscales] Processing Stages. It can autoscale such stages independently of each
The Job Master autoscales Processing Stages. It can autoscale such stages independently of each
other. If the configuration of a Job indicates that any Processing Stage is autoscalable, Mantis
will automatically add a Job Master as the initial processing stage of the Job. This is a hidden
stage that Job owners do not explicitly manage; instead, Mantis will create and configure a
`JobMasterService`. This service creates a subscription to worker metrics via the
`WorkerMetricHandler` and a `MetricsClient` which receives metrics over HTTP via [SSE] and sends
`WorkerMetricHandler` and a `MetricsClient` which receives metrics over HTTP via SSE and sends
them over to the `JobAutoScaler`.

#### Job Autoscaler
Expand Down
Loading