DLN includes services to help catalog files, monitor workflows, and troubleshoot failures associated with snakemake. Foundational to these services is the ability to collect and organize messages produced by snakemake as it executes a workflow. At this point, DLN's frontend can display a stream of logs from snakemake, but it does no processing or persistence.
Processing
To yield maximum insight, messages from snakemake need to be organized into a trace (i.e. a logical structure that correlates related events). The idea is to collect events and sort them according to which workflow execution they arise from. Snakemake emits two different type of data: logs (received via RabbitMQ) and file provenance records (written to a DB).
DLN includes services to help catalog files, monitor workflows, and troubleshoot failures associated with snakemake. Foundational to these services is the ability to collect and organize messages produced by snakemake as it executes a workflow. At this point, DLN's frontend can display a stream of logs from snakemake, but it does no processing or persistence.
Processing
To yield maximum insight, messages from snakemake need to be organized into a trace (i.e. a logical structure that correlates related events). The idea is to collect events and sort them according to which workflow execution they arise from. Snakemake emits two different type of data: logs (received via RabbitMQ) and file provenance records (written to a DB).