RAD — Rapid Analysis of Data

The Unified Database Engine

RAD is a lightweight command-line tool for generating reports from multiple databases using a single configuration file.

It connects to different data sources, executes queries in parallel, merges the results, applies transformations, and exports the final dataset into files such as CSV reports and interactive charts.

The goal of RAD is simple:

Connect to multiple data sources, run a single command, and instantly transform raw data into visual charts and Excel‑ready reports.

---

Features

• Multiple database sources (PostgreSQL, MySQL)
• Run the same query across several databases
• Parallel query execution
• Parameterized queries
• YAML-based configuration
• Dataset transformations
• CSV export
• Interactive HTML charts
• Source-aware reporting (compare multiple systems)
• Simple CLI usage

---

Architecture Overview

RAD works in four main stages:

1. Load configuration
2. Execute queries
3. Transform datasets
4. Export results

Config (YAML)
      │
      ▼
Query Runner (parallel execution)
      │
      ▼
Dataset Merge
      │
      ▼
Transformations
      │
      ▼
Exports (CSV / Charts)

---

Installation

git clone https://github.com/your-org/rad.git
cd rad
go build -o rad ./cmd

---

Usage

rad <config.yaml> <report_name> [--param=value ...]

Example:

rad configs/sample.yaml mail_traficks_per_day \
  --from_date=2025-04-01 \
  --to_date=2025-05-01

---

Configuration

RAD is fully configured using a YAML file.

Example:

version: 1

sources:
  - name: etsc
    type: postgres
    host: 172.16.1.73
    port: 5432
    database: automation
    username: automation
    password: secret

reports:
  - name: mail_traficks_per_day

    datasets:
      - name: mail_traficks_per_day

        targets:
          - sources: ["etsc", "ecic"]

            query:
              sql: |
                SELECT count(id) as total, date(created_at) as created_at
                FROM internal_mails
                WHERE created_at >= :from_date
                AND created_at <= :to_date
                GROUP BY date(created_at)
                ORDER BY created_at ASC

    outputs:
      - type: csv
        dataset: mails_statistics
        file: mails_statistics.csv

      - type: chart
        chart_type: line
        file: output/mails_statistics.html
        x_field: created_at
        y_field: total
        group_by: "_source"
        title: "Mail Traffic"

---

Data Sources

Currently supported databases:

• PostgreSQL
• MySQL

Multiple sources can be defined and queried simultaneously.

Example:

sources:
  - name: etsc
    type: postgres
    ...

  - name: ecic
    type: postgres
    ...

RAD can run the same query on multiple sources and merge the results automatically.

Each row will include an internal field:

_source

Which indicates which database the row came from.

---

Query Parameters

Queries support named parameters using the syntax:

:parameter_name

Example:

SELECT *
FROM orders
WHERE created_at >= :from_date
AND created_at <= :to_date

Parameters can be provided in CLI flags

Example:

--from_date=2025-01-01
--to_date=2025-02-01

---

Transformations

Datasets can be transformed before exporting.

Examples:

• Trim strings
• Cast fields
• Value mapping
• Expression evaluation
• Sorting and grouping

Example:

transform:
  steps:
    - type: trim
      fields: ["customer_name"]

    - type: cast
      field: total_amount
      to: float

---

Output Types

RAD currently supports:

CSV Export

- type: csv
  file: report.csv

The CSV file can be directly opened in Excel or Google Sheets.

---

Chart Export

RAD can generate interactive HTML charts.

Supported types:

• line
• bar

Example:

- type: chart
  chart_type: line
  file: output/chart.html
  x_field: created_at
  y_field: total
  group_by: "_source"

The resulting file is a self-contained HTML chart.

---

Parallel Execution

When multiple sources are defined, RAD executes queries concurrently using goroutines.

This allows large reports across many databases to complete significantly faster.

---

Project Structure

cmd/
  main.go

internal/
  config/
  db/
  export/
  report/
  runner/
  transform/

Main components:

• config – configuration parsing
• db – database connections
• runner – report execution engine
• export – CSV and chart exporters
• transform – dataset transformations

---

Example Output

RAD can generate outputs such as:

• mails_statistics.csv
• mails_statistics.html

The HTML chart can be opened directly in a browser and supports tooltips, legends, and multiple data series.

---

Roadmap

Planned improvements:

• Excel (.xlsx) export
• More chart types (pie, stacked, heatmap)
• SQL templating improvements
• Transformation pipeline extensions
• Scheduling support
• REST API mode
• Dashboard generation

---

License

MIT License

---

Philosophy

RAD follows a simple philosophy:

Data should be easy to query, combine, and visualize — regardless of where it lives.

Instead of writing custom scripts for every report, RAD allows teams to define reports declaratively and run them instantly from the command line.