Skip to content

SWORDIntel/SolidMarble

Repository files navigation

SolidMarble

String/Data Obfuscation Toolkit — Modernized for 2026

License: CC BY-NC-ND 4.0 Platform: Linux Language: C++17 Python: 3.11+

Algorithms: 126 CNSA 2.0 FIPS 140-3 SBOM: CycloneDX

gRPC Docker ML Pipeline Plugin SDK


SolidMarble is a comprehensive extension and modernization of the CIA's Marble Framework — a string/data obfuscation toolkit originally leaked via Wikileaks Vault7. This project reconstructs the missing source code, ports it to Linux, and augments it with an ML-driven pipeline for automated algorithm generation, validation, detection evasion, and reinforcement learning.


Feature Matrix

Core Framework
Feature Description
Linux Port Full C++ port with CMake + Makefile build system, 116 scramblers + 116 deobfuscators
Trinary K3 Logic 6 new trinary algorithms: XOR1, SBOX27, TRYTE_XOR8, RXOR1, BUMP1, TRIT_DIFFUSE
ML Pipeline AST feature extraction, symbolic Z3 validation, detection-aware checking, entropy camouflage
RL Training Lightweight evolutionary + full LoRA-based REINFORCE training for algorithm generation
LLVM Pass Plugin Compile-time IR-level string obfuscation via LLVM pass
Hardware-Bound Keys TPM 2.0 / SGX-backed key derivation for receipts
Distributed Registry Raft consensus-based algorithm registry for multi-node deployments
TTL Receipts Self-destructing receipts with configurable time-to-live, AES-256-GCM encrypted
2026 Modernization
Feature Description
CNSA 2.0 Compliance SHA-384 hashing, AES-256-GCM encryption, PQC readiness (ML-KEM/ML-DSA), compliance auditor
Detection Predictor ML-based detection-rate predictor (92.4% accuracy, sklearn GradientBoosting)
FastAPI Server Async REST API with Pydantic models, native OpenAPI 3.1, WebSocket streaming, Swagger UI
SBOM + Sigstore CycloneDX 1.5 SBOM generation, cosign/Sigstore binary signing for supply chain compliance
Container Deployment Multi-stage Dockerfile, docker-compose, Helm chart for Kubernetes
TUI Dashboard Rich/Textual terminal UI with algorithm browsing, CNSA audit, predictor info
Property-Based Testing Hypothesis-powered round-trip testing for all scrambler algorithms
gRPC API High-performance streaming gRPC interface with protobuf, auto-generated stubs
Plugin SDK Stable C ABI for runtime-loaded scramblers via dlopen, with Python ctypes loader
Plugin Wrappers Auto-generated C ABI wrappers for all 120 built-in algorithms — loadable as plugins
Unified Registry Merges built-in C++ algorithms with runtime plugins for transparent ML pipeline access
Adversarial Loop Automated generate→predict→mutate→re-predict red-teaming against detection
KMS Integration Key rotation & management with HashiCorp Vault, AWS KMS, and local providers
FIPS 140-3 OpenSSL FIPS provider integration for FIPS-approved crypto operations
ONNX Export Export LoRA model and detection predictor to ONNX for dependency-free inference
Integrations & Tooling
Feature Description
Web Dashboard Flask dashboard with AMOLED black/red theme, live action buttons, algorithm search
VS Code Extension Commands, configuration, and snippets for VS Code integration
CI/CD GitHub Actions workflow for build, test, LLVM pass, extension packaging, and release
REST API Standalone Flask REST API with OpenAPI spec, interactive docs, and full tool/ML pipeline access
Unified CLI marble launcher script dispatching all tools from a single entry point

Architecture

SolidMarble/
├── Marbler/              # Original Windows scrambler implementations (106 algorithms)
├── Mender/               # Post-build restoration tool
├── Validator/            # Binary string presence validator
├── Shared/               # Deobfuscator headers (116 algorithms)
├── LinuxPort/            # Linux port: CMake build, unified Mibster, LLVM pass, trinary algorithms
│   ├── Marbler/          #   Scrambler implementations (116 + 6 trinary)
│   ├── Shared/           #   Deobfuscator headers
│   ├── Validator/        #   Linux validator
│   ├── MarbleObfuscationPass.cpp   #   LLVM IR-level obfuscation pass
│   └── build_llvm_pass.sh         #   LLVM pass build script
├── LinuxTUI/             # TUI configuration tool
├── ml/                   # ML pipeline
│   ├── mibster_ml.py     #   Unified ML CLI
│   ├── dashboard.py      #   Flask web dashboard
│   ├── extract_features.py    # AST feature extraction (tree-sitter)
│   ├── symbolic_validate.py   # Z3 symbolic execution validation
│   ├── detection_check.py     # YARA-based detection evasion
│   ├── rl_train.py            # RL training (evolutionary + LoRA REINFORCE)
│   ├── rag_generate.py        # RAG-based algorithm generation
│   ├── synthetic_factory.py   # Synthetic algorithm factory
│   ├── registry_sync.py       # Algorithm registry synchronization
│   ├── hardware_key.py        # TPM/SGX hardware-bound key derivation
│   ├── distributed_registry.py # Raft consensus distributed registry
│   ├── ttl_receipts.py        # Self-destructing receipts with TTL
│   ├── build_camouflage.py    # Build fingerprint randomization
│   ├── cnsa2.py              # CNSA 2.0 compliance, PQC readiness, crypto-agility
│   ├── detection_predictor.py # ML-based detection-rate predictor
│   ├── fastapi_server.py     # FastAPI server (async, WebSocket, OpenAPI 3.1)
│   ├── sbom.py               # SBOM generation (CycloneDX) + Sigstore signing
│   ├── tui.py                # Terminal UI dashboard (Rich/Textual)
│   ├── property_test.py      # Property-based round-trip testing
│   ├── grpc_server.py        # gRPC server (streaming, protobuf)
│   ├── marble.proto          # Protobuf service definition
│   ├── plugin_loader.py      # Plugin SDK loader (ctypes/dlopen)
│   ├── adversarial.py        # Adversarial generation loop
│   ├── kms.py                # Key rotation & KMS integration
│   ├── fips.py               # FIPS 140-3 provider integration
│   ├── onnx_export.py        # ONNX export (LoRA + predictor)
│   ├── algorithm_registry.py # Unified registry (builtin + plugins)
│   ├── generate_wrappers.py  # Auto-generate plugin wrappers for builtins
│   └── ast_parser.py          # AST parsing utilities
├── ml/api.py             #   REST API server (OpenAPI 3.0)
├── vscode-extension/     # VS Code extension
│   ├── extension.js      #   Command handlers
│   ├── package.json      #   Extension manifest
│   └── snippets/         #   Code snippets
├── .github/workflows/    # CI/CD
│   └── ci.yml            #   Build, test, package, release
├── marble                # Unified launcher script
└── docs/                 # Original framework documentation

Quick Start

Build & Run

# Build the Linux port
cd LinuxPort && make

# Use the unified launcher
./marble                    # Show help
./marble list               # List all 126 algorithms
./marble scramble <file>    # Obfuscate strings in a source file
./marble mend <file>        # Restore obfuscated source
./marble validate <binary>  # Check binary for plaintext strings

ML Pipeline

# Install Python dependencies
pip install -r requirements.txt

# Core pipeline
./marble ml extract                          # Extract AST features
./marble ml validate                         # Z3 symbolic validation
./marble ml detect                           # Detection evasion checks
./marble ml generate --family XOR --count 5  # Generate new algorithms
./marble ml rltrain --episodes 100           # RL training
./marble ml novel-generate --count 20        # LoRA-based novel generation

# 2026 features
./marble ml cnsa audit                       # CNSA 2.0 compliance
./marble ml predict train                    # Train detection predictor
./marble ml sbom generate                    # CycloneDX SBOM
./marble ml tui                              # Terminal UI dashboard
./marble ml adversarial --rounds 10          # Adversarial red-teaming
./marble ml fips status                      # FIPS 140-3 status
./marble ml kms rotate                       # Key rotation
./marble ml onnx export-predictor            # ONNX export

# Plugin SDK
./marble ml plugin discover                  # Discover plugins
./marble ml plugin test ml/data/plugins/example_plugin.so
./marble ml gen-wrappers --build --parallel # Generate wrappers for all 120 algorithms
./marble ml registry stats                   # Unified registry

# Servers
./marble api                                 # REST API (Flask, port 5000)
./marble api --fastapi --port 5001           # FastAPI (async, Swagger UI)
./marble api --grpc --port 50051             # gRPC (streaming, protobuf)
./marble ml dashboard --port 8080            # Web dashboard

API Servers

Three API servers provide programmatic access — pick the one that fits your use case:

Server Port Best For
Flask REST 5000 Quick prototyping, interactive HTML docs
FastAPI 5001 Async, WebSocket streaming, Swagger UI, OpenAPI 3.1
gRPC 50051 High-performance streaming, protobuf, microservices
./marble api                           # Flask REST
./marble api --fastapi --port 5001     # FastAPI (Swagger at /docs)
./marble api --grpc --port 50051       # gRPC (protobuf streaming)
REST API Endpoints
Method Path Description
GET /api/v1/health Health check
GET /api/v1/status Full pipeline status
GET /api/v1/algorithms List all algorithms
GET /api/v1/algorithms/{name} Algorithm details
POST /api/v1/obfuscate Run mibster obfuscation
POST /api/v1/mend Run mender restoration
POST /api/v1/validate-binary Run validator
POST /api/v1/ml/generate Generate algorithms
POST /api/v1/ml/novel-generate Generate novel algorithms via LoRA
POST /api/v1/ml/rltrain Run RL training
POST /api/v1/ml/inject Inject algorithm into framework
GET /api/v1/novel-algorithms List generated novel algorithms
GET /api/v1/cnsa/audit CNSA 2.0 compliance audit
GET /api/v1/cnsa/algorithms CNSA 2.0 approved algorithm registry
GET /api/v1/cnsa/pqc-info PQC readiness info
GET /api/v1/cnsa/config Crypto suite configuration
GET /api/v1/ml/predict-detection Predict detection rate for algorithm
POST /api/v1/ml/predict-detection/train Train detection predictor
GET /api/v1/sbom Get CycloneDX SBOM
GET /api/v1/openapi.json OpenAPI 3.0 specification
GET /api/v1/docs Interactive HTML documentation

Full API documentation: docs/API.md

CNSA 2.0 Compliance

Aligned with the NSA's Commercial National Security Algorithm Suite 2.0 (post-quantum cryptography standard):

Area Algorithm Status
Hashing SHA-384 Active
HMAC HMAC-SHA-384 Active
Symmetric AES-256-GCM Active
KDF PBKDF2-HMAC-SHA-384 Active
PQC KEM ML-KEM-1024 (FIPS 203) Ready
PQC DSA ML-DSA-87 (FIPS 204) Ready
./marble ml cnsa audit          # Compliance audit
./marble ml cnsa pqc-info       # PQC readiness check
./marble ml cnsa report         # Generate report

Full documentation: docs/CNSA_2.0.md

Detection-Rate Predictor

Predicts whether an algorithm will be detected by YARA rules — before compilation. Trained on 106 algorithms with 92.4% accuracy using sklearn GradientBoosting.

./marble ml predict train                          # Train predictor
./marble ml predict predict --algorithm MBL_FORLOOP_XOR1  # Predict
./marble ml predict info                           # Model info

FastAPI Server

The modern async API server provides native OpenAPI 3.1, Pydantic v2 validation, WebSocket streaming, and Swagger UI.

./marble api --fastapi --port 5001
# Swagger UI:  http://localhost:5001/api/v1/docs
# ReDoc:       http://localhost:5001/api/v1/redoc

SBOM + Supply Chain

Generate CycloneDX 1.5 SBOMs and sign binaries with Sigstore/cosign:

./marble ml sbom generate                          # Generate SBOM
./marble ml sbom sign --binary LinuxPort/mibster   # Sign with cosign
./marble ml sbom verify --binary LinuxPort/mibster # Verify signature

Container Deployment

# Docker
docker build -t marble .
docker run -p 5000:5000 marble

# Docker Compose (API + FastAPI + dashboard)
docker compose up

# Kubernetes with Helm
helm install marble deploy/helm/marble

Terminal UI

./marble ml tui    # Launch Rich/Textual dashboard

Property-Based Testing

./marble ml ptest --count 200    # Hypothesis-powered round-trip tests

gRPC API

High-performance streaming interface alongside the REST API:

./marble api --grpc --port 50051

Features: streaming ML pipeline updates, algorithm catalog browsing, CNSA 2.0 audits, detection prediction — all via protobuf.

Plugin SDK

Runtime-loadable scrambler plugins via stable C ABI:

./marble ml plugin discover                                          # Discover plugins
./marble ml plugin test ml/data/plugins/example_plugin.so            # Load + test
./marble ml plugin list                                              # List loaded

Plugin ABI: LinuxPort/Shared/MarblePluginSDK.h

Plugin Wrappers for Built-in Algorithms

Auto-generates C ABI wrappers for all 120 built-in C++ algorithms, making them loadable as plugins:

./marble ml gen-wrappers                            # Generate source files
./marble ml gen-wrappers --build --parallel         # Generate + compile all (parallel)
./marble ml gen-wrappers --algorithm MBL_CLASS_BUMP1 --build  # Single algorithm

This makes every built-in algorithm available BOTH as compiled-in C++ (for code generation) AND as a runtime-loadable plugin (for ML pipeline testing).

Unified Algorithm Registry

Merges built-in C++ algorithms with runtime-loaded plugins into a single registry:

./marble ml registry stats                          # Registry statistics
./marble ml registry list                           # List all (builtin + plugins)
./marble ml registry plugins                        # List plugins only
./marble ml registry test-roundtrip MBL_CLASS_BUMP1 # Test round-trip
./marble ml registry search BUMP                    # Search algorithms

The registry is used by the detection predictor, adversarial loop, and API servers to query algorithms from both sources transparently.

Adversarial Generation Loop

Automated red-teaming: generate → predict detection → mutate to evade → re-predict.

./marble ml adversarial --rounds 10 --target 0.3    # Target <30% detection

Key Management (KMS)

Centralized key rotation with Vault, AWS KMS, or local provider:

./marble ml kms rotate                                              # Rotate keys (local)
./marble ml kms rotate --provider vault --url http://vault:8200 --token s.xxxx  # Vault

FIPS 140-3

./marble ml fips status     # Provider status
./marble ml fips check      # Compliance checks
./marble ml fips enable     # Enable FIPS mode

ONNX Export

Export models for dependency-free inference:

./marble ml onnx export-predictor                          # Detection predictor → ONNX
./marble ml onnx export-lora --model-path models/marble_lora  # LoRA model → ONNX

Web Dashboard

The Flask dashboard provides a real-time view of the ML pipeline:

  • Overview — Pipeline status, quick action buttons (validate, detect, generate, train)
  • Algorithms — Browse all registered algorithms with live search
  • Validation — Z3 symbolic execution results with run button
  • Detection — YARA evasion check results with run button
  • RL Training — Reward chart, episode history, training trigger
  • Receipts — TTL receipt management with cleanup
  • API — REST endpoint documentation

All buttons are functional — they trigger the underlying ML commands via POST endpoints and display results inline with toast notifications.

Windows (Visual Studio)

Open StringScrambling.sln in Visual Studio. Build produces Mibster.exe, Mender.exe, and Validator.exe. Add these to your project's pre-build and post-build steps.

Include Marble.h and your chosen deobfuscator from Shared/ in your project. Supported data types:

  • CARBLEchar[] / char*
  • WARBLEwchar_t[] / wchar_t*
  • BARBLEbyte[] / byte*

LLVM Pass (Linux)

Compile-time IR-level string obfuscation:

cd LinuxPort
./build_llvm_pass.sh
# Use with: clang -fpass-plugin=MarbleObfuscationPass.so your_file.c

Algorithm Catalog

Original Algorithms (106)

Family Count Description
XOR 12 Forward XOR with key
RXOR 12 Reverse XOR with key
BUMP 12 Bump key forward per byte
RBUMP 8 Reverse bump key
FORLOOP 12 For-loop based variants
FUNC 12 Function call variants
CLASS 36 C++ class variants
CRYPTOGRAPHIC 1 Crypto-based obfuscation
POLYMORPHIC 1 Polymorphic engine
VIRTUALIZATION 1 VM-based obfuscation

Trinary K3 Algorithms (6 new)

Algorithm Description
MBL_TRINARY_K3_XOR1 Trinary XOR on K3 field (3-valued logic)
MBL_TRINARY_SBOX27 27-entry S-box substitution over trytes
MBL_TRINARY_TRYTE_XOR8 8-tryte XOR block operation
MBL_TRINARY_K3_RXOR1 Reverse trinary XOR on K3 field
MBL_TRINARY_K3_BUMP1 Trinary bump key per trit
MBL_TRINARY_TRIT_DIFFUSE Trit-level diffusion/permutation

ML-Generated Algorithms

Algorithm Entropy Detection Triggers
MBL_GEN_0011 7.21 bits 0
MBL_GEN_0001 6.97 bits 0
MBL_GEN_0021 6.97 bits 0
MBL_GEN_0032 6.97 bits 0

Total: 126 algorithms (116 scramblers + 6 trinary + 4 ML-generated)


ML Pipeline

┌─────────────┐    ┌──────────────┐    ┌───────────────┐    ┌──────────────┐
│  Extract    │───▶│  Validate    │───▶│  Detect       │───▶│  Generate    │
│  (tree-sitter)│   │  (Z3 SMT)    │    │  (YARA rules) │    │  (RL/LoRA)   │
└─────────────┘    └──────────────┘    └───────────────┘    └──────┬───────┘
                                                                    │
                    ┌──────────────┐    ┌───────────────┐           │
                    │  Inject      │◀───│  Novel Gen    │◀──────────┘
                    │  (into .h)   │    │  (LoRA model) │
                    └──────────────┘    └───────────────┘

Feature Extraction (extract_features.py)

Uses tree-sitter to parse C++ source code and extract structural features from scrambler algorithms. Features include AST node counts, control flow complexity, data flow patterns, and entropy characteristics.

Symbolic Validation (symbolic_validate.py)

Validates round-trip correctness using Z3 SMT solver. Proves that deobfuscate(obfuscate(x)) == x for all inputs.

Detection Evasion (detection_check.py)

Runs YARA rules against generated algorithms to measure detection surface. Computes a detection score (0.0 = fully evaded, 1.0 = detected).

Entropy Camouflage (build_camouflage.py)

Randomizes build fingerprints: compiler flags, section names, timestamps, debug info.

RL Training (rl_train.py)

Trainer Strategy Use Case
Lightweight Evolutionary + reward-weighted mutation Quick iteration, no GPU
Full LoRA-based REINFORCE policy gradient High-quality novel algorithms

Reward function:

Factor Reward
Compilation success +1.0
Round-trip correctness +1.0
Z3 proof success +0.5
Entropy gate pass +0.5
Detection evasion up to +2.0
Runtime/compile errors −1.0 each

Novel Algorithm Generation (novel_generate.py)

Uses a LoRA-trained transformer model to generate structurally novel obfuscation algorithms:

  1. Model Loading — Loads LoRA adapters from models/rl_output/, or falls back to template-based generation
  2. Prompt Construction — Retrieves diverse seed algorithms, adds novelty constraints
  3. Sampling — Temperature ladder (0.6–1.1), top-p=0.95, repetition penalty
  4. Parsing — Extracts scrambler + deobfuscator class pairs from generated C++ code
  5. Anti-Collision — SHA-256 hash dedup + structural token signature topology check
  6. Validation — Full pipeline: Z3 proof + entropy gate + round-trip + detection evasion
  7. Output — Validated novel algorithms saved to data/novel/ and generated/
./marble ml novel-generate --count 20 --temperature 0.9              # High creativity
./marble ml novel-generate --count 10 --families XOR RBUMP MIXED     # Target families
./marble ml novel-generate --model-path models/rl_output --count 50  # Specific model

Advanced Modules

Hardware-Bound Key Derivation (hardware_key.py)

Derives obfuscation keys from hardware roots of trust:

  • TPM 2.0 via tpm2-tools
  • Intel SGX enclaves
  • Falls back to software-derived keys if hardware unavailable
Distributed Registry (distributed_registry.py)

Raft consensus-based registry for multi-node algorithm synchronization:

  • Leader election, log replication, snapshotting, cluster membership management
TTL Receipts (ttl_receipts.py)

Self-destructing receipts with configurable time-to-live:

  • HMAC-signed receipt generation, automatic expiration and cleanup, verification API, revocation support

VS Code Extension

Installation & Commands
cd vscode-extension
vsce package
# Install .vsix in VS Code

Commands:

  • Marble: Scramble File — Obfuscate strings in active file
  • Marble: Mend File — Restore obfuscated source
  • Marble: Validate Binary — Check for plaintext strings
  • Marble: List Algorithms — Browse all 126 algorithms
  • Marble: ML Extract Features — Run feature extraction
  • Marble: ML Validate — Run Z3 validation
  • Marble: ML Detect — Run detection checks
  • Marble: ML Generate — Generate new algorithms
  • Marble: ML Train — Launch RL training
  • Marble: Dashboard — Open web dashboard

Includes snippets for CARBLE/WARBLE/BARBLE declarations, deobfuscator functions, and full scrambler class templates.


CI/CD

GitHub Actions workflow (.github/workflows/ci.yml):

  • Build & Test — Compiles Linux port, runs round-trip tests
  • ML Pipeline Tests — Validates Python modules, runs feature extraction
  • LLVM Pass — Builds the LLVM obfuscation pass
  • VS Code Extension — Packages extension as .vsix
  • Release — Uploads artifacts on tagged releases

Dashboard REST Endpoints

Method Path Description
GET /api/status Pipeline status overview
GET /api/algorithms List all algorithms
GET /api/validation Validation report
GET /api/detection Detection report
GET /api/training RL training results
POST /api/run/validate Run validation pipeline
POST /api/run/detect Run detection check
POST /api/run/generate Generate algorithms
POST /api/run/rltrain Run RL training (3 episodes)
POST /api/run/cleanup-receipts Remove expired receipts

Background

In 2017, Wikileaks published incomplete source code for the Marble Framework as part of the Vault7 leaks. The framework was created by the CIA's Applied Engineering Department (AED) for string/data obfuscation in tool development. Several malware samples have been identified in the wild using these algorithms:

  • Worm:Win32/Takc!pz (Microsoft)
  • TROJAN-DROPPER.WIN32.DAPATO (Kaspersky)

By releasing this reconstructed and extended framework, we aim to improve detections and identification of samples using these techniques.

"The Marble Framework is designed to allow for flexible and easy-to-use obfuscation when developing tools. String obfuscation algorithms (especially those that are unique) are often used to link malware to a specific developer or development shop." — CIA Internal Wiki, Vault7

Original documentation: Wikileaks Vault7 — Marble Framework · Additional docs in docs/


License

Hacker House code additions are available under a Attribution-NonCommercial-NoDerivatives 4.0 International license.

Original Marble Framework source is attributed to the CIA AED as published via Wikileaks Vault7.


Built by Hacker House · Reconstructed from CIA Vault7 leaks

About

SolidMarble is a comprehensive extension and modernization of the CIA's Marble Framework — a string/data obfuscation toolkit originally leaked via Wikileaks Vault7

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors