feat(nexus): add Term Nexus SDK and example #75
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add optional dependencies for the Term Cloud SDK behind a `cloud` feature flag: - reqwest: HTTP client for API communication - ring: Cryptographic operations - rusqlite: Local SQLite storage for metrics buffering - directories: Cross-platform app directory detection Users who don't need cloud functionality won't have to compile these dependencies. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add the CloudConfig struct for configuring Term Cloud SDK connections. This includes: - Builder pattern with sensible defaults - Configurable endpoint, timeout, max retries, buffer size - Support for batch uploads with configurable batch size - Flush interval for background uploads - Optional offline cache path Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace plain String with SecureString for api_key field to prevent credential leakage in logs via Debug output - Remove Serialize/Deserialize derives to prevent accidental serialization of secrets - Change offline_cache_path from Option<String> to Option<PathBuf> for type-safe path handling - Update api_key getter to return &SecureString with documented expose() usage - Add tests to verify API key is not leaked in debug output Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add serializable types for the Term Cloud wire protocol: - CloudMetric: Main type for transmitting metrics - CloudResultKey: Identifies a set of metrics with dataset date and tags - CloudMetricValue: Tagged union for metric values (double, long, string, boolean, histogram) - CloudHistogram and CloudHistogramBucket: Histogram data structures - CloudMetadata: Collection metadata including timestamps and term version - CloudValidationResult and CloudValidationIssue: Validation summary types All types derive Serialize/Deserialize for JSON serialization. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add CloudError enum for Term Cloud operations with error variants for: - Authentication failures - Network errors - Rate limiting - Server errors - Request validation - Serialization issues - Buffer overflow - Cache errors - Configuration errors The key feature is the is_retryable() method which helps the upload worker decide whether to retry failed operations. Network errors, rate limiting, and server errors (5xx) are considered retryable. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add TermCloudClient for communicating with the Term Cloud API: - HMAC-SHA256 request signing using the API key - Async HTTP operations using reqwest - Error mapping from HTTP status codes to CloudError variants - Methods for health check, ingest, query, and delete operations Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add UploadWorker for asynchronous metrics upload with: - Configurable batch size and flush interval - Graceful shutdown with final flush - WorkerStats tracking (metrics_uploaded, metrics_failed, batches_sent, retries) - Exponential backoff with jitter for retries (capped at 32x) - Non-blocking retry handling to prevent flush cycle blocking - Fallible new() returning CloudResult<Self> instead of panicking Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The calculate_backoff() method existed but was never called, meaning retries occurred immediately without any backoff delay. This fix: - Add ready_at: Instant field to BufferEntry to track when entries can be retried - Modify drain() to only return entries past their ready_at time - Call calculate_backoff() in handle_retry() to set proper backoff delays - Remove #[allow(dead_code)] from calculate_backoff() since it's now used This ensures exponential backoff with jitter is actually applied during the retry flow, preventing thundering herd issues after transient failures. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add OfflineCache struct backed by SQLite for persisting metrics when network is unavailable. The cache stores metrics as JSON with retry counts for later retry attempts. Features: - File-based or in-memory cache creation - Thread-safe with Mutex<Connection> - Serialize/deserialize CloudMetric to JSON - Track retry counts for backoff strategy Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add tracing::warn! for deserialization and row read failures in load_all() to prevent silent data loss - Create CacheEntry struct that wraps BufferEntry with database ID - Update load_all() to return Vec<CacheEntry> for tracking entry IDs - Add delete_ids() method for selective deletion of uploaded entries - Add tests for delete_ids() including empty case - Export CacheEntry from cloud module Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add the main repository implementation that ties together all cloud components for metrics persistence to Term Cloud: - Implements MetricsRepository trait for integration with existing code - Uses MetricsBuffer for local buffering with async background upload - Supports offline operation with automatic sync via OfflineCache - Integrates UploadWorker for asynchronous batch uploads - Provides graceful shutdown with worker stats and cache persistence - Includes health check for Term Cloud connectivity - Converts AnalyzerContext metrics to CloudMetric wire format Key methods: - new(): Create repository and start background worker - setup_cache(): Configure offline cache at default or custom path - pending_count(): Get number of pending metrics - flush(): Force flush buffered metrics - shutdown(): Graceful shutdown with cache persistence - health_check(): Check Term Cloud connectivity - sync_offline_cache(): Upload cached offline metrics Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add webhook-based alerting for validation failures with the following components: - WebhookConfig: Builder pattern for URL, headers, details, severity threshold, and HMAC secret configuration - AlertSeverity: Info, Warning, Critical enum with Ord for comparison - AlertPayload: Structured alert with title, severity, dataset, environment, summary, details, timestamp, and dashboard URL - AlertSummary: total_checks, passed, failed, status - AlertDetail: check, constraint, level, message, metric - WebhookClient: HTTP client with severity filtering, optional HMAC-SHA256 signing, and configurable detail inclusion Includes comprehensive tests for config validation, payload creation, severity ordering, and HMAC signing. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add comprehensive integration tests covering: - Full cloud flow with repository creation, metric saving, and graceful shutdown - CloudMetric wire format serialization and JSON structure verification - Webhook alert generation from validation results with severity levels - CloudConfig builder pattern with all configuration options - Multiple metrics saving and pending count verification - CloudValidationResult serialization roundtrip - Alert severity level determination (Info, Warning, Critical) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add comprehensive documentation for Term Cloud SDK following the Diataxis how-to guide format. Covers configuration, tagging metrics, webhook alerting, offline support, and graceful shutdown. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ericpsimon
force-pushed
the
feat/cloud-sdk
branch
3 times, most recently
from
4bca929 to
4add549
Compare
Implements complete SDK for sending validation metrics to Term Nexus: - NexusConfig: Builder pattern configuration with secure API key handling - NexusClient: HTTP client with HMAC-SHA256 request signing - NexusRepository: MetricsRepository implementation with buffered uploads - MetricsBuffer: Thread-safe in-memory buffer with configurable limits - UploadWorker: Background worker with batching, retries, exponential backoff - OfflineCache: SQLite-backed persistence for network outage resilience Key features: - SecureString for API key (zeroization, masked Debug/Display) - Graceful shutdown with flush and cache persistence - Health check and metrics query endpoints - Tag validation with sensible limits - Comprehensive error types with retry guidance Rebranded from 'cloud' to 'nexus' for clearer product identity. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ericpsimon
force-pushed
the
feat/cloud-sdk
branch
from
4add549 to
b22adef
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| POST /v1/metrics HTTP/1.1 | ||
| Host: api.withterm.com | ||
| Content-Type: application/json | ||
| X-Term-Api-Key: tk_live_abc123 |
There was a problem hiding this comment.
🛑 Gitleaks has detected a secret with rule-id generic-api-key in commit b22adef.
If this secret is a true positive, please rotate the secret ASAP.
If this secret is a false positive, you can add the fingerprint below to your .gitleaksignore file and commit the change to this branch.
echo b22adef39a37938520fbf1b432efd7ccf4883dc6:docs/reference/nexus-api-spec.md:generic-api-key:450 >> .gitleaksignore
Add main.rs with core functionality: - Health check to verify Nexus connectivity - Load CSV data and run validation checks using ValidationSuite - Store metrics to Nexus with tagged ResultKey - Graceful shutdown with worker stats Also add [workspace] to Cargo.toml to exclude from parent workspace. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Set up offline cache for resilience when storing metrics to Nexus - Fix clippy::get_first warning by using .first() instead of .get(0) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Files section listing example components - Add Example Output section showing typical run output - Apply cargo fmt formatting to main.rs Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ericpsimon
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
term-guard/src/nexus/)nexus-repositoryexample demonstrating Deequ-like metrics persistenceNexus SDK Features
Example Features (
docs/examples/nexus-repository/)Test Plan
cargo fmt --all -- --checkpassescargo clippy -- -D warningspasses🤖 Generated with Claude Code