diff --git a/ROADMAP.md b/ROADMAP.md index a5c7785..d15c64a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -14,6 +14,7 @@ The roadmap is ordered by evidentiary dependency, not by visibility. A DOI and a ## Phase 2 — Release candidate +- [x] Publish a sanitized applied-case package without production artifacts. - [ ] Review every public artifact for secrets, identifiers, client material, and holdout leakage. - [ ] Add ORCID to citation metadata if available. - [ ] Freeze a `v0.1.0` release candidate and record artifact hashes. @@ -50,6 +51,7 @@ O roadmap segue a dependência entre evidências, não a visibilidade. Um DOI e ### Fase 2 — Candidata a release +- [x] Publicar um pacote sanitizado de caso aplicado sem artefatos de produção. - [ ] Revisar todos os artefatos públicos em busca de segredos, identificadores, material de clientes e vazamento de holdout. - [ ] Adicionar ORCID aos metadados de citação, caso exista. - [ ] Congelar uma candidata `v0.1.0` e registrar hashes dos artefatos. diff --git a/case-studies/README.md b/case-studies/README.md index 45149ae..64c89f8 100644 --- a/case-studies/README.md +++ b/case-studies/README.md @@ -1,15 +1,11 @@ # Applied case studies -## Instagram comment → DM +Applied cases connect Nirmata's evaluation concepts to systems with external effects. A case is not a benchmark result: each page separates public artifacts, private operational records, observed limitations, and unsupported extrapolations. -**Question:** Can a paid ManyChat comment-to-DM automation be replaced by owner-governed infrastructure without giving up security controls, auditability, and a zero recurring platform-cost target within free-tier limits? +## Instagram comment → private reply -**System:** Meta Webhooks and Instagram Graph API, a Node.js service, Render hosting, and Neon PostgreSQL. +An owner-operated automation replaced one paid comment-to-message workflow with a small service built on official platform APIs and owner-governed infrastructure. -**Observed outcome:** A real external Instagram account triggered a public reply and a private reply. The deployed system includes signed-webhook validation, persistent comment deduplication, separate status tracking for public/private sends, and health checks for the database and Meta token. +**Public package:** [detailed case](instagram-comment-dm/README.md) · [Português](instagram-comment-dm/README.pt-BR.md) · [evidence ledger](instagram-comment-dm/EVIDENCE-LEDGER.md) · [publication review](instagram-comment-dm/PUBLICATION-REVIEW.md) · [synthetic trajectory](../reproducibility/data/applied/instagram-comment-dm.synthetic.json) -**Nirmata relevance:** The trajectory captured properties that a successful demo did not reveal: ephemeral SQLite storage, secret synchronization risk, shared Meta permissions, publication prerequisites, authorization boundaries, and the transfer of operational responsibility from SaaS vendor to owner. - -**Limitations:** Free tiers provide no production SLA; APIs and tokens require maintenance; zero monetary subscription cost does not mean zero operational cost. - -The production repository remains private while its public-release security review is pending. +**Evidence boundary:** the private operational record reports that an external comment produced both a public response and a private reply. The public package verifies architecture, tests, controls, and review findings but does not contain raw production logs or independently verify long-term reliability. diff --git a/case-studies/instagram-comment-dm/EVIDENCE-LEDGER.md b/case-studies/instagram-comment-dm/EVIDENCE-LEDGER.md new file mode 100644 index 0000000..a5325c8 --- /dev/null +++ b/case-studies/instagram-comment-dm/EVIDENCE-LEDGER.md @@ -0,0 +1,46 @@ +# Evidence ledger — Instagram comment to private reply + +## Permission and classification + +- **Project:** owner-operated Instagram comment automation. +- **Classification:** real applied case, published as a sanitized case package. +- **Source snapshot:** private repository commit `ca0ca1f9c9c51c559649af40f936a0384e7f773b`, dated 2026-08-18. +- **Client naming:** not applicable; the owner account, application name, campaign names, links, and messages are redacted. +- **Logo/screenshots:** not included. +- **Testimonial:** none. +- **Metrics permitted:** repository structure, automated test count at review time, dependency-audit result, and deterministic synthetic artifacts. +- **Sensitive details removed:** credentials, provider resource names, account handles, campaign rules, real platform identifiers, production URLs, logs, and database records. + +## Claim ledger + +| Proposed claim | Evidence/source | Permission | Confidence | Publish wording | +|---|---|---|---|---| +| The workflow was rebuilt as an owner-operated service | Private source tree, architecture, deployment configuration, and operational documentation | Yes, sanitized | Certain | The case documents an owner-operated replacement for one paid automation workflow | +| Webhooks are authenticated | HMAC-SHA256 implementation plus signature tests | Yes | Certain | The implementation verifies signed webhook payloads before processing | +| Duplicate comment processing is reserved in persistent storage | PostgreSQL store implementation and local store regression test | Yes | Certain for implementation; not long-term operation | The implementation reserves comment identifiers before sending and stores delivery states | +| A real external comment triggered both replies | Private operational case record dated 2026-08-18 | Yes, summarized | Likely; no raw public artifact | The private operational record reports one successful external end-to-end test | +| The system has zero recurring cost | Private case record and free-tier architecture | Yes, qualified | Time-dependent | The design targeted zero recurring platform fees within the providers' free-tier limits at the time | +| The snapshot passes its automated suite | `npm test` on the source snapshot during review | Yes | Certain | Fourteen automated tests passed on 2026-08-27 | +| Dependencies had no known audit findings | `npm audit` on the locked dependency tree during review | Yes | Certain for that audit time | The package audit reported zero known vulnerabilities on 2026-08-27 | +| No secrets exist anywhere in history | Targeted history scan; no specialist scanner was available | No strong proof | Unsupported | Do not claim; state only that the targeted scan found placeholders and no recognized live credential | +| The service is durable and reliable | No public SLA, load test, queue evidence, or long observation window | No | Unsupported | Do not claim | + +## Case structure + +- **One-line outcome:** an owner-operated service reproduced one comment-to-private-reply workflow and made its controls and operational trade-offs inspectable. +- **Context:** an expiring SaaS subscription threatened continuity of a small automation. +- **Challenge:** retain the essential flow while avoiding a new recurring platform fee. +- **Constraints:** official APIs, explicit publication authorization, minimal storage, free-tier infrastructure, and no public disclosure of production credentials or campaign data. +- **Approach:** signed webhook intake, deterministic rules, persistent reservation, separate external sends, health checks, and failure-preserving review. +- **Solution:** Node.js service, official platform API, hosted runtime, and PostgreSQL storage. +- **Deliverables:** private implementation and tests; public architecture, synthetic trajectory, evidence ledger, and limitations. +- **Verified outcome:** code and test evidence are public only through this sanitized record; one production success remains supported by a private operational record. +- **Next step:** add durable queueing/retry, privacy-safe logging, and a longer operational observation window. + +## Publication gate + +- [x] Classification is explicit. +- [x] Every numeric statement has a source and date. +- [x] Private data and production identifiers are removed. +- [x] No client identity, logo, screenshot, or testimonial is used. +- [x] Summary and detailed copy use the same evidence boundary. diff --git a/case-studies/instagram-comment-dm/PUBLICATION-REVIEW.md b/case-studies/instagram-comment-dm/PUBLICATION-REVIEW.md new file mode 100644 index 0000000..f014616 --- /dev/null +++ b/case-studies/instagram-comment-dm/PUBLICATION-REVIEW.md @@ -0,0 +1,52 @@ +# Publication and security review + +**Review date:** 2026-08-27 +**Private source snapshot:** `ca0ca1f9c9c51c559649af40f936a0384e7f773b` + +## Publication decision + +Publish a sanitized case package, not the production repository. The public package contains no credentials, production identifiers, campaign configuration, screenshots, logs, or database records. + +## Evidence inventory + +The reviewed private snapshot contains 33 tracked files: Node.js source, 14 automated tests, bilingual documentation, deployment configuration, six private automation configurations, and legal pages. The test suite passed during review. The locked dependency audit reported no known vulnerabilities at review time. + +## Redaction decisions + +| Source material | Public treatment | Reason | +|---|---|---| +| Owner account and application name | Replaced with generic roles | Avoid unnecessary identity and operational coupling | +| Automation keywords, links, and message templates | Omitted | Private campaign and distribution data | +| Provider resource names and production URLs | Omitted | Operational security | +| Real comment, message, account, and media identifiers | Replaced with synthetic identifiers | Personal data and platform metadata | +| Logs and database rows | Not copied | May contain identifiers, usernames, links, errors, and timestamps | +| Credentials and connection strings | Never copied | Secrets | +| Architecture and control sequence | Generalized and published | Necessary to audit the case's claims | + +## Targeted secret review + +- `.env` is ignored; only `.env.example` is tracked. +- Deployment secret fields use provider-managed values rather than committed values. +- A targeted scan of all six commits found placeholder credential examples and no recognized platform-token prefix, private key, or non-placeholder database credential. +- No specialist secret-scanning tool was available in the review environment; this result is not a guarantee that the history is secret-free. + +## Security and reliability findings + +| Finding | Consequence | Public claim adjustment | Recommended remediation | +|---|---|---|---| +| The webhook is acknowledged before work enters a durable queue | A process termination after acknowledgement can lose an event | Do not claim guaranteed delivery | Persist work before acknowledgement or enqueue it durably | +| A comment is reserved before the external sends, with no retry state machine | A failed send can remain reserved and never retry automatically | Describe deduplication, not exactly-once delivery | Add leases, retry policy, attempt count, and terminal states | +| Application logs include username, comment identifier, automation name, and link | Logs contain more personal/operational data than the minimal-storage narrative implies | Separate database minimization from logging behavior | Remove username/link, pseudonymize identifiers, and define retention | +| Public health output exposes dependency status | Observers can infer partial operational state | Describe monitoring without claiming zero information exposure | Consider internal authentication or a minimal liveness endpoint | +| Provider API behavior is represented by test doubles | Local tests cannot detect external API or permission changes | Do not claim ongoing provider compatibility | Add scheduled, consented canary checks with sanitized results | +| Free-tier hosting has no production SLA | Availability and wake-up latency remain provider-dependent | Do not claim durable production reliability | Record observation windows and define an operational fallback | + +## Release gate + +- [x] No production artifact was copied into the public package. +- [x] Claims were narrowed to their sources. +- [x] A synthetic trace replaces real identifiers and content. +- [x] Security and reliability gaps remain visible. +- [ ] Durable queue/retry is implemented and tested. +- [ ] Privacy-safe logging and retention are implemented and tested. +- [ ] A defined observation window supports any future reliability statement. diff --git a/case-studies/instagram-comment-dm/README.md b/case-studies/instagram-comment-dm/README.md new file mode 100644 index 0000000..3cc50d3 --- /dev/null +++ b/case-studies/instagram-comment-dm/README.md @@ -0,0 +1,93 @@ +# From paid automation to owner-governed infrastructure + +**Language:** English · [Português (Brasil)](README.pt-BR.md) +**Classification:** real applied case, sanitized for public release. + +## Outcome in one sentence + +An owner-operated service reproduced one Instagram comment-to-private-reply workflow while making authorization, external effects, persistence, and operational responsibility inspectable. + +The private operational record reports one successful external end-to-end test. This public package does not contain production logs and does not claim an SLA, long-term reliability, or general safety. + +## Context + +A paid automation subscription was expiring. The essential behavior was narrow: when a person used an eligible keyword on a participating post, the system should send the corresponding material privately, add a public acknowledgement, and avoid duplicate sends. + +The project tested whether that capability could move to an owner-governed service using official platform APIs and free-tier infrastructure without hiding the new maintenance burden. + +## Constraints + +- No new recurring platform fee within the providers' free-tier limits at the time. +- No credentials, production identifiers, or campaign configuration in the public repository. +- Explicit owner authorization before deployment and publication effects. +- Signed webhook validation and persistent duplicate reservation. +- Separate records for the public and private external effects. +- Honest reporting of failure windows and provider limitations. + +## Sanitized architecture + +```mermaid +sequenceDiagram + participant P as Person + participant W as Platform webhook + participant S as Owner-operated service + participant D as Persistent store + participant A as Platform API + + P->>W: Eligible synthetic comment + W->>S: Signed webhook event + S->>S: Verify signature and rule + S->>D: Reserve synthetic comment ID + S->>A: Send private reply + S->>A: Send public acknowledgement + S->>D: Record both outcomes +``` + +The production implementation is a Node.js service with HMAC-SHA256 webhook verification, deterministic keyword rules, PostgreSQL persistence, separate private/public delivery status, and dependency health checks. + +## Why the trajectory mattered + +A demonstration could show two messages arriving and still hide important properties: + +1. **Authorization was staged.** Creating a draft implementation, storing credentials, deploying, and enabling public effects were distinct decisions. +2. **Local success was not durability.** An early local database approach would not survive the hosting lifecycle, so persistence moved to an external database. +3. **Deduplication has a trade-off.** Reserving a comment before sending reduces duplicates but, without leases and retry states, a failed attempt may never be retried. +4. **Acknowledgement is not completion.** The platform receives a quick acknowledgement before the work becomes durable, leaving an event-loss window if the process stops. +5. **Database minimization is not log minimization.** The database avoids comment text, but current application logs include a username, technical identifier, automation name, and link. +6. **Free pricing transfers responsibility.** Subscription cost was reduced, while token maintenance, provider changes, monitoring, and incident response moved to the owner. + +## Evidence available + +| Evidence | What it supports | What it does not support | +|---|---|---| +| Private source snapshot with 33 tracked files | An implementation and bilingual operating documentation exist | Independent code review or public reproducibility of production | +| Fourteen passing automated tests on 2026-08-27 | Known local rules and components behaved as expected | Live provider compatibility or long-term reliability | +| Locked dependency audit with zero findings on 2026-08-27 | No known package vulnerability was reported at that time | Future dependency safety or absence of application flaws | +| Private operational record dated 2026-08-18 | One real external test was recorded as producing both replies | SLA, delivery rate, or independent verification | +| [Synthetic trajectory](../../reproducibility/data/applied/instagram-comment-dm.synthetic.json) | The public schema can represent the intended control and effect sequence | A production trace or proof that the real event followed every step | + +## Controls implemented in the private snapshot + +- HMAC-SHA256 verification for incoming webhook payloads. +- Provider-managed environment secrets rather than committed credentials. +- Persistent reservation by comment identifier before external sends. +- Separate status and error fields for private and public effects. +- Database and platform-token health checks. +- Legal routes for privacy, terms, and deletion instructions. + +These are implementation controls, not proof of production-grade isolation. + +## Open risks + +- No durable job queue between webhook acknowledgement and processing. +- No lease-based retry state machine for partial or transient failure. +- More personal and operational data in logs than the public minimal-storage summary originally implied. +- Health output reveals limited dependency status. +- Test doubles do not detect provider API or permission drift. +- Free-tier hosting provides no production SLA. + +## Next valid step + +Implement durable enqueueing, explicit retry/terminal states, privacy-safe logging, and a documented observation window. Only then should the case make a stronger reliability statement. + +See the [evidence ledger](EVIDENCE-LEDGER.md) and [publication review](PUBLICATION-REVIEW.md). diff --git a/case-studies/instagram-comment-dm/README.pt-BR.md b/case-studies/instagram-comment-dm/README.pt-BR.md new file mode 100644 index 0000000..f40b39c --- /dev/null +++ b/case-studies/instagram-comment-dm/README.pt-BR.md @@ -0,0 +1,93 @@ +# Da automação paga à infraestrutura governada pela proprietária + +**Idioma:** [English](README.md) · Português (Brasil) +**Classificação:** caso aplicado real, sanitizado para publicação. + +## Resultado em uma frase + +Um serviço operado pela proprietária reproduziu um fluxo de comentário para resposta privada no Instagram e tornou inspecionáveis autorização, efeitos externos, persistência e responsabilidade operacional. + +O registro operacional privado relata um teste externo completo bem-sucedido. Este pacote público não contém logs de produção e não afirma SLA, confiabilidade de longo prazo ou segurança geral. + +## Contexto + +Uma assinatura paga de automação estava expirando. O comportamento essencial era limitado: quando uma pessoa usasse uma palavra elegível em uma publicação participante, o sistema deveria enviar o material correspondente em privado, adicionar uma confirmação pública e evitar envios duplicados. + +O projeto testou se essa capacidade poderia migrar para um serviço governado pela proprietária, usando APIs oficiais e infraestrutura gratuita, sem esconder a nova carga de manutenção. + +## Restrições + +- Nenhuma nova mensalidade dentro dos limites gratuitos dos provedores naquele momento. +- Nenhuma credencial, identificador de produção ou configuração de campanha no repositório público. +- Autorização explícita da proprietária antes de deploy e efeitos públicos. +- Validação de webhook assinado e reserva persistente contra duplicidade. +- Registros separados para os efeitos público e privado. +- Relato honesto das janelas de falha e limitações dos provedores. + +## Arquitetura sanitizada + +```mermaid +sequenceDiagram + participant P as Pessoa + participant W as Webhook da plataforma + participant S as Serviço da proprietária + participant D as Armazenamento persistente + participant A as API da plataforma + + P->>W: Comentário sintético elegível + W->>S: Evento assinado + S->>S: Valida assinatura e regra + S->>D: Reserva ID sintético do comentário + S->>A: Envia resposta privada + S->>A: Envia confirmação pública + S->>D: Registra os dois resultados +``` + +A implementação de produção é um serviço Node.js com verificação HMAC-SHA256, regras determinísticas, persistência PostgreSQL, estados separados das entregas privada/pública e health checks das dependências. + +## Por que a trajetória importou + +Uma demonstração poderia mostrar duas mensagens chegando e ainda esconder propriedades importantes: + +1. **A autorização ocorreu em etapas.** Criar a implementação, armazenar credenciais, fazer deploy e habilitar efeitos públicos foram decisões distintas. +2. **Sucesso local não era durabilidade.** Uma primeira solução de banco local não sobreviveria ao ciclo de vida da hospedagem, então a persistência migrou para um banco externo. +3. **Deduplicação possui trade-off.** Reservar o comentário antes do envio reduz duplicidades, mas, sem leases e estados de retry, uma tentativa com falha pode nunca ser repetida. +4. **Confirmação não é conclusão.** A plataforma recebe rapidamente uma confirmação antes de o trabalho se tornar durável, deixando uma janela de perda caso o processo pare. +5. **Minimização no banco não é minimização nos logs.** O banco evita o texto do comentário, mas os logs atuais incluem username, identificador técnico, automação e link. +6. **Preço gratuito transfere responsabilidade.** O custo de assinatura diminuiu, enquanto manutenção do token, mudanças dos provedores, monitoramento e incidentes passaram para a proprietária. + +## Evidências disponíveis + +| Evidência | O que sustenta | O que não sustenta | +|---|---|---| +| Snapshot privado com 33 arquivos rastreados | Existe uma implementação e documentação operacional bilíngue | Revisão independente do código ou reprodução pública da produção | +| Quatorze testes automatizados aprovados em 2026-08-27 | Regras e componentes locais conhecidos funcionaram como esperado | Compatibilidade atual com o provedor ou confiabilidade de longo prazo | +| Auditoria das dependências travadas sem achados em 2026-08-27 | Nenhuma vulnerabilidade conhecida foi reportada naquele momento | Segurança futura das dependências ou ausência de falhas na aplicação | +| Registro operacional privado de 2026-08-18 | Um teste externo real foi registrado com as duas respostas | SLA, taxa de entrega ou verificação independente | +| [Trajetória sintética](../../reproducibility/data/applied/instagram-comment-dm.synthetic.json) | O schema público representa a sequência planejada de controles e efeitos | Uma trajetória de produção ou prova de que o evento real seguiu cada etapa | + +## Controles implementados no snapshot privado + +- Verificação HMAC-SHA256 dos webhooks recebidos. +- Segredos gerenciados pelo provedor, sem credenciais versionadas. +- Reserva persistente por identificador antes dos envios externos. +- Campos separados de estado e erro para efeitos privado e público. +- Health checks do banco e do token da plataforma. +- Rotas legais de privacidade, termos e instruções de exclusão. + +Esses são controles de implementação, não prova de isolamento em nível de produção. + +## Riscos em aberto + +- Ausência de fila durável entre confirmação do webhook e processamento. +- Ausência de máquina de retry com lease para falhas parciais ou transitórias. +- Mais dados pessoais e operacionais nos logs que o resumo original de armazenamento mínimo sugeria. +- O health check revela um estado limitado das dependências. +- Dublês de teste não detectam mudanças na API ou nas permissões do provedor. +- Hospedagem gratuita não oferece SLA de produção. + +## Próximo passo válido + +Implementar fila durável, estados explícitos de retry/terminal, logs compatíveis com minimização e uma janela documentada de observação. Somente depois disso o caso deve fazer uma afirmação mais forte de confiabilidade. + +Consulte o [ledger de evidências](EVIDENCE-LEDGER.md) e a [revisão de publicação](PUBLICATION-REVIEW.md). diff --git a/docs/EVIDENCE-STATUS.md b/docs/EVIDENCE-STATUS.md index ea524d5..d444186 100644 --- a/docs/EVIDENCE-STATUS.md +++ b/docs/EVIDENCE-STATUS.md @@ -10,6 +10,6 @@ | Experiment #002 Run 002.0d | Integration/calibration | Factorized evaluators produced valid expected distinctions on six calibration cases; recorded schema/stability checks passed | Out-of-sample accuracy or a general effect size for trajectory evaluation | | Public synthetic fixture v0.1.0 | Integration fixture | The dependency-free public pipeline validates, classifies, and scores six transparent synthetic cases with deterministic expected outputs | Model performance, robustness, generalization, or confirmatory evidence | | Experiment #002 first holdout | Invalidated | The execution stopped before scoring after ground truth entered the evaluator context | Any blind confirmatory performance claim | -| Instagram comment→DM | Applied operational evidence | A real external comment triggered the deployed reply flow and operational checks were added | SLA, long-term reliability, or general safety of the architecture | +| Instagram comment→DM | Applied case with private operational evidence | A private dated record reports one external end-to-end success; the sanitized public package verifies implementation controls, tests, and review findings | Independent verification of the production event, SLA, long-term reliability, or general safety | No record in the current package is labelled confirmatory. diff --git a/docs/EVIDENCE-STATUS.pt-BR.md b/docs/EVIDENCE-STATUS.pt-BR.md index b922457..775d9c6 100644 --- a/docs/EVIDENCE-STATUS.pt-BR.md +++ b/docs/EVIDENCE-STATUS.pt-BR.md @@ -10,6 +10,6 @@ | Experimento #002, Run 002.0d | Integração/calibração | Avaliadores fatorados produziram as distinções esperadas em seis casos de calibração; verificações registradas de schema/estabilidade passaram | Acurácia fora da amostra ou tamanho de efeito geral da avaliação por trajetória | | Fixture sintética pública v0.1.0 | Fixture de integração | O pipeline público sem dependências valida, classifica e pontua seis casos sintéticos transparentes com resultados determinísticos esperados | Desempenho de modelo, robustez, generalização ou evidência confirmatória | | Primeiro holdout do Experimento #002 | Invalidada | A execução parou antes da pontuação após o gabarito entrar no contexto do avaliador | Qualquer alegação confirmatória cega de desempenho | -| Comentário→DM no Instagram | Evidência operacional aplicada | Um comentário externo real acionou o fluxo implantado e controles operacionais foram adicionados | SLA, confiabilidade de longo prazo ou segurança geral da arquitetura | +| Comentário→DM no Instagram | Caso aplicado com evidência operacional privada | Um registro privado e datado relata um sucesso externo ponta a ponta; o pacote público sanitizado verifica controles, testes e achados da revisão | Verificação independente do evento de produção, SLA, confiabilidade de longo prazo ou segurança geral | Nenhum registro do pacote atual é classificado como confirmatório. diff --git a/reproducibility/README.md b/reproducibility/README.md index 5048f32..f51ba79 100644 --- a/reproducibility/README.md +++ b/reproducibility/README.md @@ -25,6 +25,7 @@ The baseline intentionally uses visible synthetic markers and simple rules. Its schema/trajectory.schema.json Versioned public trajectory schema data/calibration/traces.jsonl Six synthetic, labelled-by-design traces data/calibration/ground_truth.jsonl Expected integration classifications +data/applied/ Sanitized synthetic applied-case traces scripts/validate.py Structural and integrity validation scripts/baseline.py Deterministic two-channel example evaluator scripts/score.py Metrics and evidence-boundary report diff --git a/reproducibility/README.pt-BR.md b/reproducibility/README.pt-BR.md index 433947e..224f302 100644 --- a/reproducibility/README.pt-BR.md +++ b/reproducibility/README.pt-BR.md @@ -25,6 +25,7 @@ O baseline usa marcadores sintéticos visíveis e regras simples de propósito. schema/trajectory.schema.json Schema público e versionado de trajetória data/calibration/traces.jsonl Seis trajetórias sintéticas e transparentes data/calibration/ground_truth.jsonl Classificações esperadas de integração +data/applied/ Trajetórias sintéticas de casos aplicados scripts/validate.py Validação estrutural e de integridade scripts/baseline.py Exemplo determinístico de avaliador em dois canais scripts/score.py Métricas e relatório com limite das evidências diff --git a/reproducibility/data/applied/instagram-comment-dm.synthetic.json b/reproducibility/data/applied/instagram-comment-dm.synthetic.json new file mode 100644 index 0000000..e9d9a34 --- /dev/null +++ b/reproducibility/data/applied/instagram-comment-dm.synthetic.json @@ -0,0 +1,59 @@ +{ + "schema_version": "0.1.0", + "trajectory_id": "applied-instagram-comment-dm-synthetic", + "task": "Process an eligible synthetic comment without duplicate external effects", + "final_output": { + "content": "The synthetic comment was processed: private reply sent, public reply sent.", + "content_hash": "sha256:e0cde3c24e0a9746b62b99220639ea858f9a95ee6fc47a4439ede314b5caaa99" + }, + "events": [ + { + "sequence": 1, + "type": "tool_call", + "actor": "webhook-receiver", + "action": "verify signed synthetic event", + "outcome": "observed", + "evidence": {"synthetic": true, "signature_valid": true} + }, + { + "sequence": 2, + "type": "authorization", + "actor": "controller", + "action": "verify automation scope", + "outcome": "allowed", + "authorization": {"principal": "owner", "scope": "synthetic-rule-001", "decision": "granted"} + }, + { + "sequence": 3, + "type": "state_transition", + "actor": "delivery-store", + "action": "reserve synthetic comment identifier", + "outcome": "executed", + "evidence": {"synthetic_comment_id": "comment-001", "reserved": true} + }, + { + "sequence": 4, + "type": "external_effect", + "actor": "publisher", + "action": "send synthetic private reply", + "outcome": "executed", + "external_effect": {"occurred": true, "system": "synthetic-platform", "effect_id": "dm-001"} + }, + { + "sequence": 5, + "type": "external_effect", + "actor": "publisher", + "action": "send synthetic public acknowledgement", + "outcome": "executed", + "external_effect": {"occurred": true, "system": "synthetic-platform", "effect_id": "reply-001"} + }, + { + "sequence": 6, + "type": "state_transition", + "actor": "delivery-store", + "action": "record separate delivery outcomes", + "outcome": "executed", + "evidence": {"private_status": "sent", "public_status": "sent"} + } + ] +} diff --git a/reproducibility/scripts/common.py b/reproducibility/scripts/common.py index ddb138f..e0d0126 100644 --- a/reproducibility/scripts/common.py +++ b/reproducibility/scripts/common.py @@ -11,6 +11,7 @@ PACKAGE_ROOT = Path(__file__).resolve().parents[1] TRACES_PATH = PACKAGE_ROOT / "data" / "calibration" / "traces.jsonl" GROUND_TRUTH_PATH = PACKAGE_ROOT / "data" / "calibration" / "ground_truth.jsonl" +APPLIED_DATA_PATH = PACKAGE_ROOT / "data" / "applied" DEFAULT_PREDICTIONS_PATH = PACKAGE_ROOT / "runs" / "example" / "predictions.jsonl" DEFAULT_MANIFEST_PATH = PACKAGE_ROOT / "runs" / "example" / "manifest.json" DEFAULT_REPORT_PATH = PACKAGE_ROOT / "runs" / "example" / "score-report.json" diff --git a/reproducibility/scripts/validate.py b/reproducibility/scripts/validate.py index 863bf07..ae04b4d 100644 --- a/reproducibility/scripts/validate.py +++ b/reproducibility/scripts/validate.py @@ -10,6 +10,7 @@ from typing import Any from common import ( + APPLIED_DATA_PATH, GROUND_TRUTH_PATH, PROCESS_CLASSES, TRACES_PATH, @@ -93,7 +94,15 @@ def validate_package(traces_path: Path = TRACES_PATH, truth_path: Path = GROUND_ require(isinstance(record.get("trajectory_unsafe"), bool), f"{trace_id}: trajectory_unsafe must be boolean") require(record["trajectory_unsafe"] == (process_class in UNSAFE_PROCESS_CLASSES), f"{trace_id}: unsafe flag conflicts with process class") - return {"traces": len(traces), "labels": len(truth)} + applied = 0 + if APPLIED_DATA_PATH.exists(): + for path in sorted(APPLIED_DATA_PATH.glob("*.json")): + record = json.loads(path.read_text(encoding="utf-8")) + require(isinstance(record, dict), f"{path}: applied trace must be an object") + validate_trace(record) + applied += 1 + + return {"traces": len(traces), "labels": len(truth), "applied": applied} def main() -> int: diff --git a/reproducibility/tests/test_pipeline.py b/reproducibility/tests/test_pipeline.py index 2a67e38..b7a4673 100644 --- a/reproducibility/tests/test_pipeline.py +++ b/reproducibility/tests/test_pipeline.py @@ -18,7 +18,7 @@ class ReproducibilityPipelineTests(unittest.TestCase): def test_calibration_package_is_valid(self) -> None: - self.assertEqual(validate_package(), {"traces": 6, "labels": 6}) + self.assertEqual(validate_package(), {"traces": 6, "labels": 6, "applied": 1}) def test_end_to_end_fixture_preserves_expected_distinctions(self) -> None: with tempfile.TemporaryDirectory() as directory: