diff --git a/.gitignore b/.gitignore index 4f56f9ab..84bfec65 100644 --- a/.gitignore +++ b/.gitignore @@ -49,3 +49,4 @@ notes/ archive-processor/downloaded-rdf archive-processor/local-kms-csv extern +serverless/certs/us-east-1-bundle.pem diff --git a/README.md b/README.md index 7e349899..85a6b50d 100644 --- a/README.md +++ b/README.md @@ -39,9 +39,15 @@ npm run localstack:start By default, `start-local` enables Redis with the local container settings from `bin/env/local_env.sh`, so the normal local startup path is: ```bash npm run redis:start +npm run documentdb:start npm run start-local ``` +The local MongoDB 8 container provides a DocumentDB-compatible metadata-correction audit store. +It listens on `localhost:27018` for host scripts and uses the shared KMS Docker network for SAM. +Starting it also creates or verifies the metadata-correction audit indexes used in AWS. +Stop it with `npm run documentdb:stop`. + If you do not need Redis for your local test, start local with Redis disabled: ```bash REDIS_ENABLED=false npm run start-local @@ -71,7 +77,8 @@ Local development intentionally splits responsibilities between SAM and LocalSta - SAM runs the API Gateway and Lambda side of KMS locally. - LocalStack emulates AWS-managed services that SAM does not model end-to-end for this repo, especially SNS and SQS. -- RDF4J and Redis remain separate local services because they are not AWS services. +- RDF4J, Redis, and the MongoDB-compatible audit database remain separate local services because + they are not modeled by SAM. We do not run the entire application stack inside LocalStack because the existing SAM flow is simpler for day-to-day Lambda/API development, while LocalStack is most useful here for the managed messaging pieces. For keyword event processing, `npm run start-local` also starts `scripts/localstack/run_bridge.sh`, which runs `scripts/localstack/bridge.js`. @@ -297,7 +304,7 @@ Internally, the correction flow is now object-first: key construction - XML and UMM delegates work from `oldKeywordObject` / `newKeywordObject` - joined `oldKeywordPath` / `newKeywordPath` strings are now primarily boundary values for Redis, - logs, and audit records + logs, and audit documents The important distinction is: @@ -440,6 +447,48 @@ Resolved corrections are also object-first now: Audit logging still derives `oldKeywordPath` / `newKeywordPath` strings for readability, but the runtime correction and delegate flow works from normalized keyword objects. +Each collection-correction run is stored as one audit document. Its `statusHistory` records the +`checked`, `pending`, and terminal `applied` or `failed` transitions. The audit document also +links to the current CMR collection record, records the prior and resulting CMR revision IDs, and +stores a bounded unified diff between the original native metadata and the corrected writeback +payload. +The audit API is: + +- `GET /metadata_correction_audit` for newest-first, token-paginated audit searches. Supported + filters include collection, keyword UUID, action, scheme, status, native format, KMS version, + source, and date range. Supplied actions and schemes must be recognized KMS values, limits must + be integers from 1 through 250, and `startDate` must not be after `endDate`. List results contain + compact collection, status, and old-to-new keyword path summaries. Add `?includeDiff=true` to + include each available native-metadata diff in the list results. +- `GET /metadata_correction_audit/{runId}` for the complete audit document. Add + `?includeDiff=true` when the native-metadata diff is needed; it is omitted by default to keep + routine responses small. +- Add `?format=html` to either endpoint for a self-contained browser view. The list is a compact + status and keyword-change summary with links to each run; add `&includeDiff=true` if list-level + native diffs are needed. The detail view includes run context, lifecycle history, diagnostics, + and the colored side-by-side native metadata diff. HTML lists default to 10 records per page. + +Publisher events carry the published KMS version through the queue into this document. Manual +correction endpoints look up the current published version before starting the run, so the +metadata-correction consumer and audit API do not query RDF4J. + +Deployed Lambdas use the public AWS `us-east-1` CA bundle to validate DocumentDB TLS connections. +The Bamboo deployment downloads the current bundle from the +[AWS certificate trust store](https://truststore.pki.rds.amazonaws.com/us-east-1/us-east-1-bundle.pem) +before building the deployment image. For an AWS deployment outside Bamboo, download it manually: + +```bash +curl --fail --location \ + https://truststore.pki.rds.amazonaws.com/us-east-1/us-east-1-bundle.pem \ + --output serverless/certs/us-east-1-bundle.pem +``` + +AWS documents the CA-bundle download requirement in +[Connecting programmatically to Amazon DocumentDB](https://docs.aws.amazon.com/documentdb/latest/developerguide/connect_programmatically.html). +That example uses the global bundle; KMS uses the equivalent regional bundle listed for +US East (N. Virginia) in the +[AWS regional certificate bundle table](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html). + ## Setting up the RDF Database for local development In order to run KMS locally, you first need to setup a RDF database. ### Prerequisites @@ -499,6 +548,9 @@ export bamboo_CMR_WRITER_TOKEN=[optional complete bearer authorization value use export bamboo_CMR_WRITEBACK_PROVIDERS=[optional provider id, comma-separated list, or ALL] export bamboo_CMR_WRITEBACK_VALIDATE_KEYWORDS=[true|false; defaults to false] export bamboo_CMR_WRITEBACK_VALIDATE_UMM_C=[true|false; defaults to false] +export bamboo_CMR_WRITEBACK_TIMEOUT_MS=[optional timeout in milliseconds; defaults to 25000] +export bamboo_METADATA_CORRECTION_RUNS_PER_MINUTE=[optional positive integer correction run rate] +export bamboo_METADATA_CORRECTION_SERVICE_RESERVED_CONCURRENCY=[optional; defaults to 5] export bamboo_CORS_ORIGIN=[comma separated list of cors origins] export bamboo_RDF4J_CONTAINER_MEMORY_LIMIT=[7168 for sit|uat, 14336 for prod] export bamboo_RDF4J_INSTANCE_TYPE=["M5.LARGE" for sit|uat, "R5.LARGE" for prod] @@ -521,6 +573,11 @@ Notes: - Leave `bamboo_CMR_WRITEBACK_PROVIDERS` empty to disable provider rollout for CMR writeback. - Set `bamboo_CMR_WRITEBACK_VALIDATE_KEYWORDS` and `bamboo_CMR_WRITEBACK_VALIDATE_UMM_C` to `true` to reject writebacks that still fail CMR keyword or UMM-C validation. +- `bamboo_CMR_WRITEBACK_TIMEOUT_MS` is capped at 45000 milliseconds so the worker can record a + failed audit before its 60-second Lambda timeout. +- Setting `bamboo_METADATA_CORRECTION_RUNS_PER_MINUTE` enables queue pacing and forces the + metadata-correction worker concurrency to `1`. When it is unset, pacing is disabled and + `bamboo_METADATA_CORRECTION_SERVICE_RESERVED_CONCURRENCY` controls concurrency. - If you are not deploying into an existing API Gateway, set `bamboo_EXISTING_API_ID` and `bamboo_ROOT_RESOURCE_ID` to empty strings. - If `bamboo_RDF4J_BACKUP_VAULT_NAME` is set, `SnapshotStack` imports that existing backup vault. This is useful when `rdf4jSnapshotStack` is being recreated after an RDF4J recovery event and you need the new stack to reuse an existing vault instead of trying to create the same vault name again. - If `bamboo_RDF4J_BACKUP_VAULT_NAME` is not set, `SnapshotStack` creates the default `rdf4j-backup-vault`. diff --git a/bin/deploy-bamboo.sh b/bin/deploy-bamboo.sh index cf8924eb..4504cdc1 100755 --- a/bin/deploy-bamboo.sh +++ b/bin/deploy-bamboo.sh @@ -18,6 +18,14 @@ config="`jq '.edl.uid = $newValue' --arg newValue $bamboo_EDL_UID <<< $config`" # overwrite static.config.json with new values echo $config > tmp.$$.json && mv tmp.$$.json static.config.json +# Download the current public AWS CA bundle before it is packaged with the Lambdas. +documentDbCaBundleUrl='https://truststore.pki.rds.amazonaws.com/us-east-1/us-east-1-bundle.pem' +documentDbCaBundlePath='serverless/certs/us-east-1-bundle.pem' +mkdir -p "$(dirname "$documentDbCaBundlePath")" +curl --fail --silent --show-error --location \ + "$documentDbCaBundleUrl" \ + --output "$documentDbCaBundlePath" + # Set up Docker image ##################### @@ -70,6 +78,8 @@ dockerRun() { --env "CMR_WRITEBACK_PROVIDERS=${bamboo_CMR_WRITEBACK_PROVIDERS:-}" \ --env "CMR_WRITEBACK_VALIDATE_KEYWORDS=${bamboo_CMR_WRITEBACK_VALIDATE_KEYWORDS:-false}" \ --env "CMR_WRITEBACK_VALIDATE_UMM_C=${bamboo_CMR_WRITEBACK_VALIDATE_UMM_C:-false}" \ + --env "CMR_WRITEBACK_TIMEOUT_MS=${bamboo_CMR_WRITEBACK_TIMEOUT_MS:-25000}" \ + --env "METADATA_CORRECTION_RUNS_PER_MINUTE=${bamboo_METADATA_CORRECTION_RUNS_PER_MINUTE:-}" \ --env "METADATA_CORRECTION_SERVICE_RESERVED_CONCURRENCY=${bamboo_METADATA_CORRECTION_SERVICE_RESERVED_CONCURRENCY:-5}" \ --env "BLOCK_PUBLISH_ON_KEYWORD_DIFF_FAILURE=${bamboo_BLOCK_PUBLISH_ON_KEYWORD_DIFF_FAILURE:-false}" \ --env "KEYWORD_SYNC_ALARM_EMAILS=${bamboo_KEYWORD_SYNC_ALARM_EMAILS:-}" \ diff --git a/bin/documentdb/start.sh b/bin/documentdb/start.sh new file mode 100755 index 00000000..f7ecf12e --- /dev/null +++ b/bin/documentdb/start.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" +# shellcheck source=bin/env/local_env.sh +source "${SCRIPT_DIR}/../env/local_env.sh" + +IMAGE="${DOCUMENTDB_LOCAL_IMAGE:-mongo:8.0}" +CONTAINER_PORT="27017" +VOLUME_NAME="${DOCUMENTDB_LOCAL_VOLUME:-kms-documentdb-local-data}" + +if ! docker network inspect "${KMS_DOCKER_NETWORK}" >/dev/null 2>&1; then + docker network create "${KMS_DOCKER_NETWORK}" >/dev/null + echo "Created docker network '${KMS_DOCKER_NETWORK}'" +fi + +existing_id="$(docker ps -aq --filter "name=^${DOCUMENTDB_CONTAINER_NAME}$")" +if [[ -n "${existing_id}" ]]; then + running_id="$(docker ps -q --filter "name=^${DOCUMENTDB_CONTAINER_NAME}$")" + if [[ -n "${running_id}" ]]; then + echo "Mongo container '${DOCUMENTDB_CONTAINER_NAME}' is already running" + else + docker start "${DOCUMENTDB_CONTAINER_NAME}" >/dev/null + echo "Started existing Mongo container '${DOCUMENTDB_CONTAINER_NAME}'" + fi +else + docker run -d \ + --name "${DOCUMENTDB_CONTAINER_NAME}" \ + --network "${KMS_DOCKER_NETWORK}" \ + --network-alias "${DOCUMENTDB_CONTAINER_NAME}" \ + -p "${DOCUMENTDB_HOST_PORT}:${CONTAINER_PORT}" \ + -v "${VOLUME_NAME}:/data/db" \ + "${IMAGE}" >/dev/null + + echo "Started Mongo container '${DOCUMENTDB_CONTAINER_NAME}' on ${DOCUMENTDB_HOST_PORT}->${CONTAINER_PORT}" +fi + +mongo_ready=false +for _ in {1..30}; do + if docker exec "${DOCUMENTDB_CONTAINER_NAME}" \ + mongosh --quiet --eval 'db.runCommand({ ping: 1 }).ok' 2>/dev/null | grep -q '1'; then + mongo_ready=true + break + fi + + sleep 1 +done + +if [[ "${mongo_ready}" != "true" ]]; then + echo "Mongo container '${DOCUMENTDB_CONTAINER_NAME}' did not become ready" >&2 + exit 1 +fi + +( + cd "${PROJECT_ROOT}" + DOCUMENTDB_URI="mongodb://127.0.0.1:${DOCUMENTDB_HOST_PORT}/?directConnection=true" \ + ./node_modules/.bin/vite-node \ + --config vite.config.js \ + scripts/local/initialize_metadata_correction_audit.mjs +) + +echo "Connect from host using mongodb://localhost:${DOCUMENTDB_HOST_PORT}" +echo "Connect from SAM using ${DOCUMENTDB_URI}" diff --git a/bin/documentdb/stop.sh b/bin/documentdb/stop.sh new file mode 100755 index 00000000..c489a645 --- /dev/null +++ b/bin/documentdb/stop.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=bin/env/local_env.sh +source "${SCRIPT_DIR}/../env/local_env.sh" + +container_id="$(docker ps -aq --filter "name=^${DOCUMENTDB_CONTAINER_NAME}$")" +if [[ -z "${container_id}" ]]; then + echo "Mongo container '${DOCUMENTDB_CONTAINER_NAME}' does not exist" + exit 0 +fi + +running_id="$(docker ps -q --filter "name=^${DOCUMENTDB_CONTAINER_NAME}$")" +if [[ -z "${running_id}" ]]; then + echo "Mongo container '${DOCUMENTDB_CONTAINER_NAME}' is already stopped" + exit 0 +fi + +docker stop "${DOCUMENTDB_CONTAINER_NAME}" >/dev/null +echo "Stopped Mongo container '${DOCUMENTDB_CONTAINER_NAME}'" diff --git a/bin/env/local_env.sh b/bin/env/local_env.sh index 30f61ba2..f98c80bc 100644 --- a/bin/env/local_env.sh +++ b/bin/env/local_env.sh @@ -19,5 +19,8 @@ export LOCALSTACK_CONTAINER_NAME="${LOCALSTACK_CONTAINER_NAME:-kms-localstack}" export LOCALSTACK_IMAGE="${LOCALSTACK_IMAGE:-localstack/localstack:3.8.1}" export LOCALSTACK_PORT="${LOCALSTACK_PORT:-4566}" export AWS_ENDPOINT_URL="${AWS_ENDPOINT_URL:-http://localstack:${LOCALSTACK_PORT}}" +export DOCUMENTDB_URI="${DOCUMENTDB_URI:-mongodb://kms-documentdb-local:27017/?directConnection=true}" +export DOCUMENTDB_CONTAINER_NAME="${DOCUMENTDB_CONTAINER_NAME:-kms-documentdb-local}" +export DOCUMENTDB_HOST_PORT="${DOCUMENTDB_HOST_PORT:-27018}" export SAM_WARM_CONTAINERS="${SAM_WARM_CONTAINERS:-LAZY}" export SAM_LOCAL_WATCH="${SAM_LOCAL_WATCH:-false}" diff --git a/bin/start-local.sh b/bin/start-local.sh index 952afd27..cd3b18bd 100755 --- a/bin/start-local.sh +++ b/bin/start-local.sh @@ -25,7 +25,7 @@ clearStaleSAMContainers() { echo "Clearing stale SAM containers..." docker ps --format '{{.ID}} {{.Image}}' \ - | awk '$2 ~ /public\.ecr\.aws\/lambda\/nodejs:22-rapid-/ { print $1 }' \ + | awk '$2 ~ /public\.ecr\.aws\/lambda\/nodejs:[0-9]+-rapid-/ { print $1 }' \ | xargs -r docker rm -f >/dev/null 2>&1 || true } diff --git a/cdk/app/lib/CmrEventProcessingStack.ts b/cdk/app/lib/CmrEventProcessingStack.ts index 7e12a0cf..3168c557 100644 --- a/cdk/app/lib/CmrEventProcessingStack.ts +++ b/cdk/app/lib/CmrEventProcessingStack.ts @@ -1,5 +1,6 @@ import * as cdk from 'aws-cdk-lib' import * as ec2 from 'aws-cdk-lib/aws-ec2' +import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager' import * as sns from 'aws-cdk-lib/aws-sns' import { Construct } from 'constructs' @@ -14,18 +15,20 @@ import { VpcSetup } from './helper/VpcSetup' export interface CmrEventProcessingStackProps extends cdk.StackProps { cmrBaseUrl: string cmrSystemTokenParameterName?: string + cmrWritebackTimeoutMs?: string metadataCorrectionRequestDelayMs?: string + metadataCorrectionRunsPerMinute?: string metadataCorrectionServiceReservedConcurrency?: string cmrWriterToken?: string cmrWritebackProviders?: string cmrWritebackValidateKeywords?: string cmrWritebackValidateUmmC?: string + metadataCorrectionAuditClientSecurityGroup?: ec2.ISecurityGroup + metadataCorrectionAuditEnvironment: Record + metadataCorrectionAuditSecret?: secretsmanager.ISecret redisEnabled?: string redisHost?: string redisPort?: string - rdf4jPassword: string - rdf4jServiceUrl: string - rdf4jUserName: string prefix: string stage: string topicArn: string @@ -66,20 +69,23 @@ export class CmrEventProcessingStack extends cdk.Stack { const metadataCorrectionSetup = new MetadataCorrectionSetup(this, 'MetadataCorrection', { cmrBaseUrl: props.cmrBaseUrl, cmrSystemTokenParameterName: props.cmrSystemTokenParameterName, + cmrWritebackTimeoutMs: props.cmrWritebackTimeoutMs, metadataCorrectionRequestDelayMs: props.metadataCorrectionRequestDelayMs, + metadataCorrectionRunsPerMinute: props.metadataCorrectionRunsPerMinute, metadataCorrectionServiceReservedConcurrency: props.metadataCorrectionServiceReservedConcurrency, cmrWriterToken: props.cmrWriterToken, cmrWritebackProviders: props.cmrWritebackProviders, cmrWritebackValidateKeywords: props.cmrWritebackValidateKeywords, cmrWritebackValidateUmmC: props.cmrWritebackValidateUmmC, + metadataCorrectionAuditClientSecurityGroup: + props.metadataCorrectionAuditClientSecurityGroup, + metadataCorrectionAuditEnvironment: props.metadataCorrectionAuditEnvironment, + metadataCorrectionAuditSecret: props.metadataCorrectionAuditSecret, prefix: props.prefix, redisEnabled: props.redisEnabled, redisHost: props.redisHost, redisPort: props.redisPort, - rdf4jPassword: props.rdf4jPassword, - rdf4jServiceUrl: props.rdf4jServiceUrl, - rdf4jUserName: props.rdf4jUserName, stage: props.stage, securityGroup: this.securityGroup, useLocalstack, diff --git a/cdk/app/lib/KmsStack.ts b/cdk/app/lib/KmsStack.ts index 9e3878aa..c44d814c 100644 --- a/cdk/app/lib/KmsStack.ts +++ b/cdk/app/lib/KmsStack.ts @@ -3,6 +3,7 @@ import * as cdk from 'aws-cdk-lib' import * as apigateway from 'aws-cdk-lib/aws-apigateway' import * as ec2 from 'aws-cdk-lib/aws-ec2' import * as iam from 'aws-cdk-lib/aws-iam' +import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager' import * as sns from 'aws-cdk-lib/aws-sns' import { Construct } from 'constructs' @@ -19,12 +20,16 @@ import { VpcSetup } from './helper/VpcSetup' */ export interface KmsStackProps extends cdk.StackProps { cmrSystemTokenParameterName?: string + cmrWritebackTimeoutMs?: string cmrWritebackProviders?: string cmrWritebackValidateKeywords?: string cmrWritebackValidateUmmC?: string cmrWriterToken?: string existingApiId: string | undefined keywordSyncAlarmEmails?: string[] + metadataCorrectionAuditClientSecurityGroup?: ec2.ISecurityGroup + metadataCorrectionAuditEnvironment: Record + metadataCorrectionAuditSecret?: secretsmanager.ISecret prefix: string rootResourceId: string | undefined stage: string @@ -124,6 +129,8 @@ export class KmsStack extends cdk.Stack { ) this.lambdaRole = iamSetup.lambdaRole + props.metadataCorrectionAuditSecret?.grantRead(this.lambdaRole) + this.keywordEventsTopic = new sns.Topic(this, 'KeywordEventsTopic', { topicName: keywordEventsTopicName }) @@ -175,12 +182,16 @@ export class KmsStack extends cdk.Stack { lambdaRole: this.lambdaRole, metadataCorrectionEnvironment: { CMR_SYSTEM_TOKEN_PARAMETER_NAME: props.cmrSystemTokenParameterName || '', + CMR_WRITEBACK_TIMEOUT_MS: props.cmrWritebackTimeoutMs || '', CMR_WRITER_TOKEN: props.cmrWriterToken || '', CMR_WRITEBACK_PROVIDERS: props.cmrWritebackProviders || '', CMR_WRITEBACK_VALIDATE_KEYWORDS: props.cmrWritebackValidateKeywords || '', CMR_WRITEBACK_VALIDATE_UMM_C: props.cmrWritebackValidateUmmC || '', - METADATA_CORRECTION_REQUESTS_TOPIC_ARN: metadataCorrectionRequestsTopicArn + METADATA_CORRECTION_REQUESTS_TOPIC_ARN: metadataCorrectionRequestsTopicArn, + ...props.metadataCorrectionAuditEnvironment }, + metadataCorrectionAuditClientSecurityGroup: + props.metadataCorrectionAuditClientSecurityGroup, prefix, securityGroup: this.securityGroup, stage: this.stage, diff --git a/cdk/app/lib/MetadataCorrectionAuditStack.ts b/cdk/app/lib/MetadataCorrectionAuditStack.ts new file mode 100644 index 00000000..83becdbe --- /dev/null +++ b/cdk/app/lib/MetadataCorrectionAuditStack.ts @@ -0,0 +1,130 @@ +import * as fs from 'fs' +import * as path from 'path' + +import * as cdk from 'aws-cdk-lib' +import * as docdb from 'aws-cdk-lib/aws-docdb' +import * as ec2 from 'aws-cdk-lib/aws-ec2' +import { NodejsFunction } from 'aws-cdk-lib/aws-lambda-nodejs' +import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager' +import * as customResources from 'aws-cdk-lib/custom-resources' +import { Construct } from 'constructs' + +import { getDocumentDbCertificateBundling } from './helper/DocumentDbLambdaConfig' +import { MetadataCorrectionAuditDatabaseSetup } from './helper/MetadataCorrectionAuditDatabaseSetup' +import { NODE_LAMBDA_RUNTIME } from './helper/NodeLambdaRuntime' + +const PROJECT_ROOT = path.join(__dirname, '../../..') +const METADATA_CORRECTION_AUDIT_INDEXES = JSON.parse(fs.readFileSync( + path.join(PROJECT_ROOT, 'config/metadataCorrectionAuditIndexes.json'), + 'utf8' +)) + +export interface MetadataCorrectionAuditStackProps extends cdk.StackProps { + localUri?: string + maxCapacity?: number + minCapacity?: number + prefix: string + stage: string + useLocalstack: boolean + vpcId: string +} + +/** + * Shared DocumentDB infrastructure for metadata-correction audit readers and writers. + */ +export class MetadataCorrectionAuditStack extends cdk.Stack { + public readonly cluster?: docdb.DatabaseCluster + + public readonly clientSecurityGroup?: ec2.SecurityGroup + + public readonly connectionEnvironment: Record + + public readonly secret?: secretsmanager.ISecret + + /** + * Creates the shared audit database resources and exposes their Lambda connection settings. + * + * @param scope Parent CDK construct. + * @param id Stack identifier. + * @param props Environment, VPC, and serverless capacity configuration. + */ + constructor( + scope: Construct, + id: string, + props: MetadataCorrectionAuditStackProps + ) { + super(scope, id, props) + + const vpc = props.useLocalstack + ? undefined + : ec2.Vpc.fromLookup(this, 'Vpc', { vpcId: props.vpcId }) + const databaseSetup = new MetadataCorrectionAuditDatabaseSetup( + this, + 'Database', + { + localUri: props.localUri, + maxCapacity: props.maxCapacity, + minCapacity: props.minCapacity, + prefix: props.prefix, + stage: props.stage, + useLocalstack: props.useLocalstack, + vpc + } + ) + + this.cluster = databaseSetup.cluster + this.clientSecurityGroup = databaseSetup.clientSecurityGroup + this.connectionEnvironment = databaseSetup.environment + this.secret = databaseSetup.secret + + if ( + !props.useLocalstack + && vpc + && this.cluster + && this.clientSecurityGroup + && this.secret + && databaseSetup.dbInstance + ) { + // Define the deployment Lambda that connects to DocumentDB and creates the indexes. + const indexInitializer = new NodejsFunction(this, 'IndexInitializer', { + functionName: `${props.prefix}-${props.stage}-metadata-correction-audit-indexes`, + entry: path.join( + PROJECT_ROOT, + 'serverless/src/initializeMetadataCorrectionAudit/handler.js' + ), + handler: 'initializeMetadataCorrectionAudit', + runtime: NODE_LAMBDA_RUNTIME, + timeout: cdk.Duration.minutes(5), + memorySize: 512, + environment: this.connectionEnvironment, + ...getDocumentDbCertificateBundling(this.connectionEnvironment), + depsLockFilePath: path.join(PROJECT_ROOT, 'package-lock.json'), + projectRoot: PROJECT_ROOT, + vpc, + vpcSubnets: { + subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS + }, + securityGroups: [this.clientSecurityGroup] + }) + this.secret.grantRead(indexInitializer) + + // Register the initializer as the handler for CloudFormation custom-resource events. + const indexProvider = new customResources.Provider(this, 'IndexProvider', { + onEventHandler: indexInitializer + }) + + // Invoke the provider when this stack creates or updates the audit index resource. + const indexResource = new cdk.CustomResource(this, 'Indexes', { + serviceToken: indexProvider.serviceToken, + properties: { + ClusterEndpoint: this.cluster.clusterEndpoint.hostname, + // Preserve numeric index directions across the string-based custom-resource boundary. + IndexDefinitions: JSON.stringify(METADATA_CORRECTION_AUDIT_INDEXES) + } + }) + indexResource.node.addDependency(databaseSetup.dbInstance) + } + } +} + +export default MetadataCorrectionAuditStack diff --git a/cdk/app/lib/helper/DocumentDbLambdaConfig.ts b/cdk/app/lib/helper/DocumentDbLambdaConfig.ts new file mode 100644 index 00000000..b2fe9801 --- /dev/null +++ b/cdk/app/lib/helper/DocumentDbLambdaConfig.ts @@ -0,0 +1,78 @@ +import * as ec2 from 'aws-cdk-lib/aws-ec2' +import { NodejsFunctionProps } from 'aws-cdk-lib/aws-lambda-nodejs' + +const DOCUMENTDB_TLS_CA_FILE_NAME = 'us-east-1-bundle.pem' + +/** + * Bundles DocumentDB Lambda dependencies and copies the CA certificate when TLS is configured. + * + * @example + * getDocumentDbCertificateBundling({ + * DOCUMENTDB_DATABASE_NAME: 'kms', + * DOCUMENTDB_TLS_CA_FILE: '/var/task/us-east-1-bundle.pem' + * }) + * // { bundling: { externalModules: [], commandHooks: { ...copy the CA... } } } + * + * @param environment Lambda environment containing the database name and optional TLS path. + * @returns Bundling props for DocumentDB-enabled Lambdas, otherwise an empty object. + */ +export const getDocumentDbCertificateBundling = ( + environment: { + DOCUMENTDB_DATABASE_NAME?: string + DOCUMENTDB_TLS_CA_FILE?: string + } +): Pick => ( + environment.DOCUMENTDB_DATABASE_NAME + ? { + bundling: { + // Node.js 24 SAM images do not supply AWS SDK packages to local Lambda containers. + externalModules: [], + loader: { + '.css': 'text' + }, + ...(environment.DOCUMENTDB_TLS_CA_FILE + ? { + commandHooks: { + beforeBundling: () => [], + beforeInstall: () => [], + afterBundling: (inputDir: string, outputDir: string) => [ + `cp "${inputDir}/serverless/certs/${DOCUMENTDB_TLS_CA_FILE_NAME}" "${outputDir}/${DOCUMENTDB_TLS_CA_FILE_NAME}"` + ] + } + } + : {}) + } + } + : {} +) + +/** + * Adds the DocumentDB client security group to a Lambda when the database is configured. + * + * @example + * getDocumentDbLambdaSecurityGroups({ + * securityGroup: baseGroup, + * clientSecurityGroup: documentDbGroup, + * environment: { DOCUMENTDB_DATABASE_NAME: 'kms' } + * }) + * // [baseGroup, documentDbGroup] + * + * @param props Security groups and DocumentDB environment for the Lambda. + * @returns The base Lambda group plus the DocumentDB client group when configured. + */ +export const getDocumentDbLambdaSecurityGroups = ({ + clientSecurityGroup, + environment, + securityGroup +}: { + clientSecurityGroup?: ec2.ISecurityGroup + environment: { DOCUMENTDB_DATABASE_NAME?: string } + securityGroup: ec2.ISecurityGroup +}): ec2.ISecurityGroup[] => [ + securityGroup, + ...(environment.DOCUMENTDB_DATABASE_NAME && clientSecurityGroup + ? [clientSecurityGroup] + : []) +] + +export default getDocumentDbCertificateBundling diff --git a/cdk/app/lib/helper/KmsLambdaFunctions.ts b/cdk/app/lib/helper/KmsLambdaFunctions.ts index f32e7300..923bba38 100644 --- a/cdk/app/lib/helper/KmsLambdaFunctions.ts +++ b/cdk/app/lib/helper/KmsLambdaFunctions.ts @@ -11,6 +11,10 @@ import { NodejsFunction, NodejsFunctionProps } from 'aws-cdk-lib/aws-lambda-node import { Construct } from 'constructs' import { ApiResources } from './ApiResources' +import { + getDocumentDbCertificateBundling, + getDocumentDbLambdaSecurityGroups +} from './DocumentDbLambdaConfig' import { NODE_LAMBDA_RUNTIME } from './NodeLambdaRuntime' /** @@ -20,12 +24,22 @@ interface LambdaFunctionsProps { api: apigateway.IRestApi; apiResources: ApiResources; lambdaRole: iam.Role; + metadataCorrectionAuditClientSecurityGroup?: ec2.ISecurityGroup; metadataCorrectionEnvironment?: { CMR_SYSTEM_TOKEN_PARAMETER_NAME?: string; + CMR_WRITEBACK_TIMEOUT_MS?: string; CMR_WRITER_TOKEN: string; CMR_WRITEBACK_PROVIDERS: string; CMR_WRITEBACK_VALIDATE_KEYWORDS: string; CMR_WRITEBACK_VALIDATE_UMM_C: string; + DOCUMENTDB_AUDIT_COLLECTION_NAME?: string; + DOCUMENTDB_DATABASE_NAME?: string; + DOCUMENTDB_HOST?: string; + DOCUMENTDB_MAX_POOL_SIZE?: string; + DOCUMENTDB_PORT?: string; + DOCUMENTDB_SECRET_ARN?: string; + DOCUMENTDB_TLS_CA_FILE?: string; + DOCUMENTDB_URI?: string; METADATA_CORRECTION_REQUESTS_TOPIC_ARN?: string; }; prefix: string; @@ -327,12 +341,29 @@ export class LambdaFunctions { ) this.createApiLambda( - scope, - 'getMetadataCorrectionAudit/handler.js', - 'get-metadata-correction-audit', - 'getMetadataCorrectionAudit', - '/metadata_correction_audit', - 'GET' + scope, // CDK construct scope + 'getMetadataCorrectionAudit/handler.js', // Lambda handler path + 'get-metadata-correction-audit', // Lambda function name + 'getMetadataCorrectionAudit', // Exported handler name + '/metadata_correction_audit', // API resource path + 'GET', // HTTP method + false, // Do not use the EDL authorizer + Duration.seconds(30), // Lambda timeout + 1024, // Lambda memory in MB + this.props.metadataCorrectionEnvironment || {} // Additional Lambda environment variables + ) + + this.createApiLambda( + scope, // CDK construct scope + 'getMetadataCorrectionAudit/handler.js', // Lambda handler path + 'get-metadata-correction-audit', // Reuse the audit Lambda + 'getMetadataCorrectionAudit', // Exported handler name + '/metadata_correction_audit/{runId}', // Detailed audit resource path + 'GET', // HTTP method + false, // Do not use the EDL authorizer + Duration.seconds(30), // Lambda timeout + 1024, // Lambda memory in MB + this.props.metadataCorrectionEnvironment || {} // Additional Lambda environment variables ) this.createApiLambda( @@ -784,13 +815,18 @@ export class LambdaFunctions { ...this.props.environment, ...additionalEnvironment }, + ...getDocumentDbCertificateBundling(additionalEnvironment), // Conditionally add VPC configuration ...(this.useLocalstack ? {} : { vpc: this.props.vpc, vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS }, - securityGroups: [this.props.securityGroup] + securityGroups: getDocumentDbLambdaSecurityGroups({ + clientSecurityGroup: this.props.metadataCorrectionAuditClientSecurityGroup, + environment: additionalEnvironment, + securityGroup: this.props.securityGroup + }) }) } diff --git a/cdk/app/lib/helper/MetadataCorrectionAuditDatabaseSetup.ts b/cdk/app/lib/helper/MetadataCorrectionAuditDatabaseSetup.ts new file mode 100644 index 00000000..7a17cf14 --- /dev/null +++ b/cdk/app/lib/helper/MetadataCorrectionAuditDatabaseSetup.ts @@ -0,0 +1,132 @@ +import * as cdk from 'aws-cdk-lib' +import * as docdb from 'aws-cdk-lib/aws-docdb' +import * as ec2 from 'aws-cdk-lib/aws-ec2' +import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager' +import { Construct } from 'constructs' + +interface MetadataCorrectionAuditDatabaseSetupProps { + databaseName?: string + localUri?: string + maxCapacity?: number + minCapacity?: number + prefix: string + stage: string + useLocalstack: boolean + vpc?: ec2.IVpc +} + +/** + * Provisions the shared DocumentDB Serverless cluster used by metadata-correction auditing. + */ +export class MetadataCorrectionAuditDatabaseSetup extends Construct { + public readonly cluster?: docdb.DatabaseCluster + + public readonly clientSecurityGroup?: ec2.SecurityGroup + + public readonly environment: Record + + public readonly secret?: secretsmanager.ISecret + + public readonly dbInstance?: docdb.DatabaseInstance + + /** + * Configures a local MongoDB URI for LocalStack, or provisions the deployed DocumentDB cluster, + * secret, TLS environment, and paired database/client security groups. + * + * @param scope Parent CDK construct. + * @param id Construct identifier. + * @param props Local or deployed database configuration. + */ + constructor( + scope: Construct, + id: string, + props: MetadataCorrectionAuditDatabaseSetupProps + ) { + super(scope, id) + + const databaseName = props.databaseName || 'kms' + const commonEnvironment = { + DOCUMENTDB_DATABASE_NAME: databaseName, + DOCUMENTDB_AUDIT_COLLECTION_NAME: 'metadataCorrectionAudits', + DOCUMENTDB_MAX_POOL_SIZE: '5' + } + + if (props.useLocalstack) { + this.environment = { + ...commonEnvironment, + DOCUMENTDB_URI: props.localUri || 'mongodb://kms-documentdb-local:27017/?directConnection=true' + } + + return + } + + if (!props.vpc) { + throw new Error('A VPC is required for a deployed DocumentDB cluster') + } + + const databaseSecurityGroup = new ec2.SecurityGroup(this, 'DatabaseSecurityGroup', { + vpc: props.vpc, + allowAllOutbound: true, + description: 'DocumentDB access for KMS metadata-correction audit Lambdas' + }) + this.clientSecurityGroup = new ec2.SecurityGroup(this, 'ClientSecurityGroup', { + vpc: props.vpc, + allowAllOutbound: true, + description: 'Shared client access to the KMS metadata-correction audit database' + }) + + databaseSecurityGroup.addIngressRule( + this.clientSecurityGroup, + ec2.Port.tcp(27017), + 'Allow metadata-correction audit clients' + ) + + this.cluster = new docdb.DatabaseCluster(this, 'Cluster', { + dbClusterName: `${props.prefix}-${props.stage}-metadata-correction-audit`, + engineVersion: '8.0.0', + masterUser: { + username: 'kms_audit', + secretName: `${props.prefix}/${props.stage}/metadata-correction-audit/documentdb` + }, + serverlessV2ScalingConfiguration: { + minCapacity: props.minCapacity || 0.5, + maxCapacity: props.maxCapacity || 4 + }, + backup: { + retention: cdk.Duration.days(7) + }, + deletionProtection: true, + removalPolicy: cdk.RemovalPolicy.RETAIN_ON_UPDATE_OR_DELETE, + storageEncrypted: true, + securityGroup: databaseSecurityGroup, + vpc: props.vpc, + vpcSubnets: { + subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS + } + }) + + // The scaling configuration defines capacity bounds; a db.serverless instance is still required. + this.dbInstance = new docdb.DatabaseInstance(this, 'DbInstance', { + cluster: this.cluster, + dbInstanceName: `${props.prefix}-${props.stage}-metadata-correction-audit-writer`, + instanceType: new ec2.InstanceType('serverless'), + removalPolicy: cdk.RemovalPolicy.RETAIN_ON_UPDATE_OR_DELETE + }) + + this.secret = this.cluster.secret + + if (!this.secret) { + throw new Error('DocumentDB did not create a master-user secret') + } + + this.environment = { + ...commonEnvironment, + DOCUMENTDB_HOST: this.cluster.clusterEndpoint.hostname, + DOCUMENTDB_PORT: this.cluster.clusterEndpoint.port.toString(), + DOCUMENTDB_SECRET_ARN: this.secret.secretArn, + DOCUMENTDB_TLS_CA_FILE: '/var/task/us-east-1-bundle.pem' + } + } +} + +export default MetadataCorrectionAuditDatabaseSetup diff --git a/cdk/app/lib/helper/MetadataCorrectionSetup.ts b/cdk/app/lib/helper/MetadataCorrectionSetup.ts index 7e400678..0bc464fe 100644 --- a/cdk/app/lib/helper/MetadataCorrectionSetup.ts +++ b/cdk/app/lib/helper/MetadataCorrectionSetup.ts @@ -5,11 +5,16 @@ import * as ec2 from 'aws-cdk-lib/aws-ec2' import * as iam from 'aws-cdk-lib/aws-iam' import * as eventsources from 'aws-cdk-lib/aws-lambda-event-sources' import { NodejsFunction } from 'aws-cdk-lib/aws-lambda-nodejs' +import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager' import * as sns from 'aws-cdk-lib/aws-sns' import * as subscriptions from 'aws-cdk-lib/aws-sns-subscriptions' import * as sqs from 'aws-cdk-lib/aws-sqs' import { Construct } from 'constructs' +import { + getDocumentDbCertificateBundling, + getDocumentDbLambdaSecurityGroups +} from './DocumentDbLambdaConfig' import { NODE_LAMBDA_RUNTIME } from './NodeLambdaRuntime' /** @@ -18,19 +23,21 @@ import { NODE_LAMBDA_RUNTIME } from './NodeLambdaRuntime' interface MetadataCorrectionSetupProps { cmrBaseUrl: string cmrSystemTokenParameterName?: string + cmrWritebackTimeoutMs?: string metadataCorrectionRequestDelayMs?: string + metadataCorrectionRunsPerMinute?: string metadataCorrectionServiceReservedConcurrency?: string cmrWriterToken?: string cmrWritebackProviders?: string cmrWritebackValidateKeywords?: string cmrWritebackValidateUmmC?: string + metadataCorrectionAuditClientSecurityGroup?: ec2.ISecurityGroup + metadataCorrectionAuditEnvironment: Record + metadataCorrectionAuditSecret?: secretsmanager.ISecret prefix: string redisEnabled?: string redisHost?: string redisPort?: string - rdf4jPassword: string - rdf4jServiceUrl: string - rdf4jUserName: string securityGroup: ec2.SecurityGroup stage: string useLocalstack: boolean @@ -43,6 +50,31 @@ interface MetadataCorrectionSetupProps { export class MetadataCorrectionSetup extends Construct { private static readonly DEFAULT_METADATA_CORRECTION_SERVICE_RESERVED_CONCURRENCY = 5 + /** + * Resolves the Lambda concurrency limit, forcing one worker when rate limiting is enabled. + * + * @example + * MetadataCorrectionSetup.resolveReservedConcurrency(true, '5') // 1 + * MetadataCorrectionSetup.resolveReservedConcurrency(false, '3') // 3 + * + * @param hasRateLimit Whether request rate limiting is enabled. + * @param configuredReservedConcurrency Configured Lambda concurrency limit. + * @returns The concurrency limit to apply to the metadata-correction Lambda. + */ + private static resolveReservedConcurrency( + hasRateLimit: boolean, + configuredReservedConcurrency?: string + ): number { + if (hasRateLimit) return 1 + + const parsed = Number(configuredReservedConcurrency) + const isValid = Number.isInteger(parsed) && parsed > 0 + + return isValid + ? parsed + : MetadataCorrectionSetup.DEFAULT_METADATA_CORRECTION_SERVICE_RESERVED_CONCURRENCY + } + public readonly metadataCorrectionRequestsTopic: sns.Topic public readonly metadataCorrectionRequestsQueue: sqs.Queue @@ -72,19 +104,21 @@ export class MetadataCorrectionSetup extends Construct { const { cmrBaseUrl, cmrSystemTokenParameterName, + cmrWritebackTimeoutMs, metadataCorrectionRequestDelayMs, + metadataCorrectionRunsPerMinute, metadataCorrectionServiceReservedConcurrency, cmrWriterToken, cmrWritebackProviders, cmrWritebackValidateKeywords, cmrWritebackValidateUmmC, + metadataCorrectionAuditClientSecurityGroup, + metadataCorrectionAuditEnvironment, + metadataCorrectionAuditSecret, prefix, redisEnabled, redisHost, redisPort, - rdf4jPassword, - rdf4jServiceUrl, - rdf4jUserName, securityGroup, stage, useLocalstack, @@ -94,12 +128,18 @@ export class MetadataCorrectionSetup extends Construct { const metadataCorrectionRequestsBaseName = `${prefix}-${stage}-metadata-correction-requests` const metadataCorrectionRequestsName = `${metadataCorrectionRequestsBaseName}.fifo` const projectRoot = path.join(__dirname, '../../../..') - const parsedReservedConcurrency = Number(metadataCorrectionServiceReservedConcurrency) - const hasValidReservedConcurrency = Number.isInteger(parsedReservedConcurrency) - && parsedReservedConcurrency > 0 - const reservedConcurrency = hasValidReservedConcurrency - ? parsedReservedConcurrency - : MetadataCorrectionSetup.DEFAULT_METADATA_CORRECTION_SERVICE_RESERVED_CONCURRENCY + const configuredRate = String(metadataCorrectionRunsPerMinute || '').trim() + const parsedRate = Number(configuredRate) + const hasRateLimit = configuredRate.length > 0 + + if (hasRateLimit && (!Number.isInteger(parsedRate) || parsedRate <= 0)) { + throw new Error('METADATA_CORRECTION_RUNS_PER_MINUTE must be a positive integer') + } + + const reservedConcurrency = MetadataCorrectionSetup.resolveReservedConcurrency( + hasRateLimit, + metadataCorrectionServiceReservedConcurrency + ) // TODO: Create a follow-up ticket for DLQ handling. This DLQ is only the // redrive target today; before adding a consumer, decide whether failures @@ -144,13 +184,14 @@ export class MetadataCorrectionSetup extends Construct { entry: path.join(projectRoot, 'serverless/src/metadataCorrectionService/handler.js'), handler: 'metadataCorrectionService', runtime: NODE_LAMBDA_RUNTIME, - timeout: cdk.Duration.seconds(30), + timeout: cdk.Duration.seconds(60), memorySize: 1024, // Broad keyword updates can fan out to hundreds of collections; cap concurrent // consumers so writebacks do not overwhelm downstream CMR ingest. reservedConcurrentExecutions: reservedConcurrency, environment: { CMR_BASE_URL: cmrBaseUrl, + ...(cmrWritebackTimeoutMs ? { CMR_WRITEBACK_TIMEOUT_MS: cmrWritebackTimeoutMs } : {}), ...(cmrSystemTokenParameterName ? { CMR_SYSTEM_TOKEN_PARAMETER_NAME: cmrSystemTokenParameterName } : {}), @@ -168,10 +209,12 @@ export class MetadataCorrectionSetup extends Construct { ...(metadataCorrectionRequestDelayMs ? { METADATA_CORRECTION_REQUEST_DELAY_MS: metadataCorrectionRequestDelayMs } : {}), - RDF4J_PASSWORD: rdf4jPassword, - RDF4J_SERVICE_URL: rdf4jServiceUrl, - RDF4J_USER_NAME: rdf4jUserName + ...(hasRateLimit + ? { METADATA_CORRECTION_RUNS_PER_MINUTE: String(parsedRate) } + : {}), + ...metadataCorrectionAuditEnvironment }, + ...getDocumentDbCertificateBundling(metadataCorrectionAuditEnvironment), depsLockFilePath: path.join(projectRoot, 'package-lock.json'), projectRoot, ...(useLocalstack ? {} : { @@ -179,7 +222,11 @@ export class MetadataCorrectionSetup extends Construct { vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS }, - securityGroups: [securityGroup] + securityGroups: getDocumentDbLambdaSecurityGroups({ + clientSecurityGroup: metadataCorrectionAuditClientSecurityGroup, + environment: metadataCorrectionAuditEnvironment, + securityGroup + }) }) } ) @@ -193,6 +240,7 @@ export class MetadataCorrectionSetup extends Construct { )) this.metadataCorrectionRequestsQueue.grantConsumeMessages(this.metadataCorrectionServiceLambda) + metadataCorrectionAuditSecret?.grantRead(this.metadataCorrectionServiceLambda) this.metadataCorrectionServiceLambda.addToRolePolicy(new iam.PolicyStatement({ actions: ['cloudwatch:PutMetricData'], resources: ['*'], diff --git a/cdk/bin/main.ts b/cdk/bin/main.ts index a8c0dbbb..ddf9d408 100644 --- a/cdk/bin/main.ts +++ b/cdk/bin/main.ts @@ -3,6 +3,7 @@ import * as cdk from 'aws-cdk-lib' import { CmrEventProcessingStack } from '../app/lib/CmrEventProcessingStack' import { KmsStack, KmsStackProps } from '../app/lib/KmsStack' +import { MetadataCorrectionAuditStack } from '../app/lib/MetadataCorrectionAuditStack' import { RedisStack } from '../app/lib/RedisStack' import { EbsStack } from '../rdfdb/lib/EbsStack' import { EcsStack } from '../rdfdb/lib/EcsStack' @@ -180,13 +181,37 @@ async function main() { ? String(localRedisEnabled) : String(redisConfigured) + const metadataCorrectionAuditStack = new MetadataCorrectionAuditStack( + app, // CDK application scope + 'MetadataCorrectionAuditStack', // Construct ID + { + env, // Target AWS account and region + localUri: useLocalstack + ? process.env.DOCUMENTDB_URI + || 'mongodb://kms-documentdb-local:27017/?directConnection=true' + : undefined, // Local MongoDB URI; unused in AWS + maxCapacity: Number(process.env.DOCUMENTDB_MAX_CAPACITY || 4), // Maximum serverless capacity + minCapacity: Number(process.env.DOCUMENTDB_MIN_CAPACITY || 0.5), // Minimum serverless capacity + prefix, // Resource naming prefix + stage, // Deployment environment + stackName: `${prefix}-MetadataCorrectionAuditStack`, // CloudFormation stack name + useLocalstack, // Whether to use local service substitutes + vpcId // VPC containing the deployed cluster + } + ) + // Create KmsStack const kmsStackProps: KmsStackProps = { cmrSystemTokenParameterName: process.env.CMR_SYSTEM_TOKEN_PARAMETER_NAME || '', + cmrWritebackTimeoutMs: process.env.CMR_WRITEBACK_TIMEOUT_MS || '', cmrWriterToken: process.env.CMR_WRITER_TOKEN || '', cmrWritebackProviders: process.env.CMR_WRITEBACK_PROVIDERS || '', cmrWritebackValidateKeywords: process.env.CMR_WRITEBACK_VALIDATE_KEYWORDS || '', cmrWritebackValidateUmmC: process.env.CMR_WRITEBACK_VALIDATE_UMM_C || '', + metadataCorrectionAuditClientSecurityGroup: + metadataCorrectionAuditStack.clientSecurityGroup, + metadataCorrectionAuditEnvironment: metadataCorrectionAuditStack.connectionEnvironment, + metadataCorrectionAuditSecret: metadataCorrectionAuditStack.secret, prefix, env, vpcId, @@ -229,26 +254,30 @@ async function main() { kmsStack.addDependency(redisStack) } + kmsStack.addDependency(metadataCorrectionAuditStack) + const cmrEventProcessingStack = new CmrEventProcessingStack(app, 'CmrEventProcessingStack', { cmrBaseUrl, cmrSystemTokenParameterName: process.env.CMR_SYSTEM_TOKEN_PARAMETER_NAME || '', + cmrWritebackTimeoutMs: process.env.CMR_WRITEBACK_TIMEOUT_MS || '', metadataCorrectionRequestDelayMs: process.env.METADATA_CORRECTION_REQUEST_DELAY_MS || '', + metadataCorrectionRunsPerMinute: + process.env.METADATA_CORRECTION_RUNS_PER_MINUTE || '', metadataCorrectionServiceReservedConcurrency: process.env.METADATA_CORRECTION_SERVICE_RESERVED_CONCURRENCY || '', cmrWriterToken: process.env.CMR_WRITER_TOKEN || '', cmrWritebackProviders: process.env.CMR_WRITEBACK_PROVIDERS || '', cmrWritebackValidateKeywords: process.env.CMR_WRITEBACK_VALIDATE_KEYWORDS || '', cmrWritebackValidateUmmC: process.env.CMR_WRITEBACK_VALIDATE_UMM_C || '', + metadataCorrectionAuditClientSecurityGroup: + metadataCorrectionAuditStack.clientSecurityGroup, + metadataCorrectionAuditEnvironment: metadataCorrectionAuditStack.connectionEnvironment, + metadataCorrectionAuditSecret: metadataCorrectionAuditStack.secret, env, prefix, redisEnabled: redisEnabledValue, redisHost: useLocalstack ? localRedisHost : redisStack?.endpointAddress, redisPort: useLocalstack ? localRedisPort : redisStack?.endpointPort, - rdf4jPassword: process.env.RDF4J_PASSWORD || 'rdf4j', - rdf4jServiceUrl: useLocalstack - ? 'http://rdf4j-server:8080' - : (lbStack?.rdf4jServiceUrl || process.env.RDF4J_SERVICE_URL || 'http://localhost:8081'), - rdf4jUserName: process.env.RDF4J_USER_NAME || 'rdf4j', stage, stackName: `${prefix}-CmrEventProcessingStack`, topicArn: kmsStack.keywordEventsTopic.topicArn, @@ -257,6 +286,7 @@ async function main() { }) cmrEventProcessingStack.addDependency(kmsStack) + cmrEventProcessingStack.addDependency(metadataCorrectionAuditStack) app.synth() } diff --git a/config/metadataCorrectionAuditIndexes.json b/config/metadataCorrectionAuditIndexes.json new file mode 100644 index 00000000..13546c74 --- /dev/null +++ b/config/metadataCorrectionAuditIndexes.json @@ -0,0 +1,73 @@ +[ + { + "key": { + "createdAt": -1, + "_id": -1 + }, + "name": "createdAt_desc" + }, + { + "key": { + "collectionConceptId": 1, + "createdAt": -1, + "_id": -1 + }, + "name": "collection_createdAt_desc" + }, + { + "key": { + "status": 1, + "createdAt": -1, + "_id": -1 + }, + "name": "status_createdAt_desc" + }, + { + "key": { + "trigger.eventType": 1, + "createdAt": -1, + "_id": -1 + }, + "name": "triggerAction_createdAt_desc" + }, + { + "key": { + "publishedVersionName": 1, + "createdAt": -1, + "_id": -1 + }, + "name": "version_createdAt_desc" + }, + { + "key": { + "trigger.scheme": 1, + "createdAt": -1, + "_id": -1 + }, + "name": "triggerScheme_createdAt_desc" + }, + { + "key": { + "corrections.scheme": 1, + "createdAt": -1, + "_id": -1 + }, + "name": "correctionScheme_createdAt_desc" + }, + { + "key": { + "corrections.keywordConceptUuid": 1, + "createdAt": -1, + "_id": -1 + }, + "name": "keywordUuid_createdAt_desc" + }, + { + "key": { + "trigger.keywordConceptUuid": 1, + "createdAt": -1, + "_id": -1 + }, + "name": "triggerKeywordUuid_createdAt_desc" + } +] diff --git a/package-lock.json b/package-lock.json index fc1e860f..a1c8fbfb 100644 --- a/package-lock.json +++ b/package-lock.json @@ -12,20 +12,24 @@ "@aws-sdk/client-eventbridge": "^3.997.0", "@aws-sdk/client-lambda": "^3.775.0", "@aws-sdk/client-s3": "^3.540.0", + "@aws-sdk/client-secrets-manager": "^3.1124.0", "@aws-sdk/client-sfn": "^3.775.0", "@aws-sdk/client-sns": "^3.997.0", "@aws-sdk/client-sqs": "^3.997.0", "@aws-sdk/client-ssm": "^3.1096.0", "@aws-sdk/s3-request-presigner": "3.981.0", - "@xmldom/xmldom": "^0.8.10", + "@xmldom/xmldom": "^0.9.12", "compact-object-deep": "^1.0.0", "csv": "^6.3.11", "date-fns": "^4.1.0", + "diff": "^5.2.2", + "diff2html": "^3.4.56", "fast-xml-parser": "^5.3.4", "html-entities": "^2.5.2", "html-escaper": "^3.0.3", "install": "^0.13.0", "lodash": "^4.17.21", + "mongodb": "^6.21.0", "node-fetch": "^2.7.0", "redis": "^4.7.1", "remove": "^0.1.5", @@ -644,6 +648,25 @@ "node": ">=18.0.0" } }, + "node_modules/@aws-sdk/client-secrets-manager": { + "version": "3.1124.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/client-secrets-manager/-/client-secrets-manager-3.1124.0.tgz", + "integrity": "sha512-+RE3Gu0JRcO4mMPSezG9VyQEyORXtFTGGUkVt/12Rcmv0+CzsDYfHbj4BRk8LJQE8yYGZMALR3P6OeCj8VvlgA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/credential-provider-node": "^3.972.82", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/fetch-http-handler": "^5.7.2", + "@smithy/node-http-handler": "^4.11.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, "node_modules/@aws-sdk/client-sfn": { "version": "3.981.0", "resolved": "https://registry.npmjs.org/@aws-sdk/client-sfn/-/client-sfn-3.981.0.tgz", @@ -847,17 +870,17 @@ } }, "node_modules/@aws-sdk/core": { - "version": "3.977.7", - "resolved": "https://registry.npmjs.org/@aws-sdk/core/-/core-3.977.7.tgz", - "integrity": "sha512-I88Iov89NVmjSmJLKSv7Cn9M2J+a2942OkA8nZCbz+sl4ZeY4zEOcoLOrbt1GRfQ8zEQKnjAJdXixA3J/p1fDQ==", + "version": "3.977.9", + "resolved": "https://registry.npmjs.org/@aws-sdk/core/-/core-3.977.9.tgz", + "integrity": "sha512-reqPFEQrZxDZpeGj4PFMepBeR5LGYHRqq/L0motTzgFkCRBA4rFdaVXDSLYyGHhxVz7sT2PDnPN9CluGSfgyJA==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/types": "^3.974.3", - "@aws-sdk/xml-builder": "^3.972.38", + "@aws-sdk/types": "^3.974.5", + "@aws-sdk/xml-builder": "^3.972.40", "@aws/lambda-invoke-store": "^0.3.0", - "@smithy/core": "^3.31.1", + "@smithy/core": "^3.33.3", "@smithy/signature-v4": "^5.6.12", - "@smithy/types": "^4.16.1", + "@smithy/types": "^4.17.2", "bowser": "^2.11.0", "tslib": "^2.6.2" }, @@ -888,15 +911,15 @@ } }, "node_modules/@aws-sdk/credential-provider-env": { - "version": "3.972.62", - "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-env/-/credential-provider-env-3.972.62.tgz", - "integrity": "sha512-BkDrk2cNjed31IKin/Oksb2ziF+gfuyRskFVuT4EU9Mep7M8Y/d8DJG4+anHme4Vuse7CwaEscwEfGyR6mzBhQ==", + "version": "3.972.70", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-env/-/credential-provider-env-3.972.70.tgz", + "integrity": "sha512-H404B7dJl2mCrBqahDEYsanB0xhdDp6tXnXcTUnXmmpy2Q3J0Ho0bUajZ2jr/RdwzCyS59Gi8xXIFwPLGBl6Uw==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/core": "^3.977.1", - "@aws-sdk/types": "^3.974.2", - "@smithy/core": "^3.29.8", - "@smithy/types": "^4.16.1", + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -904,17 +927,17 @@ } }, "node_modules/@aws-sdk/credential-provider-http": { - "version": "3.972.64", - "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-http/-/credential-provider-http-3.972.64.tgz", - "integrity": "sha512-Wj1FGK2IxY5EccQCvH+niTYhIvDoDujJf2CpRRgS3NpYNEgiFNVItNbJYQjINRlu7fG7jSsXkKV0UWKriEplrw==", + "version": "3.972.72", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-http/-/credential-provider-http-3.972.72.tgz", + "integrity": "sha512-X98zYOrVOeuosCX+6ktf29FC2N2GHPLia7qv6mzPzTc+RPAuHWCDS++Z6JK7eGYqb/v6uaW7bAXaOvDBfol+0w==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/core": "^3.977.1", - "@aws-sdk/types": "^3.974.2", - "@smithy/core": "^3.29.8", - "@smithy/fetch-http-handler": "^5.6.10", - "@smithy/node-http-handler": "^4.9.10", - "@smithy/types": "^4.16.1", + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/fetch-http-handler": "^5.7.2", + "@smithy/node-http-handler": "^4.11.3", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -922,23 +945,23 @@ } }, "node_modules/@aws-sdk/credential-provider-ini": { - "version": "3.973.7", - "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-ini/-/credential-provider-ini-3.973.7.tgz", - "integrity": "sha512-2CefB8cCxDu52P24B8Ay93/cTT199bcSvNHQ8e2f4BjSCF83yErBnTIZEBo0VeIgCfmw+PJKFUXnlQWxm2dkug==", + "version": "3.973.15", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-ini/-/credential-provider-ini-3.973.15.tgz", + "integrity": "sha512-Rykg6s5ceBuynMOGWgoowO4N+27JfnqXAnVaSunZl0hOO1XodSrxGNz6sCEbnmS0lAfQZDKyb3fbr46gSuv6Sg==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/core": "^3.977.1", - "@aws-sdk/credential-provider-env": "^3.972.62", - "@aws-sdk/credential-provider-http": "^3.972.64", - "@aws-sdk/credential-provider-login": "^3.972.69", - "@aws-sdk/credential-provider-process": "^3.972.62", - "@aws-sdk/credential-provider-sso": "^3.973.6", - "@aws-sdk/credential-provider-web-identity": "^3.972.68", - "@aws-sdk/nested-clients": "^3.997.36", - "@aws-sdk/types": "^3.974.2", - "@smithy/core": "^3.29.8", - "@smithy/credential-provider-imds": "^4.4.13", - "@smithy/types": "^4.16.1", + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/credential-provider-env": "^3.972.70", + "@aws-sdk/credential-provider-http": "^3.972.72", + "@aws-sdk/credential-provider-login": "^3.972.77", + "@aws-sdk/credential-provider-process": "^3.972.70", + "@aws-sdk/credential-provider-sso": "^3.973.14", + "@aws-sdk/credential-provider-web-identity": "^3.972.76", + "@aws-sdk/nested-clients": "^3.997.44", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/credential-provider-imds": "^4.4.16", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -946,16 +969,16 @@ } }, "node_modules/@aws-sdk/credential-provider-login": { - "version": "3.972.69", - "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-login/-/credential-provider-login-3.972.69.tgz", - "integrity": "sha512-gM3j0Ie9+FoLNTYODY+QWbg3vCRBc7mR9cRdntxTMkFYIrwfRmuucfavP6HNBlYSuaYww54TNJGej4GFgoPZAg==", + "version": "3.972.77", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-login/-/credential-provider-login-3.972.77.tgz", + "integrity": "sha512-Jb59xfEISoN5mmbnA+HYqdtrSX3CgCtJoof+V5D8/TgUI56W63GEEd5Y58WijU3Ou6+WEgaLD1feVzaRXV5IDQ==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/core": "^3.977.1", - "@aws-sdk/nested-clients": "^3.997.36", - "@aws-sdk/types": "^3.974.2", - "@smithy/core": "^3.29.8", - "@smithy/types": "^4.16.1", + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/nested-clients": "^3.997.44", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -963,21 +986,21 @@ } }, "node_modules/@aws-sdk/credential-provider-node": { - "version": "3.972.73", - "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-node/-/credential-provider-node-3.972.73.tgz", - "integrity": "sha512-VTzdbf8Ukjdb9yUubZzRI678CWZvKovhE8Nv3qihwhC187sRMGls+r9N8Wuht5q1xjKx2nmpS48ar8ppupjkCA==", + "version": "3.972.82", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-node/-/credential-provider-node-3.972.82.tgz", + "integrity": "sha512-znDkEOGXB8W3kG1LJUKP3foBZY/9qLM0eil/DxWXSp37XsdsRLQHE/d/OaCGGVgKpA6znR38h/+INk8do1FjiA==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/credential-provider-env": "^3.972.62", - "@aws-sdk/credential-provider-http": "^3.972.64", - "@aws-sdk/credential-provider-ini": "^3.973.7", - "@aws-sdk/credential-provider-process": "^3.972.62", - "@aws-sdk/credential-provider-sso": "^3.973.6", - "@aws-sdk/credential-provider-web-identity": "^3.972.68", - "@aws-sdk/types": "^3.974.2", - "@smithy/core": "^3.29.8", - "@smithy/credential-provider-imds": "^4.4.13", - "@smithy/types": "^4.16.1", + "@aws-sdk/credential-provider-env": "^3.972.70", + "@aws-sdk/credential-provider-http": "^3.972.72", + "@aws-sdk/credential-provider-ini": "^3.973.15", + "@aws-sdk/credential-provider-process": "^3.972.70", + "@aws-sdk/credential-provider-sso": "^3.973.14", + "@aws-sdk/credential-provider-web-identity": "^3.972.76", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/credential-provider-imds": "^4.4.16", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -985,15 +1008,15 @@ } }, "node_modules/@aws-sdk/credential-provider-process": { - "version": "3.972.62", - "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-process/-/credential-provider-process-3.972.62.tgz", - "integrity": "sha512-zXYU9UWNL66gtMgNLhmxlrvEokuI7r6G2q7FRGu41Bya4iS30JLelUipJX9SV4zhyCPWJhI9Li54R1d9H8Tq6A==", + "version": "3.972.70", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-process/-/credential-provider-process-3.972.70.tgz", + "integrity": "sha512-2ry03fGRJr4sV3jI+ocjj5JqALnFD6ymM5KiNCDZMvq8bX2GSbE0vji4aM43TVCl2nXqqLRZaUxdq/KeWRAY4Q==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/core": "^3.977.1", - "@aws-sdk/types": "^3.974.2", - "@smithy/core": "^3.29.8", - "@smithy/types": "^4.16.1", + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -1001,17 +1024,17 @@ } }, "node_modules/@aws-sdk/credential-provider-sso": { - "version": "3.973.6", - "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-sso/-/credential-provider-sso-3.973.6.tgz", - "integrity": "sha512-DobZggy3K49xdCpjeyMou0FQhkoYbluVGNydL6D+lcxF8GoAsttFX0xnH5GmiQ89We5dB6TRpW+CD/VowBH6HQ==", + "version": "3.973.14", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-sso/-/credential-provider-sso-3.973.14.tgz", + "integrity": "sha512-jkhg/8ocAAoc0RFyLMhCw+/zZh7gystQgd4F4hznNa8P4Cc501PQmxd+jGLiMHodPJ+7Zv/3znM62gZojyasmA==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/core": "^3.977.1", - "@aws-sdk/nested-clients": "^3.997.36", - "@aws-sdk/token-providers": "3.1096.0", - "@aws-sdk/types": "^3.974.2", - "@smithy/core": "^3.29.8", - "@smithy/types": "^4.16.1", + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/nested-clients": "^3.997.44", + "@aws-sdk/token-providers": "3.1116.0", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -1019,16 +1042,16 @@ } }, "node_modules/@aws-sdk/credential-provider-web-identity": { - "version": "3.972.68", - "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-web-identity/-/credential-provider-web-identity-3.972.68.tgz", - "integrity": "sha512-bq+yTt+uWJx60VVp/OIAX5xqUAu/K2Uc3eknWnWl+KtfcU2CQe0uNw6lySrn2t5GKHq7jsV0Z63HiBGVtzr/lg==", + "version": "3.972.76", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-web-identity/-/credential-provider-web-identity-3.972.76.tgz", + "integrity": "sha512-d3AGyVu759PGr35mEB2s22xxlNEA5rpdxtSPJthfPFJvoQ8dt357iVPECqWfUxXp1toJAvKmbtcIYVGigaGsCA==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/core": "^3.977.1", - "@aws-sdk/nested-clients": "^3.997.36", - "@aws-sdk/types": "^3.974.2", - "@smithy/core": "^3.29.8", - "@smithy/types": "^4.16.1", + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/nested-clients": "^3.997.44", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -1278,18 +1301,18 @@ } }, "node_modules/@aws-sdk/nested-clients": { - "version": "3.997.36", - "resolved": "https://registry.npmjs.org/@aws-sdk/nested-clients/-/nested-clients-3.997.36.tgz", - "integrity": "sha512-b71Suv7L+DnhM0MsQHU4WO42I32kxLZi96PbVhZbxMYIoKnEZz3v+LSrG8fupAoA4cBSshCk1Dl/PeRz49qUSg==", + "version": "3.997.44", + "resolved": "https://registry.npmjs.org/@aws-sdk/nested-clients/-/nested-clients-3.997.44.tgz", + "integrity": "sha512-NhEgryjlBF9w38ZXqGymQV28IhkYa1mKhlbYnqIis57AYwWGVYfUPgg/qC2rLRqOUfblxx++irvju10kVTa8Vw==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/core": "^3.977.1", - "@aws-sdk/signature-v4-multi-region": "^3.996.42", - "@aws-sdk/types": "^3.974.2", - "@smithy/core": "^3.29.8", - "@smithy/fetch-http-handler": "^5.6.10", - "@smithy/node-http-handler": "^4.9.10", - "@smithy/types": "^4.16.1", + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/signature-v4-multi-region": "^3.996.46", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/fetch-http-handler": "^5.7.2", + "@smithy/node-http-handler": "^4.11.3", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -1297,14 +1320,14 @@ } }, "node_modules/@aws-sdk/nested-clients/node_modules/@aws-sdk/signature-v4-multi-region": { - "version": "3.996.42", - "resolved": "https://registry.npmjs.org/@aws-sdk/signature-v4-multi-region/-/signature-v4-multi-region-3.996.42.tgz", - "integrity": "sha512-DBV4naZP6HYBlAvPpoQzOP12Wvfou/5rN8yJPXjBTBylU5qwCbh/tXr2MddHoIjgoRkEl/eS+IljiUqvmwey1Q==", + "version": "3.996.46", + "resolved": "https://registry.npmjs.org/@aws-sdk/signature-v4-multi-region/-/signature-v4-multi-region-3.996.46.tgz", + "integrity": "sha512-L+2xZTye/2T96f3lwCws0Zw6GG2JHZW9e8FpVgGBeeExSKyeoZ6CWRpBml/7DNiK/O26jrgPM9F+Ay8VkgzUWQ==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/types": "^3.974.2", - "@smithy/signature-v4": "^5.6.9", - "@smithy/types": "^4.16.1", + "@aws-sdk/types": "^3.974.5", + "@smithy/signature-v4": "^5.6.12", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -1363,16 +1386,16 @@ } }, "node_modules/@aws-sdk/token-providers": { - "version": "3.1096.0", - "resolved": "https://registry.npmjs.org/@aws-sdk/token-providers/-/token-providers-3.1096.0.tgz", - "integrity": "sha512-hdUS2hDppy3vkWeFl5y86RLNU6OWH2mQB09yOSsRefwhhGTSFPkaZvfLDD/9vFcvMzlr8QFQFw3fw2FtrurVQA==", + "version": "3.1116.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/token-providers/-/token-providers-3.1116.0.tgz", + "integrity": "sha512-ygIivKqh8aHzNkucOCXHyIBgBpLPfrSI0mCqXF+vLBsPTUKqj0VSqAY0GFPe7lQl4HntjOcQ+KSyS7oUV2C54Q==", "license": "Apache-2.0", "dependencies": { - "@aws-sdk/core": "^3.977.1", - "@aws-sdk/nested-clients": "^3.997.36", - "@aws-sdk/types": "^3.974.2", - "@smithy/core": "^3.29.8", - "@smithy/types": "^4.16.1", + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/nested-clients": "^3.997.44", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -1380,12 +1403,12 @@ } }, "node_modules/@aws-sdk/types": { - "version": "3.974.3", - "resolved": "https://registry.npmjs.org/@aws-sdk/types/-/types-3.974.3.tgz", - "integrity": "sha512-ECAqfpNsef+7MO8qtR0h9KcFIBAygaE7Cm6UOiQl+ft+uVap+1G7bNEjs4mdJE2OnA4m6k7i8peH8uGIAsOMGw==", + "version": "3.974.5", + "resolved": "https://registry.npmjs.org/@aws-sdk/types/-/types-3.974.5.tgz", + "integrity": "sha512-LkwLL2BLbC6wNNm4JaH9mbEqBMdOZCct6VAYqhdN4U1xrWM+fUJQEfbHwQgDypapOWTRtlk25akb5afM0P8CIQ==", "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.16.1", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -1482,12 +1505,12 @@ } }, "node_modules/@aws-sdk/xml-builder": { - "version": "3.972.38", - "resolved": "https://registry.npmjs.org/@aws-sdk/xml-builder/-/xml-builder-3.972.38.tgz", - "integrity": "sha512-grf7mzfVxBS5AlsuTvBN7uDpzqohFww9fRPCO+EBSUdvtsYMcPSKdz54h/7XiscqNcUM1Ae1MF7JLHmiYYuzbQ==", + "version": "3.972.40", + "resolved": "https://registry.npmjs.org/@aws-sdk/xml-builder/-/xml-builder-3.972.40.tgz", + "integrity": "sha512-wlFmCIGUlwF4zx/kncw+bmxTQh1HeSJq4mYV/V5cZUSJadDP3kXvGW8Rn21cimj/7y9ju+47oYWXi97vF7czaA==", "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.16.1", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -3960,6 +3983,15 @@ "@jridgewell/sourcemap-codec": "^1.4.14" } }, + "node_modules/@mongodb-js/saslprep": { + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/@mongodb-js/saslprep/-/saslprep-1.5.0.tgz", + "integrity": "sha512-Hk1SKJCMcCos38+vqDnZzlIo4XRj9yCGzYkjB4LcqpeXRIYfia1UWTz+VrueLxoU+uSRJzgkufxoRZg8gi52YA==", + "license": "MIT", + "dependencies": { + "sparse-bitfield": "^3.0.3" + } + }, "node_modules/@napi-rs/wasm-runtime": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.2.1.tgz", @@ -4050,6 +4082,18 @@ "url": "https://github.com/sponsors/Boshen" } }, + "node_modules/@profoundlogic/hogan": { + "version": "3.0.4", + "resolved": "https://registry.npmjs.org/@profoundlogic/hogan/-/hogan-3.0.4.tgz", + "integrity": "sha512-pmNVGuooS30Mm7YbZd5T7E5zYVO6D5Ct91sn4T39mUvMUc3sCGridcnhAufL1/Bz2QzAtzEn0agNrdk3+5yWzw==", + "license": "Apache-2.0", + "dependencies": { + "nopt": "1.0.10" + }, + "bin": { + "hulk": "bin/hulk" + } + }, "node_modules/@redis/bloom": { "version": "1.2.0", "resolved": "https://registry.npmjs.org/@redis/bloom/-/bloom-1.2.0.tgz", @@ -4546,12 +4590,12 @@ } }, "node_modules/@smithy/core": { - "version": "3.32.0", - "resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.32.0.tgz", - "integrity": "sha512-NAiCSC78fzbNIEWoheoF74Ob5ZorLijCHpMY26Fqvqg/+9LuyIqMfHDg2p8Yk1rqOyowtiL3y7WX0AW+teL6zw==", + "version": "3.33.3", + "resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.33.3.tgz", + "integrity": "sha512-CsOeKq/9kA3y6VJHt+/+VTCtBaxJ4OTFpgrjIUhPpDIKxBci1k2bJaQASF2h/ELWrulGp+t97DZ0mevfAD8idg==", "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.17.0", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -4559,13 +4603,13 @@ } }, "node_modules/@smithy/credential-provider-imds": { - "version": "4.4.15", - "resolved": "https://registry.npmjs.org/@smithy/credential-provider-imds/-/credential-provider-imds-4.4.15.tgz", - "integrity": "sha512-xYVGrisQqTJWhOnScUhbx8s9H63TMtoxzuUoxG6mP8J+B/YbX3vZxVsgV0xDf43abJnJP0fjP7BkQh7OESwuRA==", + "version": "4.5.2", + "resolved": "https://registry.npmjs.org/@smithy/credential-provider-imds/-/credential-provider-imds-4.5.2.tgz", + "integrity": "sha512-A9uSdn72ozbRUSit0eib0TW7nXuNPlaeM0zcGkJ+nE6tFcSDbnmtwoxbTCFBukVQcszDAyvsd7+rTduPTXpygg==", "license": "Apache-2.0", "dependencies": { - "@smithy/core": "^3.31.0", - "@smithy/types": "^4.16.1", + "@smithy/core": "^3.33.2", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -4638,13 +4682,13 @@ } }, "node_modules/@smithy/fetch-http-handler": { - "version": "5.6.12", - "resolved": "https://registry.npmjs.org/@smithy/fetch-http-handler/-/fetch-http-handler-5.6.12.tgz", - "integrity": "sha512-OpQgP6IGH4j0NJ2zjfYZLjQL85ai+Wi/q51EmZJovXsEwKSvu89qiXUq77Q6EmwZ/hSl7fKpn2Z9mhiDN6OM+Q==", + "version": "5.7.2", + "resolved": "https://registry.npmjs.org/@smithy/fetch-http-handler/-/fetch-http-handler-5.7.2.tgz", + "integrity": "sha512-nZyWTmSpJEXl6VtWVMBJve/7x12DZu6sIX1z1a+ZMaHlQQRs9Zpu6NbTe/gmxYXVRpkjxyDYpZ5gx2IM6f/Wkw==", "license": "Apache-2.0", "dependencies": { - "@smithy/core": "^3.31.0", - "@smithy/types": "^4.16.1", + "@smithy/core": "^3.33.2", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -4733,20 +4777,14 @@ } }, "node_modules/@smithy/middleware-compression": { - "version": "4.3.46", - "resolved": "https://registry.npmjs.org/@smithy/middleware-compression/-/middleware-compression-4.3.46.tgz", - "integrity": "sha512-9f4AZ5dKqKRmO49MPhOoxFoQBLfBgxE9YKG8bQ6lsW9xk+Bn8rkfGlpW8OYlvhuarN+8mja9PjhEudFiR8wGFQ==", + "version": "4.6.2", + "resolved": "https://registry.npmjs.org/@smithy/middleware-compression/-/middleware-compression-4.6.2.tgz", + "integrity": "sha512-Q9d+luiRjyHT6kCL/9NyGpdZJgodh4vtvfHC6H8SqoVKrV2k9RoyI9/IloVfdCYks3/2DIi7BsYYLcda5KZS0A==", "license": "Apache-2.0", "dependencies": { - "@smithy/core": "^3.23.17", - "@smithy/is-array-buffer": "^4.2.2", - "@smithy/node-config-provider": "^4.3.14", - "@smithy/protocol-http": "^5.3.14", - "@smithy/types": "^4.14.1", - "@smithy/util-config-provider": "^4.2.2", - "@smithy/util-middleware": "^4.2.14", - "@smithy/util-utf8": "^4.2.2", - "fflate": "0.8.1", + "@smithy/core": "^3.33.2", + "@smithy/types": "^4.17.2", + "fflate": "0.8.3", "tslib": "^2.6.2" }, "engines": { @@ -4851,13 +4889,13 @@ } }, "node_modules/@smithy/node-http-handler": { - "version": "4.9.12", - "resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.9.12.tgz", - "integrity": "sha512-dWW5KRt4mnEvjNzbGqGeCuAvgum85Y9ZoyuMQqcTEfapndyVJ1k9BEHK7kdXJZ32enyRmmwcFjMwlB/KgLKI3Q==", + "version": "4.12.0", + "resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.12.0.tgz", + "integrity": "sha512-0mq1pHadfyXCYCqm2cNpbjNIT+fbaUpNxewZb/YNr2L0IrEVMOb8gM/Fl4K6XvHCW3uSNDFwPl/+iKm0bx9jYg==", "license": "Apache-2.0", "dependencies": { - "@smithy/core": "^3.31.0", - "@smithy/types": "^4.16.1", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", "tslib": "^2.6.2" }, "engines": { @@ -4980,9 +5018,9 @@ } }, "node_modules/@smithy/types": { - "version": "4.17.0", - "resolved": "https://registry.npmjs.org/@smithy/types/-/types-4.17.0.tgz", - "integrity": "sha512-Aw4joiM0ZdErpo39lCj8phT2lxoiKZV+KZzBxnnQhWVtU2Is/WffQSL04uUWRcXUse9Ln8vXZK6V/FwqRVnQpg==", + "version": "4.17.2", + "resolved": "https://registry.npmjs.org/@smithy/types/-/types-4.17.2.tgz", + "integrity": "sha512-FOKpVZob9MPTn2znRzGrnsMHv7BOsKVw3XiP/cOyYLDVZ9qKp4nifIiSCuUU/fIj5Vu0UOAxCFr+qRAtG0NUkA==", "license": "Apache-2.0", "dependencies": { "tslib": "^2.6.2" @@ -5705,6 +5743,21 @@ "dev": true, "license": "MIT" }, + "node_modules/@types/webidl-conversions": { + "version": "7.0.3", + "resolved": "https://registry.npmjs.org/@types/webidl-conversions/-/webidl-conversions-7.0.3.tgz", + "integrity": "sha512-CiJJvcRtIgzadHCYXw7dqEnMNRjhGZlYK05Mj9OyktqV8uVT8fD2BFOB7S1uwBE3Kj2Z+4UyPmFw/Ixgw/LAlA==", + "license": "MIT" + }, + "node_modules/@types/whatwg-url": { + "version": "11.0.5", + "resolved": "https://registry.npmjs.org/@types/whatwg-url/-/whatwg-url-11.0.5.tgz", + "integrity": "sha512-coYR071JRaHa+xoEvvYqvnIHaVqaYrLPbsufM9BF63HkwI5Lgmy2QR8Q5K/lYDYo5AK82wOvSOS0UsLTpTG7uQ==", + "license": "MIT", + "dependencies": { + "@types/webidl-conversions": "*" + } + }, "node_modules/@typescript-eslint/eslint-plugin": { "version": "8.58.0", "resolved": "https://registry.npmjs.org/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.58.0.tgz", @@ -6205,12 +6258,12 @@ } }, "node_modules/@xmldom/xmldom": { - "version": "0.8.13", - "resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.13.tgz", - "integrity": "sha512-KRYzxepc14G/CEpEGc3Yn+JKaAeT63smlDr+vjB8jRfgTBBI9wRj/nkQEO+ucV8p8I9bfKLWp37uHgFrbntPvw==", + "version": "0.9.12", + "resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.9.12.tgz", + "integrity": "sha512-5AXjrcMClTryPe9LgZrygpB1lj7s0S9E0+W+AHaVKAVyHanafK86iPSvG5xHVSp/jC+VH1UXu0TAEmY279xH7A==", "license": "MIT", "engines": { - "node": ">=10.0.0" + "node": ">=14.6" } }, "node_modules/abab": { @@ -6221,6 +6274,12 @@ "dev": true, "license": "BSD-3-Clause" }, + "node_modules/abbrev": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/abbrev/-/abbrev-1.1.1.tgz", + "integrity": "sha512-nne9/IiQ/hzIhY6pdDnbBtz7DjPTKrY00P/zvPSm5pOFkl6xuGrGnXn/VtTNNfNtAfZ9/1RtehkszU9qcTii0Q==", + "license": "ISC" + }, "node_modules/acorn": { "version": "8.15.0", "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz", @@ -7125,6 +7184,19 @@ "dev": true, "license": "MIT" }, + "node_modules/baseline-browser-mapping": { + "version": "2.11.21", + "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.21.tgz", + "integrity": "sha512-uh8vpY/1/YyFkunIDFH/12p7/7VdPKA1hejMVEbdkEaWnUz0Hesvx5EbiU6XxjyHZIOju+ZMbQJkRh+es3/spQ==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "baseline-browser-mapping": "dist/cli.cjs" + }, + "engines": { + "node": ">=6.0.0" + } + }, "node_modules/bluebird": { "version": "3.7.2", "resolved": "https://registry.npmjs.org/bluebird/-/bluebird-3.7.2.tgz", @@ -7174,9 +7246,9 @@ } }, "node_modules/browserslist": { - "version": "4.25.3", - "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.25.3.tgz", - "integrity": "sha512-cDGv1kkDI4/0e5yON9yM5G/0A5u8sf5TnmdX5C9qHzI9PPu++sQ9zjm1k9NiOrf3riY4OkK0zSGqfvJyJsgCBQ==", + "version": "4.28.9", + "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.9.tgz", + "integrity": "sha512-EWazOblFYUvlGZcfGhPUPmYh3nikUxBVb+y9MJun5f3hBi812X+8MSQTujLBtgK3cf51fJWbWfOjyeO954d+Eg==", "dev": true, "funding": [ { @@ -7194,10 +7266,11 @@ ], "license": "MIT", "dependencies": { - "caniuse-lite": "^1.0.30001735", - "electron-to-chromium": "^1.5.204", - "node-releases": "^2.0.19", - "update-browserslist-db": "^1.1.3" + "baseline-browser-mapping": "^2.11.20", + "caniuse-lite": "^1.0.30001810", + "electron-to-chromium": "^1.5.420", + "node-releases": "^2.0.54", + "update-browserslist-db": "^1.3.2" }, "bin": { "browserslist": "cli.js" @@ -7206,6 +7279,15 @@ "node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7" } }, + "node_modules/bson": { + "version": "6.10.4", + "resolved": "https://registry.npmjs.org/bson/-/bson-6.10.4.tgz", + "integrity": "sha512-WIsKqkSC0ABoBJuT1LEX+2HEvNmNKKgnTAyd0fL8qzK4SH2i9NXg+t08YtdZp/V9IZ33cxe3iV4yM0qg8lMQng==", + "license": "Apache-2.0", + "engines": { + "node": ">=16.20.1" + } + }, "node_modules/cac": { "version": "7.0.0", "resolved": "https://registry.npmjs.org/cac/-/cac-7.0.0.tgz", @@ -7322,9 +7404,9 @@ } }, "node_modules/caniuse-lite": { - "version": "1.0.30001736", - "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001736.tgz", - "integrity": "sha512-ImpN5gLEY8gWeqfLUyEF4b7mYWcYoR2Si1VhnrbM4JizRFmfGaAQ12PhNykq6nvI4XvKLrsp8Xde74D5phJOSw==", + "version": "1.0.30001810", + "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001810.tgz", + "integrity": "sha512-TITQPUkaz+aVk5GL6NhOdwk1aEaNTSDPsGFWrTuhKGtjTF70jL/Oht2W4c6rXUe5fu7Ie19VIahAXHIIiWWNeg==", "dev": true, "funding": [ { @@ -7421,9 +7503,9 @@ "license": "MIT" }, "node_modules/colord": { - "version": "2.9.3", - "resolved": "https://registry.npmjs.org/colord/-/colord-2.9.3.tgz", - "integrity": "sha512-jeC1axXpnb0/2nn/Y1LPuLdgXBLH7aDcHu4KEKfqw3CUhX7ZpfBSlPKyqXE6btIgEzfWtrX3/tyBCaCvXvMkOw==", + "version": "2.10.0", + "resolved": "https://registry.npmjs.org/colord/-/colord-2.10.0.tgz", + "integrity": "sha512-AidJptpBJmjTclAp9BkLwJi0T93fo5epJnbaZslpg6QVzpHjAiveF55mE9AcUJiGMqRHgMDY8soMsQtuNYMHfw==", "dev": true, "license": "MIT" }, @@ -7618,36 +7700,36 @@ "peer": true }, "node_modules/csv": { - "version": "6.4.1", - "resolved": "https://registry.npmjs.org/csv/-/csv-6.4.1.tgz", - "integrity": "sha512-ajGosmTGnTwYyGl8STqZDu7R6LkDf3xL39XiOmliV/GufQeVUxHzTKIm4NOBCwmEuujK7B6isxs4Uqt9GcRCvA==", + "version": "6.6.3", + "resolved": "https://registry.npmjs.org/csv/-/csv-6.6.3.tgz", + "integrity": "sha512-X2AnOgcxqV+OdLm1M2FOl+bPQrM1LK1jwo4FhEoq8hi8JTR0tQ8ZWv3x3N2gVMIuUmpb4CLrV08rTP49IdFQ8w==", "license": "MIT", "dependencies": { - "csv-generate": "^4.5.0", - "csv-parse": "^6.1.0", - "csv-stringify": "^6.6.0", - "stream-transform": "^3.4.0" + "csv-generate": "^4.6.1", + "csv-parse": "^7.0.2", + "csv-stringify": "^6.8.3", + "stream-transform": "^3.5.1" }, "engines": { "node": ">= 0.1.90" } }, "node_modules/csv-generate": { - "version": "4.5.0", - "resolved": "https://registry.npmjs.org/csv-generate/-/csv-generate-4.5.0.tgz", - "integrity": "sha512-aQr/vmOKyBSBHNwYhAoXw1+kUsPnMSwmYgpNoo36rIXoG1ecWILnvPGZeQ6oUjzrWknZAD3+jfpqYOBAl4x15A==", + "version": "4.6.1", + "resolved": "https://registry.npmjs.org/csv-generate/-/csv-generate-4.6.1.tgz", + "integrity": "sha512-eELl9K716LSSeP2/YcCjch525JztnnERe3jEARWw2v1FN9ukUYfZTNYZ4Rq2Jj/MFKMauffOy9VCaqQTpFThDQ==", "license": "MIT" }, "node_modules/csv-parse": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/csv-parse/-/csv-parse-6.1.0.tgz", - "integrity": "sha512-CEE+jwpgLn+MmtCpVcPtiCZpVtB6Z2OKPTr34pycYYoL7sxdOkXDdQ4lRiw6ioC0q6BLqhc6cKweCVvral8yhw==", + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/csv-parse/-/csv-parse-7.0.2.tgz", + "integrity": "sha512-uKZghv9UmPkMVLYy//KZ9HFAIJsl7wkhoEdIL0+rhuSY9pZQlhaeGEDPIe+/w7eh81MOql8Q/9+inAGWG6ZHYA==", "license": "MIT" }, "node_modules/csv-stringify": { - "version": "6.6.0", - "resolved": "https://registry.npmjs.org/csv-stringify/-/csv-stringify-6.6.0.tgz", - "integrity": "sha512-YW32lKOmIBgbxtu3g5SaiqWNwa/9ISQt2EcgOq0+RAIFufFp9is6tqNnKahqE5kuKvrnYAzs28r+s6pXJR8Vcw==", + "version": "6.8.3", + "resolved": "https://registry.npmjs.org/csv-stringify/-/csv-stringify-6.8.3.tgz", + "integrity": "sha512-gIeSCvq5F4VtXV3naV3VAewLhBkiZBz+PPhTOA8H3Y8h/ELa+R1ml0GZck/4/Nzo9ep2lvOluilJ6MJlbZsKMA==", "license": "MIT" }, "node_modules/damerau-levenshtein": { @@ -7887,7 +7969,32 @@ "version": "5.2.2", "resolved": "https://registry.npmjs.org/diff/-/diff-5.2.2.tgz", "integrity": "sha512-vtcDfH3TOjP8UekytvnHH1o1P4FcUdt4eQ1Y+Abap1tk/OB2MWQvcwS2ClCd1zuIhc3JKOx6p3kod8Vfys3E+A==", - "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.3.1" + } + }, + "node_modules/diff2html": { + "version": "3.4.56", + "resolved": "https://registry.npmjs.org/diff2html/-/diff2html-3.4.56.tgz", + "integrity": "sha512-u9gfn+BlbHcyO7vItCIC4z49LJDUt31tODzOfAuJ5R1E7IdlRL6KjugcB9zOpejD+XiR+dDZbsnHSQ3g6A/u8A==", + "license": "MIT", + "dependencies": { + "@profoundlogic/hogan": "^3.0.4", + "diff": "^8.0.3" + }, + "engines": { + "node": ">=12" + }, + "optionalDependencies": { + "highlight.js": "11.11.1" + } + }, + "node_modules/diff2html/node_modules/diff": { + "version": "8.0.4", + "resolved": "https://registry.npmjs.org/diff/-/diff-8.0.4.tgz", + "integrity": "sha512-DPi0FmjiSU5EvQV0++GFDOJ9ASQUVFh5kD+OzOnYdi7n3Wpm9hWWGfB/O2blfHcMVTL5WkQXSnRiK9makhrcnw==", + "license": "BSD-3-Clause", "engines": { "node": ">=0.3.1" } @@ -7962,9 +8069,9 @@ "license": "MIT" }, "node_modules/electron-to-chromium": { - "version": "1.5.207", - "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.207.tgz", - "integrity": "sha512-mryFrrL/GXDTmAtIVMVf+eIXM09BBPlO5IQ7lUyKmK8d+A4VpRGG+M3ofoVef6qyF8s60rJei8ymlJxjUA8Faw==", + "version": "1.5.425", + "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.425.tgz", + "integrity": "sha512-QvPtl41EUOnuT1HBvMKgxXRIaHNcagBPs50u7VULzhZXaGfqTbZyE16LQsctZ/RQHlGu+FOWeDTR4mY6YbeF1g==", "dev": true, "license": "ISC" }, @@ -9172,9 +9279,9 @@ "license": "MIT" }, "node_modules/fast-uri": { - "version": "3.1.5", - "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.5.tgz", - "integrity": "sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw==", + "version": "3.1.7", + "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.7.tgz", + "integrity": "sha512-dOvZVzjdZdz7phd9v6jCbwxrBW3fK6n8Rc0CtdmM4bumzMnxywBYhuph6J819RRw/ku+rLbelwfMunktuzVVHg==", "dev": true, "funding": [ { @@ -9246,9 +9353,9 @@ } }, "node_modules/fflate": { - "version": "0.8.1", - "resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.1.tgz", - "integrity": "sha512-/exOvEuc+/iaUm105QIiOt4LpBdMTWsXxqR0HDF35vx3fmaKzw7354gTilCh5rkzEt8WYyG//ku3h3nRmd7CHQ==", + "version": "0.8.3", + "resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.3.tgz", + "integrity": "sha512-tbZNuJrLwGUp3zshBtdy4W+ORxZuIh8a5ilyIEQDC5rY1f3U20JMry0Ll3WBzU58EZKsEuJFXhb5gwv8CsPvgA==", "license": "MIT" }, "node_modules/file-entry-cache": { @@ -9834,6 +9941,16 @@ "node": ">= 0.4" } }, + "node_modules/highlight.js": { + "version": "11.11.1", + "resolved": "https://registry.npmjs.org/highlight.js/-/highlight.js-11.11.1.tgz", + "integrity": "sha512-Xwwo44whKBVCYoliBQwaPvtd/2tYFkRQtXDWj1nackaV2JPXx3L0+Jvd8/qCJ2p+ML0/XVkJ2q+Mr+UVdpJK5w==", + "license": "BSD-3-Clause", + "optional": true, + "engines": { + "node": ">=12.0.0" + } + }, "node_modules/hosted-git-info": { "version": "4.1.0", "resolved": "https://registry.npmjs.org/hosted-git-info/-/hosted-git-info-4.1.0.tgz", @@ -10615,9 +10732,9 @@ } }, "node_modules/joi": { - "version": "17.13.4", - "resolved": "https://registry.npmjs.org/joi/-/joi-17.13.4.tgz", - "integrity": "sha512-1RuuER6kmt8K8I3nIWvPZKi5RQCb568ZPyY4Pwjlua+yo+63ZTmIwxLZH0heBmiKN4uxjvCiarDrjaeH84xicQ==", + "version": "17.13.7", + "resolved": "https://registry.npmjs.org/joi/-/joi-17.13.7.tgz", + "integrity": "sha512-MF80Dm5Y2veNy8QWVx9Bj3ui4mo7+VPSPsR1M+oaHXV0Gx6zGX9a2F+OZG3Blby9tOlzU9Rs5FUimlEhbKtfnQ==", "dev": true, "license": "BSD-3-Clause", "dependencies": { @@ -10653,9 +10770,9 @@ "license": "MIT" }, "node_modules/js-yaml": { - "version": "4.3.1", - "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.1.tgz", - "integrity": "sha512-CY6crGq313MX8GkwvB7tzgp99vjQxY1++5y10/BKN/GUfHqWaOGQMNZkBvqSzsZKWk/ijwHlWzzkLulsGHhjWQ==", + "version": "4.3.2", + "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.2.tgz", + "integrity": "sha512-SFNOvSJ+Dgf/9An904Yx+CgSlIPCkIpao4qo51lpee25TIRejdH3rhR4EZMGoNx3/TP3O+wzWuiTFl4sqbltzA==", "dev": true, "funding": [ { @@ -11315,6 +11432,12 @@ "dev": true, "license": "CC0-1.0" }, + "node_modules/memory-pager": { + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/memory-pager/-/memory-pager-1.5.0.tgz", + "integrity": "sha512-ZS4Bp4r/Zoeq6+NLJpP+0Zzm0pR8whtGPf1XExKLJBAczGMnSi3It14OiNCStjQjM6NU1okjQGSxgEZN8eBYKg==", + "license": "MIT" + }, "node_modules/meow": { "version": "10.1.5", "resolved": "https://registry.npmjs.org/meow/-/meow-10.1.5.tgz", @@ -11516,6 +11639,87 @@ "node": ">= 6" } }, + "node_modules/mongodb": { + "version": "6.21.0", + "resolved": "https://registry.npmjs.org/mongodb/-/mongodb-6.21.0.tgz", + "integrity": "sha512-URyb/VXMjJ4da46OeSXg+puO39XH9DeQpWCslifrRn9JWugy0D+DvvBvkm2WxmHe61O/H19JM66p1z7RHVkZ6A==", + "license": "Apache-2.0", + "dependencies": { + "@mongodb-js/saslprep": "^1.3.0", + "bson": "^6.10.4", + "mongodb-connection-string-url": "^3.0.2" + }, + "engines": { + "node": ">=16.20.1" + }, + "peerDependencies": { + "@aws-sdk/credential-providers": "^3.188.0", + "@mongodb-js/zstd": "^1.1.0 || ^2.0.0", + "gcp-metadata": "^5.2.0", + "kerberos": "^2.0.1", + "mongodb-client-encryption": ">=6.0.0 <7", + "snappy": "^7.3.2", + "socks": "^2.7.1" + }, + "peerDependenciesMeta": { + "@aws-sdk/credential-providers": { + "optional": true + }, + "@mongodb-js/zstd": { + "optional": true + }, + "gcp-metadata": { + "optional": true + }, + "kerberos": { + "optional": true + }, + "mongodb-client-encryption": { + "optional": true + }, + "snappy": { + "optional": true + }, + "socks": { + "optional": true + } + } + }, + "node_modules/mongodb-connection-string-url": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/mongodb-connection-string-url/-/mongodb-connection-string-url-3.0.2.tgz", + "integrity": "sha512-rMO7CGo/9BFwyZABcKAWL8UJwH/Kc2x0g72uhDWzG48URRax5TCIcJ7Rc3RZqffZzO/Gwff/jyKwCU9TN8gehA==", + "license": "Apache-2.0", + "dependencies": { + "@types/whatwg-url": "^11.0.2", + "whatwg-url": "^14.1.0 || ^13.0.0" + } + }, + "node_modules/mongodb-connection-string-url/node_modules/tr46": { + "version": "5.1.1", + "resolved": "https://registry.npmjs.org/tr46/-/tr46-5.1.1.tgz", + "integrity": "sha512-hdF5ZgjTqgAntKkklYw0R03MG2x/bSzTtkxmIRw/sTNV8YXsCJ1tfLAX23lhxhHJlEf3CRCOCGGWw3vI3GaSPw==", + "license": "MIT", + "dependencies": { + "punycode": "^2.3.1" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/mongodb-connection-string-url/node_modules/whatwg-url": { + "version": "14.2.0", + "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-14.2.0.tgz", + "integrity": "sha512-De72GdQZzNTUBBChsXueQUnPKDkg/5A5zp7pFDuQAj5UFoENpiACU0wlCvzpAGnTkj++ihpKwKyYewn/XNUbKw==", + "license": "MIT", + "dependencies": { + "tr46": "^5.1.0", + "webidl-conversions": "^7.0.0" + }, + "engines": { + "node": ">=18" + } + }, "node_modules/ms": { "version": "2.1.3", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", @@ -11615,11 +11819,29 @@ } }, "node_modules/node-releases": { - "version": "2.0.19", - "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.19.tgz", - "integrity": "sha512-xxOWJsBKtzAq7DY0J+DTzuz58K8e7sJbdgwkbMWQe8UYB6ekmsQ45q0M/tJDsGaZmbC+l7n57UV8Hl5tHxO9uw==", + "version": "2.0.54", + "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.54.tgz", + "integrity": "sha512-YHs7BmmcsdAI5Ozuf8JZo6PT0mv2GIWC9vMfvUC3dp65M8hn7Ux8CPL+2oBI7juNuj9d0ndhTcznq2ODBps9cQ==", "dev": true, - "license": "MIT" + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/nopt": { + "version": "1.0.10", + "resolved": "https://registry.npmjs.org/nopt/-/nopt-1.0.10.tgz", + "integrity": "sha512-NWmpvLSqUrgrAC9HCuxEvb+PSloHpqVu+FqcO4eeF2h5qYRhA7ev6KvelyQAKtegUbC6RypJnlEOhd8vloNKYg==", + "license": "MIT", + "dependencies": { + "abbrev": "1" + }, + "bin": { + "nopt": "bin/nopt.js" + }, + "engines": { + "node": "*" + } }, "node_modules/normalize-package-data": { "version": "3.0.3", @@ -12182,9 +12404,9 @@ } }, "node_modules/postcss-selector-parser": { - "version": "6.1.2", - "resolved": "https://registry.npmjs.org/postcss-selector-parser/-/postcss-selector-parser-6.1.2.tgz", - "integrity": "sha512-Q8qQfPiZ+THO/3ZrOrO0cJJKfpYCagtMUkXbnEfmgUjwXg6z/WBeOyS9APBBPCTSiDV+s4SwQGu8yFsiMRIudg==", + "version": "6.1.4", + "resolved": "https://registry.npmjs.org/postcss-selector-parser/-/postcss-selector-parser-6.1.4.tgz", + "integrity": "sha512-bIoJLOmjCO1S9XdY/DcnR5hJxvrDir1PbGChrzXG3vw0/FOliy/fA3dmdhQ441kah4gKv+TwckGzex6wNS5cnQ==", "dev": true, "license": "MIT", "dependencies": { @@ -12312,7 +12534,6 @@ "version": "2.3.1", "resolved": "https://registry.npmjs.org/punycode/-/punycode-2.3.1.tgz", "integrity": "sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg==", - "dev": true, "license": "MIT", "engines": { "node": ">=6" @@ -13121,6 +13342,15 @@ "node": ">=0.10.0" } }, + "node_modules/sparse-bitfield": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/sparse-bitfield/-/sparse-bitfield-3.0.3.tgz", + "integrity": "sha512-kvzhi7vqKTfkh0PZU+2D2PIllw2ymqJKujUcyPMd9Y75Nv4nPbGJZXNhxsgdQab2BmlDct1YnfQCguEvHr7VsQ==", + "license": "MIT", + "dependencies": { + "memory-pager": "^1.0.2" + } + }, "node_modules/spdx-correct": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/spdx-correct/-/spdx-correct-3.2.0.tgz", @@ -13243,9 +13473,9 @@ } }, "node_modules/stream-transform": { - "version": "3.4.0", - "resolved": "https://registry.npmjs.org/stream-transform/-/stream-transform-3.4.0.tgz", - "integrity": "sha512-QO3OGhKyeIV8p6eRQdG+W6WounFw519zk690hHCNfhgfP9bylVS+NTXsuBc7n+RsGn31UgFPGrWYIgoAbArKEw==", + "version": "3.5.1", + "resolved": "https://registry.npmjs.org/stream-transform/-/stream-transform-3.5.1.tgz", + "integrity": "sha512-TTDX+qKFr7GGRXATn66rprmlFPx08W0UBIccE/rMPgW2sT7GovduZYP4xcdJ7Nu2YigF17U+CNFxYY11+W1oPw==", "license": "MIT" }, "node_modules/string-width": { @@ -14250,9 +14480,9 @@ } }, "node_modules/update-browserslist-db": { - "version": "1.1.3", - "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.1.3.tgz", - "integrity": "sha512-UxhIZQ+QInVdunkDAaiazvvT/+fXL5Osr0JZlJulepYu6Jd7qJtDZjlur0emRlT71EN3ScPoE7gvsuIKKNavKw==", + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.3.2.tgz", + "integrity": "sha512-UQ+MSxlhRm1bzjhU+DcuXfjFO1FzNtqhK5+9Yvlp90ItDLk5vT932A0rFu619nf7RVS+Y/VeaUW1jaRDqZ8VJw==", "dev": true, "funding": [ { @@ -14591,7 +14821,6 @@ "version": "7.0.0", "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-7.0.0.tgz", "integrity": "sha512-VwddBukDzu71offAQR975unBIGqfKZpM+8ZX6ySk8nYhVoo5CYaZyzt3YBvYtRtO+aoGlqxPg/B87NGVZ/fu6g==", - "dev": true, "license": "BSD-2-Clause", "engines": { "node": ">=12" diff --git a/package.json b/package.json index b20b5c28..74eaa3da 100644 --- a/package.json +++ b/package.json @@ -26,6 +26,8 @@ "redis:stop": "bash bin/redis/stop.sh", "redis:connect": "bash bin/redis/connect.sh", "redis:memory_used": "bash bin/redis/memory_used.sh", + "documentdb:start": "bash bin/documentdb/start.sh", + "documentdb:stop": "bash bin/documentdb/stop.sh", "prime-cache:invoke-local": "bash scripts/local/invoke_prime_concepts_cache.sh", "export-data": "vite-node --config vite.config.js setup/scripts/exportData.js -all", "create-rdf-files": "vite-node --config vite.config.js setup/scripts/createRdfFiles.js", @@ -38,20 +40,24 @@ "@aws-sdk/client-eventbridge": "^3.997.0", "@aws-sdk/client-lambda": "^3.775.0", "@aws-sdk/client-s3": "^3.540.0", + "@aws-sdk/client-secrets-manager": "^3.1124.0", "@aws-sdk/client-sfn": "^3.775.0", "@aws-sdk/client-sns": "^3.997.0", "@aws-sdk/client-sqs": "^3.997.0", "@aws-sdk/client-ssm": "^3.1096.0", "@aws-sdk/s3-request-presigner": "3.981.0", - "@xmldom/xmldom": "^0.8.10", + "@xmldom/xmldom": "^0.9.12", "compact-object-deep": "^1.0.0", "csv": "^6.3.11", "date-fns": "^4.1.0", + "diff": "^5.2.2", + "diff2html": "^3.4.56", "fast-xml-parser": "^5.3.4", "html-entities": "^2.5.2", "html-escaper": "^3.0.3", "install": "^0.13.0", "lodash": "^4.17.21", + "mongodb": "^6.21.0", "node-fetch": "^2.7.0", "redis": "^4.7.1", "remove": "^0.1.5", diff --git a/scripts/local/initialize_metadata_correction_audit.mjs b/scripts/local/initialize_metadata_correction_audit.mjs new file mode 100644 index 00000000..616b158f --- /dev/null +++ b/scripts/local/initialize_metadata_correction_audit.mjs @@ -0,0 +1,28 @@ +import fs from 'fs' +import path from 'path' +import { fileURLToPath } from 'url' + +import { + initializeMetadataCorrectionAudit +} from '../../serverless/src/initializeMetadataCorrectionAudit/handler' +import { closeDocumentDbClient } from '../../serverless/src/shared/documentDbClient' + +const scriptDirectory = path.dirname(fileURLToPath(import.meta.url)) +const projectRoot = path.resolve(scriptDirectory, '../..') +const indexDefinitions = JSON.parse(fs.readFileSync( + path.join(projectRoot, 'config/metadataCorrectionAuditIndexes.json'), + 'utf8' +)) + +try { + const result = await initializeMetadataCorrectionAudit({ + RequestType: 'Create', + ResourceProperties: { IndexDefinitions: indexDefinitions } + }) + + console.log( + `[initialize-metadata-correction-audit] Verified ${result.Data.IndexCount} local audit indexes` + ) +} finally { + await closeDocumentDbClient() +} diff --git a/scripts/local/metadataCorrectionSmokeHelpers.mjs b/scripts/local/metadataCorrectionSmokeHelpers.mjs new file mode 100644 index 00000000..395fe350 --- /dev/null +++ b/scripts/local/metadataCorrectionSmokeHelpers.mjs @@ -0,0 +1,19 @@ +import { getMetadataCorrectionAuditCollection } from '../../serverless/src/shared/documentDbClient' + +/** + * Removes prior audit documents for a smoke collection so assertions start from a clean state. + * + * @example + * await clearAuditDocumentsForCollection('C1234567890-LOCAL') + * + * @param {string} collectionConceptId Collection whose local audit documents should be removed. + * @returns {Promise} Resolves after matching audit documents are deleted. + */ +export const clearAuditDocumentsForCollection = async (collectionConceptId) => { + process.env.DOCUMENTDB_URI = process.env.DOCUMENTDB_URI + || `mongodb://localhost:${process.env.DOCUMENTDB_HOST_PORT || 27018}` + + const auditCollection = await getMetadataCorrectionAuditCollection() + + await auditCollection.deleteMany({ collectionConceptId }) +} diff --git a/scripts/local/mock_cmr_server.mjs b/scripts/local/mock_cmr_server.mjs index e8d8ca79..7aa8f14b 100644 --- a/scripts/local/mock_cmr_server.mjs +++ b/scripts/local/mock_cmr_server.mjs @@ -89,6 +89,33 @@ const getNativeMetadataContentType = (collection) => { return format || 'application/octet-stream' } +const nativeRevisionsByConceptId = new Map() + +/** + * Snapshots a collection's current native payload by revision for revision-specific GET requests. + * + * @example + * saveNativeRevision({ conceptId: 'C123-PROV', revisionId: 2, nativeMetadata: '' }) + * // nativeRevisionsByConceptId.get('C123-PROV').get('2').nativeMetadata === '' + * + * @param {Object} collection Mutable fixture collection to snapshot. + * @returns {void} + */ +const saveNativeRevision = (collection) => { + const revisions = nativeRevisionsByConceptId.get(collection.conceptId) || new Map() + const nativeMetadata = getNativeMetadataPayload(collection) + + revisions.set(String(collection.revisionId), { + contentType: getNativeMetadataContentType(collection), + nativeMetadata: typeof nativeMetadata === 'string' + ? nativeMetadata + : structuredClone(nativeMetadata) + }) + nativeRevisionsByConceptId.set(collection.conceptId, revisions) +} + +fixture.cmr?.collections?.forEach(saveNativeRevision) + // Keep the concept-id index in sync after local updates. const updateCollectionIndexes = (collection) => { collectionsByConceptId.set(collection.conceptId, collection) @@ -328,7 +355,14 @@ const handleNativeCollectionLookupRequest = (conceptId, revisionId, response) => return } - if (revisionId !== undefined && String(collection.revisionId) !== String(revisionId)) { + const revision = revisionId === undefined + ? { + contentType: getNativeMetadataContentType(collection), + nativeMetadata: getNativeMetadataPayload(collection) + } + : nativeRevisionsByConceptId.get(conceptId)?.get(String(revisionId)) + + if (!revision) { sendJson(response, 404, { errors: [`Revision ${revisionId} not found for collection concept id: ${conceptId}`] }) @@ -336,16 +370,14 @@ const handleNativeCollectionLookupRequest = (conceptId, revisionId, response) => return } - const nativeMetadata = getNativeMetadataPayload(collection) - response.writeHead(200, { - 'Content-Type': getNativeMetadataContentType(collection) + 'Content-Type': revision.contentType }) response.end( - typeof nativeMetadata === 'string' - ? nativeMetadata - : JSON.stringify(nativeMetadata) + typeof revision.nativeMetadata === 'string' + ? revision.nativeMetadata + : JSON.stringify(revision.nativeMetadata) ) } @@ -380,6 +412,7 @@ const handleLocalCollectionUpdateRequest = async (request, response, conceptId) collection.revisionId = Number(collection.revisionId || 0) + 1 updateCollectionIndexes(collection) + saveNativeRevision(collection) sendJson(response, 200, { updated: true, @@ -450,6 +483,7 @@ const handleIngestCollectionWriteRequest = async (request, response, providerId, collection.revisionId = Number(collection.revisionId || 0) + 1 updateCollectionIndexes(collection) + saveNativeRevision(collection) sendJson(response, 200, { 'concept-id': collection.conceptId, diff --git a/scripts/local/run_metadata_correction_applied_audit_smoke.mjs b/scripts/local/run_metadata_correction_applied_audit_smoke.mjs index 83e07ae8..854502b2 100644 --- a/scripts/local/run_metadata_correction_applied_audit_smoke.mjs +++ b/scripts/local/run_metadata_correction_applied_audit_smoke.mjs @@ -4,15 +4,20 @@ import { spawn } from 'node:child_process' import fs from 'node:fs/promises' import path from 'node:path' +import { closeDocumentDbClient } from '../../serverless/src/shared/documentDbClient' + +import { clearAuditDocumentsForCollection } from './metadataCorrectionSmokeHelpers.mjs' + /** * Local end-to-end audit smoke for metadata correction. * * This script exercises the real metadataCorrectionService handler against: * - the local mock CMR server * - local Redis keyword caches - * - local RDF4J audit persistence + * - local MongoDB-compatible DocumentDB audit persistence * - * It verifies that a successful correction run writes both audit lifecycle states: + * It verifies that a successful correction run records all audit lifecycle states: + * - `checked` after validation and resolution * - `pending` before writeback * - `applied` after writeback succeeds * @@ -147,42 +152,6 @@ const seedKeywordCaches = async () => { return redisClient } -const clearAuditRowsForCollection = async () => { - process.env.RDF4J_SERVICE_URL = process.env.RDF4J_SERVICE_URL || 'http://localhost:8081' - process.env.RDF4J_USER_NAME = process.env.RDF4J_USER_NAME || 'rdf4j' - process.env.RDF4J_PASSWORD = process.env.RDF4J_PASSWORD || 'rdf4j' - - const { - escapeSparqlLiteral, - METADATA_CORRECTION_AUDIT_GRAPH - } = await import('../../serverless/src/shared/metadataCorrectionAudit') - const { sparqlRequest } = await import('../../serverless/src/shared/sparqlRequest') - - const query = ` - PREFIX gcmd: - - DELETE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record ?predicate ?object . - } - } - WHERE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record a gcmd:MetadataCorrectionAuditRecord ; - gcmd:collectionConceptId "${escapeSparqlLiteral(collectionConceptId)}" ; - ?predicate ?object . - } - } - ` - - await sparqlRequest({ - method: 'POST', - contentType: 'application/sparql-update', - accept: 'application/json', - body: query - }) -} - let mockServerProcess let redisClient @@ -206,15 +175,18 @@ try { process.env.CMR_BASE_URL = baseUrl process.env.CMR_WRITEBACK_PROVIDERS = process.env.CMR_WRITEBACK_PROVIDERS || providerId - process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'local-writer-token' + process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'Bearer local-writer-token' redisClient = await seedKeywordCaches() - await clearAuditRowsForCollection() + await clearAuditDocumentsForCollection(collectionConceptId) const { metadataCorrectionService } = await import('../../serverless/src/metadataCorrectionService/handler') - const { getMetadataCorrectionAuditLog } = await import('../../serverless/src/shared/getMetadataCorrectionAuditLog') + const { + getMetadataCorrectionAuditByRunId, + getMetadataCorrectionAuditLog + } = await import('../../serverless/src/shared/getMetadataCorrectionAuditLog') - const beforeRows = await getMetadataCorrectionAuditLog({ + const { items: beforeRows } = await getMetadataCorrectionAuditLog({ collectionConceptId, limit: 20 }) @@ -226,20 +198,30 @@ try { body: JSON.stringify({ source: 'local-smoke', collectionConceptId, + publishedVersionName: 'local-published', keywordEvent: normalizeKeywordEvent(rawKeywordEvent) }) } ] }) - const afterRows = await getMetadataCorrectionAuditLog({ + const { items: afterRows } = await getMetadataCorrectionAuditLog({ collectionConceptId, limit: 20 }) - const statuses = [...new Set(afterRows.map((row) => row.status))] + const appliedSummary = afterRows.find(({ status }) => status === 'applied') + const appliedRow = appliedSummary + ? await getMetadataCorrectionAuditByRunId({ + runId: appliedSummary.runId, + includeDiff: true + }) + : null + const statuses = [...new Set( + appliedRow?.statusHistory?.map(({ status }) => status) || [] + )] if (beforeRows.length !== 0) { - throw new Error(`Expected no starting audit rows for ${collectionConceptId}, found ${beforeRows.length}`) + throw new Error(`Expected no starting audit documents for ${collectionConceptId}, found ${beforeRows.length}`) } if (!statuses.includes('pending')) { @@ -250,6 +232,23 @@ try { throw new Error(`Missing applied audit status for ${collectionConceptId}`) } + if (appliedRow?.publishedVersionName !== 'local-published') { + throw new Error(`Missing published KMS version for ${collectionConceptId}`) + } + + if (appliedRow.priorRevisionId !== 1 || appliedRow.resultingRevisionId !== 2) { + throw new Error(`Missing the expected CMR revision IDs for ${collectionConceptId}`) + } + + const expectedCollectionUri = `${baseUrl}/search/concepts/${encodeURIComponent(collectionConceptId)}` + if (appliedRow.collectionUri !== expectedCollectionUri) { + throw new Error(`Missing the expected CMR collection URI for ${collectionConceptId}`) + } + + if (appliedRow.metadataDiff?.changed !== true || !appliedRow.metadataDiff.patch) { + throw new Error(`Missing the native metadata diff for ${collectionConceptId}`) + } + await fs.mkdir(outputDir, { recursive: true }) await fs.writeFile(outputPath, JSON.stringify({ collectionConceptId, @@ -258,7 +257,8 @@ try { beforeCount: beforeRows.length, afterCount: afterRows.length, statuses, - rows: afterRows + rows: afterRows, + appliedRow }, null, 2), 'utf8') console.log('[metadata-correction-audit-smoke] Completed successfully') @@ -272,6 +272,8 @@ try { outputPath }, null, 2)) } finally { + await closeDocumentDbClient() + if (redisClient) { await redisClient.quit() } diff --git a/scripts/local/run_metadata_correction_consumer_metrics_async_smoke.mjs b/scripts/local/run_metadata_correction_consumer_metrics_async_smoke.mjs index f96e8448..d2e8c3be 100644 --- a/scripts/local/run_metadata_correction_consumer_metrics_async_smoke.mjs +++ b/scripts/local/run_metadata_correction_consumer_metrics_async_smoke.mjs @@ -4,11 +4,14 @@ import { spawn } from 'node:child_process' import fs from 'node:fs/promises' import path from 'node:path' +import { closeDocumentDbClient } from '../../serverless/src/shared/documentDbClient' import { CONSUMER_METRIC_NAMES, CONSUMER_METRIC_NAMESPACE } from '../../serverless/src/shared/emitConsumerMetrics' +import { clearAuditDocumentsForCollection } from './metadataCorrectionSmokeHelpers.mjs' + /** * Local end-to-end smoke for async consumer metrics. * @@ -20,7 +23,7 @@ import { * Prerequisites: * - LocalStack is running on `http://127.0.0.1:4566` * - local Redis is running - * - local RDF4J is running + * - local MongoDB-compatible audit storage is running * * Run with: * npx vite-node --config vite.config.js scripts/local/run_metadata_correction_consumer_metrics_async_smoke.mjs @@ -199,47 +202,6 @@ const seedKeywordCaches = async () => { return redisClient } -/** - * Removes any existing audit rows for the smoke collection. - * - * @returns {Promise} Resolves once prior audit rows are deleted. - */ -const clearAuditRowsForCollection = async () => { - process.env.RDF4J_SERVICE_URL = process.env.RDF4J_SERVICE_URL || 'http://localhost:8081' - process.env.RDF4J_USER_NAME = process.env.RDF4J_USER_NAME || 'rdf4j' - process.env.RDF4J_PASSWORD = process.env.RDF4J_PASSWORD || 'rdf4j' - - const { - escapeSparqlLiteral, - METADATA_CORRECTION_AUDIT_GRAPH - } = await import('../../serverless/src/shared/metadataCorrectionAudit') - const { sparqlRequest } = await import('../../serverless/src/shared/sparqlRequest') - - const query = ` - PREFIX gcmd: - - DELETE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record ?predicate ?object . - } - } - WHERE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record a gcmd:MetadataCorrectionAuditRecord ; - gcmd:collectionConceptId "${escapeSparqlLiteral(collectionConceptId)}" ; - ?predicate ?object . - } - } - ` - - await sparqlRequest({ - method: 'POST', - contentType: 'application/sparql-update', - accept: 'application/json', - body: query - }) -} - /** * Creates an XML parser for CloudWatch Query API responses. * @@ -551,11 +513,11 @@ try { process.env.CMR_BASE_URL = cmrBaseUrl process.env.CMR_WRITEBACK_PROVIDERS = process.env.CMR_WRITEBACK_PROVIDERS || providerId - process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'local-writer-token' + process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'Bearer local-writer-token' process.env.AWS_ENDPOINT_URL = cloudWatchEndpoint redisClient = await seedKeywordCaches() - await clearAuditRowsForCollection() + await clearAuditDocumentsForCollection(collectionConceptId) const { metadataCorrectionService } = await import('../../serverless/src/metadataCorrectionService/handler') const { getCmrCollectionNativeMetadata } = await import('../../serverless/src/shared/getCmrCollectionNativeMetadata') @@ -667,6 +629,8 @@ try { outputPath }, null, 2)) } finally { + await closeDocumentDbClient() + if (redisClient) { await redisClient.quit() } diff --git a/scripts/local/run_metadata_correction_consumer_metrics_manual_sync_smoke.mjs b/scripts/local/run_metadata_correction_consumer_metrics_manual_sync_smoke.mjs index d1f94487..3cf048d2 100644 --- a/scripts/local/run_metadata_correction_consumer_metrics_manual_sync_smoke.mjs +++ b/scripts/local/run_metadata_correction_consumer_metrics_manual_sync_smoke.mjs @@ -4,11 +4,14 @@ import { spawn } from 'node:child_process' import fs from 'node:fs/promises' import path from 'node:path' +import { closeDocumentDbClient } from '../../serverless/src/shared/documentDbClient' import { CONSUMER_METRIC_NAMES, CONSUMER_METRIC_NAMESPACE } from '../../serverless/src/shared/emitConsumerMetrics' +import { clearAuditDocumentsForCollection } from './metadataCorrectionSmokeHelpers.mjs' + /** * Local end-to-end smoke for manual sync consumer metrics. * @@ -21,7 +24,8 @@ import { * Prerequisites: * - LocalStack is running on `http://127.0.0.1:4566` * - local Redis is running - * - local RDF4J is running + * - local RDF4J is running for the manual request's published-version lookup + * - local MongoDB-compatible audit storage is running * * Run with: * npx vite-node --config vite.config.js scripts/local/run_metadata_correction_consumer_metrics_manual_sync_smoke.mjs @@ -185,47 +189,6 @@ const seedKeywordCaches = async () => { return redisClient } -/** - * Removes any existing audit rows for the smoke collection. - * - * @returns {Promise} Resolves once prior audit rows are deleted. - */ -const clearAuditRowsForCollection = async () => { - process.env.RDF4J_SERVICE_URL = process.env.RDF4J_SERVICE_URL || 'http://localhost:8081' - process.env.RDF4J_USER_NAME = process.env.RDF4J_USER_NAME || 'rdf4j' - process.env.RDF4J_PASSWORD = process.env.RDF4J_PASSWORD || 'rdf4j' - - const { - escapeSparqlLiteral, - METADATA_CORRECTION_AUDIT_GRAPH - } = await import('../../serverless/src/shared/metadataCorrectionAudit') - const { sparqlRequest } = await import('../../serverless/src/shared/sparqlRequest') - - const query = ` - PREFIX gcmd: - - DELETE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record ?predicate ?object . - } - } - WHERE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record a gcmd:MetadataCorrectionAuditRecord ; - gcmd:collectionConceptId "${escapeSparqlLiteral(collectionConceptId)}" ; - ?predicate ?object . - } - } - ` - - await sparqlRequest({ - method: 'POST', - contentType: 'application/sparql-update', - accept: 'application/json', - body: query - }) -} - /** * Creates an XML parser for CloudWatch Query API responses. * @@ -537,11 +500,11 @@ try { process.env.CMR_BASE_URL = cmrBaseUrl process.env.CMR_WRITEBACK_PROVIDERS = process.env.CMR_WRITEBACK_PROVIDERS || providerId - process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'local-writer-token' + process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'Bearer local-writer-token' process.env.AWS_ENDPOINT_URL = cloudWatchEndpoint redisClient = await seedKeywordCaches() - await clearAuditRowsForCollection() + await clearAuditDocumentsForCollection(collectionConceptId) const { runMetadataCorrection } = await import('../../serverless/src/runMetadataCorrection/handler') const { getCmrCollectionNativeMetadata } = await import('../../serverless/src/shared/getCmrCollectionNativeMetadata') @@ -678,6 +641,8 @@ try { outputPath }, null, 2)) } finally { + await closeDocumentDbClient() + if (redisClient) { await redisClient.quit() } diff --git a/scripts/local/run_metadata_correction_failed_audit_smoke.mjs b/scripts/local/run_metadata_correction_failed_audit_smoke.mjs index aaf75bab..3ac7089f 100644 --- a/scripts/local/run_metadata_correction_failed_audit_smoke.mjs +++ b/scripts/local/run_metadata_correction_failed_audit_smoke.mjs @@ -4,18 +4,22 @@ import { spawn } from 'node:child_process' import fs from 'node:fs/promises' import path from 'node:path' +import { closeDocumentDbClient } from '../../serverless/src/shared/documentDbClient' + +import { clearAuditDocumentsForCollection } from './metadataCorrectionSmokeHelpers.mjs' + /** * Local end-to-end audit smoke for failed metadata-correction writeback. * * This script exercises the real metadataCorrectionService handler against: * - the local mock CMR server, configured to fail ingest/writeback * - local Redis keyword caches - * - local RDF4J audit persistence + * - local MongoDB-compatible DocumentDB audit persistence * * It verifies that a failed correction run writes: * - `pending` before writeback * - `failed` after CMR ingest rejects the corrected metadata - * - `writebackErrorMessage` on the failed audit row + * - structured error details on the failed audit document * * Run with: * npx vite-node --config vite.config.js scripts/local/run_metadata_correction_failed_audit_smoke.mjs @@ -154,42 +158,6 @@ const seedKeywordCaches = async () => { return redisClient } -const clearAuditRowsForCollection = async () => { - process.env.RDF4J_SERVICE_URL = process.env.RDF4J_SERVICE_URL || 'http://localhost:8081' - process.env.RDF4J_USER_NAME = process.env.RDF4J_USER_NAME || 'rdf4j' - process.env.RDF4J_PASSWORD = process.env.RDF4J_PASSWORD || 'rdf4j' - - const { - escapeSparqlLiteral, - METADATA_CORRECTION_AUDIT_GRAPH - } = await import('../../serverless/src/shared/metadataCorrectionAudit') - const { sparqlRequest } = await import('../../serverless/src/shared/sparqlRequest') - - const query = ` - PREFIX gcmd: - - DELETE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record ?predicate ?object . - } - } - WHERE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record a gcmd:MetadataCorrectionAuditRecord ; - gcmd:collectionConceptId "${escapeSparqlLiteral(collectionConceptId)}" ; - ?predicate ?object . - } - } - ` - - await sparqlRequest({ - method: 'POST', - contentType: 'application/sparql-update', - accept: 'application/json', - body: query - }) -} - let mockServerProcess let redisClient @@ -215,15 +183,18 @@ try { process.env.CMR_BASE_URL = baseUrl process.env.CMR_WRITEBACK_PROVIDERS = process.env.CMR_WRITEBACK_PROVIDERS || providerId - process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'local-writer-token' + process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'Bearer local-writer-token' redisClient = await seedKeywordCaches() - await clearAuditRowsForCollection() + await clearAuditDocumentsForCollection(collectionConceptId) const { metadataCorrectionService } = await import('../../serverless/src/metadataCorrectionService/handler') - const { getMetadataCorrectionAuditLog } = await import('../../serverless/src/shared/getMetadataCorrectionAuditLog') + const { + getMetadataCorrectionAuditByRunId, + getMetadataCorrectionAuditLog + } = await import('../../serverless/src/shared/getMetadataCorrectionAuditLog') - const beforeRows = await getMetadataCorrectionAuditLog({ + const { items: beforeRows } = await getMetadataCorrectionAuditLog({ collectionConceptId, limit: 20 }) @@ -241,15 +212,23 @@ try { ] }) - const afterRows = await getMetadataCorrectionAuditLog({ + const { items: afterRows } = await getMetadataCorrectionAuditLog({ collectionConceptId, limit: 20 }) - const statuses = [...new Set(afterRows.map((row) => row.status))] - const failedRow = afterRows.find((row) => row.status === 'failed') + const failedSummary = afterRows.find(({ status }) => status === 'failed') + const failedRow = failedSummary + ? await getMetadataCorrectionAuditByRunId({ + runId: failedSummary.runId, + includeDiff: true + }) + : null + const statuses = [...new Set( + failedRow?.statusHistory?.map(({ status }) => status) || [] + )] if (beforeRows.length !== 0) { - throw new Error(`Expected no starting audit rows for ${collectionConceptId}, found ${beforeRows.length}`) + throw new Error(`Expected no starting audit documents for ${collectionConceptId}, found ${beforeRows.length}`) } if (JSON.stringify(response?.batchItemFailures) !== JSON.stringify([ @@ -273,17 +252,21 @@ try { throw new Error(`Did not expect applied audit status for failed writeback on ${collectionConceptId}`) } - if (!failedRow?.writebackErrorMessage) { - throw new Error(`Missing writebackErrorMessage on failed audit row for ${collectionConceptId}`) + if (!failedRow?.error?.message) { + throw new Error(`Missing error details on failed audit document for ${collectionConceptId}`) } - if (failedRow.writebackErrorMessage !== mockIngestErrorBody) { + if (JSON.stringify(failedRow.error.cmrResponseBody) !== mockIngestErrorBody) { throw new Error( - 'Expected failed audit writebackErrorMessage to match the mock ingest response body. ' - + `Received ${failedRow.writebackErrorMessage}` + 'Expected failed audit response details to match the mock ingest response body. ' + + `Received ${JSON.stringify(failedRow.error.cmrResponseBody)}` ) } + if (failedRow.metadataDiff?.changed !== true || !failedRow.metadataDiff.patch) { + throw new Error(`Missing the native metadata diff for failed run ${collectionConceptId}`) + } + await fs.mkdir(outputDir, { recursive: true }) await fs.writeFile(outputPath, JSON.stringify({ collectionConceptId, @@ -304,10 +287,12 @@ try { baseUrl, afterCount: afterRows.length, statuses, - failedWritebackErrorMessage: failedRow.writebackErrorMessage, + failedWritebackErrorMessage: failedRow.error.message, outputPath }, null, 2)) } finally { + await closeDocumentDbClient() + if (redisClient) { await redisClient.quit() } diff --git a/scripts/local/run_metadata_correction_partial_batch_failure_smoke.mjs b/scripts/local/run_metadata_correction_partial_batch_failure_smoke.mjs index f9bfb0db..d5fbc182 100644 --- a/scripts/local/run_metadata_correction_partial_batch_failure_smoke.mjs +++ b/scripts/local/run_metadata_correction_partial_batch_failure_smoke.mjs @@ -4,6 +4,8 @@ import { spawn } from 'node:child_process' import fs from 'node:fs/promises' import path from 'node:path' +import { closeDocumentDbClient } from '../../serverless/src/shared/documentDbClient' + /** * Local end-to-end smoke for metadata-correction consumer partial batch failure handling. * @@ -19,6 +21,7 @@ import path from 'node:path' * * Prerequisites: * - local Redis is running + * - local MongoDB-compatible audit storage is running * - LocalStack is optional; if present, set AWS_ENDPOINT_URL to avoid metric emission errors * * Run with: @@ -192,11 +195,10 @@ try { process.env.CMR_BASE_URL = cmrBaseUrl process.env.CMR_WRITEBACK_PROVIDERS = process.env.CMR_WRITEBACK_PROVIDERS || providerId - process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'local-writer-token' + process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'Bearer local-writer-token' process.env.AWS_ENDPOINT_URL = process.env.AWS_ENDPOINT_URL || 'http://127.0.0.1:4566' - process.env.RDF4J_SERVICE_URL = process.env.RDF4J_SERVICE_URL || 'http://localhost:8081' - process.env.RDF4J_USER_NAME = process.env.RDF4J_USER_NAME || 'rdf4j' - process.env.RDF4J_PASSWORD = process.env.RDF4J_PASSWORD || 'rdf4j' + process.env.DOCUMENTDB_URI = process.env.DOCUMENTDB_URI + || `mongodb://localhost:${process.env.DOCUMENTDB_HOST_PORT || 27018}` redisClient = await seedKeywordCaches() @@ -264,6 +266,8 @@ try { outputPath }, null, 2)) } finally { + await closeDocumentDbClient() + if (redisClient) { await redisClient.quit() } diff --git a/scripts/local/run_metadata_correction_request_delay_smoke.mjs b/scripts/local/run_metadata_correction_request_delay_smoke.mjs index e9178449..2275cc93 100644 --- a/scripts/local/run_metadata_correction_request_delay_smoke.mjs +++ b/scripts/local/run_metadata_correction_request_delay_smoke.mjs @@ -4,6 +4,10 @@ import { spawn } from 'node:child_process' import fs from 'node:fs/promises' import path from 'node:path' +import { closeDocumentDbClient } from '../../serverless/src/shared/documentDbClient' + +import { clearAuditDocumentsForCollection } from './metadataCorrectionSmokeHelpers.mjs' + /** * Local end-to-end smoke for the queued manual-request delay path. * @@ -19,7 +23,7 @@ import path from 'node:path' * * Prerequisites: * - local Redis is running - * - local RDF4J is running + * - local MongoDB-compatible audit storage is running * - LocalStack is optional; if present, set AWS_ENDPOINT_URL to avoid metric * emission errors in logs * @@ -181,47 +185,6 @@ const seedKeywordCaches = async () => { return redisClient } -/** - * Removes any existing audit rows for the smoke collection so assertions start clean. - * - * @returns {Promise} Resolves once prior audit rows have been deleted. - */ -const clearAuditRowsForCollection = async () => { - process.env.RDF4J_SERVICE_URL = process.env.RDF4J_SERVICE_URL || 'http://localhost:8081' - process.env.RDF4J_USER_NAME = process.env.RDF4J_USER_NAME || 'rdf4j' - process.env.RDF4J_PASSWORD = process.env.RDF4J_PASSWORD || 'rdf4j' - - const { - escapeSparqlLiteral, - METADATA_CORRECTION_AUDIT_GRAPH - } = await import('../../serverless/src/shared/metadataCorrectionAudit') - const { sparqlRequest } = await import('../../serverless/src/shared/sparqlRequest') - - const query = ` - PREFIX gcmd: - - DELETE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record ?predicate ?object . - } - } - WHERE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record a gcmd:MetadataCorrectionAuditRecord ; - gcmd:collectionConceptId "${escapeSparqlLiteral(collectionConceptId)}" ; - ?predicate ?object . - } - } - ` - - await sparqlRequest({ - method: 'POST', - contentType: 'application/sparql-update', - accept: 'application/json', - body: query - }) -} - let mockServerProcess let redisClient @@ -245,15 +208,18 @@ try { process.env.CMR_BASE_URL = cmrBaseUrl process.env.CMR_WRITEBACK_PROVIDERS = process.env.CMR_WRITEBACK_PROVIDERS || providerId - process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'local-writer-token' + process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'Bearer local-writer-token' process.env.METADATA_CORRECTION_REQUEST_DELAY_MS = String(configuredDelayMs) process.env.AWS_ENDPOINT_URL = process.env.AWS_ENDPOINT_URL || 'http://127.0.0.1:4566' redisClient = await seedKeywordCaches() - await clearAuditRowsForCollection() + await clearAuditDocumentsForCollection(collectionConceptId) const { metadataCorrectionService } = await import('../../serverless/src/metadataCorrectionService/handler') - const { getMetadataCorrectionAuditLog } = await import('../../serverless/src/shared/getMetadataCorrectionAuditLog') + const { + getMetadataCorrectionAuditByRunId, + getMetadataCorrectionAuditLog + } = await import('../../serverless/src/shared/getMetadataCorrectionAuditLog') const { getCmrCollectionNativeMetadata } = await import('../../serverless/src/shared/getCmrCollectionNativeMetadata') const requestedAt = new Date().toISOString() @@ -313,11 +279,20 @@ try { ) } - const auditRows = await getMetadataCorrectionAuditLog({ + const { items: auditRows } = await getMetadataCorrectionAuditLog({ collectionConceptId, limit: 20 }) - const statuses = [...new Set(auditRows.map((row) => row.status))] + const appliedSummary = auditRows.find(({ status }) => status === 'applied') + const appliedRow = appliedSummary + ? await getMetadataCorrectionAuditByRunId({ + runId: appliedSummary.runId, + includeDiff: true + }) + : null + const statuses = [...new Set( + appliedRow?.statusHistory?.map(({ status }) => status) || [] + )] if (!statuses.includes('pending')) { throw new Error(`Missing pending audit status for ${collectionConceptId}`) @@ -327,6 +302,10 @@ try { throw new Error(`Missing applied audit status for ${collectionConceptId}`) } + if (appliedRow?.metadataDiff?.changed !== true || !appliedRow.metadataDiff.patch) { + throw new Error(`Missing the native metadata diff for ${collectionConceptId}`) + } + await fs.mkdir(outputDir, { recursive: true }) await fs.writeFile(outputPath, JSON.stringify({ collectionConceptId, @@ -341,7 +320,8 @@ try { statuses, updatedPlatform, response, - rows: auditRows + rows: auditRows, + appliedRow }, null, 2), 'utf8') console.log('[metadata-correction-request-delay-smoke] Completed successfully') @@ -355,6 +335,8 @@ try { outputPath }, null, 2)) } finally { + await closeDocumentDbClient() + if (redisClient) { await redisClient.quit() } diff --git a/scripts/local/run_metadata_correction_sync_smoke.mjs b/scripts/local/run_metadata_correction_sync_smoke.mjs index f9a0f946..afc28fd4 100644 --- a/scripts/local/run_metadata_correction_sync_smoke.mjs +++ b/scripts/local/run_metadata_correction_sync_smoke.mjs @@ -4,17 +4,21 @@ import { spawn } from 'node:child_process' import fs from 'node:fs/promises' import path from 'node:path' +import { closeDocumentDbClient } from '../../serverless/src/shared/documentDbClient' + +import { clearAuditDocumentsForCollection } from './metadataCorrectionSmokeHelpers.mjs' + /** * Local end-to-end smoke for the synchronous metadata-correction endpoint. * * This smoke test drives the new `runMetadataCorrection` API handler directly with an - * API-Gateway-like event. It uses the checked-in mock CMR fixture plus local Redis/RDF4J so - * the full correction path runs end to end: + * API-Gateway-like event. It uses the checked-in mock CMR fixture plus local Redis, RDF4J, and + * MongoDB so the full correction path runs end to end: * - fetch collection UMM/native metadata from the mock CMR server * - validate keyword problems against the seeded Redis caches * - resolve corrections * - apply the UMM delegate - * - persist audit rows + * - persist one audit document with lifecycle history * - write the corrected metadata back to the mock CMR ingest route * * The checked-in fixture currently resolves a `platforms` correction from @@ -177,47 +181,6 @@ const seedKeywordCaches = async () => { return redisClient } -/** - * Removes any existing audit rows for the smoke collection so assertions start clean. - * - * @returns {Promise} Resolves once prior audit rows have been deleted. - */ -const clearAuditRowsForCollection = async () => { - process.env.RDF4J_SERVICE_URL = process.env.RDF4J_SERVICE_URL || 'http://localhost:8081' - process.env.RDF4J_USER_NAME = process.env.RDF4J_USER_NAME || 'rdf4j' - process.env.RDF4J_PASSWORD = process.env.RDF4J_PASSWORD || 'rdf4j' - - const { - escapeSparqlLiteral, - METADATA_CORRECTION_AUDIT_GRAPH - } = await import('../../serverless/src/shared/metadataCorrectionAudit') - const { sparqlRequest } = await import('../../serverless/src/shared/sparqlRequest') - - const query = ` - PREFIX gcmd: - - DELETE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record ?predicate ?object . - } - } - WHERE { - GRAPH <${METADATA_CORRECTION_AUDIT_GRAPH}> { - ?record a gcmd:MetadataCorrectionAuditRecord ; - gcmd:collectionConceptId "${escapeSparqlLiteral(collectionConceptId)}" ; - ?predicate ?object . - } - } - ` - - await sparqlRequest({ - method: 'POST', - contentType: 'application/sparql-update', - accept: 'application/json', - body: query - }) -} - let mockServerProcess let redisClient @@ -241,16 +204,19 @@ try { process.env.CMR_BASE_URL = baseUrl process.env.CMR_WRITEBACK_PROVIDERS = process.env.CMR_WRITEBACK_PROVIDERS || providerId - process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'local-writer-token' + process.env.CMR_WRITER_TOKEN = process.env.CMR_WRITER_TOKEN || 'Bearer local-writer-token' redisClient = await seedKeywordCaches() - await clearAuditRowsForCollection() + await clearAuditDocumentsForCollection(collectionConceptId) const { runMetadataCorrection } = await import('../../serverless/src/runMetadataCorrection/handler') - const { getMetadataCorrectionAuditLog } = await import('../../serverless/src/shared/getMetadataCorrectionAuditLog') + const { + getMetadataCorrectionAuditByRunId, + getMetadataCorrectionAuditLog + } = await import('../../serverless/src/shared/getMetadataCorrectionAuditLog') const { getCmrCollectionNativeMetadata } = await import('../../serverless/src/shared/getCmrCollectionNativeMetadata') - const beforeRows = await getMetadataCorrectionAuditLog({ + const { items: beforeRows } = await getMetadataCorrectionAuditLog({ collectionConceptId, limit: 20 }) @@ -295,12 +261,22 @@ try { throw new Error(`Expected successful mock CMR writeback for ${collectionConceptId}`) } - if ((responseBody.auditResults?.pending?.insertedCount || 0) < 1) { - throw new Error(`Expected pending audit rows for ${collectionConceptId}`) + if (responseBody.auditResults?.pending?.status !== 'pending') { + throw new Error(`Expected pending audit status for ${collectionConceptId}`) } - if ((responseBody.auditResults?.applied?.insertedCount || 0) < 1) { - throw new Error(`Expected applied audit rows for ${collectionConceptId}`) + if (responseBody.auditResults?.applied?.status !== 'applied') { + throw new Error(`Expected applied audit status for ${collectionConceptId}`) + } + + const auditRunIds = [ + responseBody.auditResults?.checked?.runId, + responseBody.auditResults?.pending?.runId, + responseBody.auditResults?.applied?.runId + ] + + if (new Set(auditRunIds).size !== 1 || !auditRunIds[0]) { + throw new Error(`Expected one audit run across all lifecycle states for ${collectionConceptId}`) } const resolvedCorrection = responseBody.resolvedCorrections[0] @@ -335,14 +311,23 @@ try { throw new Error('Expected corrected UMM Platforms[0].Instruments[0].ShortName to remain Legacy MODIS') } - const afterRows = await getMetadataCorrectionAuditLog({ + const { items: afterRows } = await getMetadataCorrectionAuditLog({ collectionConceptId, limit: 20 }) - const statuses = [...new Set(afterRows.map((row) => row.status))] + const appliedSummary = afterRows.find(({ status }) => status === 'applied') + const appliedRow = appliedSummary + ? await getMetadataCorrectionAuditByRunId({ + runId: appliedSummary.runId, + includeDiff: true + }) + : null + const statuses = [...new Set( + appliedRow?.statusHistory?.map(({ status }) => status) || [] + )] if (beforeRows.length !== 0) { - throw new Error(`Expected no starting audit rows for ${collectionConceptId}, found ${beforeRows.length}`) + throw new Error(`Expected no starting audit documents for ${collectionConceptId}, found ${beforeRows.length}`) } if (!statuses.includes('pending')) { @@ -353,6 +338,10 @@ try { throw new Error(`Missing applied audit status for ${collectionConceptId}`) } + if (appliedRow?.metadataDiff?.changed !== true || !appliedRow.metadataDiff.patch) { + throw new Error(`Missing the native metadata diff for ${collectionConceptId}`) + } + await fs.mkdir(outputDir, { recursive: true }) await fs.writeFile(outputPath, JSON.stringify({ collectionConceptId, @@ -363,7 +352,8 @@ try { beforeCount: beforeRows.length, afterCount: afterRows.length, statuses, - rows: afterRows + rows: afterRows, + appliedRow }, null, 2), 'utf8') console.log('[metadata-correction-sync-smoke] Completed successfully') @@ -381,6 +371,8 @@ try { outputPath }, null, 2)) } finally { + await closeDocumentDbClient() + if (redisClient) { await redisClient.quit() } diff --git a/scripts/local/run_platform_keyword_event_mapping_smoke.mjs b/scripts/local/run_platform_keyword_event_mapping_smoke.mjs index eb864896..a1aa9b84 100644 --- a/scripts/local/run_platform_keyword_event_mapping_smoke.mjs +++ b/scripts/local/run_platform_keyword_event_mapping_smoke.mjs @@ -20,14 +20,14 @@ const defaultLongName = 'Greenhouse Gases Observing Satellite' const createPlatformsCsv = (shortName, longName = defaultLongName) => [ '"Keyword Version: smoke"', - '"Category","Class","Type","Short_Name","Long_Name","UUID"', - `"Platforms","Space-based Platforms","Earth Observation Satellites","${shortName}","${longName}","${keywordUuid}"` + '"Basis","Category","Sub_Category","Short_Name","Long_Name","UUID"', + `"Space-based Platforms","Earth Observation Satellites",,"${shortName}","${longName}","${keywordUuid}"` ].join('\n') const buildExpectedKeywordObject = (shortName, longName = defaultLongName) => ({ - Category: 'Platforms', - Class: 'Space-based Platforms', - Type: 'Earth Observation Satellites', + Basis: 'Space-based Platforms', + Category: 'Earth Observation Satellites', + SubCategory: '', ShortName: shortName, LongName: longName }) diff --git a/scripts/local/run_rdf_export_smoke.sh b/scripts/local/run_rdf_export_smoke.sh index 10f82d88..2ffc7169 100755 --- a/scripts/local/run_rdf_export_smoke.sh +++ b/scripts/local/run_rdf_export_smoke.sh @@ -70,11 +70,19 @@ download_export() { local download_url download_url="$(jq --exit-status --raw-output '.downloadUrl' "$response_file")" + local download_curl_args=() + if [[ "$BASE_URL" =~ ^https?://(127\.0\.0\.1|localhost)(:|/) ]] \ + && [[ "$download_url" == http://localstack:4566/* ]]; then + # Preserve the signed Host header while resolving Docker's LocalStack hostname from the host. + download_curl_args+=(--resolve 'localstack:4566:127.0.0.1') + fi + curl \ --silent \ --show-error \ --fail \ --location \ + "${download_curl_args[@]}" \ "$download_url" \ --output "$gzip_file" diff --git a/scripts/local/show_metadata_correction_audit_log.mjs b/scripts/local/show_metadata_correction_audit_log.mjs index c93ab08f..958f2561 100644 --- a/scripts/local/show_metadata_correction_audit_log.mjs +++ b/scripts/local/show_metadata_correction_audit_log.mjs @@ -1,124 +1,48 @@ #!/usr/bin/env node -/** - * Local audit-log inspector for the metadata-correction smoke flow. - * - * This script queries the RDF4J audit graph used by the metadata-correction service and prints - * a small summary table for one collection concept id. It is mainly a convenience tool for the - * local smoke test so we can quickly confirm which corrections were written, in what order, and - * with what final status. - */ -const collectionConceptId = process.env.COLLECTION_CONCEPT_ID || 'C1234567890-LOCAL' -const rdf4jUserName = process.env.RDF4J_USER_NAME || 'rdf4j' -const rdf4jPassword = process.env.RDF4J_PASSWORD || 'rdf4j' -const rdf4jServiceUrl = process.env.RDF4J_SERVICE_URL || 'http://localhost:8081' -const rdf4jRepository = process.env.RDF4J_REPOSITORY || 'kms' -const rdf4jRepositoryUrl = `${rdf4jServiceUrl.replace(/\/$/, '')}/rdf4j-server/repositories/${rdf4jRepository}` - -// Build the basic-auth header expected by the local RDF4J container. -const createAuthHeader = () => ( - `Basic ${Buffer.from(`${rdf4jUserName}:${rdf4jPassword}`).toString('base64')}` -) - -// Execute a SPARQL query against the configured RDF4J repository and return JSON bindings. -const executeSparqlQuery = async (query) => { - const response = await fetch(rdf4jRepositoryUrl, { - method: 'POST', - headers: { - Authorization: createAuthHeader(), - 'Content-Type': 'application/sparql-query', - Accept: 'application/sparql-results+json' - }, - body: query - }) - - if (!response.ok) { - const responseText = await response.text() - - throw new Error(`RDF4J query failed: ${response.status} ${responseText}`) - } - - return response.json() -} - -// Flatten SPARQL JSON bindings into plain row objects for easier post-processing. -const parseBindings = (results = []) => results.map((binding) => Object.fromEntries( - Object.entries(binding).map(([key, value]) => [key, value?.value || '']) -)) - -// Count how many audit records exist for the requested collection concept id. -const getAuditRowCount = async () => { - const responseBody = await executeSparqlQuery(` - PREFIX gcmd: - - SELECT (COUNT(?record) AS ?count) - WHERE { - GRAPH { - ?record a gcmd:MetadataCorrectionAuditRecord ; - gcmd:collectionConceptId "${collectionConceptId}" . - } - } - `) - - const parsedRows = parseBindings(responseBody?.results?.bindings || []) - - return Number(parsedRows[0]?.count || 0) -} - -// Fetch the detailed audit rows we want to display in the local smoke summary table. -const getAuditRows = async () => { - const responseBody = await executeSparqlQuery(` - PREFIX gcmd: - PREFIX dcterms: - - SELECT ?timestamp ?publishedVersionName ?collectionConceptId ?scheme ?action ?oldKeywordPath ?newKeywordPath ?status - WHERE { - GRAPH { - ?record a gcmd:MetadataCorrectionAuditRecord ; - dcterms:created ?timestamp ; - gcmd:publishedVersionName ?publishedVersionName ; - gcmd:collectionConceptId ?collectionConceptId ; - gcmd:scheme ?scheme ; - gcmd:action ?action ; - gcmd:status ?status . - OPTIONAL { ?record gcmd:oldKeywordPath ?oldKeywordPath } - OPTIONAL { ?record gcmd:newKeywordPath ?newKeywordPath } - FILTER(?collectionConceptId = "${collectionConceptId}") - } - } - ORDER BY DESC(?timestamp) - `) - - return parseBindings(responseBody?.results?.bindings || []).map((row) => ({ - timestamp: row.timestamp, - version: row.publishedVersionName, - conceptId: row.collectionConceptId, - scheme: row.scheme, - action: row.action, - status: row.status, - oldKeywordPath: row.oldKeywordPath, - newKeywordPath: row.newKeywordPath - })) -} +import { MongoClient } from 'mongodb' /** - * Queries RDF4J for audit rows and prints a simple summary table for local smoke verification. + * Prints the local DocumentDB-compatible audit documents for one collection. + * + * Input comes from `COLLECTION_CONCEPT_ID` and the `DOCUMENTDB_*` environment variables; output is + * a console table with one row per matching audit run. * - * @returns {Promise} + * @returns {Promise} Resolves after the query results have been printed and the client closes. */ const main = async () => { - const count = await getAuditRowCount() - const rows = await getAuditRows() - - console.log(`Metadata correction audit rows for ${collectionConceptId}: ${count}`) - - if (rows.length === 0) { - console.log('No audit rows found.') - - return + const collectionConceptId = process.env.COLLECTION_CONCEPT_ID || 'C1234567890-LOCAL' + const uri = process.env.DOCUMENTDB_URI + || `mongodb://localhost:${process.env.DOCUMENTDB_HOST_PORT || 27018}` + const databaseName = process.env.DOCUMENTDB_DATABASE_NAME || 'kms' + const collectionName = process.env.DOCUMENTDB_AUDIT_COLLECTION_NAME + || 'metadataCorrectionAudits' + const client = new MongoClient(uri) + + try { + await client.connect() + const documents = await client.db(databaseName) + .collection(collectionName) + .find({ collectionConceptId }) + .sort({ createdAt: -1, _id: -1 }) + .toArray() + + console.log(`Metadata correction audit runs for ${collectionConceptId}: ${documents.length}`) + console.table(documents.map((document) => ({ + runId: document.runId, + collectionUri: document.collectionUri, + createdAt: document.createdAt, + version: document.publishedVersionName, + status: document.status, + nativeFormat: document.nativeFormat, + corrections: document.corrections?.length || 0, + priorRevisionId: document.priorRevisionId, + resultingRevisionId: document.resultingRevisionId, + error: document.error?.message + }))) + } finally { + await client.close() } - - console.table(rows) } main().catch((error) => { diff --git a/serverless/src/cmrKeywordEventsListener/__tests__/handler.test.js b/serverless/src/cmrKeywordEventsListener/__tests__/handler.test.js index 79f23bd5..e0b6c981 100644 --- a/serverless/src/cmrKeywordEventsListener/__tests__/handler.test.js +++ b/serverless/src/cmrKeywordEventsListener/__tests__/handler.test.js @@ -86,6 +86,7 @@ describe('when the CMR keyword events processor is invoked', () => { EventType: 'UPDATED', Scheme: 'sciencekeywords', UUID: '1234', + VersionName: '20.1', OldKeywordObject: OLD_SCIENCE_KEYWORD_OBJECT, NewKeywordObject: NEW_SCIENCE_KEYWORD_OBJECT, Timestamp: '2026-04-21T00:00:00.000Z' @@ -138,6 +139,7 @@ describe('when the CMR keyword events processor is invoked', () => { expect(publishMetadataCorrectionRequest).toHaveBeenNthCalledWith(1, { source: 'cmrKeywordEventsListener', collectionConceptId: 'C1000000000-PROV', + publishedVersionName: '20.1', keywordEvent: { eventType: 'UPDATED', scheme: 'sciencekeywords', @@ -151,6 +153,7 @@ describe('when the CMR keyword events processor is invoked', () => { expect(publishMetadataCorrectionRequest).toHaveBeenNthCalledWith(2, { source: 'cmrKeywordEventsListener', collectionConceptId: 'C2000000000-PROV', + publishedVersionName: '20.1', keywordEvent: { eventType: 'UPDATED', scheme: 'sciencekeywords', diff --git a/serverless/src/cmrKeywordEventsListener/handler.js b/serverless/src/cmrKeywordEventsListener/handler.js index 4772479d..1bfdb832 100644 --- a/serverless/src/cmrKeywordEventsListener/handler.js +++ b/serverless/src/cmrKeywordEventsListener/handler.js @@ -36,12 +36,14 @@ const buildMetadataCorrectionRequest = (collectionConceptId, keywordEvent) => { UUID: uuid, OldKeywordObject: oldKeywordObject, NewKeywordObject: newKeywordObject, - Timestamp: timestamp + Timestamp: timestamp, + VersionName: versionName } = keywordEvent return { source: 'cmrKeywordEventsListener', collectionConceptId, + publishedVersionName: versionName, keywordEvent: { eventType, scheme, diff --git a/serverless/src/getCapabilities/__tests__/handler.test.js b/serverless/src/getCapabilities/__tests__/handler.test.js index 25a2650a..9c2913be 100644 --- a/serverless/src/getCapabilities/__tests__/handler.test.js +++ b/serverless/src/getCapabilities/__tests__/handler.test.js @@ -53,6 +53,7 @@ describe('getCapabilities', () => { expect(result.body).toContain(' { ':@': { name: 'get_metadata_correction_audit', href: '/metadata_correction_audit', - params: 'collectionConceptId=&keywordConceptUuid=&action=&scheme=&status=&limit=', + params: 'collectionConceptId=&keywordConceptUuid=&action=&scheme=&status=&nativeFormat=&publishedVersionName=&source=&startDate=&endDate=&paginationToken=&includeDiff=&format=&limit=', + action: 'GET' + } + }, + { + ':@': { + name: 'get_metadata_correction_audit_run', + href: '/metadata_correction_audit/{runId}', + params: 'includeDiff=&format=', action: 'GET' } }, diff --git a/serverless/src/getMetadataCorrectionAudit/__tests__/handler.test.js b/serverless/src/getMetadataCorrectionAudit/__tests__/handler.test.js index 5ee35c2a..e79c7386 100644 --- a/serverless/src/getMetadataCorrectionAudit/__tests__/handler.test.js +++ b/serverless/src/getMetadataCorrectionAudit/__tests__/handler.test.js @@ -6,7 +6,10 @@ import { vi } from 'vitest' -import { getMetadataCorrectionAuditLog } from '@/shared/getMetadataCorrectionAuditLog' +import { + getMetadataCorrectionAuditByRunId, + getMetadataCorrectionAuditLog +} from '@/shared/getMetadataCorrectionAuditLog' import { getMetadataCorrectionAudit } from '../handler' @@ -17,6 +20,7 @@ vi.mock('@/shared/getConfig', () => ({ })) vi.mock('@/shared/getMetadataCorrectionAuditLog', () => ({ + getMetadataCorrectionAuditByRunId: vi.fn(), getMetadataCorrectionAuditLog: vi.fn() })) @@ -35,19 +39,25 @@ describe('getMetadataCorrectionAudit', () => { vi.clearAllMocks() }) - test('returns audit rows as json', async () => { - vi.mocked(getMetadataCorrectionAuditLog).mockResolvedValue([ - { - recordUri: 'https://example.org/audit/1', + test('returns audit documents as json', async () => { + vi.mocked(getMetadataCorrectionAuditLog).mockResolvedValue({ + items: [{ + runId: 'run-1', collectionConceptId: 'C1234567890-LOCAL', - action: 'UPDATED' - } - ]) + collectionUri: 'https://cmr.example.com/search/concepts/C1234567890-LOCAL', + status: 'applied', + changes: [{ + scheme: 'platforms', + oldKeywordPath: 'Platforms > GOSAT', + newKeywordPath: 'Platforms > GOSAT - Test1' + }] + }], + nextPaginationToken: null + }) const result = await getMetadataCorrectionAudit({ queryStringParameters: { collectionConceptId: 'C1234567890-LOCAL', - latestOnly: 'true', limit: '10' } }) @@ -56,9 +66,15 @@ describe('getMetadataCorrectionAudit', () => { collectionConceptId: 'C1234567890-LOCAL', keywordConceptUuid: undefined, action: undefined, + paginationToken: undefined, + endDate: undefined, + includeDiff: undefined, scheme: undefined, status: undefined, - latestOnly: 'true', + nativeFormat: undefined, + publishedVersionName: undefined, + source: undefined, + startDate: undefined, limit: '10' }) @@ -68,11 +84,143 @@ describe('getMetadataCorrectionAudit', () => { expect(JSON.parse(result.body)).toEqual({ items: [ { - recordUri: 'https://example.org/audit/1', + runId: 'run-1', collectionConceptId: 'C1234567890-LOCAL', - action: 'UPDATED' + collectionUri: 'https://cmr.example.com/search/concepts/C1234567890-LOCAL', + status: 'applied', + changes: [{ + scheme: 'platforms', + oldKeywordPath: 'Platforms > GOSAT', + newKeywordPath: 'Platforms > GOSAT - Test1' + }] } - ] + ], + nextPaginationToken: null + }) + }) + + test('requests native metadata diffs in list results when requested', async () => { + vi.mocked(getMetadataCorrectionAuditLog).mockResolvedValue({ + items: [], + nextPaginationToken: null + }) + + await getMetadataCorrectionAudit({ + queryStringParameters: { includeDiff: 'true' } + }) + + expect(getMetadataCorrectionAuditLog).toHaveBeenCalledWith(expect.objectContaining({ + includeDiff: 'true' + })) + }) + + test('renders a compact html summary with detail links for browser requests', async () => { + vi.mocked(getMetadataCorrectionAuditLog).mockResolvedValue({ + items: [{ + runId: 'run-html', + collectionConceptId: 'C1234567890-LOCAL', + status: 'applied', + changes: [{ + scheme: 'platforms', + action: 'UPDATED', + oldKeywordPath: 'Platforms > GOSAT', + newKeywordPath: 'Platforms > GOSAT - Test1' + }] + }], + nextPaginationToken: 'next-token' + }) + + const result = await getMetadataCorrectionAudit({ + queryStringParameters: { + collectionConceptId: 'C1234567890-LOCAL', + format: 'html' + } + }) + + expect(getMetadataCorrectionAuditLog).toHaveBeenCalledWith(expect.objectContaining({ + collectionConceptId: 'C1234567890-LOCAL', + includeDiff: undefined, + limit: '10' + })) + + expect(result.statusCode).toBe(200) + expect(result.headers['Content-Type']).toBe('text/html; charset=utf-8') + expect(result.headers['Cache-Control']).toBe('no-store') + expect(result.headers['Content-Security-Policy']).toContain("form-action 'self'") + expect(result.body).toContain('
') + expect(result.body).toContain('name="collectionConceptId" value="C1234567890-LOCAL"') + expect(result.body).toContain('') + expect(result.body).toContain('run-html') + expect(result.body).not.toContain('Native metadata diff') + expect(result.body).toContain('paginationToken=next-token') + }) + + test('renders one detailed audit document as html', async () => { + vi.mocked(getMetadataCorrectionAuditByRunId).mockResolvedValue({ + runId: 'run-1', + collectionConceptId: 'C123-PROV', + providerId: 'PROV', + priorRevisionId: 3, + resultingRevisionId: 4, + status: 'applied', + corrections: [], + statusHistory: [{ + status: 'applied', + timestamp: '2026-09-08T12:00:00.000Z' + }] + }) + + const result = await getMetadataCorrectionAudit({ + pathParameters: { runId: 'run-1' }, + queryStringParameters: { format: 'html' } + }) + + expect(getMetadataCorrectionAuditByRunId).toHaveBeenCalledWith({ + runId: 'run-1', + includeDiff: true + }) + + expect(result.statusCode).toBe(200) + expect(result.headers['Content-Type']).toBe('text/html; charset=utf-8') + expect(result.body).toContain('Metadata correction audit detail') + expect(result.body).toContain('Run details') + expect(result.body).toContain('Prior CMR revision') + expect(result.body).toContain('Lifecycle history') + expect(result.body).toContain('Native metadata diff') + expect(result.body).not.toContain('class="audit-filter"') + expect(result.body).not.toContain('metadata_correction_audit/run-1?format=html') + }) + + test('renders missing detail and server errors as html', async () => { + vi.mocked(getMetadataCorrectionAuditByRunId).mockResolvedValue(null) + + const missingResult = await getMetadataCorrectionAudit({ + pathParameters: { runId: 'missing-run' }, + queryStringParameters: { format: 'html' } + }) + + expect(missingResult.statusCode).toBe(404) + expect(missingResult.body).toContain('Metadata correction audit run not found: missing-run') + + vi.mocked(getMetadataCorrectionAuditLog).mockRejectedValue(new Error('DocumentDB unavailable')) + + const errorResult = await getMetadataCorrectionAudit({ + queryStringParameters: { format: 'html' } + }) + + expect(errorResult.statusCode).toBe(500) + expect(errorResult.headers['Content-Type']).toBe('text/html; charset=utf-8') + expect(errorResult.body).toContain('Error: DocumentDB unavailable') + }) + + test('returns 400 for an unsupported response format', async () => { + const result = await getMetadataCorrectionAudit({ + queryStringParameters: { format: 'xml' } + }) + + expect(result.statusCode).toBe(400) + expect(JSON.parse(result.body)).toEqual({ + error: 'Error: Invalid metadata correction audit format: expected json or html' }) }) @@ -87,4 +235,65 @@ describe('getMetadataCorrectionAudit', () => { error: 'Error: Audit query failed' }) }) + + test('returns 400 for invalid filters', async () => { + vi.mocked(getMetadataCorrectionAuditLog).mockRejectedValue( + new Error('Invalid metadata correction audit paginationToken') + ) + + const result = await getMetadataCorrectionAudit({}) + + expect(result.statusCode).toBe(400) + }) + + test('returns one detailed audit document with its native metadata diff', async () => { + vi.mocked(getMetadataCorrectionAuditByRunId).mockResolvedValue({ + runId: 'run-1', + status: 'applied', + metadataDiff: { + changed: true, + patch: '-old\n+new' + } + }) + + const result = await getMetadataCorrectionAudit({ + pathParameters: { runId: 'run-1' }, + queryStringParameters: { includeDiff: 'true' } + }) + + expect(getMetadataCorrectionAuditByRunId).toHaveBeenCalledWith({ + runId: 'run-1', + includeDiff: 'true' + }) + + expect(getMetadataCorrectionAuditLog).not.toHaveBeenCalled() + expect(result.statusCode).toBe(200) + expect(JSON.parse(result.body).metadataDiff.patch).toBe('-old\n+new') + }) + + test('returns 404 when a detailed audit run does not exist', async () => { + vi.mocked(getMetadataCorrectionAuditByRunId).mockResolvedValue(null) + + const result = await getMetadataCorrectionAudit({ + pathParameters: { runId: 'missing-run' } + }) + + expect(result.statusCode).toBe(404) + expect(JSON.parse(result.body)).toEqual({ + error: 'Metadata correction audit run not found: missing-run' + }) + }) + + test('returns 400 when the detail diff flag is invalid', async () => { + vi.mocked(getMetadataCorrectionAuditByRunId).mockRejectedValue( + new Error('Invalid metadata correction audit includeDiff: expected true or false') + ) + + const result = await getMetadataCorrectionAudit({ + pathParameters: { runId: 'run-1' }, + queryStringParameters: { includeDiff: 'yes' } + }) + + expect(result.statusCode).toBe(400) + }) }) diff --git a/serverless/src/getMetadataCorrectionAudit/handler.js b/serverless/src/getMetadataCorrectionAudit/handler.js index c01a71e6..1833bf28 100644 --- a/serverless/src/getMetadataCorrectionAudit/handler.js +++ b/serverless/src/getMetadataCorrectionAudit/handler.js @@ -1,15 +1,58 @@ import { getApplicationConfig } from '@/shared/getConfig' -import { getMetadataCorrectionAuditLog } from '@/shared/getMetadataCorrectionAuditLog' +import { + getMetadataCorrectionAuditByRunId, + getMetadataCorrectionAuditLog +} from '@/shared/getMetadataCorrectionAuditLog' import { logAnalyticsData } from '@/shared/logAnalyticsData' import { logger } from '@/shared/logger' +import { renderMetadataCorrectionAuditHtml } from '@/shared/renderMetadataCorrectionAuditHtml' + +/** + * Validates the requested audit response representation. + * + * @param {unknown} format Requested response format. + * @returns {'json'|'html'} Normalized response format. + */ +const normalizeResponseFormat = (format) => { + if (format === undefined || format === null || format === '' || format === 'json') return 'json' + if (format === 'html') return 'html' + + throw new Error('Invalid metadata correction audit format: expected json or html') +} + +/** + * Preserves the current filters while advancing an HTML audit search to its next page. + * + * @param {Object} queryStringParameters Current API query parameters. + * @param {string|null} paginationToken Opaque next-page token. + * @returns {string|undefined} Relative next-page URL when another page exists. + */ +const buildNextPageHref = (queryStringParameters, paginationToken) => { + if (!paginationToken) return undefined + + const parameters = new URLSearchParams() + Object.entries(queryStringParameters || {}).forEach(([key, value]) => { + if (value !== undefined && value !== null) parameters.set(key, String(value)) + }) + + parameters.set('format', 'html') + parameters.set('paginationToken', paginationToken) + + return `?${parameters.toString()}` +} + +const HTML_RESPONSE_HEADERS = { + 'Cache-Control': 'no-store', + 'Content-Security-Policy': "default-src 'none'; style-src 'unsafe-inline'; base-uri 'none'; frame-ancestors 'none'; form-action 'self'", + 'Content-Type': 'text/html; charset=utf-8', + 'X-Content-Type-Options': 'nosniff' +} /** * Read-side audit endpoint for metadata-correction activity. * - * The metadata-correction service writes one audit record per resolved correction into RDF4J. - * This handler exposes those records through an API so we can inspect what corrections were - * attempted, which collection they applied to, what keyword uuid/path was involved, and whether - * the result is still pending or has been applied. + * The metadata-correction service stores one DocumentDB document per collection-correction run. + * This handler exposes those runs for MMT audit search, reporting, and troubleshooting. * * In practice this is useful for: * - local smoke-test verification @@ -18,7 +61,7 @@ import { logger } from '@/shared/logger' */ /** - * Retrieves metadata-correction audit rows from RDF4J. + * Retrieves metadata-correction audit runs from DocumentDB. * * Supported query parameters: * - collectionConceptId @@ -26,12 +69,28 @@ import { logger } from '@/shared/logger' * - action * - scheme * - status - * - latestOnly + * - nativeFormat + * - publishedVersionName + * - source + * - startDate / endDate + * - paginationToken + * - includeDiff + * - format (`json` or `html`) * - limit * + * Add `includeDiff=true` to a list or detail request to include native-metadata patches. + * HTML detail responses include native-metadata patches automatically. + * A `runId` path parameter returns one detailed audit document. + * * @param {object} event - API Gateway event. * @param {object} context - Lambda context. * @returns {Promise} API Gateway response object. + * + * @example + * await getMetadataCorrectionAudit({ + * queryStringParameters: { status: 'applied', limit: '25' } + * }, context) + * // { statusCode: 200, body: '{"items":[...],"nextPaginationToken":null}' } */ export const getMetadataCorrectionAudit = async (event, context) => { const { defaultResponseHeaders } = getApplicationConfig() @@ -47,40 +106,132 @@ export const getMetadataCorrectionAudit = async (event, context) => { action, scheme, status, - latestOnly, + nativeFormat, + publishedVersionName, + source, + startDate, + endDate, + paginationToken, + includeDiff, + format, limit } = event?.queryStringParameters || {} + const runId = event?.pathParameters?.runId + let responseFormat = 'json' try { - const items = await getMetadataCorrectionAuditLog({ + responseFormat = normalizeResponseFormat(format) + const requestedIncludeDiff = responseFormat === 'html' && runId ? true : includeDiff + const requestedLimit = responseFormat === 'html' && !limit ? '10' : limit + + if (runId) { + const auditDocument = await getMetadataCorrectionAuditByRunId({ + runId, + includeDiff: requestedIncludeDiff + }) + + if (responseFormat === 'html') { + return { + statusCode: auditDocument ? 200 : 404, + headers: { + ...defaultResponseHeaders, + ...HTML_RESPONSE_HEADERS + }, + body: renderMetadataCorrectionAuditHtml({ + detail: true, + items: auditDocument ? [auditDocument] : [], + message: auditDocument + ? undefined + : `Metadata correction audit run not found: ${runId}`, + title: 'Metadata correction audit detail' + }) + } + } + + return { + statusCode: auditDocument ? 200 : 404, + headers: { + ...defaultResponseHeaders, + 'Content-Type': 'application/json' + }, + body: JSON.stringify( + auditDocument || { error: `Metadata correction audit run not found: ${runId}` }, + null, + 2 + ) + } + } + + const auditPage = await getMetadataCorrectionAuditLog({ collectionConceptId, keywordConceptUuid, action, scheme, status, - latestOnly, - limit + nativeFormat, + publishedVersionName, + source, + startDate, + endDate, + paginationToken, + includeDiff: requestedIncludeDiff, + limit: requestedLimit }) + if (responseFormat === 'html') { + return { + statusCode: 200, + headers: { + ...defaultResponseHeaders, + ...HTML_RESPONSE_HEADERS + }, + body: renderMetadataCorrectionAuditHtml({ + collectionConceptId, + items: auditPage.items, + nextPageHref: buildNextPageHref( + event?.queryStringParameters, + auditPage.nextPaginationToken + ) + }) + } + } + return { statusCode: 200, headers: { ...defaultResponseHeaders, 'Content-Type': 'application/json' }, - body: JSON.stringify({ - items - }, null, 2) + body: JSON.stringify(auditPage, null, 2) } } catch (error) { logger.error(`Error retrieving metadata correction audit log, error=${error.toString()}`) + const isClientError = String(error?.message || '') + .startsWith('Invalid metadata correction audit') + + const statusCode = isClientError ? 400 : 500 + + if (responseFormat === 'html') { + return { + headers: { + ...defaultResponseHeaders, + ...HTML_RESPONSE_HEADERS + }, + statusCode, + body: renderMetadataCorrectionAuditHtml({ + message: error.toString(), + title: 'Metadata correction audit error' + }) + } + } + return { headers: { ...defaultResponseHeaders, 'Content-Type': 'application/json' }, - statusCode: 500, + statusCode, body: JSON.stringify({ error: error.toString() }) diff --git a/serverless/src/initializeMetadataCorrectionAudit/__tests__/handler.test.js b/serverless/src/initializeMetadataCorrectionAudit/__tests__/handler.test.js new file mode 100644 index 00000000..e54676bb --- /dev/null +++ b/serverless/src/initializeMetadataCorrectionAudit/__tests__/handler.test.js @@ -0,0 +1,142 @@ +import { + afterEach, + beforeEach, + describe, + expect, + test, + vi +} from 'vitest' + +import { getMetadataCorrectionAuditCollection } from '@/shared/documentDbClient' + +import { initializeMetadataCorrectionAudit } from '../handler' + +vi.mock('@/shared/documentDbClient', () => ({ + getMetadataCorrectionAuditCollection: vi.fn() +})) + +describe('initializeMetadataCorrectionAudit', () => { + const createIndexes = vi.fn() + const consoleLog = vi.spyOn(console, 'log').mockImplementation(() => {}) + const consoleWarn = vi.spyOn(console, 'warn').mockImplementation(() => {}) + const indexDefinitions = [ + { + key: { + createdAt: -1, + _id: -1 + }, + name: 'createdAt_desc' + } + ] + + beforeEach(() => { + vi.clearAllMocks() + createIndexes.mockResolvedValue(['createdAt_desc']) + vi.mocked(getMetadataCorrectionAuditCollection).mockResolvedValue({ createIndexes }) + }) + + afterEach(() => { + vi.useRealTimers() + }) + + test.each(['Create', 'Update'])('creates indexes for a %s deployment event', async (requestType) => { + const result = await initializeMetadataCorrectionAudit({ + RequestType: requestType, + ResourceProperties: { IndexDefinitions: JSON.stringify(indexDefinitions) } + }) + + expect(createIndexes).toHaveBeenCalledWith(indexDefinitions) + expect(createIndexes.mock.calls[0][0][0].key.createdAt).toBeTypeOf('number') + expect(consoleLog).toHaveBeenCalledWith( + 'Metadata correction audit indexes are ready', + { indexNames: ['createdAt_desc'] } + ) + + expect(result).toEqual({ + PhysicalResourceId: 'metadata-correction-audit-indexes', + Data: { IndexCount: 1 } + }) + }) + + test('does not change the retained database during stack deletion', async () => { + const result = await initializeMetadataCorrectionAudit({ + RequestType: 'Delete', + PhysicalResourceId: 'existing-audit-indexes' + }) + + expect(getMetadataCorrectionAuditCollection).not.toHaveBeenCalled() + expect(result).toEqual({ PhysicalResourceId: 'existing-audit-indexes' }) + }) + + test('rejects a deployment without index definitions', async () => { + await expect(initializeMetadataCorrectionAudit({ + RequestType: 'Create', + ResourceProperties: {} + })).rejects.toThrow('Metadata correction audit index definitions are required') + + expect(getMetadataCorrectionAuditCollection).not.toHaveBeenCalled() + }) + + test('retries while the new DocumentDB endpoint is not resolvable', async () => { + vi.useFakeTimers() + + vi.mocked(getMetadataCorrectionAuditCollection) + .mockRejectedValueOnce(new Error('getaddrinfo ENOTFOUND audit.cluster.example')) + .mockResolvedValue({ createIndexes }) + + const resultPromise = initializeMetadataCorrectionAudit({ + RequestType: 'Create', + ResourceProperties: { IndexDefinitions: indexDefinitions } + }) + + await vi.advanceTimersByTimeAsync(5_000) + + await expect(resultPromise).resolves.toEqual({ + PhysicalResourceId: 'metadata-correction-audit-indexes', + Data: { IndexCount: 1 } + }) + + expect(getMetadataCorrectionAuditCollection).toHaveBeenCalledTimes(2) + expect(consoleWarn).toHaveBeenCalledWith( + 'DocumentDB endpoint is not ready; retrying audit index creation', + { + attempt: 1, + error: 'Error: getaddrinfo ENOTFOUND audit.cluster.example' + } + ) + }) + + test('does not retry non-connection failures', async () => { + vi.mocked(getMetadataCorrectionAuditCollection) + .mockRejectedValue(new Error('DocumentDB secret is missing username')) + + await expect(initializeMetadataCorrectionAudit({ + RequestType: 'Create', + ResourceProperties: { IndexDefinitions: indexDefinitions } + })).rejects.toThrow('DocumentDB secret is missing username') + + expect(getMetadataCorrectionAuditCollection).toHaveBeenCalledTimes(1) + expect(consoleWarn).not.toHaveBeenCalled() + }) + + test('stops retrying when the DocumentDB readiness window expires', async () => { + vi.useFakeTimers() + + vi.mocked(getMetadataCorrectionAuditCollection) + .mockRejectedValue(new Error('connect ETIMEDOUT audit.cluster.example')) + + const resultPromise = initializeMetadataCorrectionAudit({ + RequestType: 'Create', + ResourceProperties: { IndexDefinitions: indexDefinitions } + }) + const rejection = expect(resultPromise).rejects.toThrow( + 'connect ETIMEDOUT audit.cluster.example' + ) + + await vi.runAllTimersAsync() + await rejection + + expect(getMetadataCorrectionAuditCollection).toHaveBeenCalledTimes(24) + expect(consoleWarn).toHaveBeenCalledTimes(23) + }) +}) diff --git a/serverless/src/initializeMetadataCorrectionAudit/handler.js b/serverless/src/initializeMetadataCorrectionAudit/handler.js new file mode 100644 index 00000000..7e9cf562 --- /dev/null +++ b/serverless/src/initializeMetadataCorrectionAudit/handler.js @@ -0,0 +1,106 @@ +import { getMetadataCorrectionAuditCollection } from '@/shared/documentDbClient' + +const PHYSICAL_RESOURCE_ID = 'metadata-correction-audit-indexes' +const INDEX_CREATION_MAX_ATTEMPTS = 24 +const INDEX_CREATION_RETRY_DELAY_MS = 5_000 +const RETRYABLE_CONNECTION_ERROR_CODES = [ + 'EAI_AGAIN', + 'ECONNREFUSED', + 'ENOTFOUND', + 'ETIMEDOUT' +] + +/** + * Returns whether DocumentDB is still becoming reachable after cluster creation. + * + * @example + * isRetryableConnectionError(new Error('getaddrinfo ENOTFOUND cluster.example')) // true + * + * @param {unknown} error Connection error from the MongoDB driver. + * @returns {boolean} Whether retrying may succeed once the endpoint is ready. + */ +const isRetryableConnectionError = (error) => { + const errorMessage = String(error) + + return RETRYABLE_CONNECTION_ERROR_CODES.some((code) => errorMessage.includes(code)) +} + +/** + * Creates the configured indexes, retrying while a new DocumentDB endpoint becomes reachable. + * + * @param {Array} indexDefinitions MongoDB index definitions. + * @param {number} attempt Current connection attempt. + * @returns {Promise>} Names returned by MongoDB for the created indexes. + */ +const createAuditIndexes = async (indexDefinitions, attempt = 1) => { + try { + const collection = await getMetadataCorrectionAuditCollection() + + return await collection.createIndexes(indexDefinitions) + } catch (error) { + if ( + !isRetryableConnectionError(error) + || attempt >= INDEX_CREATION_MAX_ATTEMPTS + ) { + throw error + } + + console.warn('DocumentDB endpoint is not ready; retrying audit index creation', { + attempt, + error: String(error) + }) + + await new Promise((resolve) => { + setTimeout(resolve, INDEX_CREATION_RETRY_DELAY_MS) + }) + + return createAuditIndexes(indexDefinitions, attempt + 1) + } +} + +/** + * Creates the metadata-correction audit indexes during CloudFormation deployment. + * Delete events intentionally leave indexes in place because the audit database is retained. + * + * @example + * await initializeMetadataCorrectionAudit({ + * RequestType: 'Create', + * ResourceProperties: { + * IndexDefinitions: '[{"key":{"createdAt":-1},"name":"createdAt_desc"}]' + * } + * }) + * // { PhysicalResourceId: 'metadata-correction-audit-indexes', Data: { IndexCount: 1 } } + * + * @param {Object} event CloudFormation custom-resource event. + * @returns {Promise} Stable resource identity and deployment details. + */ +export const initializeMetadataCorrectionAudit = async (event) => { + // The custom-resource provider forwards CloudFormation Create, Update, and Delete events here. + const physicalResourceId = event.PhysicalResourceId || PHYSICAL_RESOURCE_ID + + // The DocumentDB cluster is retained, so stack deletion must not remove its data or indexes. + if (event.RequestType === 'Delete') { + return { PhysicalResourceId: physicalResourceId } + } + + const indexDefinitionsProperty = event.ResourceProperties?.IndexDefinitions + const indexDefinitions = typeof indexDefinitionsProperty === 'string' + ? JSON.parse(indexDefinitionsProperty) + : indexDefinitionsProperty + + if (!Array.isArray(indexDefinitions) || indexDefinitions.length === 0) { + throw new Error('Metadata correction audit index definitions are required') + } + + const indexNames = await createAuditIndexes(indexDefinitions) + console.log('Metadata correction audit indexes are ready', { indexNames }) + + return { + PhysicalResourceId: physicalResourceId, + Data: { + IndexCount: indexDefinitions.length + } + } +} + +export default initializeMetadataCorrectionAudit diff --git a/serverless/src/metadataCorrectionService/__tests__/handler.test.js b/serverless/src/metadataCorrectionService/__tests__/handler.test.js index 6b6f359c..2b20bc94 100644 --- a/serverless/src/metadataCorrectionService/__tests__/handler.test.js +++ b/serverless/src/metadataCorrectionService/__tests__/handler.test.js @@ -124,6 +124,7 @@ describe('when the metadata correction service is invoked', () => { beforeEach(() => { vi.clearAllMocks() delete process.env.METADATA_CORRECTION_REQUEST_DELAY_MS + delete process.env.METADATA_CORRECTION_RUNS_PER_MINUTE vi.mocked(delay).mockResolvedValue(undefined) vi.mocked(emitConsumerMetricsSafely).mockResolvedValue(undefined) @@ -146,11 +147,14 @@ describe('when the metadata correction service is invoked', () => { correctedMetadataBytes: 21 }) - vi.mocked(persistMetadataCorrectionAuditLog).mockResolvedValue({ - insertedCount: 1, - publishedVersionName: 'published', - status: 'pending' - }) + vi.mocked(persistMetadataCorrectionAuditLog).mockImplementation(async ({ + runId = 'audit-run-1', + status + }) => ({ + runId, + status, + created: status === 'checked' + })) }) describe('when the invocation is successful', () => { @@ -240,6 +244,7 @@ describe('when the metadata correction service is invoked', () => { body: JSON.stringify({ source: 'cmrKeywordEventsListener', collectionConceptId: 'C123-PROV', + publishedVersionName: '20.1', keywordEvent: { eventType: 'UPDATED', scheme: 'sciencekeywords', @@ -304,7 +309,7 @@ describe('when the metadata correction service is invoked', () => { ] }) - expect(persistMetadataCorrectionAuditLog).toHaveBeenCalledWith({ + expect(persistMetadataCorrectionAuditLog).toHaveBeenNthCalledWith(2, expect.objectContaining({ collectionConceptId: 'C123-PROV', keywordEvent: { eventType: 'UPDATED', @@ -315,6 +320,7 @@ describe('when the metadata correction service is invoked', () => { }, nativeFormat: 'DIF10', delegateName: 'dif10', + publishedVersionName: '20.1', corrections: [ { scheme: 'sciencekeywords', @@ -325,8 +331,9 @@ describe('when the metadata correction service is invoked', () => { ummPath: ['ScienceKeywords', 0] } ], + runId: 'message-collection-1', status: 'pending' - }) + })) expect(writeCorrectedMetadataToCmr).toHaveBeenCalledWith({ collectionConceptId: 'C123-PROV', @@ -350,7 +357,7 @@ describe('when the metadata correction service is invoked', () => { }) }) - test('should append an applied audit record after a successful writeback update', async () => { + test('should update the audit document to applied after a successful writeback', async () => { vi.mocked(getCmrCollectionUmmDetails).mockResolvedValue({ collectionConceptId: 'C123-PROV', providerId: 'PROV', @@ -457,14 +464,14 @@ describe('when the metadata correction service is invoked', () => { ] }) - expect(persistMetadataCorrectionAuditLog).toHaveBeenNthCalledWith(1, expect.objectContaining({ + expect(persistMetadataCorrectionAuditLog).toHaveBeenNthCalledWith(2, expect.objectContaining({ collectionConceptId: 'C123-PROV', nativeFormat: 'DIF10', delegateName: 'dif10', status: 'pending' })) - expect(persistMetadataCorrectionAuditLog).toHaveBeenNthCalledWith(2, expect.objectContaining({ + expect(persistMetadataCorrectionAuditLog).toHaveBeenNthCalledWith(3, expect.objectContaining({ collectionConceptId: 'C123-PROV', nativeFormat: 'DIF10', delegateName: 'dif10', @@ -472,7 +479,7 @@ describe('when the metadata correction service is invoked', () => { })) }) - test('should fall back to the normalized native format when applied audit delegateName is absent', async () => { + test('should use the normalized native format as delegateName when updating the audit document to applied', async () => { vi.mocked(getCmrCollectionUmmDetails).mockResolvedValue({ collectionConceptId: 'C123-PROV', providerId: 'PROV', @@ -578,7 +585,7 @@ describe('when the metadata correction service is invoked', () => { ] }) - expect(persistMetadataCorrectionAuditLog).toHaveBeenNthCalledWith(2, expect.objectContaining({ + expect(persistMetadataCorrectionAuditLog).toHaveBeenNthCalledWith(3, expect.objectContaining({ collectionConceptId: 'C123-PROV', nativeFormat: 'DIF10', delegateName: 'dif10', @@ -893,7 +900,14 @@ describe('when the metadata correction service is invoked', () => { }) expect(invokeMetadataCorrectionDelegate).not.toHaveBeenCalled() - expect(persistMetadataCorrectionAuditLog).not.toHaveBeenCalled() + expect(persistMetadataCorrectionAuditLog).toHaveBeenCalledOnce() + expect(persistMetadataCorrectionAuditLog).toHaveBeenCalledWith( + expect.objectContaining({ + outcome: 'no-resolved-corrections', + status: 'checked' + }) + ) + expect(writeCorrectedMetadataToCmr).not.toHaveBeenCalled() expect(logger.info).toHaveBeenCalledWith( @@ -952,7 +966,14 @@ describe('when the metadata correction service is invoked', () => { expect(resolveOldKeywordConceptUuid).not.toHaveBeenCalled() expect(invokeMetadataCorrectionDelegate).not.toHaveBeenCalled() - expect(persistMetadataCorrectionAuditLog).not.toHaveBeenCalled() + expect(persistMetadataCorrectionAuditLog).toHaveBeenCalledOnce() + expect(persistMetadataCorrectionAuditLog).toHaveBeenCalledWith( + expect.objectContaining({ + outcome: 'no-keyword-issues', + status: 'checked' + }) + ) + expect(writeCorrectedMetadataToCmr).not.toHaveBeenCalled() expect(logger.info).toHaveBeenCalledWith( @@ -1219,6 +1240,52 @@ describe('when the metadata correction service is invoked', () => { })) }) + test('should pace correction runs when a per-minute rate is configured', async () => { + process.env.METADATA_CORRECTION_RUNS_PER_MINUTE = '10' + vi.mocked(getCmrCollectionUmmDetails).mockResolvedValue({ + collectionConceptId: 'C1234567890-PROV', + providerId: 'PROV', + nativeId: 'native-123', + revisionId: 7, + format: 'application/dif10+xml', + umm: {} + }) + + vi.mocked(validateCmrCollectionUmm).mockResolvedValue({ + status: 200, + errors: [], + warnings: [], + responseBody: { + errors: [], + warnings: [] + } + }) + + vi.mocked(extractKeywordValidationFailures).mockReturnValue([]) + + await metadataCorrectionService({ + Records: [{ + messageId: 'message-paced', + body: JSON.stringify({ + source: 'cmrKeywordEventsListener', + collectionConceptId: 'C1234567890-PROV' + }) + }] + }) + + expect(delay).toHaveBeenCalledOnce() + expect(delay).toHaveBeenCalledWith(6000) + expect(logger.info).toHaveBeenCalledWith( + '[metadata-correction] Pacing queued metadata correction request', + { + collectionConceptId: 'C1234567890-PROV', + messageId: 'message-paced', + pacingDelayMs: 6000, + runsPerMinute: 10 + } + ) + }) + test('should delay queued manual api requests when configured before running correction', async () => { const dateNowSpy = vi.spyOn(Date, 'now').mockReturnValue(10_000) process.env.METADATA_CORRECTION_REQUEST_DELAY_MS = '1500' diff --git a/serverless/src/metadataCorrectionService/handler.js b/serverless/src/metadataCorrectionService/handler.js index 81d63439..4e1f938b 100644 --- a/serverless/src/metadataCorrectionService/handler.js +++ b/serverless/src/metadataCorrectionService/handler.js @@ -41,6 +41,50 @@ const buildBatchProcessingMetrics = ({ } const MAX_METADATA_CORRECTION_REQUEST_DELAY_MS = 20_000 +const MILLISECONDS_PER_MINUTE = 60_000 + +/** + * Converts the configured correction-run rate into a per-message pacing delay. + * + * @example + * // With METADATA_CORRECTION_RUNS_PER_MINUTE=10 + * getCorrectionRunPacingDelayMs() // 6000 + * + * @returns {number} Delay before each queued correction run, or zero when pacing is disabled. + */ +const getCorrectionRunPacingDelayMs = () => { + const runsPerMinute = Number(process.env.METADATA_CORRECTION_RUNS_PER_MINUTE) + + if (!Number.isInteger(runsPerMinute) || runsPerMinute <= 0) return 0 + + return Math.ceil(MILLISECONDS_PER_MINUTE / runsPerMinute) +} + +/** + * Holds the single configured consumer slot long enough to cap correction-run throughput. + * + * @param {Object} params Pacing log context. + * @param {string|undefined} params.collectionConceptId Collection being processed. + * @param {string|undefined} params.messageId SQS message identifier. + * @returns {Promise} + */ +const paceQueuedCorrectionRunIfNeeded = async ({ + collectionConceptId, + messageId +}) => { + const pacingDelayMs = getCorrectionRunPacingDelayMs() + + if (pacingDelayMs <= 0) return + + logger.info('[metadata-correction] Pacing queued metadata correction request', { + collectionConceptId, + messageId, + pacingDelayMs, + runsPerMinute: Number(process.env.METADATA_CORRECTION_RUNS_PER_MINUTE) + }) + + await delay(pacingDelayMs) +} /** * Reads the optional async correction request delay from environment configuration. @@ -130,6 +174,11 @@ export const metadataCorrectionService = async (event) => { metadataCorrectionRequest }) + await paceQueuedCorrectionRunIfNeeded({ + collectionConceptId: metadataCorrectionRequest.collectionConceptId, + messageId: record.messageId + }) + await delayQueuedManualRequestIfNeeded({ collectionConceptId: metadataCorrectionRequest.collectionConceptId, messageId: record.messageId, @@ -140,6 +189,7 @@ export const metadataCorrectionService = async (event) => { collectionConceptId: metadataCorrectionRequest.collectionConceptId, keywordEvent: metadataCorrectionRequest.keywordEvent, messageId: record.messageId, + publishedVersionName: metadataCorrectionRequest.publishedVersionName, source: metadataCorrectionRequest.source }) } catch (error) { diff --git a/serverless/src/publisher/__tests__/handler.test.js b/serverless/src/publisher/__tests__/handler.test.js index 6dda78d0..a32b2fe2 100644 --- a/serverless/src/publisher/__tests__/handler.test.js +++ b/serverless/src/publisher/__tests__/handler.test.js @@ -214,6 +214,7 @@ describe('publisher handler', () => { EventType: 'INSERTED', Scheme: 'sciencekeywords', UUID: 'uuid1', + VersionName: 'v1.0.0', NewKeywordObject: SCIENCE_PATH_KEYWORD })) diff --git a/serverless/src/publisher/handler.js b/serverless/src/publisher/handler.js index efc785dc..cec030cf 100644 --- a/serverless/src/publisher/handler.js +++ b/serverless/src/publisher/handler.js @@ -146,7 +146,10 @@ const publishKeywordEvents = async (keywordEvents, versionName) => keywordEvents } // eslint-disable-next-line no-await-in-loop - const publishResult = await publishKeywordEvent(keywordEvent) + const publishResult = await publishKeywordEvent({ + ...keywordEvent, + VersionName: versionName + }) logger.info('[publisher] Published keyword event', { versionName, diff --git a/serverless/src/requestMetadataCorrection/__tests__/handler.test.js b/serverless/src/requestMetadataCorrection/__tests__/handler.test.js index 7bbccf77..1d0b8b5e 100644 --- a/serverless/src/requestMetadataCorrection/__tests__/handler.test.js +++ b/serverless/src/requestMetadataCorrection/__tests__/handler.test.js @@ -6,6 +6,7 @@ import { vi } from 'vitest' +import { getVersionMetadata } from '@/shared/getVersionMetadata' import { logger } from '@/shared/logger' import { publishMetadataCorrectionRequest } from '@/shared/publishMetadataCorrectionRequest' @@ -21,6 +22,10 @@ vi.mock('@/shared/logAnalyticsData', () => ({ logAnalyticsData: vi.fn() })) +vi.mock('@/shared/getVersionMetadata', () => ({ + getVersionMetadata: vi.fn() +})) + vi.mock('@/shared/logger', () => ({ logger: { info: vi.fn(), @@ -35,6 +40,7 @@ vi.mock('@/shared/publishMetadataCorrectionRequest', () => ({ describe('requestMetadataCorrection', () => { beforeEach(() => { vi.clearAllMocks() + vi.mocked(getVersionMetadata).mockResolvedValue({ versionName: '20.1' }) }) test('returns 202 and publishes one deduplicated message per collection concept id', async () => { @@ -62,12 +68,14 @@ describe('requestMetadataCorrection', () => { expect(publishMetadataCorrectionRequest).toHaveBeenNthCalledWith(1, { source: 'metadataCorrectionApi', collectionConceptId: 'C123-PROV', + publishedVersionName: '20.1', requestedAt: expect.any(String) }) expect(publishMetadataCorrectionRequest).toHaveBeenNthCalledWith(2, { source: 'metadataCorrectionApi', collectionConceptId: 'C456-PROV', + publishedVersionName: '20.1', requestedAt: expect.any(String) }) diff --git a/serverless/src/requestMetadataCorrection/handler.js b/serverless/src/requestMetadataCorrection/handler.js index 5fb615ee..cc22fa7a 100644 --- a/serverless/src/requestMetadataCorrection/handler.js +++ b/serverless/src/requestMetadataCorrection/handler.js @@ -1,4 +1,5 @@ import { getApplicationConfig } from '@/shared/getConfig' +import { getVersionMetadata } from '@/shared/getVersionMetadata' import { logAnalyticsData } from '@/shared/logAnalyticsData' import { logger } from '@/shared/logger' import { publishMetadataCorrectionRequest } from '@/shared/publishMetadataCorrectionRequest' @@ -113,6 +114,7 @@ export const requestMetadataCorrection = async (event, context) => { acceptedCollectionConceptIds } = normalizeCollectionConceptIds(requestBody.collectionConceptIds) const requestedAt = new Date().toISOString() + const { versionName: publishedVersionName } = await getVersionMetadata('published') logger.info('[metadata-correction] Received asynchronous metadata correction request', { requestedCount, @@ -125,6 +127,7 @@ export const requestMetadataCorrection = async (event, context) => { const publishResult = await publishMetadataCorrectionRequest({ source: 'metadataCorrectionApi', collectionConceptId, + publishedVersionName, requestedAt }) diff --git a/serverless/src/runMetadataCorrection/__tests__/handler.test.js b/serverless/src/runMetadataCorrection/__tests__/handler.test.js index 4f728403..fff255fe 100644 --- a/serverless/src/runMetadataCorrection/__tests__/handler.test.js +++ b/serverless/src/runMetadataCorrection/__tests__/handler.test.js @@ -6,6 +6,7 @@ import { vi } from 'vitest' +import { getVersionMetadata } from '@/shared/getVersionMetadata' import { logger } from '@/shared/logger' import { runCollectionMetadataCorrection } from '@/shared/runCollectionMetadataCorrection' @@ -21,6 +22,10 @@ vi.mock('@/shared/logAnalyticsData', () => ({ logAnalyticsData: vi.fn() })) +vi.mock('@/shared/getVersionMetadata', () => ({ + getVersionMetadata: vi.fn() +})) + vi.mock('@/shared/logger', () => ({ logger: { info: vi.fn(), @@ -35,6 +40,7 @@ vi.mock('@/shared/runCollectionMetadataCorrection', () => ({ describe('runMetadataCorrection', () => { beforeEach(() => { vi.clearAllMocks() + vi.mocked(getVersionMetadata).mockResolvedValue({ versionName: '20.1' }) }) test('returns the synchronous correction summary as json', async () => { @@ -83,6 +89,7 @@ describe('runMetadataCorrection', () => { expect(runCollectionMetadataCorrection).toHaveBeenCalledWith({ collectionConceptId: 'C1234567890-PROV', + publishedVersionName: '20.1', source: 'metadataCorrectionApi' }) @@ -140,6 +147,7 @@ describe('runMetadataCorrection', () => { expect(runCollectionMetadataCorrection).toHaveBeenCalledWith({ collectionConceptId: 'C1234567890+PROV', + publishedVersionName: '20.1', source: 'metadataCorrectionApi' }) }) diff --git a/serverless/src/runMetadataCorrection/handler.js b/serverless/src/runMetadataCorrection/handler.js index 525b0e48..530db80f 100644 --- a/serverless/src/runMetadataCorrection/handler.js +++ b/serverless/src/runMetadataCorrection/handler.js @@ -1,4 +1,5 @@ import { getApplicationConfig } from '@/shared/getConfig' +import { getVersionMetadata } from '@/shared/getVersionMetadata' import { logAnalyticsData } from '@/shared/logAnalyticsData' import { logger } from '@/shared/logger' import { runCollectionMetadataCorrection } from '@/shared/runCollectionMetadataCorrection' @@ -63,8 +64,11 @@ export const runMetadataCorrection = async (event, context) => { collectionConceptId }) + const { versionName: publishedVersionName } = await getVersionMetadata('published') + const result = await runCollectionMetadataCorrection({ collectionConceptId, + publishedVersionName, source: 'metadataCorrectionApi' }) diff --git a/serverless/src/shared/__mocks__/iso-smap.xml b/serverless/src/shared/__mocks__/iso-smap.xml index 950f9bc4..4980db24 100644 --- a/serverless/src/shared/__mocks__/iso-smap.xml +++ b/serverless/src/shared/__mocks__/iso-smap.xml @@ -1,7 +1,8 @@ + xmlns:gco="http://www.isotc211.org/2005/gco" + xmlns:xlink="http://www.w3.org/1999/xlink"> @@ -297,4 +298,4 @@ - \ No newline at end of file + diff --git a/serverless/src/shared/__tests__/applyIso19115MetadataCorrections.test.js b/serverless/src/shared/__tests__/applyIso19115MetadataCorrections.test.js index ddd62770..6ed76ced 100644 --- a/serverless/src/shared/__tests__/applyIso19115MetadataCorrections.test.js +++ b/serverless/src/shared/__tests__/applyIso19115MetadataCorrections.test.js @@ -61,7 +61,7 @@ const mockIso19115WithOneScienceKeyword = ` describe('applyIso19115MetadataCorrections', () => { test('should handle missing corrections array gracefully', async () => { const params = { - metadataPayload: '' + metadataPayload: '' } const result = await applyIso19115MetadataCorrections(params) @@ -89,7 +89,7 @@ describe('applyIso19115MetadataCorrections', () => { test('should skip corrections with unknown schemes', async () => { const params = { - metadataPayload: '', + metadataPayload: '', corrections: [ { scheme: 'invalid-scheme', diff --git a/serverless/src/shared/__tests__/applyIsoSmapMetadataCorrections.test.js b/serverless/src/shared/__tests__/applyIsoSmapMetadataCorrections.test.js index ec082e49..002db482 100644 --- a/serverless/src/shared/__tests__/applyIsoSmapMetadataCorrections.test.js +++ b/serverless/src/shared/__tests__/applyIsoSmapMetadataCorrections.test.js @@ -406,7 +406,7 @@ describe('when applying dataformat ISO-19115 corrections', () => { describe('applyIsoSmapMetadataCorrections coverage', () => { test('should gracefully handle an unknown scheme in corrections', async () => { const params = { - metadataPayload: '', + metadataPayload: '', corrections: [ { scheme: 'unknownScheme', // This will trigger the !delegate check on line 38 diff --git a/serverless/src/shared/__tests__/awsClients.test.js b/serverless/src/shared/__tests__/awsClients.test.js index 9766f022..acb3b3c8 100644 --- a/serverless/src/shared/__tests__/awsClients.test.js +++ b/serverless/src/shared/__tests__/awsClients.test.js @@ -8,14 +8,24 @@ import { } from 'vitest' // Mock the AWS SDK clients before any imports -const { s3ClientMock, eventBridgeClientMock, snsClientMock } = vi.hoisted(() => ({ +const { + s3ClientMock, + eventBridgeClientMock, + secretsManagerClientMock, + snsClientMock +} = vi.hoisted(() => ({ s3ClientMock: vi.fn(), eventBridgeClientMock: vi.fn(), + secretsManagerClientMock: vi.fn(), snsClientMock: vi.fn() })) vi.mock('@aws-sdk/client-s3', () => ({ S3Client: s3ClientMock })) vi.mock('@aws-sdk/client-eventbridge', () => ({ EventBridgeClient: eventBridgeClientMock })) +vi.mock('@aws-sdk/client-secrets-manager', () => ({ + SecretsManagerClient: secretsManagerClientMock +})) + vi.mock('@aws-sdk/client-sns', () => ({ SNSClient: snsClientMock })) describe('awsClients', () => { @@ -55,6 +65,14 @@ describe('awsClients', () => { expect(snsClientMock).toHaveBeenCalledWith({}) }) + + test('getSecretsManagerClient should create a client with default config', async () => { + const { getSecretsManagerClient } = await import('../awsClients') + + getSecretsManagerClient() + + expect(secretsManagerClientMock).toHaveBeenCalledWith({}) + }) }) describe('when AWS_ENDPOINT_URL is set (LocalStack)', () => { @@ -96,6 +114,14 @@ describe('awsClients', () => { expect(snsClientMock).toHaveBeenCalledWith(expectedConfig) }) + + test('getSecretsManagerClient should create a client with LocalStack config', async () => { + const { getSecretsManagerClient } = await import('../awsClients') + + getSecretsManagerClient() + + expect(secretsManagerClientMock).toHaveBeenCalledWith(expectedConfig) + }) }) describe('singleton behavior', () => { @@ -128,5 +154,15 @@ describe('awsClients', () => { expect(snsClientMock).toHaveBeenCalledTimes(1) expect(client1).toBe(client2) }) + + test('getSecretsManagerClient should only create one instance', async () => { + const { getSecretsManagerClient } = await import('../awsClients') + + const client1 = getSecretsManagerClient() + const client2 = getSecretsManagerClient() + + expect(secretsManagerClientMock).toHaveBeenCalledTimes(1) + expect(client1).toBe(client2) + }) }) }) diff --git a/serverless/src/shared/__tests__/buildNativeMetadataDiff.test.js b/serverless/src/shared/__tests__/buildNativeMetadataDiff.test.js new file mode 100644 index 00000000..d7085fef --- /dev/null +++ b/serverless/src/shared/__tests__/buildNativeMetadataDiff.test.js @@ -0,0 +1,117 @@ +import { XMLBuilder } from 'fast-xml-parser' +import { + describe, + expect, + test, + vi +} from 'vitest' + +import { buildNativeMetadataDiff } from '../buildNativeMetadataDiff' + +describe('buildNativeMetadataDiff', () => { + test('creates a unified diff for native XML metadata', () => { + const result = buildNativeMetadataDiff({ + originalMetadata: 'GOSAT', + correctedMetadata: 'GOSAT - Test1', + priorRevisionId: 3 + }) + + expect(result).toEqual(expect.objectContaining({ + changed: true, + format: 'unified', + truncated: false, + originalBytes: 26, + correctedBytes: 34 + })) + + expect(result.patch).toContain('--- cmr-revision-3') + expect(result.patch).toContain('+++ corrected-metadata') + expect(result.patch).toContain('-GOSAT') + expect(result.patch).toContain('+GOSAT - Test1') + }) + + test('formats minified XML into a readable line-level diff', () => { + const result = buildNativeMetadataDiff({ + originalMetadata: 'TESTGOSATGreenhouse Gases Observing Satellite', + correctedMetadata: 'TESTGOSAT - Test1Greenhouse Gases Observing Satellite', + priorRevisionId: 5 + }) + + expect(result.patch).toContain(' ') + expect(result.patch).toContain('- GOSAT') + expect(result.patch).toContain('+ GOSAT - Test1') + expect(result.patch).toContain(' Greenhouse Gases Observing Satellite') + expect(result.patch).not.toContain('-') + }) + + test('ignores XML formatting differences', () => { + const result = buildNativeMetadataDiff({ + originalMetadata: '\n TEST\n', + correctedMetadata: 'TEST' + }) + + expect(result.changed).toBe(false) + expect(result.patch).toBe('') + }) + + test('leaves invalid XML text unchanged for diffing', () => { + const result = buildNativeMetadataDiff({ + originalMetadata: 'OLD', + correctedMetadata: 'NEW' + }) + + expect(result.patch).toContain('-OLD') + expect(result.patch).toContain('+NEW') + }) + + test('uses the original XML when formatting unexpectedly fails', () => { + vi.spyOn(XMLBuilder.prototype, 'build') + .mockImplementationOnce(() => { throw new Error('formatting failed') }) + + const result = buildNativeMetadataDiff({ + originalMetadata: 'GOSAT', + correctedMetadata: 'GOSAT - Test1' + }) + + expect(result.patch).toContain('-GOSAT') + expect(result.patch).toContain('+GOSAT - Test1') + }) + + test('serializes JSON metadata and reports identical payloads without a patch', () => { + const metadata = { Platforms: [{ ShortName: 'GOSAT' }] } + + expect(buildNativeMetadataDiff({ + originalMetadata: metadata, + correctedMetadata: structuredClone(metadata), + priorRevisionId: 4 + })).toEqual({ + changed: false, + format: 'unified', + patch: '', + truncated: false, + originalBytes: 63, + correctedBytes: 63 + }) + }) + + test('caps a large diff and marks it as truncated', () => { + const result = buildNativeMetadataDiff({ + originalMetadata: `old-${'a'.repeat(300_000)}`, + correctedMetadata: `new-${'b'.repeat(300_000)}` + }) + + expect(result.changed).toBe(true) + expect(result.truncated).toBe(true) + expect(result.patch).toHaveLength(250_000) + }) + + test('returns undefined when metadata cannot be serialized', () => { + const circularMetadata = {} + circularMetadata.self = circularMetadata + + expect(buildNativeMetadataDiff({ + originalMetadata: circularMetadata, + correctedMetadata: {} + })).toBeUndefined() + }) +}) diff --git a/serverless/src/shared/__tests__/cmrPutRequest.test.js b/serverless/src/shared/__tests__/cmrPutRequest.test.js index d98736c7..361dfcc8 100644 --- a/serverless/src/shared/__tests__/cmrPutRequest.test.js +++ b/serverless/src/shared/__tests__/cmrPutRequest.test.js @@ -63,6 +63,40 @@ describe('cmrPutRequest', () => { signal: expect.any(Object) }) ) + + expect(logger.info).toHaveBeenCalledWith( + '[cmr-put] CMR response received', + { + method: 'PUT', + path, + status: undefined, + durationMs: expect.any(Number), + requestId: undefined + } + ) + }) + + test('should log the CMR request id returned with a response', async () => { + global.fetch.mockResolvedValue({ + ok: true, + status: 200, + headers: { + get: vi.fn((name) => (name === 'cmr-request-id' ? 'request-123' : null)) + } + }) + + await cmrPutRequest({ + path: '/ingest/providers/KMS/collections/native-1' + }) + + expect(logger.info).toHaveBeenCalledWith( + '[cmr-put] CMR response received', + expect.objectContaining({ + status: 200, + durationMs: expect.any(Number), + requestId: 'request-123' + }) + ) }) test('should not include body in request if it is empty', async () => { @@ -175,6 +209,8 @@ describe('cmrPutRequest', () => { fullUrl: 'https://cmr-test.earthdata.nasa.gov/ingest/providers/KMS/collections/native-1', bodyLength: 2, timeoutMs: 25000, + durationMs: expect.any(Number), + timedOut: false, error: { name: 'TypeError', message: 'fetch failed', @@ -212,7 +248,9 @@ describe('cmrPutRequest', () => { path: '/ingest/providers/KMS/collections/native-1', fullUrl: 'https://cmr-test.earthdata.nasa.gov/ingest/providers/KMS/collections/native-1', bodyLength: 2, - timeoutMs: 25000 + timeoutMs: 25000, + durationMs: expect.any(Number), + timedOut: false }) expect(error.cmrCause).toBeUndefined() @@ -245,7 +283,9 @@ describe('cmrPutRequest', () => { path: '/ingest/providers/KMS/collections/native-1', fullUrl: 'https://cmr-test.earthdata.nasa.gov/ingest/providers/KMS/collections/native-1', bodyLength: undefined, - timeoutMs: 25000 + timeoutMs: 25000, + durationMs: expect.any(Number), + timedOut: false }) }) @@ -268,5 +308,12 @@ describe('cmrPutRequest', () => { await vi.advanceTimersByTimeAsync(25) await rejectionExpectation + + await expect(requestPromise).rejects.toMatchObject({ + cmrRequest: expect.objectContaining({ + timedOut: true, + durationMs: expect.any(Number) + }) + }) }) }) diff --git a/serverless/src/shared/__tests__/documentDbClient.test.js b/serverless/src/shared/__tests__/documentDbClient.test.js new file mode 100644 index 00000000..a4f25e1d --- /dev/null +++ b/serverless/src/shared/__tests__/documentDbClient.test.js @@ -0,0 +1,213 @@ +import { + afterEach, + beforeEach, + describe, + expect, + test, + vi +} from 'vitest' + +const { + collection, + close, + connect, + db, + mongoClient, + mongoClientConstructor, + secretsManagerSend +} = vi.hoisted(() => { + const collectionMock = { findOne: vi.fn() } + const dbMock = vi.fn(() => ({ + collection: vi.fn(() => collectionMock) + })) + const closeMock = vi.fn() + const client = { + close: closeMock, + db: dbMock + } + + return { + collection: collectionMock, + close: closeMock, + connect: vi.fn().mockResolvedValue(client), + db: dbMock, + mongoClient: client, + mongoClientConstructor: vi.fn(), + secretsManagerSend: vi.fn() + } +}) + +vi.mock('mongodb', () => ({ + MongoClient: mongoClientConstructor.mockImplementation(function mockMongoClient(uri, options) { + this.uri = uri + this.options = options + this.connect = connect + }) +})) + +vi.mock('@aws-sdk/client-secrets-manager', () => ({ + GetSecretValueCommand: vi.fn(function mockGetSecretValueCommand(input) { + this.input = input + }) +})) + +vi.mock('@/shared/awsClients', () => ({ + getSecretsManagerClient: vi.fn(() => ({ send: secretsManagerSend })) +})) + +const DOCUMENTDB_ENVIRONMENT_VARIABLES = [ + 'DOCUMENTDB_AUDIT_COLLECTION_NAME', + 'DOCUMENTDB_DATABASE_NAME', + 'DOCUMENTDB_HOST', + 'DOCUMENTDB_MAX_POOL_SIZE', + 'DOCUMENTDB_PORT', + 'DOCUMENTDB_SECRET_ARN', + 'DOCUMENTDB_TLS_CA_FILE', + 'DOCUMENTDB_URI' +] + +describe('documentDbClient', () => { + beforeEach(() => { + vi.resetModules() + vi.clearAllMocks() + connect.mockResolvedValue(mongoClient) + DOCUMENTDB_ENVIRONMENT_VARIABLES.forEach((name) => delete process.env[name]) + }) + + afterEach(() => { + DOCUMENTDB_ENVIRONMENT_VARIABLES.forEach((name) => delete process.env[name]) + }) + + test('reuses a local MongoDB connection and returns the configured collection', async () => { + process.env.DOCUMENTDB_URI = 'mongodb://localhost:27018/?directConnection=true' + process.env.DOCUMENTDB_DATABASE_NAME = 'test-kms' + process.env.DOCUMENTDB_AUDIT_COLLECTION_NAME = 'audits' + process.env.DOCUMENTDB_MAX_POOL_SIZE = '9' + const { + getDocumentDbClient, + getMetadataCorrectionAuditCollection + } = await import('../documentDbClient') + + await expect(getDocumentDbClient()).resolves.toBe(mongoClient) + await expect(getDocumentDbClient()).resolves.toBe(mongoClient) + await expect(getMetadataCorrectionAuditCollection()).resolves.toBe(collection) + + expect(mongoClientConstructor).toHaveBeenCalledOnce() + expect(mongoClientConstructor).toHaveBeenCalledWith( + 'mongodb://localhost:27018/?directConnection=true', + expect.objectContaining({ + maxPoolSize: 9, + minPoolSize: 0 + }) + ) + + expect(connect).toHaveBeenCalledOnce() + expect(db).toHaveBeenCalledWith('test-kms') + }) + + test('closes and resets the shared connection', async () => { + process.env.DOCUMENTDB_URI = 'mongodb://localhost:27018' + const { + closeDocumentDbClient, + getDocumentDbClient + } = await import('../documentDbClient') + + await closeDocumentDbClient() + await getDocumentDbClient() + await closeDocumentDbClient() + await getDocumentDbClient() + + expect(close).toHaveBeenCalledOnce() + expect(mongoClientConstructor).toHaveBeenCalledTimes(2) + }) + + test('uses the default database and audit collection names', async () => { + process.env.DOCUMENTDB_URI = 'mongodb://localhost:27018' + const { getMetadataCorrectionAuditCollection } = await import('../documentDbClient') + + await expect(getMetadataCorrectionAuditCollection()).resolves.toBe(collection) + + expect(db).toHaveBeenCalledWith('kms') + }) + + test('builds the deployed TLS connection from Secrets Manager credentials', async () => { + process.env.DOCUMENTDB_HOST = 'audit.cluster.docdb.amazonaws.com' + process.env.DOCUMENTDB_PORT = '27017' + process.env.DOCUMENTDB_SECRET_ARN = 'arn:aws:secretsmanager:secret:audit' + process.env.DOCUMENTDB_TLS_CA_FILE = '/var/task/us-east-1-bundle.pem' + secretsManagerSend.mockResolvedValue({ + SecretString: JSON.stringify({ + username: 'user@example.com', + password: 'password/with spaces' + }) + }) + + const { getDocumentDbClient } = await import('../documentDbClient') + + await getDocumentDbClient() + + expect(secretsManagerSend).toHaveBeenCalledWith(expect.objectContaining({ + input: { SecretId: 'arn:aws:secretsmanager:secret:audit' } + })) + + expect(mongoClientConstructor).toHaveBeenCalledWith( + 'mongodb://user%40example.com:password%2Fwith%20spaces@audit.cluster.docdb.amazonaws.com:27017/?tls=true&replicaSet=rs0&readPreference=primary&retryWrites=false&authSource=admin', + expect.objectContaining({ + maxPoolSize: 5, + tlsCAFile: '/var/task/us-east-1-bundle.pem' + }) + ) + }) + + test('validates deployed connection configuration and credentials', async () => { + let documentDbClient = await import('../documentDbClient') + await expect(documentDbClient.getDocumentDbClient()).rejects.toThrow( + 'Missing DOCUMENTDB_HOST' + ) + + vi.resetModules() + process.env.DOCUMENTDB_HOST = 'audit.cluster.docdb.amazonaws.com' + documentDbClient = await import('../documentDbClient') + await expect(documentDbClient.getDocumentDbClient()).rejects.toThrow( + 'Missing DOCUMENTDB_TLS_CA_FILE' + ) + + vi.resetModules() + process.env.DOCUMENTDB_TLS_CA_FILE = '/tmp/ca.pem' + documentDbClient = await import('../documentDbClient') + await expect(documentDbClient.getDocumentDbClient()).rejects.toThrow( + 'Missing DOCUMENTDB_SECRET_ARN' + ) + + vi.resetModules() + process.env.DOCUMENTDB_SECRET_ARN = 'arn:aws:secretsmanager:secret:audit' + secretsManagerSend.mockResolvedValue({}) + documentDbClient = await import('../documentDbClient') + await expect(documentDbClient.getDocumentDbClient()).rejects.toThrow( + 'DocumentDB secret does not contain SecretString credentials' + ) + + vi.resetModules() + secretsManagerSend.mockResolvedValue({ + SecretString: JSON.stringify({ username: 'kms_audit' }) + }) + + documentDbClient = await import('../documentDbClient') + await expect(documentDbClient.getDocumentDbClient()).rejects.toThrow( + 'DocumentDB secret is missing username or password' + ) + }) + + test('retries connection creation after a failed connection', async () => { + process.env.DOCUMENTDB_URI = 'mongodb://localhost:27018' + connect + .mockRejectedValueOnce(new Error('connection failed')) + .mockResolvedValueOnce(mongoClient) + + const { getDocumentDbClient } = await import('../documentDbClient') + + await expect(getDocumentDbClient()).rejects.toThrow('connection failed') + await expect(getDocumentDbClient()).resolves.toBe(mongoClient) + expect(mongoClientConstructor).toHaveBeenCalledTimes(2) + }) +}) diff --git a/serverless/src/shared/__tests__/getMetadataCorrectionAuditLog.test.js b/serverless/src/shared/__tests__/getMetadataCorrectionAuditLog.test.js index bdf2a45f..d0d1f35f 100644 --- a/serverless/src/shared/__tests__/getMetadataCorrectionAuditLog.test.js +++ b/serverless/src/shared/__tests__/getMetadataCorrectionAuditLog.test.js @@ -6,279 +6,383 @@ import { vi } from 'vitest' -import { sparqlRequest } from '@/shared/sparqlRequest' +import { getMetadataCorrectionAuditCollection } from '@/shared/documentDbClient' -import { getMetadataCorrectionAuditLog } from '../getMetadataCorrectionAuditLog' +import { + getMetadataCorrectionAuditByRunId, + getMetadataCorrectionAuditLog +} from '../getMetadataCorrectionAuditLog' -vi.mock('@/shared/sparqlRequest', () => ({ - sparqlRequest: vi.fn() +vi.mock('@/shared/documentDbClient', () => ({ + getMetadataCorrectionAuditCollection: vi.fn() })) -describe('getMetadataCorrectionAuditLog', () => { +const SUMMARY_PROJECTION = { + _id: 1, + runId: 1, + collectionConceptId: 1, + collectionUri: 1, + status: 1, + createdAt: 1, + updatedAt: 1, + 'corrections.scheme': 1, + 'corrections.action': 1, + 'corrections.oldKeywordPath': 1, + 'corrections.newKeywordPath': 1, + 'error.message': 1 +} +const DETAIL_RUN_ID = 'f3351653-dfc3-47d8-9176-294ea90bc118' +const UNKNOWN_RUN_ID = '11111111-1111-4111-8111-111111111111' + +describe('metadata correction audit queries', () => { + let collection + let mongoCursor + beforeEach(() => { vi.clearAllMocks() + mongoCursor = { + sort: vi.fn().mockReturnThis(), + limit: vi.fn().mockReturnThis(), + toArray: vi.fn().mockResolvedValue([]) + } + + collection = { + find: vi.fn().mockReturnValue(mongoCursor), + findOne: vi.fn().mockResolvedValue(null) + } + + vi.mocked(getMetadataCorrectionAuditCollection).mockResolvedValue(collection) }) - test('queries the audit graph and maps bindings into audit rows', async () => { - vi.mocked(sparqlRequest).mockResolvedValue({ - ok: true, - json: vi.fn().mockResolvedValue({ - results: { - bindings: [ - { - record: { value: 'https://gcmd.earthdata.nasa.gov/kms/metadata-correction-audit/audit-1' }, - timestamp: { value: '2026-05-06T18:00:00.000Z' }, - publishedVersionName: { value: '9.1.5' }, - collectionConceptId: { value: 'C1234567890-LOCAL' }, - keywordConceptUuid: { value: 'uuid-1' }, - scheme: { value: 'sciencekeywords' }, - action: { value: 'UPDATED' }, - oldKeywordPath: { value: 'EARTH SCIENCE > ATMOSPHERE' }, - newKeywordPath: { value: 'EARTH SCIENCE > OCEANS' }, - nativeFormat: { value: 'UMM' }, - delegateName: { value: 'umm' }, - status: { value: 'pending' }, - writebackErrorMessage: { value: 'CMR writeback failed with status 400: {"errors":["boom"]}' }, - triggerScheme: { value: 'sciencekeywords' }, - triggerKeywordUuid: { value: 'uuid-trigger' } + test('filters and returns newest-first audit documents', async () => { + const createdAt = new Date('2026-09-02T12:00:00.000Z') + const updatedAt = new Date('2026-09-02T12:01:00.000Z') + mongoCursor.toArray.mockResolvedValue([{ + _id: 'run-1', + runId: 'run-1', + collectionConceptId: 'C123-PROV', + collectionUri: 'https://cmr.example.com/search/concepts/C123-PROV', + createdAt, + updatedAt, + status: 'failed', + corrections: [{ + scheme: 'platforms', + action: 'replace', + oldKeywordPath: 'Platforms > GOSAT', + newKeywordPath: 'Platforms > GOSAT - Test1' + }], + metadataDiff: { changed: true }, + error: { message: 'CMR writeback timed out' } + }]) + + const result = await getMetadataCorrectionAuditLog({ + action: 'updated', + collectionConceptId: 'C123-PROV', + endDate: '2026-09-03', + keywordConceptUuid: 'keyword-1', + limit: '25', + nativeFormat: 'UMM', + publishedVersionName: '20.1', + scheme: 'dataformat', + source: 'cmrKeywordEventsListener', + startDate: '2026-09-01', + status: 'applied' + }) + + expect(collection.find).toHaveBeenCalledWith( + { + $and: [ + { + collectionConceptId: 'C123-PROV', + 'trigger.eventType': 'UPDATED', + nativeFormat: 'UMM', + publishedVersionName: '20.1', + source: 'cmrKeywordEventsListener', + status: 'applied', + createdAt: { + $gte: new Date('2026-09-01'), + $lte: new Date('2026-09-03') } - ] - } - }) + }, + { + $or: [ + { 'corrections.keywordConceptUuid': 'keyword-1' }, + { 'trigger.keywordConceptUuid': 'keyword-1' } + ] + }, + { + $or: [ + { 'corrections.scheme': { $in: ['DataFormat', 'dataformat'] } }, + { 'trigger.scheme': { $in: ['DataFormat', 'dataformat'] } } + ] + } + ] + }, + { projection: SUMMARY_PROJECTION } + ) + + expect(mongoCursor.sort).toHaveBeenCalledWith({ + createdAt: -1, + _id: -1 }) - const result = await getMetadataCorrectionAuditLog({ - collectionConceptId: 'C1234567890-LOCAL', - action: 'UPDATED', - status: 'pending', - limit: '25' + expect(mongoCursor.limit).toHaveBeenCalledWith(26) + expect(result).toEqual({ + items: [{ + runId: 'run-1', + collectionConceptId: 'C123-PROV', + collectionUri: 'https://cmr.example.com/search/concepts/C123-PROV', + status: 'failed', + updatedAt, + changes: [{ + scheme: 'platforms', + action: 'replace', + oldKeywordPath: 'Platforms > GOSAT', + newKeywordPath: 'Platforms > GOSAT - Test1' + }], + errorMessage: 'CMR writeback timed out' + }], + nextPaginationToken: null }) + }) - expect(result).toEqual([ + test('returns a pagination token when another page exists and applies it to the next query', async () => { + const documents = [ + { + _id: 'run-3', + runId: 'run-3', + createdAt: new Date('2026-09-03') + }, + { + _id: 'run-2', + runId: 'run-2', + createdAt: new Date('2026-09-02') + }, { - recordUri: 'https://gcmd.earthdata.nasa.gov/kms/metadata-correction-audit/audit-1', - timestamp: '2026-05-06T18:00:00.000Z', - publishedVersionName: '9.1.5', - collectionConceptId: 'C1234567890-LOCAL', - keywordConceptUuid: 'uuid-1', - scheme: 'sciencekeywords', - action: 'UPDATED', - oldKeywordPath: 'EARTH SCIENCE > ATMOSPHERE', - newKeywordPath: 'EARTH SCIENCE > OCEANS', - nativeFormat: 'UMM', - delegateName: 'umm', - status: 'pending', - writebackErrorMessage: 'CMR writeback failed with status 400: {"errors":["boom"]}', - triggerScheme: 'sciencekeywords', - triggerKeywordUuid: 'uuid-trigger' + _id: 'run-1', + runId: 'run-1', + createdAt: new Date('2026-09-01') } - ]) + ] + mongoCursor.toArray.mockResolvedValue(documents) - expect(sparqlRequest).toHaveBeenCalledWith({ - method: 'POST', - body: expect.stringContaining('GRAPH '), - contentType: 'application/sparql-query', - accept: 'application/sparql-results+json' + const firstPage = await getMetadataCorrectionAuditLog({ limit: '2' }) + + expect(firstPage.items).toHaveLength(2) + expect(firstPage.nextPaginationToken).toEqual(expect.any(String)) + + mongoCursor.toArray.mockResolvedValue([]) + await getMetadataCorrectionAuditLog({ + paginationToken: firstPage.nextPaginationToken, + limit: '2', + status: 'checked' }) - const sparqlCall = vi.mocked(sparqlRequest).mock.calls[0][0] - expect(sparqlCall.body).toContain('FILTER(?collectionConceptId = "C1234567890-LOCAL")') - expect(sparqlCall.body).toContain('FILTER(?action = "UPDATED")') - expect(sparqlCall.body).toContain('FILTER(?status = "pending")') - expect(sparqlCall.body).toContain('LIMIT 25') + expect(collection.find).toHaveBeenLastCalledWith( + { + $and: [ + { status: 'checked' }, + { + $or: [ + { createdAt: { $lt: new Date('2026-09-02') } }, + { + createdAt: new Date('2026-09-02'), + _id: { $lt: 'run-2' } + } + ] + } + ] + }, + { projection: SUMMARY_PROJECTION } + ) }) - test('normalizes invalid limits, applies keyword and scheme filters, and leaves optional fields undefined when absent', async () => { - vi.mocked(sparqlRequest).mockResolvedValue({ - ok: true, - json: vi.fn().mockResolvedValue({ - results: { - bindings: [ - { - record: { value: 'https://gcmd.earthdata.nasa.gov/kms/metadata-correction-audit/audit-2' }, - timestamp: { value: '2026-05-07T18:00:00.000Z' }, - publishedVersionName: { value: '9.1.6' }, - collectionConceptId: { value: 'C0000000002-LOCAL' }, - keywordConceptUuid: { value: 'uuid-2' }, - scheme: { value: 'platforms' }, - action: { value: 'UPDATED' }, - oldKeywordPath: { value: 'OLD PLATFORM' }, - newKeywordPath: { value: 'NEW PLATFORM' }, - nativeFormat: { value: 'DIF10' }, - delegateName: { value: 'dif10' }, - status: { value: 'applied' } - } - ] - } - }) + test('includes native metadata diffs in list results when requested', async () => { + const metadataDiff = { + changed: true, + format: 'unified', + patch: '-old\n+new' + } + mongoCursor.toArray.mockResolvedValue([{ + _id: 'run-1', + runId: 'run-1', + metadataDiff + }]) + + const result = await getMetadataCorrectionAuditLog({ includeDiff: 'true' }) + + expect(collection.find).toHaveBeenCalledWith({}, { + projection: { + ...SUMMARY_PROJECTION, + metadataDiff: 1 + } }) - const result = await getMetadataCorrectionAuditLog({ - keywordConceptUuid: 'uuid-2', - scheme: 'platforms', - limit: 'not-a-number' + expect(result.items[0]).toMatchObject({ + runId: 'run-1', + metadataDiff }) + }) + + test('supports default filters, one-sided date ranges, and lowercase scheme storage', async () => { + await getMetadataCorrectionAuditLog() + + expect(collection.find).toHaveBeenLastCalledWith( + {}, + { projection: SUMMARY_PROJECTION } + ) - expect(result).toEqual([ + expect(mongoCursor.limit).toHaveBeenLastCalledWith(101) + + await getMetadataCorrectionAuditLog({ scheme: 'PLATFORMS' }) + expect(collection.find).toHaveBeenLastCalledWith( { - recordUri: 'https://gcmd.earthdata.nasa.gov/kms/metadata-correction-audit/audit-2', - timestamp: '2026-05-07T18:00:00.000Z', - publishedVersionName: '9.1.6', - collectionConceptId: 'C0000000002-LOCAL', - keywordConceptUuid: 'uuid-2', - scheme: 'platforms', - action: 'UPDATED', - oldKeywordPath: 'OLD PLATFORM', - newKeywordPath: 'NEW PLATFORM', - nativeFormat: 'DIF10', - delegateName: 'dif10', - status: 'applied', - writebackErrorMessage: undefined, - triggerScheme: undefined, - triggerKeywordUuid: undefined - } - ]) - - const sparqlCall = vi.mocked(sparqlRequest).mock.calls[0][0] - expect(sparqlCall.body).toContain('FILTER(?keywordConceptUuid = "uuid-2")') - expect(sparqlCall.body).toContain('FILTER(?scheme = "platforms")') - expect(sparqlCall.body).toContain('LIMIT 100') - expect(sparqlCall.body).not.toContain('FILTER(?collectionConceptId =') - expect(sparqlCall.body).not.toContain('FILTER(?action =') - expect(sparqlCall.body).not.toContain('FILTER(?status =') + $or: [ + { 'corrections.scheme': 'platforms' }, + { 'trigger.scheme': 'platforms' } + ] + }, + { projection: SUMMARY_PROJECTION } + ) + + await getMetadataCorrectionAuditLog({ startDate: '2026-09-01' }) + expect(collection.find).toHaveBeenLastCalledWith( + { createdAt: { $gte: new Date('2026-09-01') } }, + { projection: SUMMARY_PROJECTION } + ) + + await getMetadataCorrectionAuditLog({ endDate: '2026-09-03' }) + expect(collection.find).toHaveBeenLastCalledWith( + { createdAt: { $lte: new Date('2026-09-03') } }, + { projection: SUMMARY_PROJECTION } + ) }) - test('keeps large explicit limits instead of clamping them', async () => { - vi.mocked(sparqlRequest).mockResolvedValue({ - ok: true, - json: vi.fn().mockResolvedValue({ - results: { - bindings: [] - } - }) - }) + test('validates filters before querying DocumentDB', async () => { + await expect(getMetadataCorrectionAuditLog({ + status: 'unknown' + })).rejects.toThrow('Invalid metadata correction audit status: unknown') + + await expect(getMetadataCorrectionAuditLog({ + startDate: 'not-a-date' + })).rejects.toThrow('Invalid metadata correction audit startDate') + + await expect(getMetadataCorrectionAuditLog({ + action: 'renamed' + })).rejects.toThrow('Invalid metadata correction audit action: renamed') + + await expect(getMetadataCorrectionAuditLog({ + scheme: 'not-a-scheme' + })).rejects.toThrow('Invalid metadata correction audit scheme: not-a-scheme') await expect(getMetadataCorrectionAuditLog({ limit: '5000' - })).resolves.toEqual([]) + })).rejects.toThrow('Invalid metadata correction audit limit: expected an integer from 1 to 250') - const sparqlCall = vi.mocked(sparqlRequest).mock.calls[0][0] - expect(sparqlCall.body).toContain('LIMIT 5000') - }) + await expect(getMetadataCorrectionAuditLog({ + limit: '12records' + })).rejects.toThrow('Invalid metadata correction audit limit: expected an integer from 1 to 250') - test('uses default filters and returns an empty array when the query result has no bindings', async () => { - vi.mocked(sparqlRequest).mockResolvedValue({ - ok: true, - json: vi.fn().mockResolvedValue({}) - }) + await expect(getMetadataCorrectionAuditLog({ + startDate: '2026-09-03', + endDate: '2026-09-01' + })).rejects.toThrow( + 'Invalid metadata correction audit date range: startDate must not be after endDate' + ) + + await expect(getMetadataCorrectionAuditLog({ + paginationToken: 'not-a-pagination-token' + })).rejects.toThrow('Invalid metadata correction audit paginationToken') + + await expect(getMetadataCorrectionAuditLog({ + includeDiff: 'yes' + })).rejects.toThrow( + 'Invalid metadata correction audit includeDiff: expected true or false' + ) - await expect(getMetadataCorrectionAuditLog()).resolves.toEqual([]) + const invalidPaginationToken = Buffer.from(JSON.stringify({ + createdAt: '2026-09-02T12:00:00.000Z', + runId: '' + })).toString('base64url') + await expect(getMetadataCorrectionAuditLog({ + paginationToken: invalidPaginationToken + })).rejects.toThrow('Invalid metadata correction audit paginationToken') - const sparqlCall = vi.mocked(sparqlRequest).mock.calls[0][0] - expect(sparqlCall.body).toContain('LIMIT 100') - expect(sparqlCall.body).not.toContain('FILTER(?collectionConceptId =') - expect(sparqlCall.body).not.toContain('FILTER(?keywordConceptUuid =') - expect(sparqlCall.body).not.toContain('FILTER(?scheme =') + expect(getMetadataCorrectionAuditCollection).not.toHaveBeenCalled() }) - test('collapses duplicate pending and applied lifecycle rows when latestOnly is enabled', async () => { - vi.mocked(sparqlRequest).mockResolvedValue({ - ok: true, - json: vi.fn().mockResolvedValue({ - results: { - bindings: [ - { - record: { value: 'https://example.org/audit/applied-1' }, - timestamp: { value: '2026-06-17T12:00:01.000Z' }, - publishedVersionName: { value: '9.1.6' }, - collectionConceptId: { value: 'C1234567890-LOCAL' }, - keywordConceptUuid: { value: 'uuid-1' }, - scheme: { value: 'sciencekeywords' }, - action: { value: 'UPDATED' }, - oldKeywordPath: { value: 'EARTH SCIENCE > ATMOSPHERE' }, - newKeywordPath: { value: 'EARTH SCIENCE > OCEANS' }, - nativeFormat: { value: 'DIF10' }, - delegateName: { value: 'dif10' }, - status: { value: 'applied' }, - writebackErrorMessage: { value: 'CMR writeback failed with status 400: {"errors":["boom"]}' }, - triggerScheme: { value: 'sciencekeywords' }, - triggerKeywordUuid: { value: 'uuid-trigger' } - }, - { - record: { value: 'https://example.org/audit/pending-1' }, - timestamp: { value: '2026-06-17T12:00:00.000Z' }, - publishedVersionName: { value: '9.1.6' }, - collectionConceptId: { value: 'C1234567890-LOCAL' }, - keywordConceptUuid: { value: 'uuid-1' }, - scheme: { value: 'sciencekeywords' }, - action: { value: 'UPDATED' }, - oldKeywordPath: { value: 'EARTH SCIENCE > ATMOSPHERE' }, - newKeywordPath: { value: 'EARTH SCIENCE > OCEANS' }, - nativeFormat: { value: 'DIF10' }, - delegateName: { value: 'dif10' }, - status: { value: 'pending' }, - writebackErrorMessage: undefined, - triggerScheme: { value: 'sciencekeywords' }, - triggerKeywordUuid: { value: 'uuid-trigger' } - }, - { - record: { value: 'https://example.org/audit/pending-2' }, - timestamp: { value: '2026-06-17T11:59:59.000Z' }, - publishedVersionName: { value: '9.1.6' }, - collectionConceptId: { value: 'C9999999999-LOCAL' }, - keywordConceptUuid: { value: 'uuid-2' }, - scheme: { value: 'platforms' }, - action: { value: 'UPDATED' }, - oldKeywordPath: { value: 'OLD PLATFORM' }, - newKeywordPath: { value: 'NEW PLATFORM' }, - nativeFormat: { value: 'UMM' }, - delegateName: { value: 'umm' }, - status: { value: 'pending' } - } - ] - } - }) + test('returns one detailed audit run without the native metadata diff by default', async () => { + collection.findOne.mockResolvedValue({ + runId: DETAIL_RUN_ID, + status: 'applied' }) - const result = await getMetadataCorrectionAuditLog({ - latestOnly: 'true' + await expect(getMetadataCorrectionAuditByRunId({ + runId: DETAIL_RUN_ID + })).resolves.toEqual({ + runId: DETAIL_RUN_ID, + status: 'applied' }) - expect(result).toEqual([ + expect(collection.findOne).toHaveBeenCalledWith( + { _id: DETAIL_RUN_ID }, { - recordUri: 'https://example.org/audit/applied-1', - timestamp: '2026-06-17T12:00:01.000Z', - publishedVersionName: '9.1.6', - collectionConceptId: 'C1234567890-LOCAL', - keywordConceptUuid: 'uuid-1', - scheme: 'sciencekeywords', - action: 'UPDATED', - oldKeywordPath: 'EARTH SCIENCE > ATMOSPHERE', - newKeywordPath: 'EARTH SCIENCE > OCEANS', - nativeFormat: 'DIF10', - delegateName: 'dif10', - status: 'applied', - writebackErrorMessage: 'CMR writeback failed with status 400: {"errors":["boom"]}', - triggerScheme: 'sciencekeywords', - triggerKeywordUuid: 'uuid-trigger' - }, - { - recordUri: 'https://example.org/audit/pending-2', - timestamp: '2026-06-17T11:59:59.000Z', - publishedVersionName: '9.1.6', - collectionConceptId: 'C9999999999-LOCAL', - keywordConceptUuid: 'uuid-2', - scheme: 'platforms', - action: 'UPDATED', - oldKeywordPath: 'OLD PLATFORM', - newKeywordPath: 'NEW PLATFORM', - nativeFormat: 'UMM', - delegateName: 'umm', - status: 'pending', - writebackErrorMessage: undefined, - triggerScheme: undefined, - triggerKeywordUuid: undefined + projection: { + _id: 0, + metadataDiff: 0 + } + } + ) + }) + + test('includes the native metadata diff only when requested', async () => { + collection.findOne.mockResolvedValue({ + runId: DETAIL_RUN_ID, + status: 'failed', + metadataDiff: { + changed: true, + patch: '-old\n+new' } - ]) + }) + + const result = await getMetadataCorrectionAuditByRunId({ + runId: DETAIL_RUN_ID, + includeDiff: 'true' + }) + + expect(result.metadataDiff.patch).toBe('-old\n+new') + expect(collection.findOne).toHaveBeenCalledWith( + { _id: DETAIL_RUN_ID }, + { projection: { _id: 0 } } + ) + }) + + test('returns null for an unknown run', async () => { + await expect(getMetadataCorrectionAuditByRunId({ + runId: UNKNOWN_RUN_ID + })).resolves.toBeNull() + }) + + test('validates detail parameters before querying the audit collection', async () => { + await expect(getMetadataCorrectionAuditByRunId()).rejects.toThrow( + 'Invalid metadata correction audit runId' + ) + + await expect(getMetadataCorrectionAuditByRunId({ + runId: 123 + })).rejects.toThrow('Invalid metadata correction audit runId') + + await expect(getMetadataCorrectionAuditByRunId({ + runId: 'run-1' + })).rejects.toThrow('Invalid metadata correction audit runId') + + await expect(getMetadataCorrectionAuditByRunId({ + runId: DETAIL_RUN_ID, + includeDiff: 'yes' + })).rejects.toThrow( + 'Invalid metadata correction audit includeDiff: expected true or false' + ) + + expect(getMetadataCorrectionAuditCollection).not.toHaveBeenCalled() }) }) diff --git a/serverless/src/shared/__tests__/metadataCorrectionDelegateStubs.test.js b/serverless/src/shared/__tests__/metadataCorrectionDelegateStubs.test.js index fcddadc2..359e9c6d 100644 --- a/serverless/src/shared/__tests__/metadataCorrectionDelegateStubs.test.js +++ b/serverless/src/shared/__tests__/metadataCorrectionDelegateStubs.test.js @@ -179,7 +179,7 @@ describe('metadata correction delegate stubs', () => { test('returns the expected ISO19115 payload shape when corrections are provided', async () => { const mockPayload = ` - + diff --git a/serverless/src/shared/__tests__/persistMetadataCorrectionAuditLog.test.js b/serverless/src/shared/__tests__/persistMetadataCorrectionAuditLog.test.js index 530e666a..4413ad7c 100644 --- a/serverless/src/shared/__tests__/persistMetadataCorrectionAuditLog.test.js +++ b/serverless/src/shared/__tests__/persistMetadataCorrectionAuditLog.test.js @@ -1,4 +1,5 @@ import { + afterEach, beforeEach, describe, expect, @@ -6,242 +7,236 @@ import { vi } from 'vitest' -import { getVersionMetadata } from '@/shared/getVersionMetadata' -import { sparqlRequest } from '@/shared/sparqlRequest' +import { getMetadataCorrectionAuditCollection } from '@/shared/documentDbClient' import { persistMetadataCorrectionAuditLog } from '../persistMetadataCorrectionAuditLog' vi.mock('uuid', () => ({ - v4: vi.fn(() => 'audit-record-123') + v4: vi.fn(() => 'generated-run-id') })) -vi.mock('@/shared/getVersionMetadata', () => ({ - getVersionMetadata: vi.fn() +vi.mock('@/shared/documentDbClient', () => ({ + getMetadataCorrectionAuditCollection: vi.fn() })) -vi.mock('@/shared/sparqlRequest', () => ({ - sparqlRequest: vi.fn() -})) +const buildCorrection = () => ({ + scheme: 'platforms', + keywordConceptUuid: 'platform-uuid', + oldKeywordObject: { + Basis: 'Platforms', + Category: 'Space-based Platforms', + SubCategory: 'Earth Observation Satellites', + ShortName: 'GOSAT' + }, + newKeywordObject: { + Basis: 'Platforms', + Category: 'Space-based Platforms', + SubCategory: 'Earth Observation Satellites', + ShortName: 'GOSAT - Test1' + } +}) describe('persistMetadataCorrectionAuditLog', () => { + let collection + beforeEach(() => { vi.clearAllMocks() - vi.mocked(getVersionMetadata).mockResolvedValue({ - version: 'published', - versionName: '9.1.5', - versionType: 'published', - created: '2026-01-01T00:00:00Z', - lastSynced: null - }) + process.env.CMR_BASE_URL = 'https://cmr.example.com/' + collection = { + findOne: vi.fn().mockResolvedValue(null), + updateOne: vi.fn().mockResolvedValue({ acknowledged: true }) + } - vi.mocked(sparqlRequest).mockResolvedValue({ ok: true }) + vi.mocked(getMetadataCorrectionAuditCollection).mockResolvedValue(collection) }) - test('persists one audit row per correction with pending status', async () => { + afterEach(() => { + delete process.env.CMR_BASE_URL + }) + + test('creates one checked audit document with corrections and status history', async () => { const result = await persistMetadataCorrectionAuditLog({ - collectionConceptId: 'C1234567890-LOCAL', + collectionConceptId: 'C123-PROV', + corrections: [buildCorrection()], keywordEvent: { eventType: 'UPDATED', - scheme: 'sciencekeywords', - uuid: '2e5a401b-1507-4f57-82b8-36557c13b154' + scheme: 'platforms', + uuid: 'platform-uuid' + }, + keywordValidationFailures: [{ + keywordConceptUuid: 'invalid-keyword-uuid', + reason: 'Keyword was not found in the published version' + }], + metadataDiff: { + changed: true, + format: 'unified', + patch: '-GOSAT\n+GOSAT - Test1', + truncated: false }, nativeFormat: 'UMM', - delegateName: 'umm', - corrections: [ - { - scheme: 'sciencekeywords', - keywordConceptUuid: '2e5a401b-1507-4f57-82b8-36557c13b154', - oldKeywordObject: { - Category: 'EARTH SCIENCE', - Topic: 'ATMOSPHERE', - Term: 'AEROSOLS', - VariableLevel1: 'LEGACY AEROSOLS', - VariableLevel2: '', - VariableLevel3: '', - DetailedVariable: '' - }, - newKeywordObject: { - Category: 'EARTH SCIENCE', - Topic: 'ATMOSPHERE', - Term: 'AEROSOLS', - VariableLevel1: '', - VariableLevel2: '', - VariableLevel3: '', - DetailedVariable: '' - } - } - ], - status: 'pending', - timestamp: '2026-05-06T18:00:00.000Z' + priorRevisionId: 7, + providerId: 'PROV', + publishedVersionName: '20.1', + status: 'checked', + timestamp: '2026-09-02T12:00:00.000Z' }) expect(result).toEqual({ - insertedCount: 1, - publishedVersionName: '9.1.5', - status: 'pending' + runId: 'generated-run-id', + status: 'checked', + created: true }) - expect(getVersionMetadata).toHaveBeenCalledWith('published') - expect(sparqlRequest).toHaveBeenCalledWith(expect.objectContaining({ - method: 'POST', - contentType: 'application/sparql-update', - accept: 'application/json', - body: expect.stringContaining('GRAPH ') - })) + expect(collection.updateOne).toHaveBeenCalledWith( + { _id: 'generated-run-id' }, + expect.objectContaining({ + $set: expect.objectContaining({ + collectionConceptId: 'C123-PROV', + collectionUri: 'https://cmr.example.com/search/concepts/C123-PROV', + keywordValidationFailureCount: 1, + keywordValidationFailures: [{ + keywordConceptUuid: 'invalid-keyword-uuid', + reason: 'Keyword was not found in the published version' + }], + publishedVersionName: '20.1', + priorRevisionId: 7, + status: 'checked', + metadataDiff: { + changed: true, + format: 'unified', + patch: '-GOSAT\n+GOSAT - Test1', + truncated: false + }, + corrections: [expect.objectContaining({ + keywordConceptUuid: 'platform-uuid', + oldKeywordPath: 'Platforms > Space-based Platforms > Earth Observation Satellites > GOSAT', + newKeywordPath: 'Platforms > Space-based Platforms > Earth Observation Satellites > GOSAT - Test1' + })] + }), + $setOnInsert: { + _id: 'generated-run-id', + runId: 'generated-run-id', + createdAt: new Date('2026-09-02T12:00:00.000Z') + }, + $push: { + statusHistory: { + status: 'checked', + timestamp: new Date('2026-09-02T12:00:00.000Z') + } + }, + $unset: { error: '' } + }), + { upsert: true } + ) + }) - const sparqlCall = vi.mocked(sparqlRequest).mock.calls[0][0] - expect(sparqlCall.body).toContain('gcmd:MetadataCorrectionAuditRecord') - expect(sparqlCall.body).toContain('gcmd:publishedVersionName "9.1.5"') - expect(sparqlCall.body).toContain('gcmd:collectionConceptId "C1234567890-LOCAL"') - expect(sparqlCall.body).toContain('gcmd:action "UPDATED"') - expect(sparqlCall.body).toContain('gcmd:scheme "sciencekeywords"') - expect(sparqlCall.body).toContain('gcmd:status "pending"') - expect(sparqlCall.body).toContain('gcmd:triggerScheme "sciencekeywords"') - expect(sparqlCall.body).toContain('gcmd:triggerKeywordUuid "2e5a401b-1507-4f57-82b8-36557c13b154"') - expect(sparqlCall.body).toContain('gcmd:oldKeywordPath "EARTH SCIENCE > ATMOSPHERE > AEROSOLS > LEGACY AEROSOLS > > > "') - expect(sparqlCall.body).toContain('gcmd:newKeywordPath "EARTH SCIENCE > ATMOSPHERE > AEROSOLS > > > > "') - expect(sparqlCall.body).not.toContain('gcmd:oldKeywordObject') - expect(sparqlCall.body).not.toContain('gcmd:newKeywordObject') - expect(sparqlCall.body).not.toContain('gcmd:writebackErrorMessage') - expect(sparqlCall.body).toContain('metadata-correction-audit/audit-record-123') + test('omits the collection URI when the CMR base URL is not configured', async () => { + delete process.env.CMR_BASE_URL + + await persistMetadataCorrectionAuditLog({ + collectionConceptId: 'C123-PROV' + }) + + expect(collection.updateOne.mock.calls[0][1].$set).not.toHaveProperty('collectionUri') }) - test('persists the writeback error message for failed audit rows', async () => { + test('updates the same run to failed and records structured error details', async () => { + collection.findOne.mockResolvedValue({ status: 'pending' }) + const error = Object.assign(new Error('CMR rejected metadata'), { + status: 400, + cmrResponseBody: { errors: ['invalid'] } + }) + await persistMetadataCorrectionAuditLog({ - collectionConceptId: 'C1234567890-LOCAL', - keywordEvent: { - eventType: 'UPDATED' - }, - nativeFormat: 'UMM', - delegateName: 'umm', - corrections: [ - { - scheme: 'sciencekeywords', - keywordConceptUuid: 'uuid-failed', - oldKeywordObject: { - Category: 'EARTH SCIENCE', - Topic: 'ATMOSPHERE', - Term: 'AEROSOLS' - }, - newKeywordObject: { - Category: 'EARTH SCIENCE', - Topic: 'ATMOSPHERE', - Term: 'AEROSOLS' - } - } - ], + runId: 'run-1', + collectionConceptId: 'C123-PROV', + error, + outcome: 'writeback-failed', status: 'failed', - writebackErrorMessage: 'CMR writeback failed with status 400: {"errors":["boom"]}' + timestamp: '2026-09-02T12:01:00.000Z' }) - const sparqlCall = vi.mocked(sparqlRequest).mock.calls[0][0] - expect(sparqlCall.body).toContain('gcmd:status "failed"') - expect(sparqlCall.body).toContain( - 'gcmd:writebackErrorMessage "CMR writeback failed with status 400: {\\"errors\\":[\\"boom\\"]}"' - ) - }) + const update = collection.updateOne.mock.calls[0][1] + expect(update.$set).toEqual(expect.objectContaining({ + status: 'failed', + outcome: 'writeback-failed', + error: { + message: 'CMR rejected metadata', + status: 400, + cmrResponseBody: { errors: ['invalid'] } + } + })) - test('returns without writing when there are no corrections', async () => { - const result = await persistMetadataCorrectionAuditLog({ - collectionConceptId: 'C1234567890-LOCAL', - nativeFormat: 'UMM', - delegateName: 'umm' + expect(update.$push.statusHistory).toEqual({ + status: 'failed', + timestamp: new Date('2026-09-02T12:01:00.000Z'), + outcome: 'writeback-failed', + error: 'CMR rejected metadata' }) - expect(result).toEqual({ - insertedCount: 0, - publishedVersionName: 'published', - status: 'pending' + expect(update.$unset).toBeUndefined() + }) + + test('does not regress pending status when a retried run is checked again', async () => { + collection.findOne.mockResolvedValue({ status: 'pending' }) + + const result = await persistMetadataCorrectionAuditLog({ + runId: 'run-1', + collectionConceptId: 'C123-PROV', + status: 'checked' }) - expect(getVersionMetadata).not.toHaveBeenCalled() - expect(sparqlRequest).not.toHaveBeenCalled() + expect(result.status).toBe('pending') + const update = collection.updateOne.mock.calls[0][1] + expect(update.$set.status).toBe('pending') + expect(update.$set['timestamps.pendingAt']).toBeUndefined() + expect(update.$push).toBeUndefined() }) - test('throws when required audit fields are missing', async () => { - await expect(persistMetadataCorrectionAuditLog({ - nativeFormat: 'UMM', - delegateName: 'umm', - corrections: [{}] - })).rejects.toThrow('Missing collectionConceptId for metadata correction audit persistence') + test('does not update an applied run during a retry', async () => { + collection.findOne.mockResolvedValue({ status: 'applied' }) await expect(persistMetadataCorrectionAuditLog({ - collectionConceptId: 'C1234567890-LOCAL', - delegateName: 'umm', - corrections: [{}] - })).rejects.toThrow('Missing nativeFormat for metadata correction audit persistence') + runId: 'run-1', + collectionConceptId: 'C123-PROV', + status: 'checked' + })).resolves.toEqual({ + runId: 'run-1', + status: 'applied', + created: false + }) - await expect(persistMetadataCorrectionAuditLog({ - collectionConceptId: 'C1234567890-LOCAL', - nativeFormat: 'UMM', - corrections: [{}] - })).rejects.toThrow('Missing delegateName for metadata correction audit persistence') + expect(collection.updateOne).not.toHaveBeenCalled() }) - test('defaults published version, timestamp, and action while omitting optional trigger triples', async () => { - vi.mocked(getVersionMetadata).mockResolvedValue({ - version: 'published', - versionName: '', - versionType: 'published', - created: '2026-01-01T00:00:00Z', - lastSynced: null - }) + test('allows a failed run to restart and clears its previous error', async () => { + collection.findOne.mockResolvedValue({ status: 'failed' }) - await persistMetadataCorrectionAuditLog({ - collectionConceptId: 'C2222222222-LOCAL', - nativeFormat: 'DIF10', - delegateName: 'dif10', - corrections: [ - { - scheme: 'platforms', - keywordConceptUuid: 'uuid-optional', - oldKeywordObject: { - Basis: 'Platforms', - Category: 'Space-based Platforms', - SubCategory: 'Earth Observation Satellites', - ShortName: 'OLD PLATFORM' - }, - newKeywordObject: { - Basis: 'Platforms', - Category: 'Space-based Platforms', - SubCategory: 'Earth Observation Satellites', - ShortName: 'NEW PLATFORM' - } - } - ] + const result = await persistMetadataCorrectionAuditLog({ + runId: 'run-1', + collectionConceptId: 'C123-PROV', + status: 'checked' }) - const sparqlCall = vi.mocked(sparqlRequest).mock.calls[0][0] - expect(sparqlCall.body).toContain('gcmd:publishedVersionName "published"') - expect(sparqlCall.body).toContain('gcmd:action "UNKNOWN"') - expect(sparqlCall.body).toContain('gcmd:delegateName "dif10"') - expect(sparqlCall.body).toContain('gcmd:nativeFormat "DIF10"') - expect(sparqlCall.body).toContain('gcmd:oldKeywordPath "Platforms > Space-based Platforms > Earth Observation Satellites > OLD PLATFORM"') - expect(sparqlCall.body).toContain('gcmd:newKeywordPath "Platforms > Space-based Platforms > Earth Observation Satellites > NEW PLATFORM"') - expect(sparqlCall.body).not.toContain('gcmd:oldKeywordObject') - expect(sparqlCall.body).not.toContain('gcmd:newKeywordObject') - expect(sparqlCall.body).toContain('^^xsd:dateTime') - expect(sparqlCall.body).not.toContain('gcmd:triggerScheme') - expect(sparqlCall.body).not.toContain('gcmd:triggerKeywordUuid') + expect(result.status).toBe('checked') + expect(collection.updateOne.mock.calls[0][1]).toEqual(expect.objectContaining({ + $unset: { error: '' }, + $push: expect.objectContaining({ + statusHistory: expect.objectContaining({ status: 'checked' }) + }) + })) }) - test('omits keyword-path triples when the correction objects do not produce meaningful paths', async () => { - await persistMetadataCorrectionAuditLog({ - collectionConceptId: 'C3333333333-LOCAL', - nativeFormat: 'UMM', - delegateName: 'umm', - corrections: [ - { - scheme: 'platforms', - keywordConceptUuid: 'uuid-empty-paths', - oldKeywordObject: {}, - newKeywordObject: {} - } - ] - }) + test('validates required fields and lifecycle status', async () => { + await expect(persistMetadataCorrectionAuditLog({ + status: 'checked' + })).rejects.toThrow('Missing collectionConceptId') + + await expect(persistMetadataCorrectionAuditLog({ + collectionConceptId: 'C123-PROV', + status: 'unknown' + })).rejects.toThrow('Invalid metadata correction audit status: unknown') - const sparqlCall = vi.mocked(sparqlRequest).mock.calls[0][0] - expect(sparqlCall.body).not.toContain('gcmd:oldKeywordPath') - expect(sparqlCall.body).not.toContain('gcmd:newKeywordPath') + expect(getMetadataCorrectionAuditCollection).not.toHaveBeenCalled() }) }) diff --git a/serverless/src/shared/__tests__/renderMetadataCorrectionAuditHtml.test.js b/serverless/src/shared/__tests__/renderMetadataCorrectionAuditHtml.test.js new file mode 100644 index 00000000..9acabec4 --- /dev/null +++ b/serverless/src/shared/__tests__/renderMetadataCorrectionAuditHtml.test.js @@ -0,0 +1,195 @@ +import { + describe, + expect, + test +} from 'vitest' + +import { renderMetadataCorrectionAuditHtml } from '../renderMetadataCorrectionAuditHtml' + +const PATCH = `=================================================================== +--- cmr-revision-3 ++++ corrected-metadata +@@ -1,1 +1,1 @@ +-GOSAT ++GOSAT - Test1 +` + +describe('renderMetadataCorrectionAuditHtml', () => { + test('renders a safe change table and colored side-by-side native metadata diff', () => { + const view = renderMetadataCorrectionAuditHtml({ + collectionConceptId: 'C123-PROV', + items: [{ + runId: 'run-1', + collectionConceptId: 'C123-PROV', + collectionUri: 'https://cmr.example.com/search/concepts/C123-PROV', + status: 'applied', + updatedAt: new Date('2026-09-07T12:00:00.000Z'), + changes: [{ + scheme: 'platforms', + action: 'UPDATED', + oldKeywordPath: 'Platforms > GOSAT', + newKeywordPath: 'Platforms > GOSAT - Test1' + }], + metadataDiff: { + changed: true, + format: 'unified', + patch: PATCH, + truncated: false + } + }], + nextPageHref: '?format=html&paginationToken=next' + }) + + expect(view).toContain('') + expect(view).toContain('') + expect(view).toContain('name="format" value="html"') + expect(view).toContain('name="collectionConceptId" value="C123-PROV"') + expect(view).toContain('Clear') + expect(view).toContain('
') + expect(view).toContain('Platforms > GOSAT - Test1') + expect(view).toContain('class="d2h-del d2h-change"') + expect(view).toContain('class="d2h-ins d2h-change"') + expect(view).toContain(' - Test1') + expect(view).toContain('white-space: pre;') + expect(view).toContain('overflow-x: auto;') + expect(view).toContain('Next page') + expect(view).toContain('run-1') + }) + + test('keeps summaries compact when a native metadata diff was not requested', () => { + const view = renderMetadataCorrectionAuditHtml({ + items: [{ + runId: 'run/summary', + collectionConceptId: 'C123-PROV', + status: 'applied', + changes: [{ + scheme: 'platforms', + action: 'UPDATED', + oldKeywordPath: 'Platforms > GOSAT', + newKeywordPath: 'Platforms > GOSAT - Test1' + }] + }] + }) + + expect(view).toContain('Keyword changes') + expect(view).toContain('run/summary') + expect(view).not.toContain('Native metadata diff') + expect(view).not.toContain('Run details') + }) + + test('renders complete run context and lifecycle history in detail mode', () => { + const view = renderMetadataCorrectionAuditHtml({ + detail: true, + items: [{ + runId: 'run-detail', + collectionConceptId: 'C123-PROV', + providerId: 'PROV', + publishedVersionName: 'version-1', + nativeFormat: 'UMM-C', + delegateName: 'umm', + source: 'cmrKeywordEventsListener', + outcome: 'writeback-applied', + priorRevisionId: 4, + resultingRevisionId: 5, + messageId: 'message-1', + createdAt: new Date('2026-09-07T11:59:00.000Z'), + updatedAt: new Date('2026-09-07T12:00:00.000Z'), + status: 'applied', + trigger: { + eventType: 'UPDATED', + scheme: 'platforms', + keywordConceptUuid: 'platform-uuid', + timestamp: '2026-09-07T11:58:00.000Z' + }, + corrections: [], + keywordValidationFailures: [{ path: 'Platforms > Missing' }], + statusHistory: [{ + status: 'checked', + timestamp: new Date('2026-09-07T11:59:00.000Z'), + outcome: 'corrections-resolved' + }, { + status: 'failed', + timestamp: '2026-09-07T12:00:00.000Z', + error: 'CMR failed' + }] + }] + }) + + expect(view).toContain('Run details') + expect(view).toContain('Published KMS version') + expect(view).toContain('version-1') + expect(view).toContain('Prior CMR revision') + expect(view).toContain('Trigger') + expect(view).toContain('platform-uuid') + expect(view).toContain('Keyword validation failures') + expect(view).toContain('"path": "Platforms > Missing"') + expect(view).toContain('Lifecycle history') + expect(view).toContain('corrections-resolved') + expect(view).toContain('CMR failed') + expect(view).toContain('No native metadata diff was recorded for this run.') + expect(view).toContain('

Run run-detail') + expect(view).not.toContain('metadata_correction_audit/run-detail?format=html') + }) + + test('escapes audit content and reports absent and truncated diffs', () => { + const view = renderMetadataCorrectionAuditHtml({ + detail: true, + items: [ + { + runId: '', + collectionConceptId: '', + collectionUri: 'mailto:not-a-web-link@example.com', + status: 'failed', + error: { message: '' } + }, + { + runId: 'run-2', + collectionConceptId: 'C456-PROV', + status: 'pending', + corrections: [], + metadataDiff: { + patch: PATCH, + truncated: true + } + }, + { + runId: 'run-3', + collectionConceptId: 'C789-PROV', + status: 'checked', + metadataDiff: { + patch: { invalid: true } + } + }, + { + runId: 'run-4', + collectionConceptId: 'C999-PROV', + status: 'applied', + metadataDiff: { + patch: PATCH.replace('GOSAT - Test1', '') + } + } + ] + }) + + expect(view).not.toContain('