From 1812cc7737ad296b113d5e1e40b73a2ac34431b6 Mon Sep 17 00:00:00 2001 From: "@royashbrook" Date: Thu, 5 Nov 2026 09:24:00 -0500 Subject: [PATCH] add the http destination and pin Send-FileViaEmail 2.1.0.0 for body mode, 0.11.0 (fix #34) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015ZRSvwh5rc6izCELestiVq --- .github/workflows/test.yml | 2 +- DataAgent/CAPABILITIES.md | 6 +++-- DataAgent/DataAgent.psd1 | 6 ++--- DataAgent/dst/email.ps1 | 2 +- DataAgent/dst/http.ps1 | 11 +++++++++ README.md | 4 ++-- examples/job.ps1 | 2 +- testing/DataAgent.Test/DataAgent.Test.psd1 | 4 ++-- tests/acceptance.ps1 | 9 ++++++-- tests/transfers.ps1 | 26 ++++++++++++++++++++++ 10 files changed, 58 insertions(+), 14 deletions(-) create mode 100644 DataAgent/dst/http.ps1 diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 914d56c..2ae7f3f 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -16,7 +16,7 @@ jobs: run: | Install-Module Add-PrefixForLogging -RequiredVersion 1.0.0.3 -Scope CurrentUser -Force -Repository PSGallery Install-Module Clear-Files -RequiredVersion 1.2.0.0 -Scope CurrentUser -Force -Repository PSGallery - Install-Module Send-FileViaEmail -RequiredVersion 2.0.0.0 -Scope CurrentUser -Force -Repository PSGallery + Install-Module Send-FileViaEmail -RequiredVersion 2.1.0.0 -Scope CurrentUser -Force -Repository PSGallery Install-Module ImportExcel -RequiredVersion 7.8.10 -Scope CurrentUser -Force -Repository PSGallery - name: Offline acceptance shell: pwsh diff --git a/DataAgent/CAPABILITIES.md b/DataAgent/CAPABILITIES.md index c71990b..ce2d329 100644 --- a/DataAgent/CAPABILITIES.md +++ b/DataAgent/CAPABILITIES.md @@ -37,6 +37,7 @@ owned options are the keys the adapter interprets itself. other arguments belong | `dst/sftp` | `connect`, `send`, `attempts`, `delay` | `New-SFTPSession`, `Set-SFTPItem` | | `dst/ftp` | `Uri`, `Credential` | `FtpWebRequest` | | `dst/ftps` | `Uri`, `Credential` | `FtpWebRequest` | +| `dst/http` | `uri`, `headers`, `nameheader`, `contenttype` | `Invoke-RestMethod` | - **sql:** arguments pass to Invoke-Sqlcmd, including connection strings and authentication settings. DataTable results emit their rows; empty tables emit nothing. multiple result sets are flattened, not separate artifacts. use compatible schemas. - **csv source:** arguments pass directly to Import-Csv. relative paths resolve in the job directory. @@ -45,7 +46,8 @@ owned options are the keys the adapter interprets itself. other arguments belong - **xlsx:** all arguments pass to Export-Excel, including Path and layout options. an existing file at Path is replaced, never written into, so a shorter run leaves no rows from the last one. verify cells/layout, not ZIP hashes. AutoSize may need native support on non-Windows hosts. - **custom format bridge:** Module is a path to a module exporting `ConvertTo-Custom($dt)`. the bridge passes a DataTable and saves returned text to Path as UTF-8 without BOM. this bridge is for existing text converters, not binary ZIP output. - **invoice formats:** pdi writes H/D/I records; x12-210 writes an interchange with one transaction per invoice; dtn takes one object per invoice with nested Detail records. baseline templates ship in fmt/invoice/*.psd1. Values supplies company identifiers, addresses, terms and other constants. every template or field selector can be overridden by its listed option. substitutions support `{field}`, `{a.b}`, `{field,-15}` and `{field:0.00}`. Values wins over row fields. missing fields become empty, values are not escaped or truncated, and an alignment width is a minimum. these are configured layouts, not general CSV quoting or X12/DTN certification. prove the receiver's exact bytes before adoption. x12-210 defaults ISA ids to 15-wide, YYMMDD, a 9-digit interchange number and matching IEA; GS/GE use the same unpadded group number. ST/SE and segment counts are generated; SeExtra permits an explicit legacy count offset. the default control number is the extracted hour-of-year convention, not uniqueness across jobs or reruns. Values.now and Values.control can supply a filing time and number instead. dtn Values.dtn supplies user/pass/site for the file's login block; keep those values in env: variables, not committed config. its Header and Line are column templates, so a numeric precision variant needs no alternate formatter. -- **email:** one filename, one Send-FileViaEmail call with `file` supplied by the adapter. other arguments pass through, such as `cfg` and `contentType`. multiple filenames throw before sending, never fan out. use a filename in the job directory: the helper uses it as both path and attachment name. no body/multi-attachment API or new size policy is provided here. +- **email:** one filename, one Send-FileViaEmail call with `file` supplied by the adapter. other arguments pass through, such as `cfg`, `contentType` and `bodyType`. `bodyType` (Html or Text) is the helper's: the file's text becomes the message body and nothing is attached. multiple filenames throw before sending, never fan out. use a filename in the job directory: the helper uses it as both path and attachment name. no multi-attachment API or new size policy is provided here. +- **http:** each file's text is POSTed to `uri` as the request body, one request per file. `headers` is a hashtable sent as is, so a key belongs in an `env:` value there. `nameheader`, when set, names a header that carries the file name. `contenttype` sets the request content type, otherwise Invoke-RestMethod chooses. a non-success status throws and stops the job; the response is written to the log. no retry, auth flow or multipart upload. - **sftp:** connect is splatted to New-SFTPSession, send to Set-SFTPItem. the adapter supplies the session and local Path; the session closes even on failure. a connect that fails is tried again, up to `attempts` tries in all (3) `delay` seconds apart (60), and the last error is the one thrown. choose the intended host-key/overwrite policy. connection Force bypasses host-key validation; prefer verified trusted hosts. - **ftp / ftps:** independent scripts. Uri is a remote directory (`ftp://host/path/`); Credential is a PSCredential or .NET NetworkCredential. uploads replace normally. FTPS uses explicit TLS, not implicit FTPS. plain FTP sends credentials and data unencrypted. @@ -77,7 +79,7 @@ adapters must not change location. the runner moves PowerShell's location, not t ## dependencies and limits -the runner does not install dependencies. every helper is imported at a pinned version, the one the tests ran against: Add-PrefixForLogging 1.0.0.3, Clear-Files 1.2.0.0, SqlServer 22.4.5.1, Send-FileViaEmail 2.0.0.0, ImportExcel 7.8.10, Posh-SSH 4.0.0. install exactly those versions on the runner, only for the helpers the job uses; a newer install beside them changes nothing, a missing pinned version stops the run. +the runner does not install dependencies. every helper is imported at a pinned version, the one the tests ran against: Add-PrefixForLogging 1.0.0.3, Clear-Files 1.2.0.0, SqlServer 22.4.5.1, Send-FileViaEmail 2.1.0.0, ImportExcel 7.8.10, Posh-SSH 4.0.0. install exactly those versions on the runner, only for the helpers the job uses; a newer install beside them changes nothing, a missing pinned version stops the run. no cache/deduplication, acknowledgment tracking, retry engine, receipts, scheduler or conditional routing. cleanup remains Clear-Files behavior; choose deliberate patterns in a dedicated output directory. provider limits still apply. before adopting a feed, prove its output/layout, runner identity, dependencies, cleanup and scoped delivery behavior. this brief is not a live-provider test. diff --git a/DataAgent/DataAgent.psd1 b/DataAgent/DataAgent.psd1 index 850da87..9b61430 100644 --- a/DataAgent/DataAgent.psd1 +++ b/DataAgent/DataAgent.psd1 @@ -1,6 +1,6 @@ @{ RootModule = 'DataAgent.psm1' - ModuleVersion = '0.10.0' + ModuleVersion = '0.11.0' GUID = '32662c0b-0a5d-49ac-8c62-8f7ae05f79c0' Author = 'Roy Ashbrook' Copyright = '(c) 2026 Roy Ashbrook. MIT License.' @@ -10,13 +10,13 @@ CmdletsToExport = @() VariablesToExport = @() AliasesToExport = @() - FileList = @('DataAgent.psd1', 'DataAgent.psm1', 'LICENSE', 'CAPABILITIES.md', 'src/sql.ps1', 'src/csv.ps1', 'src/cosmos.ps1', 'fmt/csv.ps1', 'fmt/xlsx.ps1', 'fmt/custom.ps1', 'fmt/pdi.ps1', 'fmt/x12-210.ps1', 'fmt/dtn.ps1', 'fmt/invoice/template.ps1', 'fmt/invoice/pdi.psd1', 'fmt/invoice/x12-210.psd1', 'fmt/invoice/dtn.psd1', 'fmt/text/frame.ps1', 'dst/email.ps1', 'dst/sftp.ps1', 'dst/ftp.ps1', 'dst/ftps.ps1') + FileList = @('DataAgent.psd1', 'DataAgent.psm1', 'LICENSE', 'CAPABILITIES.md', 'src/sql.ps1', 'src/csv.ps1', 'src/cosmos.ps1', 'fmt/csv.ps1', 'fmt/xlsx.ps1', 'fmt/custom.ps1', 'fmt/pdi.ps1', 'fmt/x12-210.ps1', 'fmt/dtn.ps1', 'fmt/invoice/template.ps1', 'fmt/invoice/pdi.psd1', 'fmt/invoice/x12-210.psd1', 'fmt/invoice/dtn.psd1', 'fmt/text/frame.ps1', 'dst/email.ps1', 'dst/sftp.ps1', 'dst/ftp.ps1', 'dst/ftps.ps1', 'dst/http.ps1') PrivateData = @{ PSData = @{ Tags = @('ETL', 'Data', 'CSV', 'Automation', 'PSEdition_Core', 'Windows', 'Linux', 'MacOS') LicenseUri = 'https://github.com/royashbrook/DataAgent/blob/main/LICENSE' ProjectUri = 'https://github.com/royashbrook/DataAgent' - ReleaseNotes = 'Preserve invoice input order. Add configured x12 conditions, control/date schemes, separators, terminators and legacy counts. Add text prepend/append with optional distinct-key sums, and per-invoice footer templates. No helper pin or destination changes. Re-prove receiver output before adoption.' + ReleaseNotes = 'Add the http destination: post each file to a uri with headers. Pin Send-FileViaEmail 2.1.0.0, whose bodyType sends the file as the message body; install it on the runner before upgrading.' } } } diff --git a/DataAgent/dst/email.ps1 b/DataAgent/dst/email.ps1 index 6033f7a..d45d732 100644 --- a/DataAgent/dst/email.ps1 +++ b/DataAgent/dst/email.ps1 @@ -1,4 +1,4 @@ param([string[]] $Data, [hashtable] $Options) if ($Data.Count -ne 1) { throw 'email requires one file; multiple attachments are not supported' } -Import-Module Send-FileViaEmail -RequiredVersion 2.0.0.0 -ErrorAction Stop +Import-Module Send-FileViaEmail -RequiredVersion 2.1.0.0 -ErrorAction Stop Send-FileViaEmail -file $Data[0] @Options diff --git a/DataAgent/dst/http.ps1 b/DataAgent/dst/http.ps1 new file mode 100644 index 0000000..ed8bdda --- /dev/null +++ b/DataAgent/dst/http.ps1 @@ -0,0 +1,11 @@ +param([string[]] $Data, [hashtable] $Options) +foreach ($file in $Data) { + $headers = if ($Options.headers) { @{} + $Options.headers } else { @{} } + # some receivers take the file's name in a header they name themselves + if ($Options.nameheader) { $headers[$Options.nameheader] = $file } + $post = @{ Method = 'Post'; Uri = $Options.uri; Headers = $headers; Body = Get-Content -LiteralPath $file -Raw } + if ($Options.contenttype) { $post.ContentType = $Options.contenttype } + # a non-2xx answer throws here with the receiver's status, so a refused post fails the run + $response = Invoke-RestMethod @post + Write-Host ("Response: {0}" -f $response) +} diff --git a/README.md b/README.md index 3463858..9eed727 100644 --- a/README.md +++ b/README.md @@ -3,13 +3,13 @@ the repeated part of a feed job: get data, format it, send it. use the tools you already use. ```powershell -Install-Module DataAgent -RequiredVersion 0.10.0 -Scope CurrentUser +Install-Module DataAgent -RequiredVersion 0.11.0 -Scope CurrentUser ``` ## the job ```powershell -Import-Module DataAgent -RequiredVersion 0.10.0 +Import-Module DataAgent -RequiredVersion 0.11.0 Invoke-DataAgent "$PSScriptRoot/settings.json" ``` diff --git a/examples/job.ps1 b/examples/job.ps1 index 8be3f2f..c1cf34f 100644 --- a/examples/job.ps1 +++ b/examples/job.ps1 @@ -1,2 +1,2 @@ -Import-Module DataAgent -RequiredVersion 0.10.0 -ErrorAction Stop +Import-Module DataAgent -RequiredVersion 0.11.0 -ErrorAction Stop Invoke-DataAgent "$PSScriptRoot/settings.json" diff --git a/testing/DataAgent.Test/DataAgent.Test.psd1 b/testing/DataAgent.Test/DataAgent.Test.psd1 index af13005..39d7f72 100644 --- a/testing/DataAgent.Test/DataAgent.Test.psd1 +++ b/testing/DataAgent.Test/DataAgent.Test.psd1 @@ -1,6 +1,6 @@ @{ RootModule = 'DataAgent.Test.psm1' - ModuleVersion = '0.10.0' + ModuleVersion = '0.11.0' GUID = '3d16fcb0-a991-4c65-a6e4-3bfcd360688d' Author = 'Roy Ashbrook' Copyright = '(c) 2026 Roy Ashbrook. MIT License.' @@ -10,7 +10,7 @@ CmdletsToExport = @() VariablesToExport = @() AliasesToExport = @() - RequiredModules = @(@{ ModuleName = 'DataAgent'; RequiredVersion = '0.10.0' }) + RequiredModules = @(@{ ModuleName = 'DataAgent'; RequiredVersion = '0.11.0' }) FileList = @('DataAgent.Test.psd1', 'DataAgent.Test.psm1', 'LICENSE', 'synthetic.csv') PrivateData = @{ PSData = @{ Tags = @('ETL', 'DataAgent'); ProjectUri = 'https://github.com/royashbrook/DataAgent'; LicenseUri = 'https://github.com/royashbrook/DataAgent/blob/main/LICENSE' } } } diff --git a/tests/acceptance.ps1 b/tests/acceptance.ps1 index dddcafa..3bf998e 100644 --- a/tests/acceptance.ps1 +++ b/tests/acceptance.ps1 @@ -278,7 +278,7 @@ $null = Run 'xlsx' $cfg Assert (@(Import-Excel "$root/xlsx/output.xlsx").Count -eq 1) 'XLSX replaces the file: a shorter run leaves none of the last run''s rows' # The real email helper runs; only its HTTP boundary is replaced in this process. -Import-Module Send-FileViaEmail -RequiredVersion 2.0.0.0 +Import-Module Send-FileViaEmail -RequiredVersion 2.1.0.0 & (Get-Module Send-FileViaEmail) { $script:messages = [Collections.Generic.List[object]]::new() function script:Invoke-RestMethod { @@ -302,6 +302,11 @@ Assert (($log -join "`n") -notmatch 'synthetic secret|synthetic connection') 'ru Refuses { & "$repo/DataAgent/dst/email.ps1" -Data @('one.csv','two.csv') -Options $cfg.dst.args } 'one file' Assert (@(& (Get-Module Send-FileViaEmail) { $script:messages.ToArray() }).Count -eq 1) 'multiple files refused before any email' Assert ((Get-Location).Path -eq (Join-Path $root 'email')) 'email does not change directory' +$cfg = Config 'email-body' +$cfg.dst = @{ adapter = 'email'; args = @{ bodyType = 'Html'; cfg = @{ msgraph = @{ tenant_id = 'example'; client_id = 'example'; client_secret = 'synthetic secret' }; mail = @{ from = 'from@example.invalid'; to = @('to@example.invalid'); subject = 'report' } } } } +$null = Run 'email-body' $cfg +$body = @(& (Get-Module Send-FileViaEmail) { $script:messages.ToArray() })[-1].message +Assert ($body.body.contentType -eq 'Html' -and $body.body.content -eq (Get-Content "$root/email-body/output.csv" -Raw) -and !$body.attachments) 'bodyType passes through: the file is the message body' $cfg = Config 'extension' @' @@ -383,7 +388,7 @@ foreach ($package in @('DataAgent','testing/DataAgent.Test')) { } Remove-Module DataAgent -Force $env:PSModulePath = (@("$root/staged", $priorModules) -join [IO.Path]::PathSeparator) -Import-Module DataAgent.Test -RequiredVersion 0.10.0 +Import-Module DataAgent.Test -RequiredVersion 0.11.0 $result = @(Test-DataAgent) Assert (@($result | Where-Object { $_ -is [IO.FileInfo] -and $_.Name -eq 'output.csv' }).Count -eq 1) 'staged optional test package exercises bundled formatter' $env:PSModulePath = $priorModules diff --git a/tests/transfers.ps1 b/tests/transfers.ps1 index 7992612..e41792f 100644 --- a/tests/transfers.ps1 +++ b/tests/transfers.ps1 @@ -75,3 +75,29 @@ Assert ([TestFtpFactory]::Requests[-1].EnableSsl) 'FTPS selector enables TLS wit [TestFtpRequest]::Fail=$true Refuses { Run 'ftp' $cfg } 'transfer failed' Assert ([TestFtpFactory]::Requests[-1].Aborted) 'FTP releases request after failure' + +# the http boundary is replaced for this process only; a function wins over the cmdlet of the same name +$global:httpPosts = [Collections.Generic.List[object]]::new() +$global:httpRefuse = $false +function global:Invoke-RestMethod { + param($Method, $Uri, $Headers, $Body, $ContentType) + if ($global:httpRefuse) { throw 'Response status code does not indicate success: 400 (Bad Request).' } + $global:httpPosts.Add(@{ Method = $Method; Uri = $Uri; Headers = $Headers; Body = $Body; ContentType = $ContentType }); 'accepted' +} +$env:DATAAGENT_TEST_KEY = 'synthetic key' +$cfg = Config 'http' +$cfg.dst = @{ adapter = 'http'; args = @{ uri = 'https://dataagent.invalid/upload'; headers = @{ 'Api-Key' = 'env:DATAAGENT_TEST_KEY' }; nameheader = 'filename'; contenttype = 'text/plain' } } +$log = Run 'http' $cfg +$post = $global:httpPosts[-1] +Assert ($post.Method -eq 'Post' -and $post.Uri -eq 'https://dataagent.invalid/upload' -and $post.ContentType -eq 'text/plain') 'http posts to the uri with the content type' +Assert ($post.Headers['Api-Key'] -eq 'synthetic key' -and $post.Headers.filename -eq 'output.csv') 'http headers resolve env: and carry the file name' +Assert ($post.Body -eq (Get-Content "$root/http/output.csv" -Raw)) 'http body is the file text' +Assert (($log -join "`n") -match 'Response: accepted' -and ($log -join "`n") -notmatch 'synthetic key') 'http logs the answer, never the key' +$cfg.dst.args.Remove('nameheader') +$null = Run 'http' $cfg +Assert ($global:httpPosts.Count -eq 2 -and !$global:httpPosts[-1].Headers.ContainsKey('filename')) 'no nameheader, no file name header' +$global:httpRefuse = $true +Refuses { Run 'http' $cfg } '400' +Remove-Item function:global:Invoke-RestMethod +Remove-Variable httpPosts, httpRefuse -Scope Global +Remove-Item Env:DATAAGENT_TEST_KEY