fix: no mail data in the errors of an encrypted mail job

CHANGELOG.md

@@ -1,5 +1,6 @@
 ## Unreleased
 
+- Fix: with `Mail.queue(message, encrypt: true)`, a send error could name the recipient (eg. an SMTP reply `550 <ann@example.com>: mailbox unavailable`). The worker wrote that error to its log and to the failed jobs in plain text. Now the encrypted mail job throws a `RedactedMailError`. Its text has only the error type and the SMTP or HTTP code. The full error text, with the answer of the server, is encrypted with the app key (at most the first 64K characters, counted in UTF-16 code units). Read it with `RedactedMailError.decrypt(job.error)`. When the error can not be encrypted, only the type and the code are kept. Mail jobs without `encrypt` keep the full error, as before.
 - New: a built-in job for `Mail.queue()` and `Mail.later()` of dox_mailer. `Queue.configure()` turns it on. Apps do not add it to `lib/generated/jobs.g.dart`: every `JobRegistry` finds the built-in jobs (`JobRegistry.builtIn`) after its own jobs. With `encrypt: true`, the mail is saved like a `ShouldBeEncrypted` job.
 - Now depends on dox_mailer.
 - Deploy the queue worker together with the web app. An old worker does not know the new mail job, so it fails the queued emails.

README.md

@@ -277,6 +277,9 @@ dox_queue has a built-in job for it. It is not in `lib/generated/jobs.g.dart`, a
 
 Deploy the worker together with the web app. An old worker does not know this job, so it fails the queued emails.
 
+With `encrypt: true`, a send error is logged and saved as its type and code only, eg. `RedactedMailError: MailException (code 550). Details are encrypted: ...`.
+The full error text, with the answer of the server, is encrypted with the app key (at most the first 64K characters, counted in UTF-16 code units). To read it, take a `FailedJob` from `Queue.driver.failed()` and call `RedactedMailError.decrypt(job.error)`.
+
 ## Run the worker
 
 ```bash

lib/dox_queue.dart

@@ -14,4 +14,5 @@ export 'src/queue_driver.dart';
 export 'src/queue_provider.dart';
 export 'src/queue_tables.dart';
 export 'src/queue_worker.dart';
+export 'src/send_queued_mail.dart' show RedactedMailError;
 export 'src/worker.dart';

lib/src/send_queued_mail.dart

@@ -1,4 +1,6 @@
-import 'package:dox_mailer/dox_mailer.dart' show Mail, MailQueue, Message;
+import 'package:dox_core/dox_core.dart' show AESEncryptor, Dox;
+import 'package:dox_mailer/dox_mailer.dart'
+    show Mail, MailException, MailQueue, Message;
 
 import 'job.dart';
 import 'job_registry.dart';
@@ -62,6 +64,19 @@ class SendEncryptedQueuedMail extends SendQueuedMail with ShouldBeEncrypted {
     _decode,
   );
 
+  /// Sends the mail. When the send fails, it throws a [RedactedMailError]
+  /// in place of the error. An SMTP reply or a [MailException] can name
+  /// an address, and the worker writes the error to its log and to the
+  /// failed jobs in plain text.
+  @override
+  Future<void> handle() async {
+    try {
+      await super.handle();
+    } catch (error, stack) {
+      Error.throwWithStackTrace(RedactedMailError.of(error), stack);
+    }
+  }
+
   static DoxJob _decode(Map<String, Object?> data) {
     return SendEncryptedQueuedMail(
       mailer: data['mailer'] as String,
@@ -72,6 +87,107 @@ class SendEncryptedQueuedMail extends SendQueuedMail with ShouldBeEncrypted {
   }
 }
 
+/// The error of a mail of `Mail.queue(message, encrypt: true)` that
+/// could not be sent. The worker logs and saves it in place of the
+/// first error, so no address or text of the mail is written in plain
+/// text.
+///
+/// Its text has only the type of the first error and the SMTP or HTTP
+/// code. The full text of the first error (and the answer of the server,
+/// `MailException.body`) is encrypted with the app key, like the job
+/// data. At most the first 64K characters (UTF-16 code units) are
+/// encrypted, so a huge error does not fill the log. Read it with
+/// [decrypt].
+/// ```
+/// RedactedMailError: MailException (code 550). Details are encrypted: <base64>
+/// ```
+class RedactedMailError implements Exception {
+  RedactedMailError(this.type, {this.statusCode, this.details});
+
+  /// [error] with its text encrypted. When that fails, only the type
+  /// (and the code) is kept, never the text. Everything is read from
+  /// [error] inside the `try`, so an error that throws can not leak.
+  factory RedactedMailError.of(Object error) {
+    String type = 'unknown';
+    int? code;
+    String? details;
+    try {
+      type = '${error.runtimeType}';
+      code = error is MailException ? error.statusCode : null;
+      String text = '$error';
+      if (error is MailException && (error.body ?? '').isNotEmpty) {
+        text = '$text\nbody: ${error.body}';
+      }
+      details = Queue.encryptData(_cut(text), _purpose);
+    } catch (_) {
+      details = null;
+    }
+    return RedactedMailError(type, statusCode: code, details: details);
+  }
+
+  /// the most characters (UTF-16 code units) of the error that are
+  /// encrypted
+  static const int maxDetails = 64 * 1024;
+
+  /// the first [maxDetails] characters of [text]. It does not cut an
+  /// emoji (a surrogate pair) in two.
+  static String _cut(String text) {
+    if (text.length <= maxDetails) return text;
+    int end = maxDetails;
+    int last = text.codeUnitAt(end - 1);
+    if (last >= 0xD800 && last <= 0xDBFF) end--;
+    return text.substring(0, end);
+  }
+
+  /// bound to the encrypted details, so job data can not be read as them
+  static const String _purpose = 'dox_queue:RedactedMailError';
+
+  static const String _marker = 'Details are encrypted: ';
+
+  /// the class of the first error, eg. `MailException`
+  final String type;
+
+  /// the SMTP reply code or the HTTP status, when there is one
+  final int? statusCode;
+
+  /// the encrypted text of the first error, or null when it could not
+  /// be encrypted
+  final String? details;
+
+  /// The text of the first error, from a saved or logged [error] text
+  /// (eg. the `error` of a failed job). Needs the same app key, or one
+  /// of `AppConfig.previousKeys`. Null when it has no details. Throws a
+  /// `DecryptException` when the key is wrong, or the details are not
+  /// real encrypted details.
+  /// ```
+  /// for (FailedJob job in await Queue.driver.failed()) {
+  ///   print(RedactedMailError.decrypt(job.error));
+  /// }
+  /// ```
+  static String? decrypt(String error) {
+    int at = error.indexOf(_marker);
+    if (at < 0) return null;
+    String rest = error.substring(at + _marker.length);
+    String details = rest.split(RegExp(r'\s')).first;
+    return AESEncryptor.decode(
+      details,
+      Dox().config.appKey,
+      associatedData: _purpose,
+      purpose: Queue.encryptPurpose,
+      previousSecrets: Dox().config.previousKeys,
+    );
+  }
+
+  @override
+  String toString() {
+    String code = statusCode == null ? '' : ' (code $statusCode)';
+    String text = details == null
+        ? 'details could not be encrypted'
+        : '$_marker$details';
+    return 'RedactedMailError: $type$code. $text';
+  }
+}
+
 /// Puts the mails of `Mail.queue()` on this queue. `Queue.configure()`
 /// sets it with `Mail.useQueue()`.
 class QueueMailQueue implements MailQueue {

test/queue_mail_test.dart

@@ -1,3 +1,6 @@
+import 'dart:async';
+
+import 'package:dox_core/dox_core.dart' show AppConfig, Dox, LoggerConfig;
 import 'package:dox_mailer/dox_mailer.dart';
 import 'package:dox_queue/dox_queue.dart';
 import 'package:test/test.dart';
@@ -20,6 +23,49 @@ class BrokenTransport extends Transport {
 
 late BrokenTransport brokenMailer;
 
+/// a mail server that says no, and names the recipient and the subject,
+/// like many SMTP servers do
+class RejectTransport extends Transport {
+  @override
+  Future<SentMessage> send(Message message) async {
+    throw MailException(
+      '550 5.1.1 <ANN@example.com>: mailbox unavailable '
+      '(subject "Your invoice", for Ann, forwarded to ann.lee@other.org)',
+      statusCode: 550,
+    );
+  }
+}
+
+/// a mail server that quotes the whole body in its answer
+class QuoteTransport extends Transport {
+  @override
+  Future<SentMessage> send(Message message) async {
+    throw MailException(
+      '552 message refused: ${message.html}',
+      statusCode: 552,
+    );
+  }
+}
+
+/// every line the worker logs while [body] runs
+Future<String> _logOf(Future<void> Function() body) async {
+  StringBuffer log = StringBuffer();
+  void keep(Zone self, ZoneDelegate parent, Zone zone, String line) {
+    log.writeln(line);
+  }
+
+  await runZoned(body, zoneSpecification: ZoneSpecification(print: keep));
+  return log.toString();
+}
+
+/// the parts of the test mail that must not be in a log
+const List<String> _secrets = <String>[
+  'ann@example.com',
+  'ann.lee@other.org',
+  'Your invoice',
+  'Ann',
+];
+
 Message _message() => Message(
   to: <Address>[const Address('ann@example.com', 'Ann')],
   bcc: <Address>[const Address('boss@example.com')],
@@ -45,7 +91,17 @@ Future<JobPayload> _work([String queue = 'default', int tries = 1]) async {
 }
 
 void main() {
-  setUpAll(setUpTestApp);
+  setUpAll(() {
+    setUpTestApp();
+    // log on, so the tests can read what the worker logs
+    Dox().initialize(
+      AppConfig(
+        appKey: 'k' * 32,
+        totalIsolate: 1,
+        logger: LoggerConfig(enabled: true),
+      ),
+    );
+  });
 
   setUp(() {
     Mail.restore();
@@ -60,6 +116,8 @@ void main() {
           'main': mainMailer,
           'other': otherMailer,
           'broken': brokenMailer,
+          'reject': RejectTransport(),
+          'quote': QuoteTransport(),
         },
       ),
     );
@@ -123,6 +181,47 @@ void main() {
         expect(await Queue.driver.pop('default'), isNull);
       });
 
+      test('encrypt: true: a send error shows no mail data', () async {
+        await Mail.mailer('reject').queue(_message(), encrypt: true);
+        String log = await _logOf(_retryThenFail);
+        List<FailedJob> failed = await Queue.driver.failed();
+        String error = failed.single.error;
+        for (String secret in _secrets) {
+          expect(log, isNot(contains(secret)));
+          expect(error, isNot(contains(secret)));
+        }
+        // still enough to find the problem
+        printOnFailure(log);
+        expect(log, contains('failed try 1 of 2'));
+        expect(log, contains('RedactedMailError: MailException (code 550).'));
+        expect(error, contains('RedactedMailError: MailException (code 550).'));
+        // the app key gives back the full error
+        expect(
+          RedactedMailError.decrypt(error),
+          startsWith('MailException: 550 5.1.1 <ANN@example.com>'),
+        );
+      });
+
+      test('encrypt: true: a 100 KB body is not in the error', () async {
+        String html = '<p>Your reset code is 918273</p>${'x' * 100 * 1024}';
+        Message message = _message().copyWith(html: html);
+        await Mail.mailer('quote').queue(message, encrypt: true);
+        String log = await _logOf(_retryThenFail);
+        String error = (await Queue.driver.failed()).single.error;
+        for (String text in <String>[log, error]) {
+          expect(text, isNot(contains('918273')));
+          expect(text, isNot(contains('xxxx')));
+          expect(text, contains('MailException (code 552). Details are'));
+        }
+      });
+
+      test('a plain mail job keeps the full error, as before', () async {
+        await Mail.mailer('reject').queue(_message());
+        await _logOf(_retryThenFail);
+        List<FailedJob> failed = await Queue.driver.failed();
+        expect(failed.single.error, contains('<ANN@example.com>'));
+      });
+
       test('encrypt: true saves the mail encrypted', () async {
         await Mail.queue(_message(), encrypt: true);
         JobPayload payload = await _work();
@@ -135,3 +234,9 @@ void main() {
     });
   }
 }
+
+/// two tries: the first is released, the second fails
+Future<void> _retryThenFail() async {
+  await _work('default', 2);
+  await _work('default', 2);
+}