[17.0][ADD] edi_exchange_file_db_oca

edi_exchange_file_db_oca/README.rst

@@ -0,0 +1,230 @@
+=============================
+EDI Exchange File in Database
+=============================
+
+.. 
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! This file is generated by oca-gen-addon-readme !!
+   !! changes will be overwritten.                   !!
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! source digest: sha256:8e72ea86ccfb18fc3d93d3f584a29464a8538c75b2f56e0ed471067e3fd41571
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
+.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png
+    :target: https://odoo-community.org/page/development-status
+    :alt: Beta
+.. |badge2| image:: https://img.shields.io/badge/licence-LGPL--3-blue.png
+    :target: http://www.gnu.org/licenses/lgpl-3.0-standalone.html
+    :alt: License: LGPL-3
+.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fedi--framework-lightgray.png?logo=github
+    :target: https://github.com/OCA/edi-framework/tree/17.0/edi_exchange_file_db_oca
+    :alt: OCA/edi-framework
+.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
+    :target: https://translation.odoo-community.org/projects/edi-framework-17-0/edi-framework-17-0-edi_exchange_file_db_oca
+    :alt: Translate me on Weblate
+.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png
+    :target: https://runboat.odoo-community.org/builds?repo=OCA/edi-framework&target_branch=17.0
+    :alt: Try me on Runboat
+
+|badge1| |badge2| |badge3| |badge4| |badge5|
+
+This module keeps the content of EDI exchange files in the database
+instead of the filestore, so a database restored without its filestore
+still holds every exchange payload. The **Exchange file** of an exchange
+record is still an attachment: only the place where that attachment
+keeps its content changes, from a filestore file to the database, so
+everything that reads, writes or searches exchange files works as
+before. It is enabled for every exchange type on installation, can be
+disabled per type, and a size limit keeps big files in the filestore.
+The exchange record form also shows the content of text files in a
+**File Content** tab, with JSON indented.
+
+**Table of contents**
+
+.. contents::
+   :local:
+
+Use Cases / Context
+===================
+
+Databases are often restored without their filestore for development,
+debugging or testing: on a busy instance it can weigh far more than the
+database, and copying it makes every restore slower and heavier. Such a
+database keeps every exchange record but none of their files, so an
+exchange can no longer be inspected, reprocessed or replayed to
+reproduce an issue. EDI payloads are usually small and a negligible
+share of the filestore, so keeping their content in the database costs
+little.
+
+Configuration
+=============
+
+No configuration is required: on installation every exchange type stores
+the content of its files in the database.
+
+To change it for one exchange type:
+
+1. Go to *EDI > Config > Exchange types* and open the type.
+2. Untick **Store File Content in Database** to keep the content of its
+   files in the filestore, or set **Max File Content Size in Database
+   (KB)** so that bigger files keep their content in the filestore. 0
+   uses the global limit.
+
+To set the global limit, go to *Settings > Technical > Parameters >
+System Parameters* (developer mode) and edit
+``edi_exchange_file_db_oca.max_size_kb``, in KB. The module creates it
+with ``0``, which means no limit.
+
+Both settings apply when a file is written: the files already stored
+keep their content where it is (see **Usage**).
+
+Usage
+=====
+
+Exchange files are handled as before: the module only changes where
+their content is kept.
+
+To read the content of an exchange file:
+
+1. Go to *EDI > Exchanges* and open an exchange record.
+2. Open the **File Content** tab. JSON is indented, and JSON and XML are
+   highlighted; other text, such as CSV or EDIFACT, is shown as it is.
+   Text that does not match the encoding of the exchange type is decoded
+   with the encoding detected from its content.
+
+Binary files, such as PDF, images or spreadsheets, and files above 1 MB
+are not displayed: the tab shows their type, or their size limit, and
+they are downloaded from the **Exchange file** field.
+
+On installation, the content of the existing exchange files is moved
+from the filestore to the database, following the type settings. Files
+missing from the filestore, or whose content does not match their
+checksum, are left untouched. The server log reports how many files were
+moved and skipped. The moved filestore files are deleted by the regular
+filestore garbage collection.
+
+To move the existing files again later, for instance after enabling a
+type or raising its limit, run from an Odoo shell:
+
+.. code:: python
+
+   from odoo.addons.edi_exchange_file_db_oca.hooks import (
+       move_exchange_file_contents_to_db,
+   )
+
+   move_exchange_file_contents_to_db(env)
+   env.cr.commit()
+
+Uninstalling the module keeps every file readable, since Odoo reads the
+content of an attachment from the database or from the filestore,
+wherever it is; new files go to the filestore again. To move the stored
+content back to the filestore as well, run
+``env["ir.attachment"].force_storage()`` with the system parameter
+``ir_attachment.location`` set to ``file``.
+
+Development
+===========
+
+Odoo decides where an attachment keeps its content in
+``ir.attachment._get_datas_related_values``, from ``_storage()``, which
+knows neither the attachment's model nor its field. The decision is
+therefore taken by the exchange record: ``edi.exchange.record`` writes
+its **Exchange file** under the ``edi_exchange_file_content_in_db``
+context key, for which ``ir.attachment._storage()`` returns ``db``.
+Records are grouped by decision and each group is written with the key
+set to ``True`` or ``False``, so a value inherited from the caller never
+applies.
+
+The key is scoped to the writing of the file only:
+
+-  on creation, ``_create`` is overridden rather than ``create``, so the
+   key covers the insertion of the rows and of their attachments, but
+   not the quick execution of the exchange that ``edi_oca`` runs right
+   after;
+-  on update, ``write`` writes the other values first, then the file
+   alone.
+
+``edi.exchange.type._should_store_file_content_in_db(size)`` is the
+single decision point: it combines the type's toggle with its size limit
+(the type's own one, otherwise the global system parameter).
+
+Content written to the attachment without going through the field, such
+as ``ir.attachment.force_storage()``, follows the configured storage
+instead.
+
+``hooks.move_exchange_file_contents_to_db`` moves the existing content.
+It selects the candidates in SQL, by batches, and asks their exchange
+type. It reads each file, checks it against the stored checksum, and
+writes the content with a batched SQL update: the checksum, size,
+mimetype and index of the attachment stay valid since the content does
+not change, and going through the ORM would recompute them for every
+attachment.
+
+The **File Content** tab decides between text and binary before
+decoding: a file is binary when ``guess_mimetype`` recognises a binary
+signature (PDF, images, Office documents, archives), or when its first
+KB holds a NUL byte (except for UTF-16 and UTF-32 types). Decoding
+success alone is not a reliable signal, since a type in ``latin-1``, or
+with the ``replace`` error handler, decodes any byte sequence. The
+display limit is ``_get_exchange_file_display_max_size()``.
+
+Known issues / Roadmap
+======================
+
+-  A default on the backend, to configure every type of a backend at
+   once.
+-  From Odoo 19.0, writing an attachment with the database storage no
+   longer marks its previous filestore file for garbage collection: the
+   forward port must mark it itself.
+-  A preview of PDF and image exchange files in the exchange record
+   form.
+
+Bug Tracker
+===========
+
+Bugs are tracked on `GitHub Issues <https://github.com/OCA/edi-framework/issues>`_.
+In case of trouble, please check there if your issue has already been reported.
+If you spotted it first, help us to smash it by providing a detailed and welcomed
+`feedback <https://github.com/OCA/edi-framework/issues/new?body=module:%20edi_exchange_file_db_oca%0Aversion:%2017.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
+
+Do not contact contributors directly about support or help with technical issues.
+
+Credits
+=======
+
+Authors
+-------
+
+* ForgeFlow
+
+Contributors
+------------
+
+-  `ForgeFlow <https://www.forgeflow.com>`__:
+
+   -  Guillem Casassas <guillem.casassas@forgeflow.com>
+
+Maintainers
+-----------
+
+This module is maintained by the OCA.
+
+.. image:: https://odoo-community.org/logo.png
+   :alt: Odoo Community Association
+   :target: https://odoo-community.org
+
+OCA, or the Odoo Community Association, is a nonprofit organization whose
+mission is to support the collaborative development of Odoo features and
+promote its widespread use.
+
+.. |maintainer-GuillemCForgeFlow| image:: https://github.com/GuillemCForgeFlow.png?size=40px
+    :target: https://github.com/GuillemCForgeFlow
+    :alt: GuillemCForgeFlow
+
+Current `maintainer <https://odoo-community.org/page/maintainer-role>`__:
+
+|maintainer-GuillemCForgeFlow| 
+
+This module is part of the `OCA/edi-framework <https://github.com/OCA/edi-framework/tree/17.0/edi_exchange_file_db_oca>`_ project on GitHub.
+
+You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.

edi_exchange_file_db_oca/__init__.py

@@ -0,0 +1,2 @@
+from . import models
+from .hooks import post_init_hook

edi_exchange_file_db_oca/__manifest__.py

@@ -0,0 +1,21 @@
+# Copyright 2026 ForgeFlow S.L.
+# License LGPL-3.0 or later (http://www.gnu.org/licenses/lgpl).
+{
+    "name": "EDI Exchange File in Database",
+    "summary": """
+    Store the files of EDI exchange records in the database instead of the
+    filestore.
+    """,
+    "version": "17.0.1.0.0",
+    "license": "LGPL-3",
+    "website": "https://github.com/OCA/edi-framework",
+    "author": "ForgeFlow, Odoo Community Association (OCA)",
+    "maintainers": ["GuillemCForgeFlow"],
+    "depends": ["edi_oca"],
+    "data": [
+        "data/ir_config_parameter.xml",
+        "views/edi_exchange_record_views.xml",
+        "views/edi_exchange_type_views.xml",
+    ],
+    "post_init_hook": "post_init_hook",
+}

edi_exchange_file_db_oca/data/ir_config_parameter.xml

@@ -0,0 +1,7 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<odoo noupdate="1">
+    <record id="max_size_kb_parameter" model="ir.config_parameter">
+        <field name="key">edi_exchange_file_db_oca.max_size_kb</field>
+        <field name="value">0</field>
+    </record>
+</odoo>

edi_exchange_file_db_oca/hooks.py

@@ -0,0 +1,118 @@
+# Copyright 2026 ForgeFlow S.L.
+# License LGPL-3.0 or later (http://www.gnu.org/licenses/lgpl).
+
+import logging
+import os
+
+import psycopg2
+
+from odoo.tools import SQL, human_size
+
+_logger = logging.getLogger(__name__)
+
+
+def _get_exchange_files_to_move(env, last_id, limit):
+    record_model = env["edi.exchange.record"]
+    env.cr.execute(
+        SQL(
+            """
+            SELECT a.id, r.type_id, a.file_size, a.store_fname, a.checksum
+            FROM ir_attachment a
+            JOIN %(record_table)s r ON r.id = a.res_id
+            WHERE a.res_model = %(res_model)s
+                AND a.res_field = %(res_field)s
+                AND a.type = 'binary'
+                AND a.store_fname IS NOT NULL
+                AND a.id > %(last_id)s
+            ORDER BY a.id
+            LIMIT %(limit)s
+            """,
+            record_table=SQL.identifier(record_model._table),
+            res_model=record_model._name,
+            res_field="exchange_file",
+            last_id=last_id,
+            limit=limit,
+        )
+    )
+    return env.cr.fetchall()
+
+
+def _read_exchange_file(env, fname, checksum):
+    attachment_model = env["ir.attachment"]
+    full_path = attachment_model._full_path(fname)
+    if not os.path.isfile(full_path):
+        return None, "missing"
+    with open(full_path, "rb") as file:
+        data = file.read()
+    if attachment_model._compute_checksum(data) != checksum:
+        return None, "mismatched"
+    return data, "moved"
+
+
+def move_exchange_file_contents_to_db(env, batch_size=1000):
+    attachment_model = env["ir.attachment"]
+    attachment_model.flush_model(
+        [
+            "res_model",
+            "res_field",
+            "res_id",
+            "type",
+            "file_size",
+            "store_fname",
+            "checksum",
+        ]
+    )
+    env["edi.exchange.record"].flush_model(["type_id"])
+    counts = {"moved": 0, "excluded": 0, "missing": 0, "mismatched": 0}
+    moved_size = 0
+    last_id = 0
+    while rows := _get_exchange_files_to_move(env, last_id, batch_size):
+        last_id = rows[-1][0]
+        exchange_types = {
+            exchange_type.id: exchange_type
+            for exchange_type in env["edi.exchange.type"].browse(
+                list({row[1] for row in rows})
+            )
+        }
+        values, fnames = [], []
+        for attachment_id, type_id, file_size, fname, checksum in rows:
+            if not exchange_types[type_id]._should_store_file_content_in_db(file_size):
+                counts["excluded"] += 1
+                continue
+            data, status = _read_exchange_file(env, fname, checksum)
+            counts[status] += 1
+            if data is None:
+                continue
+            values.append((attachment_id, psycopg2.Binary(data)))
+            fnames.append(fname)
+            moved_size += len(data)
+        if not values:
+            continue
+        env.cr.execute_values(
+            """
+            UPDATE ir_attachment a
+            SET db_datas = v.data, store_fname = NULL
+            FROM (VALUES %s) AS v(id, data)
+            WHERE a.id = v.id
+            """,
+            values,
+            page_size=batch_size,
+        )
+        for fname in fnames:
+            attachment_model._file_delete(fname)
+    attachment_model.invalidate_model(["db_datas", "store_fname", "raw", "datas"])
+    _logger.info(
+        "EDI exchange file contents moved to the database: %s (%s). Skipped: %s "
+        "excluded by their exchange type, %s missing from the filestore, %s not "
+        "matching their checksum.",
+        counts["moved"],
+        human_size(moved_size) or "0 bytes",
+        counts["excluded"],
+        counts["missing"],
+        counts["mismatched"],
+    )
+    return counts
+
+
+def post_init_hook(env):
+    move_exchange_file_contents_to_db(env)

edi_exchange_file_db_oca/models/__init__.py

@@ -0,0 +1,3 @@
+from . import edi_exchange_record
+from . import edi_exchange_type
+from . import ir_attachment