ECKIT-680: Add target library, tags and lazy loading of plugins

.gitignore

@@ -13,3 +13,4 @@ build/
 tests/geo/eckit_geo_cache
 _build
 *.ccls-cache
+Testing/

docs/content/plugins.rst

@@ -0,0 +1,184 @@
+Plugins
+=======
+
+eckit supports loading additional functionality at runtime via plugins:
+shared libraries that derive from :cpp:class:`eckit::system::Plugin` and
+self-register with :cpp:class:`eckit::system::LibraryManager` as their
+static instance is constructed.
+
+Plugins are described by YAML manifest files. The loader discovers
+manifests, validates them against the running process and ``dlopen``\ s
+the corresponding shared object on demand.
+
+Once loaded, the `init()` method of the plugin is called. Prior to unloading
+the plugin, during rollback of a failed load, or during clean shutdown of
+the process, the plugin's `finalise()` method is called.
+
+Many plugins create global resources, which self-register with appropriate
+global factories (in eckit or elsewhere). The self-registering resources
+should be instantiated in the plugin's `init()` method, and cleaned up in
+`finalise()`, rather than relying on static (global) initialisation and
+destruction, if this is at all possible.
+
+Manifest discovery
+------------------
+
+When ``LibraryManager`` scans for manifests it visits the following
+directories, in this order, stopping at the first match per fully
+qualified plugin name:
+
+#. Each directory listed in ``$PLUGINS_MANIFEST_PATH``
+   (or the ``pluginManifestPath`` resource), ``:``-separated.
+#. ``<path>/share/plugins`` for every directory registered with
+   :cpp:func:`eckit::system::LibraryManager::addPluginSearchPath`.
+#. The directories returned by
+   :cpp:func:`eckit::system::Library::pluginManifestPaths` for each
+   currently registered library (default: ``~<libname>/share/plugins``).
+#. ``~eckit/share/plugins``
+#. ``~/share/plugins``
+
+Symlinks resolve to canonical paths and the same manifest directory is
+never scanned twice. The first manifest seen for a fully qualified name
+``namespace.name`` wins; subsequent duplicates are logged and ignored.
+
+Manifest schema
+---------------
+
+A manifest is a YAML file with a single top-level ``plugin:`` mapping.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 15 65
+
+   * - Key
+     - Required
+     - Meaning
+   * - ``name``
+     - yes
+     - Plugin name (matches the argument passed to the
+       :cpp:class:`Plugin <eckit::system::Plugin>` constructor in C++).
+   * - ``namespace``
+     - yes
+     - Namespace used to build the fully qualified name (example int.ecmwf)
+       ``namespace.name``.
+   * - ``library``
+     - yes
+     - Shared library file stem, without the ``lib`` prefix or
+       platform-specific extension.
+   * - ``for-library``
+     - no
+     - Owning library name. When set, the plugin is *scoped* and is
+       only loaded by an explicit request from that library or by
+       fully-qualified name (see below). When absent the plugin is
+       *global*.
+   * - ``min-version``
+     - no
+     - Minimum version of ``for-library`` required for the plugin to
+       load. Compared as semver-ish numeric ``major.minor[.patch]``;
+       if the registered library's version is lower throw ``BadValue``.
+   * - ``version``
+     - no
+     - Pinned exact value of the plugin's self-reported
+       :cpp:func:`Plugin::version`. Compared verbatim (string equality,
+       not semver) against the version reported by the Plugin object
+       at runtime. On mismatch the loader unloads the offending shared
+       object again and throws ``BadValue`` without calling the Plugin's
+       ``init()`` function.
+   * - ``tags``
+     - no
+     - Free-form list of labels used by ``loadPluginsFor`` for filtering.
+
+Example:
+
+.. code-block:: yaml
+
+   plugin:
+     name: my-plugin
+     namespace: int.ecmwf
+     library: my-plugin
+     for-library: eckit
+     min-version: 1.20.0
+     version: 0.4.2
+     tags: [grids, io]
+
+Loading model
+-------------
+
+There are three orthogonal ways a plugin can be loaded.
+
+**Global auto-load (Main startup)**
+   Plugins without ``for-library`` are loaded by :cpp:class:`eckit::Main`
+   on construction when both:
+
+   * the ``Main`` constructor's ``autoLoadPlugins`` parameter is
+     ``true`` (the default), and
+   * the resource ``$AUTO_LOAD_PLUGINS`` (default ``true``) is true.
+
+   Scoped plugins are *not* loaded by this path.
+
+   ***Note: This mechanism is deprecated and will be removed in the future.***
+   Plugins should be explicitly loaded by their owning library via the scoped
+   mechanism described below.
+
+**Scoped load (library-driven)**
+   A library opts in to loading its scoped plugins by calling
+   :cpp:func:`eckit::system::LibraryManager::loadPluginsFor` from its
+   initialisation routine, or lazily during execution when required, e.g.::
+
+       LibraryManager::loadPluginsFor("my-lib", {"grids"});
+
+   Only manifests whose ``for-library`` matches and whose ``tags``
+   include *all* of the requested tags are considered. An empty tag
+   list matches every scoped manifest for the library. ``min-version``
+   and ``version`` checks apply.
+
+**Explicit list**
+   Setting ``$LOAD_PLUGINS`` (or the ``loadPlugins`` resource) to a
+   comma-separated list of fully qualified names loads exactly those
+   plugins, scoped or not, at ``Main`` startup. A name with no matching
+   manifest, or a scoped entry whose owning library is not registered,
+   throws ``BadValue``.
+
+Disabling plugins
+-----------------
+
+Setting ``$DISABLE_PLUGINS`` (or the ``disablePlugins`` resource) to a
+comma- or colon-separated list of fully qualified plugin names
+suppresses both auto-load and scoped load for those plugins.
+
+Subclassing ``Main``
+--------------------
+
+A subclass that should not perform global plugin auto-load passes
+``false`` for the ``autoLoadPlugins`` constructor argument::
+
+   class MyMain : public eckit::Main {
+   public:
+       MyMain(int argc, char** argv) :
+           Main(argc, argv, /*homeenv=*/nullptr, /*autoLoadPlugins=*/false) {}
+   };
+
+The default value of ``true`` is expected to flip to ``false`` in a
+future release once existing plugins have migrated to scoped
+(``for-library``) manifests.
+
+Migration notes
+---------------
+
+Behavioural changes introduced with the scoped-plugin support:
+
+* ``LibraryManager::autoLoadPlugins({})`` no longer loads scoped
+  plugins. Previously every discovered manifest was loaded
+  indiscriminately. Library authors must now load their scoped plugins
+  via ``loadPluginsFor`` from their library's initialisation path.
+* A name listed in ``$LOAD_PLUGINS`` that has no matching manifest now
+  throws ``BadValue``. Previously the loader emitted a warning and
+  continued.
+* A manifest declaring ``version:`` whose value differs from the
+  installed shared object's :cpp:func:`Plugin::version` now causes the
+  shared object to be unloaded again and ``BadValue`` is thrown. This
+  helps detect cases where a manifest on disk no longer matches the
+  binary that satisfies its ``library:`` entry.
+* :cpp:func:`Plugin::finalise` may be invoked even when
+  :cpp:func:`Plugin::init` was never called (the rollback path of a
+  failed ``version:`` pin check). Implementations must tolerate this.

docs/content/string_tools.rst

@@ -0,0 +1,201 @@
+String tools
+============
+
+:cpp:class:`eckit::StringTools` is a small static utility class that
+collects the ``std::string`` manipulation primitives the rest of eckit
+relies on. It lives in ``eckit/utils/StringTools.h`` and is part of the
+core ``eckit`` library — no extra dependency is needed to use it.
+
+The class has a deleted default constructor: every entry point is a
+``static`` member. Most operations return a new ``std::string`` and leave
+their inputs untouched; the lazy streaming helper :ref:`stringtools-joinostream`
+is the one exception and returns a non-owning proxy instead.
+
+Quick reference
+---------------
+
+.. list-table::
+   :widths: 30 70
+   :header-rows: 1
+
+   * - Function
+     - Purpose
+   * - :cpp:func:`eckit::StringTools::upper`,
+       :cpp:func:`eckit::StringTools::lower`
+     - ASCII case conversion (returns a new string).
+   * - :cpp:func:`eckit::StringTools::trim`,
+       :cpp:func:`eckit::StringTools::front_trim`,
+       :cpp:func:`eckit::StringTools::back_trim`
+     - Strip whitespace (or a custom set of characters) from both ends,
+       the front, or the back of a string.
+   * - :cpp:func:`eckit::StringTools::startsWith`,
+       :cpp:func:`eckit::StringTools::beginsWith`,
+       :cpp:func:`eckit::StringTools::endsWith`
+     - Prefix / suffix tests; ``startsWith`` and ``beginsWith`` are
+       synonyms.
+   * - :cpp:func:`eckit::StringTools::split`
+     - Split a string on every occurrence of a delimiter character set,
+       returning a ``std::vector<std::string>``.
+   * - :cpp:func:`eckit::StringTools::join`
+     - Concatenate a sequence of strings into a single ``std::string``,
+       inserting a delimiter between consecutive elements.
+   * - :cpp:func:`eckit::StringTools::joinOstream`
+     - As ``join``, but lazily streamed to an existing ``std::ostream``
+       without ever materialising a temporary ``std::string``.
+   * - :cpp:func:`eckit::StringTools::substitute`,
+       :cpp:func:`eckit::StringTools::listVariables`
+     - Substitute ``{name}`` placeholders from a dictionary; or list the
+       placeholders found in a template.
+   * - :cpp:func:`eckit::StringTools::isQuoted`,
+       :cpp:func:`eckit::StringTools::unQuote`
+     - Detect and strip matching surrounding quote characters.
+
+Case conversion and trimming
+----------------------------
+
+.. code-block:: cpp
+
+   #include "eckit/utils/StringTools.h"
+
+   const std::string s = "  Hello, World!  ";
+
+   eckit::StringTools::trim(s);                  // "Hello, World!"
+   eckit::StringTools::front_trim(s);            // "Hello, World!  "
+   eckit::StringTools::back_trim(s);             // "  Hello, World!"
+
+   eckit::StringTools::trim("xxxhelloxxx", "x"); // "hello"
+   eckit::StringTools::lower("Hello");           // "hello"
+   eckit::StringTools::upper("Hello");           // "HELLO"
+
+The two-argument trim variants take a string of *characters to strip*,
+not a substring; ``trim("0001000", "0") == "1"``.
+
+Searching and splitting
+-----------------------
+
+.. code-block:: cpp
+
+   eckit::StringTools::startsWith("/etc/passwd", "/etc/");  // true
+   eckit::StringTools::endsWith("file.tar.gz",   ".gz");    // true
+
+   auto parts = eckit::StringTools::split(":/", "a:b/c:d");
+   // parts == { "a", "b", "c", "d" }
+
+The delimiter passed to ``split`` is treated as a *set of characters*:
+any character in the delimiter string ends the current token. Empty
+tokens between consecutive delimiters are collapsed.
+
+Joining
+-------
+
+``StringTools::join`` builds a single ``std::string`` from any iterable
+container of strings:
+
+.. code-block:: cpp
+
+   std::vector<std::string> v{"alpha", "beta", "gamma"};
+
+   eckit::StringTools::join(", ", v);
+   // "alpha, beta, gamma"
+
+   eckit::StringTools::join("/", v.begin() + 1, v.end());
+   // "beta/gamma"
+
+Both the iterator-pair and the container overloads are templates, so a
+``std::list``, ``std::set``, or any other range-iterable container of
+strings works without conversion.
+
+.. _stringtools-joinostream:
+
+Lazy joining for streams
+------------------------
+
+When the result of a join is going to be written to an ``std::ostream``
+(such as a log channel or a ``std::ostringstream``), allocating a
+temporary ``std::string`` first is wasteful.
+:cpp:func:`eckit::StringTools::joinOstream` returns a small proxy object
+that streams the elements one at a time, separated by the delimiter, the
+moment it is itself streamed:
+
+.. code-block:: cpp
+
+   #include "eckit/log/Log.h"
+   #include "eckit/utils/StringTools.h"
+
+   std::vector<std::string> values{"alpha", "beta", "gamma"};
+
+   eckit::Log::error() << "values: ["
+                       << eckit::StringTools::joinOstream(values, ", ")
+                       << "]" << std::endl;
+   // values: [alpha, beta, gamma]
+
+The proxy is generic over both the container type and the element type;
+anything streamable with ``operator<<`` works, so numeric containers and
+custom types are equally fine:
+
+.. code-block:: cpp
+
+   std::set<std::string> s{"gamma", "alpha", "beta"};
+   std::vector<int> nums{1, 2, 3, 4};
+
+   std::ostringstream oss;
+   oss << eckit::StringTools::joinOstream(s,    "-")  << '\n'
+       << eckit::StringTools::joinOstream(nums, ", ");
+   // alpha-beta-gamma
+   // 1, 2, 3, 4
+
+The delimiter is a ``std::string_view``, so ``const char *``,
+``std::string``, and any ``string_view``-convertible type are all
+accepted with no copy.
+
+The proxy holds *references* to the container and the delimiter, so it
+must be consumed in the same full-expression in which it is created.
+This is the natural ``os << joinOstream(...)`` idiom; the type is
+intentionally non-copyable to make accidental misuse harder. **Do not**
+store the proxy in a variable that outlives the source container.
+
+Template substitution
+---------------------
+
+``StringTools::substitute`` performs ``{name}``-style placeholder
+expansion against a ``std::map``. ``listVariables`` returns the names
+that appear in such a template, in left-to-right order with duplicates
+preserved:
+
+.. code-block:: cpp
+
+   std::map<std::string, std::string> m{
+       {"user", "alice"},
+       {"host", "example.org"},
+   };
+
+   eckit::StringTools::substitute("{user}@{host}", m);
+   // "alice@example.org"
+
+   eckit::StringTools::listVariables("{user}@{host}/{user}");
+   // { "user", "host", "user" }
+
+Unbalanced braces, nested braces, and references to keys missing from
+the dictionary all raise an exception derived from
+:cpp:class:`eckit::Exception`.
+
+Quoting
+-------
+
+``isQuoted`` returns true when a string starts and ends with a matching
+``'`` or ``"`` character; ``unQuote`` strips one such pair off:
+
+.. code-block:: cpp
+
+   eckit::StringTools::isQuoted("\"hello\"");  // true
+   eckit::StringTools::isQuoted("'hello'");    // true
+   eckit::StringTools::isQuoted("'hello\"");   // false (mismatched)
+   eckit::StringTools::unQuote("\"hello\"");   // "hello"
+
+Mismatched or absent quotes leave the input unchanged.
+
+API reference
+-------------
+
+.. doxygenclass:: eckit::StringTools
+   :members:

docs/index.rst

@@ -9,4 +9,6 @@ Welcome to EcKit’s Documentation!
    :caption: Contents
 
    content/concepts
+   content/plugins
+   content/string_tools
    content/reference