ETF Improvement Proposal (EIP)
Background and Motivation:
As part of the creation of a cloud deployment of the ETF using AWS, we put in place a load balancer to automatically create new instances of the application ir order to manage heavy workloads. To avoid consistency problems, and for every user to have the same information, we established a centralized data storage, to mount it on every ETF container.
However, the BaseX process locks down the data storage folder, and an exception is thrown if any other process tries to access it. This prevents the addition of new instance from the load balancer.
Proposed change
As proposed on the BaseX documentation, we propose to add an optional configuration for ETF deployment, from an embedded library to a client-server architecture. This way the database can be used concurrently by multiple ETF instances. The server can be hosted on a separate container, and the ETF instances can be configured to interact with this database through the configuration files.
This alternate interface should be developed on the data storage module of the ETF, to direct all the CRUD operations through the REST API of BaseX.
Alternatives
An alternative could be to deploy separated folders for each ETF instance, and establish a way to synchronize all the folder to avoid consistency issue across all the instances. This could be complex, and would mean to have a synchronization job scheduled with a high frequency, consuming a non trivial amount of resources.
Funding
N/A
Additional information
ETF Improvement Proposal (EIP)
Background and Motivation:
As part of the creation of a cloud deployment of the ETF using AWS, we put in place a load balancer to automatically create new instances of the application ir order to manage heavy workloads. To avoid consistency problems, and for every user to have the same information, we established a centralized data storage, to mount it on every ETF container.
However, the BaseX process locks down the data storage folder, and an exception is thrown if any other process tries to access it. This prevents the addition of new instance from the load balancer.
Proposed change
As proposed on the BaseX documentation, we propose to add an optional configuration for ETF deployment, from an embedded library to a client-server architecture. This way the database can be used concurrently by multiple ETF instances. The server can be hosted on a separate container, and the ETF instances can be configured to interact with this database through the configuration files.
This alternate interface should be developed on the data storage module of the ETF, to direct all the CRUD operations through the REST API of BaseX.
Alternatives
An alternative could be to deploy separated folders for each ETF instance, and establish a way to synchronize all the folder to avoid consistency issue across all the instances. This could be complex, and would mean to have a synchronization job scheduled with a high frequency, consuming a non trivial amount of resources.
Funding
N/A
Additional information