feat(clients): add transaction support #134
Conversation
|
Hey @ibilux , you're a machine! I would love to get the transactions feature into the clients. Thanks for blazing the trail and keep pushing. I don't currently have my computer, but I gave it a quick skim. While currently grinding in another area, I've been thinking about this a bit in the back of my head. We've also touched a little bit on it already in terms of type-safety and such. I think you're proposed approach: var batch = client.Transaction();
batch.Api("name").Create(json);
await batch.send();is very valid. The other approach I was contemplating was to reuse the RecordApi "builders" var api = client.Records("Name");
var op = api.CreateOp(json);
client.commit([op]);(Phone coding is the best. Btw I only picked c# because of lexicographical sorting by suffix, please use whatever you like best. Maybe not python 😅). Not sure if it makes much of a difference. Maybe if folks keep long-lived builders around 🤷♀️. Maybe we wouldn't have to wonder about lifecycle of the transaction object for failed transactions, repeated sends or post send appenditures 🤷♀️ I was also wondering if there's any value in making operations generics (probably not) or type-erase them right away as you suggest (probably makes sense). I could only see this make sense in the context of function composition, e.g. void BuildCommonTxForMyBusiness(Client client, CreateOp<Type> mainOp) {
client.commit([
// ... aux changes
mainOp,
])
}In which case it would probably make sense to (also) have the API id in the op signature more so than the data type 🤷♀️ Again, thanks so much for being on top of this 🙏 |
|
Hello @ignatz,
I started with TypeScript, and it felt perfect at first. But once I tried to mirror it in other clients, I had this moment of: “oooh man, did I just over-engineer this?!”
Definitely not python or swift 😅.
While writing the code I was thinking about the lifecycle concerns too (failed transactions, reuse, appending after send, and especially repeated sends). I even wondered whether we should clear the operations array after a successful commit ?
I also leaned toward type-erased operations for my suggested approach (generics felt like overkill). However, your suggestion of reusing the let api = client.Records("Name");
let op1 = api.CreateOp(json); // ✅ belongs in transactions
let op2 = api.Create(json); // ❌ not for transactions
client.commit([op1]); // ✅ works
client.commit([op1, op2]); // ❌ invalid, but looks plausibleThere is possible solutions to make the intent clearer:
// direct API calls
client.records("users").create({ name: "Alice" });
// operation builders
client.ops("users").create({ name: "Alice" });
api.create({ ... }) // executes immediately
api.buildCreate({ ... }) // builds an Operation
type Operation = { ... };
client.commit([Operation, ...]); // only accepts Operations
let op1 = client.records("users").operation.create({ name: "Alice" });
client.commit([op1]); |
👍
This should be the case independent of where we put the API (below). "List of operations" will be supported by any typed language (even python's annotations :hide:) To sum up your proposals:
I think they're all workable. Just thinking out loud, I could see a slight advantage for (1) and (3) in the context of mocking/testability. I could see a slight advantage for (2) and (3) if you want to pass There's a 4th backwards-incompatible option (which is fine if we thought this is the way to go), where we'd make operations executable, i.e. const api = client.records("myapi");
const op = api.create(/*...*/);
if (useTx) {
await client.commit([op]);
} else {
await op.execute();
}At the moment I can't see a clear winner. With a good argument I could be swayed towards any of the options. Do you have a preference? Thanks 🙏 |
|
Having simmered over this a little more, I sort of like the 4th option because it completely eliminates duplication. However, it introduces new issues/asymmetries, i.e. there are no read/list operations at least for transactions. We could have client-side read/list operation representations, however they shouldn't be accepted by |
I was hoping you might provide some compelling arguments as you usually do 😅 — you always think ahead of all possible outcomes. I think we shouldn't try to squeeze both records and transactions into the same surface. It feels cleaner to separate them into their own dedicated namespaces. const tx = client.transaction();
tx.records("users").create(record);
tx.records("posts").update(id, data);
await tx.commit(); // or tx.execute()Alternatively, instead of letting the transaction builder manage the lifecycle directly, we could separate the builder from the commitment, like: const tx = client.transaction();
const op1 = tx.records("users").create(record);
const op2 = tx.records("posts").update(id, data);
await client.commit([op1, op2]); |
Appreciated
I wish 😅
I think that's a good goal. To tally up all discussed options that meet this goal (I guess the previous (2) and (3) wouldn't fit) and avoid Transaction-object lifecycle concerns:
I'm going to say that (1) and (2) are virtually the same if you fold I've been careful not to push too hard for (3) because it would be the biggest change (not even that much more work but also for users). I do think, it's probably the most intuitive, i.e. you build and operation and then you either execute it right away or you add it to a transaction. Rather than using two different APIs. Similarly, there's currently a bulk creation, which isn't a transaction. We could do away with it entirely and have an (EDIT: looking at prior art many SQL client implementations do have transaction objects. However, they're needed since a transaction isn't bulk, e.g. you'd have a |
Yes, in this case
The only downside here is the major breaking change and the asymmetry around If we go with deferred operations, each operation could be executed immediately via export class DeferredOperation {
constructor(
private client: Client,
private apiName: string,
// private recordId: string
// private record: Record<string, unknown>
) {}
// Immediate execution
async execute(): Promise<string> {
const response = await this.client.fetch(...);
return (await response.json()).ids[0]; // Or should we return an array?
}
// For transaction/bulk execution
toOperation(): Operation { // or toJSON()
return {
// ...
};
}
}const api = client.records("users");
const op = api.create(record); // Always deferred
// Either execute immediately
await op.execute();
// OR add to transaction
await client.execute([op], { transaction: false });If we adapt this model, export interface RecordApi<T = Record<string, unknown>> {
create(record: T): DeferredCreateOperation;
update(id: string | number, record: Partial<T>): DeferredUpdateOperation;
delete(id: string | number): DeferredDeleteOperation;
}The open question is how to handle
export interface RecordApi<T = Record<string, unknown>> {
// ... Deferred Create/Update/Delete Operations
// Read operations remain immediate since they don't participate in transactions
read<T = Record<string, unknown>>(id: string | number): Promise<T>;
list<T = Record<string, unknown>>(opts?: ListOptions): Promise<ListResponse<T>>;
}
Another open issue: bulk mode semantics.
✓ My personal take on this is:
|
|
Thanks for the write-up, its' very much appreciated and super useful 🙏
I can very much see your point: if you can't use the deferred operations for anything why have them? That said, I'm note sure if the asymmetry is actually helping semantics, UX or simplicity. Simplicity on our end for sure but for users? I worry that for users it will mostly stick out as an inconsistency - user-case or not. (Independently, one could try to fabricate uses, where users build higher level abstractions or persist read operations to then periodically carry them out,... arguably I'm scraping the bottom of the barrel here - I'm mostly wondering about how intuitive and pleasant the APIs would be. Anything that's not intuitive requires extra documentation).
Agreed. I would suggest to keep this in mind when designing the API but postponing the implementation. I.e only support create for now and then support the rest in the future. Just to keep the scope on transactions for now.
I love this. This will set us op to get rid of To extend your code above, I could imagine something like: class DeferredOperation<ResponseType> {
constructor(private api: RecordApi<ResponseType>) {}
// Immediate execution
async query(): Promise<ResponseType>;
}
class DeferredMutation<ResponseType> extends DeferredOperation<ResponseType> {
// For transaction/bulk execution
toJson(): Operation;
}
export class CreateOperation<T> extends DeferredMutation<RecordId> {
// ...
}
export class ReadOperation<T> extends DeferredOperation<T> {
// ...
}
class Client {
// ...
// Only supports create for now. Happy to name this bulk or differently
async create(ops: CreateOperation<unknown>[]);
// Happy to rename this
async execute(ops: DeferredMutation<unknown>[]);
}
// ....
const simpleApi : RecordApi<Simple> = client.records("simple");
const otherApi : RecordApi<Other> = client.records("other");
const results = await client.transaction([
simpleApi.create({ simple: 5, /* ... */ }),
otherApi.update(id, { other: "hi", /* ...*/ }),
]);
const simple : Simple = await simpleApi.read(results[0].id).query();I don't hate it 😅 . One thing that jumps out after jamming, is that we need two different methods What do you think? (NOTE: another symmetry that follows from treating read operations and mutations symmetrically would be that |
|
I agree that UX should be the top priority. Personally, I’m okay with all the solutions and suggestions we discussed, but the main goal here is to deliver the best experience for users (I’m not considering myself a user for now to avoid bias 😅). I think immediate execution should be named And Since we’re already going through major changes, introducing a Ultimately, I think we need users opinion and feedback on this before finalizing the approach ? |
That would be ideal but is sadly hard to come by. I'm very open to pull folks in, any suggestions?
I think you should. You're taking all this time out of your day to look at this problem and combined with your experience, your opinion matters (a lot) and will always be more informed than some's responding to a survey on the toilet :) (Just as a reminder, we aren't blocked. We can decide and run as long as it makes sense to us. That said, more eyeball more better when available)
I see where you're coming from and am happy either way. (In the context of databases,
Feedback taken. Then let's update the transaction handler to support both execution modes. (Happy to give it a go, just let me know) |
|
So the API would look something like this: class DeferredOperation<ResponseType> {
async query(): Promise<ResponseType>;
}
class DeferredMutation<ResponseType> extends DeferredOperation<ResponseType> {
toJSON(): Operation;
}
export class CreateOperation<T> extends DeferredMutation<RecordId> {}
export class UpdateOperation<T> extends DeferredMutation<void> {}
export class DeleteOperation extends DeferredMutation<void> {}
export class ReadOperation<T> extends DeferredOperation<T> {}
export class ListOperation<T> extends DeferredOperation<ListResponse<T>> {}
export interface RecordApi<T = Record<string, unknown>> {
// Write operations
create(record: T): CreateOperation<T>;
update(id: RecordId, record: Partial<T>): UpdateOperation<T>;
delete(id: RecordId): DeleteOperation;
// Read operations
read(id: RecordId, opt?: { expand?: string[] }): ReadOperation<T>;
list(opts?: ListOptions): ListOperation<T>;
// Subscribe
subscribe(id: RecordId): Promise<ReadableStream<Event>>;
}
class Client {
async execute(
ops: DeferredMutation[],
options?: { transaction?: boolean },
): Promise<RecordId[]>;
}The only slightly awkward bit is this pattern: const simple : Simple = await api.read(id).query();The asymmetry is unavoidable because we can’t defer a subscription anyway ( I can see a benefit to making const simple : Simple = await api.read(id).query({ expand: ["author"] }); // Options passed to query()Although this may open the door for another asymmetry. For the bulk operation on the server side, is the idea simply to iterate through the operations like this? #[derive(Clone,Debug,Deserialize,Serialize,ToSchema)]
pub struct TransactionRequest{
operations : Vec < Operation >,
transaction : Option < bool >, // Default to true or false?
}
let ids = if request.transaction.unwrap_or(true){
state.conn().call(move | conn : & mut rusqlite : : Connection |{
let tx = conn.transaction()?;
let mut ids : Vec < String > = vec ! [];
for op in operations{
if let Some(id)= op(& tx)?{
ids.push(id);
}
}
tx.commit()?;
Ok(ids)
}).await ?
}else{
// New non-transactional bulk mode
let mut ids : Vec < String > = vec ! [];
for op in operations{
let result = state.conn().call(move | conn : & mut rusqlite : : Connection |{
op(conn)
}).await ?;
if let Some(id)= result{
ids.push(id);
}
}
ids
}; |
|
Hey @ibilux, sorry for the slow reply. Wasn't feeling so hot but now back on my feet. ClientI love your proposed client-side API - very clean ❤️ . As for your points on asymmetry, I could again be swayed either way. Personally, I don't mind that read/list are deferred too much. To riff a bit more on your proposal, if you moved expand into the const read : ReadOperation<Simple> = api.read(id);
const simple = await read.query({ expand: ["author"] });
const stream = read.subscribe();On the bright side, it would remove the asymmetry but it may make subscribe a bit less discoverable. Also none of the other Spelling it out here, I feel that To sum up:
None of them strikes me as "terrible". From a mere symmetry argument, I'm actually leaning (1). Thinking further, maybe Do you feel like we're happy enough and at a point where we could jump to implementation? FWIW, I'm very exited how this is shaping up 🙏 Server
Still paging in, where do you see the main difference between your sketch and |
Signed-off-by: Bilux <i.bilux@gmail.com>
Signed-off-by: Bilux <i.bilux@gmail.com>
Signed-off-by: Bilux <i.bilux@gmail.com>
Signed-off-by: Bilux <i.bilux@gmail.com>
I have updated the code for
From may understanding to the code (correct me if I'm wrong) the Shouldn't a similar approach be implemented for transaction API ? |
You're a machine 👏
After writing it out, how do you feel about it? I would love to hear your unbiased feedback. Yet, I'll share my thoughts to minimize round-trips... I love the implementation. Everything is extremely symmetric, composable and mockable. Yet, seeing it spelled out it's clearly more cumbersome then before when you're not using transactions. I'm wondering if we could just leave the implementation as is and add short-hands, e.g.
Yikes, if that's not already the case, I would consider it a bug. You're absolutely right that also for transaction we need to cleanup overriden files or files after a transaction fails. I'll take a look. |
crates/assets/js/client/src/index.ts
Outdated
| expand?: string[]; | ||
| }, | ||
| ): Promise<T>; | ||
| export interface Operation { |
There was a problem hiding this comment.
I'm not the most proficient TS programmer, is this something better modeled as is or as a union CreateOperation<unknown>| UpdateOperation<unknown> | DeleteOperation<unknown>?
There was a problem hiding this comment.
We can use Types:
interface CreateOp {
Create: { api_name: string; value: Record<string, unknown> };
}
interface UpdateOp {
Update: { api_name: string; record_id: string | number; value: Record<string, unknown> };
}
interface DeleteOp {
Delete: { api_name: string; record_id: string | number };
}
type Operation = CreateOp | UpdateOp | DeleteOp;
export class CreateOperation<T> implements DeferredMutation<string | number>
{
toJSON(): CreateOp { ... }
}Or use generic helper:
type OperationKind = "Create" | "Update" | "Delete";
type OperationMap = {
Create: { api_name: string; value: Record<string, unknown> };
Update: { api_name: string; record_id: string | number; value: Record<string, unknown> };
Delete: { api_name: string; record_id: string | number };
};
type Operation<K extends OperationKind> = { [P in K]: OperationMap[P] };
export class CreateOperation<T> implements DeferredMutation<string | number>
{
toJSON(): Operation<"Create"> { ... }
}There was a problem hiding this comment.
Naively, I like some notion of union since it expresses AnyOf<...> rather than AnyOfOrNone<...>. I couldn't tell you if that's idiomatic or which implementation is best. I trust you.
Wouldn't my original proposal work as a minimal change with the existing abstractions?
const ops : (CreateOperation<unknown> | UpdateOperation<unknown> | DeleteOperation<unknown>)[] = [
api.createOp(...),
// ...
];
const operations: Operation[] = ops.map((o) => o.toJSON());There was a problem hiding this comment.
You’re right, the current Operation interface allows AnyOfOrNone, which means you could accidentally construct an invalid shape (none or multiple at once). Switching to a union (CreateOp | UpdateOp | DeleteOp) enforces exactly one and avoids that issue.
the
You mean use two APIs |
Agreed - like that too. I'm not suggesting to change that
With symmetry intact, from a mere convenience perspective and looking at the new test-cases I wouldn't mind them co-existing on RecordApi, i.e. const result = await api.read(...);
const deferredRead = api.readOp(...);
setInterval(() => deferredRead.query().then(update), 1000);You probably have better names, e.g. deferredRead, readQuery, ... WDYT? |
This does introduce some API surface duplication, which could cause the confusion we talked about earlier. On the other hand, it maintains backward compatibility and gives flexibility for different use cases. I think this is the best course of action for a smooth migration. We can always revisit and consolidate later once we see how developers actually use the two flavors in practice. |
|
Hello @ignatz, I have updated the PR. |
ignatz
left a comment
ignatz
left a comment
There was a problem hiding this comment.
This looks phenomenal - only minor nits (the accidental looking changes to .pre-commit-config.yaml and config.textproto are probably just stale?).
I would love to merge this. Thank you so much, both for the work and bearing with me 🙏
| record_id: string | number; | ||
| }; | ||
| } | ||
|
|
There was a problem hiding this comment.
:nit: In-line into DeferredMutation?
There was a problem hiding this comment.
Agreed, inlining makes it cleaner.
| count?: boolean; | ||
| expand?: string[]; | ||
| }): Promise<ListResponse<T>>; | ||
| export interface CreateOp { |
There was a problem hiding this comment.
(Please leave as is, in the future we could probably also just use ts_rs code-gen to generate these types).
We could probably reduce the public API surface, if we simply removed DeferredMutation (or at least made it private). Note that
async execute(
operations: (CreateOperation | UpdateOperation | DeleteOperation)[],
transaction: boolean = true,
): Promise<(string | number)[]> {doesn't need DeferredMuration and you can safely call toJSON simply due to all union members having it. (structural typing for the win, other languages won't treat us so nicely)
We could then make these internal type definitions here private as well.
There was a problem hiding this comment.
I was planning to ask you after we settled on the implementation about which classes should remain exported and which should not.
I’ve removed the export. But, I think we should keep DeferredOperation and DeferredMutation only as a reference for other clients implementation.
crates/assets/js/client/src/index.ts
Outdated
| private readonly apiName: string, | ||
| private readonly record: Partial<T>, | ||
| ) {} | ||
| async query(): Promise<string> { |
There was a problem hiding this comment.
Should this match the T in DeferredMutation<T>?
There was a problem hiding this comment.
Good catch, I meant to fix that, but, somehow got distracted. My plan was to introduce a dedicated RecordId type:
export type RecordId = string | number;| expand?: string[]; | ||
| }): ListOperation<T>; | ||
|
|
||
| readOp( |
There was a problem hiding this comment.
:super nitty: just from a discoverability point of view, i'd group and Op next to each other.
crates/assets/js/client/src/index.ts
Outdated
| private readonly client: Client, | ||
| private readonly name: string, | ||
| ) { | ||
| this._path = `${recordApiBasePath}/${this.name}`; |
There was a problem hiding this comment.
I don't think this is needed anymore 🎉
There was a problem hiding this comment.
It was used in subscribe method, but it’s better to remove and replace it.
Thanks @ignatz! I’m really glad I could help, and I’ve enjoyed our conversations along the way. Regarding the next steps, would it make sense to close this PR and open separate ones to keep the git history cleaner? I’m thinking:
|
Much much much appreciated 🙏
pfew 😅
I leave this 100% up to you. Ultimately it's just cosmetics and doesn't take away from the quality of your work. Just let me know. (I won't be available this weekend, only bringing this up in case you're keen to get this in before. Want to be mindful of your time) |
Got it, thanks @ignatz 🙏. My vacation just ended and I’m back to work, so I’ll follow up with separate PRs for the other clients when I can. |
👍
🙏 thanks for committing so much effort... really really appreciated I did a few minor tweaks, mostly just adding I've squashed and merged your change into upstream dev I really love how it turned out 🙏 |
|
Quick update: I published a new npm package for the client and cutting a new server release 🙏 |
|
In my humble and belated opinion as a TrailBase novice... const tx = client.transaction();
tx.records("users").create(record);
tx.records("posts").update(id, data);
await tx.commit(); // or tx.rollback()But you are experts at this, and you have my admiration and gratitude for the excellent work you offer us for free. |
@aMarCruz Welcome! - and thanks for the input. I agree that it may feel very natural. At the same time, what we ultimately settled on is a bit closer to what actually happens under the hood allowing for a layered approach. Specifically, it would be easy today for a first or third-party to implement an abstraction like: const tx = transaction(client);
tx.records("users").create(record);
tx.records("posts").update(id, data);
await tx.commit();Maybe fun fact: this is very similar to the "raw" SQLite API you get when you run JS/TS in a WASM guest: https://github.com/trailbaseio/trailbase/blob/main/client/testfixture/guests/typescript/src/index.ts#L59-L72. I'm not sure how misleading such an API would be on the network client. In practice, it's really more of a batch API for mutations. It would be very problematic for clients to lock the remote DB and then iterate on it. So, |
|
@ignatz thanks
Yes, I thought of it as a batch of mutations collected by An alternative I like is how Firestore does it, which I have some experience with, and I think import { runTransaction } from "firebase/firestore";
try {
await runTransaction(db, async (transaction) => {
const sfDoc = await transaction.get(sfDocRef);
if (!sfDoc.exists()) {
throw "Document does not exist!";
}
const newPopulation = sfDoc.data().population + 1;
transaction.update(sfDocRef, { population: newPopulation });
});
console.log("Transaction successfully committed!");
} catch (e) {
console.log("Transaction failed: ", e);
}Anyway, I'm sure this API will be as good as the rest of TrailBase. I'll use it with TanStack and SolidJS to develop an multiuser SPA with client-first collections (multi-database support would be great for this). Thank you again! |
Curious question, since I'm currently looking into multi-DB. Are you thinking of a small, more-or-less fixed number of DBs or a 1:1 DB to user mapping (~multi-tenancy)? The former is pretty close, the latter will need a bit more thought. |
Yes, a 1:1 DB per customer. Each customer can have zero or more users accessing a synchronized copy of the same BD. |
I'm no expert on Electric's offering, maybe you can help me out, it looks like there's a single central DB with some policy engine that decides which changes get streamed out to which on-device databases. What I first understood and am working on is: multiple SQLite DBs on the server. From multi-tenancy I would expect data silos, i.e. users can only ever access (read & write) data in their silos (e.g. 1:1 mapping for users or something more complex like members of a customer, as you're asking). FWIW, something like Electric's policy-based streaming one can already model on the existing building blocks. Specifically, you can have a group membership table to associate records in another table with a group. Then you can setup access rules that only group members can access said records, in which case any user subscribing for changes will only see records they have access to. At least on Web you can use TanStack DB as a client-side queryable, synced DB. My intuition is that one always wants to have more fine-grained access on streaming then just, stream this DB. Even in a multi-tenancy world, you might only want certain updates. Note that TB subscriptions also support client-side filters to further narrow the changes, this is on-top of the server-side access control. For example, you could subscribe to a stream of geo coordinates associated with your customer, which are within a specific bounding box. Maybe you could talk a bit more to what you're trying to do to then work our way backwards and how to best model it. |
|
Excuse my intrusion, but I’ve been looking forward to multi-database support since I saw the roadmap. My understanding was that ‘multi-database’ in TrailBase meant multiple SQLite DBs, each with its own schema. And I was curious how the separation of these DBs would work in terms of selection, migration, storage, ... etc. I think multi-tenancy could already be handled on top of this using TrailBase |
Arguably we're intruding on your PR
Currently, I'm aiming for the minimal viable product to get things going and iron out the kinks. Specifically, allowing to register additional databases, which map to I'm currently also attaching all DBs to the same connection, which has the benefit that cross-DB EDIT: spelling things out, I'm starting to think that cross-DB VIEWs is maybe a feature that can wait in favor of more independent DBs 🤷♀️
Above limitations aside, it depends what you expect it to do. If you have a 1000 tenants, every API would become 1000 APIs if they needed to be backed by independent DBs. If you want the same schema for each tenant, you'd also need some bootstrapping and maybe a few more lifecycle hooks, e.g. to clean up deleted tenants. Would love to hear your thoughts 🙏 |
Yes, it is a single DB. a table, such as items I'm not an expert in Electric, but the concept is simple: they're SQL queries that expose a set of records to the client as if it were the entire table. It's what some RDBMSs called "Views", but mutables.
Yes, 1 SQLite DB for customer (the root user for this db).
My app will be like any other administrative App, think of one for restaurants, for example... A client registers their restaurant in the App. At that moment, a new DB is created on the server. This DB will act as a backup and synchronization point for all restaurant users (cashiers, POS systems, accountants, etc.). The app needs to know the DB URL and whether the subscription is active. The other restaurants are not accessible to the App. This simplifies everything and minimizes database locks. On the server, a master DB (not directly accessible from the app) will contain records of restaurant owners, their account type, subscribed modules, balance, etc. The database filename corresponds to the owner's ID and maintains a read-only record with their data in a table. On the server, it's only necessary to create the new DB when a restaurant registers and grant access to its owner. The rest of the users and their roles in that restaurant are controlled by the App. Of course, this idea isn't new. Over 20 years ago, I created a similar app in Visual FoxPro 6 with local DBFs synchronized with an NT4 server, and it worked very well. Implementing it with MS SQL would have taken us twice as much time and money. Sorry for rambling on. I hope I answered correctly; English isn't my first language, and I barely understand it. |
This is one of the problems to be solved by the implementation; I can imagine a successful app requiring 1,000 DBs or more, although not all connected at the same time.
One App, one schema. |
Multiple databases with the same schema and migrations. |
Same boat. Your english is great. Your insights and 20 year war stories are greatly appreciated. Everything you say makes a lot of sense. I'm mostly just trying to keep "multi-tenancy" and "syncing" as two separate features, with possibly synergistic effects. Specifically, Electric with shapes seems to market itself mostly on read-only syncing from a single master Postgres instance. (I did find some reference to support for multiple but independent PG DBs in code: electric-sql/electric@3775f1c - but not when skimming the docs). Syncing
Skimming Electric's docs, this is the one part where I'm not sure. Please correct me if I'm wrong. My understanding is that the docs use "client" and "app" pretty interchangeably. So, a sync-client might exist either in one of your backends or may be directly running on your users' devices as part of an app. Either way, the new DB is created on the client side and not within the Electric's stack (PG + Sync). This client-side DB may then be backed by SQLite, PGLite, TanStack DB, ... . In other words, whether that per-client (which may correspond to a "tenant") DB is "on the server" (as you say) or somewhere else, depends on your setup and is not Electric's responsibility.... then again, I may be in the weeds. If this understanding is roughly correct, many similar building blocks are already in place. Specifically, you can sync your table contents into TanStack DB, whether it's running on the server or in your web app. (Btw. I'm sure Electric is great, I'm not trying to trash talk, I'm mostly trying to understand and then think about what aspects or features similar to or different from prior art may make sense in TrailBase). I'm wondering if maybe you're thinking ahead, e.g. unlike Electric, breaking up per-tenant data into separate DBs could allow for a more 1:1 sync between a server-side DB shard and a client? Maybe even enable write-path syncing? Multi-Tenancy
This is what I would understand as multi-tenancy, too. IIUC, Electric doesn't help with this either? Relatedly, you could also have a multi-environment setup, e.g. test, QA, and prod data physically separated, as opposed to data for different tenants physically separated. Either way, either would be cool to support. As you point out there's some extra work for lifecycle management. The first step right now for me is to support multiple independent databases to allow different data to be kept in different physical locations and overcome some limitations (e.g. locking). This wouldn't yet allow to auto-magically route different tenants to different DBs but you could manually break by topic or establish a manual routing based on policy requirements (e.g. EMEA vs north-america 🤷♀️).
Absolutely and will be an absolute requirement for true multi-tenancy... baby steps :) |
This matches exactly what I originally understood by multi-database support — independent SQLite DBs, each with their own schema and folders, and callable via fully-qualified table names.
I agree that cross-DB VIEWs are nice to have, but supporting more than 124 DBs is more valuable than automatic cross-DB VIEWs.
Yes, cross-DB VIEWs definitely feel like something that can wait, especially if they complicate scaling past the 124-DB attachment limit (Since it's an SQLite limit, not TrailBase).
I think the key question here is: (And by the way, I genuinely appreciate dense replies full of insight and new information! 🙌) |
Agreed - certainly whenever possible. Meanwhile trying to fill the gaps with looking at prior art. That's why it's so fascinating to see what Electric an others have done.
Taking this as customer satisfaction 😅
Something like that. Besides views, there's also a question of how to deal with it in the SQL editor. Maybe users should just attach DBs manually, e.g. Despite the extra challenges, I agree that an MVP should probably have more sophisticated connection management to avoid later surprises. |
Yes, customer satisfaction achieved 😂. I was actually planning to start a discussion about multi-DB last month, but I didn’t want to rush things.
Both approaches are viable, and i think neither is a blocker for moving forward.
Totally agree, given the long-term implications, having smarter connection management in the MVP feels like the best direction. And yes, referencing how previous multi-DB architectures handled this in real-world systems will definitely help guide the design. On a side note (just brainstorming here, I realize this might introduce breaking changes), what if the API structure itself included the database name? Something like: /api/records/v2/<db_name>/<api_name>This would allow each database to behave like TrailBase does now, but with its own “silo” of attached DBs under the hood. The directory structure might then evolve into something like: |
You're right, I started the implementation two days ago, and indeed, Shapes are not mutable. Although their documentation includes references to Partitions, which are, they recommend maintaining a local table with the changes until they can be written to the primary DB.
Nowadays, there's a lot of flexibility in all of this, but for client-first solutions, the database (or part of it if it's multi-tenant) is replicated on the client side.
Right.
Well, after years of not working on the back, I see that RDBMSs haven't changed that much. They were designed to be monolithic, and due to their high demand on operational resources, I think for a single schema, it's better to use a multi-tenant, single-DB approach. (Following the example of a commercial app for restaurants, by "tenants" I'll refer to the subscribers of that app.) With SQLite is different, and I believe using 1 DB per tenant is appropriate due to its lightweight nature and lack of embedded security. Each approach has its advantages/disadvantages; the best one will be determined by the app type, the target environment, and our preferences.
Yes, it's multi-tenant, although I think of that type as single-tenant DBs... 
Electric doesn't help, but it can be solved with Shapes or Partitions in a multi-tenant DB.
In this, SQLite shines.
👍🏼
Excellent! Regarding the difference between client and app... Client and application are not the same, nor are users and terminals, but depending on the context, they can be interchangeable. In a client-first SPA, a terminal, or more accurately, the software installed on a device, is the app and acts as the client. This app has a local DB that can serve any number and type of users, although typically only one is active at any given time. Synchronization with the primary DB ("on the server") is delegated to other modules that handle data transport and transformation in both directions. Another module can manage persistence, which in a PWA is usually done with IndexedDB or localStorage, and in a desktop app with SQLite. Yet another module can be used for querying and modifying the local DB. An SPA with this level of abstraction provides almost the same advantages as working in single-user mode, with local data in a single format, although in reality, the data, as well as the modifications, can come from the primary database and/or multiple internal and external sources, all of which are stored in a store to whose state changes the UI reacts. A lightweight toolchain like Electric + TanStack DB + TanStack Query + SolidJS fits into this model, and TrailBase, with support for multi-DB and transactions, would be an even better fit, achieving a simplicity and efficiency that's hard to match. Problems to solve in this model, mostly in the client:
This is why SQLite with a decoupled persistence module, fits naturally into this architecture, especially in projects like the one I'm currently working on, which is a commercial app for small teams. I am starting implementations with 2 toolchains and in the coming weeks, as I progress, I will have concrete experiences with this architecture. Un saludo desde México. |
I think a solution for the SQL editor is blocking for an MVP. Not being able to interact with
I'm not sure why there would be multiple *.db files under |
I'm not sure how PG partitions help with writes. IIUC, writes always have to go to the primary server. They may be opportunistically applied on the client but would need to be rolled back if rejected by the primary. BTW that's what TanStack DB does for you.
Got ya, the parenthesis carry a lot of weight. That's the synergy between sync and partitioning, you're hoping for.
Right. For better or worse, TB is also loosely coppled, i.e. it's not prescriptive on the client-side and you can use TanStack DB or build something else on top of the filtered SSE change streaming (~shapes).
I see now, "PG partitions" aren't per se helping with syncing (and writes) but may help to mitigate some concerns like physical data isolation (and thus shared access bottlenecks) 👍
I think we're in an agreement here.
Agreed. It's maybe worth pointing out that from the view of the client-side app, multi-tenancy or not is an implementation detail. We're arguing architectural simplicity, e.g. establishing 1:1 mappings wherever possible, which is absolutely important but not the client's concern.
I'm not sure I understand this part. Security isn't a client responsibility. Conflict resolutions depends on the use-case. Data lifecycle management is a shared responsibility.
I agree that loose coupling can help to work with a variety of client-implementations (like electric does) to optimize for a variety of use-cases. However, this loose coupling also implies that there's no need for both the server and the client to use SQLite. The server can use DB X, client A can use storage solution Y, and client B can use storage solution Z. Empirically, Electric uses Postgres on the server and supports SQLite, PGLite and TanStack DB on the client. Electric could choose to support other server-side options in the future.
That's awesome. I really do appreciate our discussion. Looking forward to more 🙏 |
Unfortunately, in the client-first model, just like the data itself, the security problem is also mirrored on the client side. With local databases lacking embedded security, the data is exposed on every terminal, potentially accessible to multiple users/employees. I remember how absurdly easy it was to read the dBASE format; things aren't so different today. Copy an SQLite DB onto a USB and you could have years' worth of an entire company's data in your pocket. And yes, conflict resolution is shared, although it can occur before the network is accessible, for example, a user making simultaneous changes to more than one tab. In any case, client-first Apps must be designed to work without a server for days. For those of us who are starting out in a world w/o Internet, this is not so difficult ;-) |
|
Quick update here, just publishing v0.22.0 with multi-DB support. Sorry it took that long, there were more moving parts than expected - delusions are a powerful tool :hide: |
3 participants
Hello,
This aims to add transaction support to Trailbase clients:
.pre-commit-config.yamlbecause Swift tests were taking forever to complete.The code definitely should be reviewed (because even thought all the high-level programming languages have some kind of an abstract similarities, but each one may have some language-specific improvements or performance idioms).
I can see that you are currently focused on
wasmso this can wait whenever you have time for it.Cheers,