Discussion around the project

[backnight]

Hello,

From what I understand of this library, it replaces others ORM like TypeORM (you have entities, stores, drivers, etc) in order to be able to fetch data from a JSON API and I was thinking: why not developing a JSON API driver for TypeORM for example?

Because in the end, you have developed a JsonApiEntityProvider but you will have to do one for SQL Database, isn't it?

[dpimonov]

Hello,

Thanks for the great question!

About replacement of TypeORM and others the answer is yes and no. :)

TypeORM is a great library and targets applications which use databases. EntityStore in the first round focused on backend API's. At first glance it may seem that they solve similar issues but if you read deaper TypeManager documentation which is the core of this package you may notice some differences.

You are right that it can provide a replacement for TypeORM after some efforts but this is currently not on the roadmap. For databases there are many issues to solve like migrations, column mapping and others to really compete on a good level and not just create another clone.

The basic idea about EntityStore is reusability when working with different datasources. Today I have to work with JSON:API, tomorrow with GraphQL, another day with ODATA and so on. I want to have one universal API which provides me first class model support with sorting, filtering, pagination and data transformation so I don't have to worry about underlying service and simply make configuration by switching providers. Besides I want ORM with API as close as possible to native JavaScript with proper code references and type checks.

There are also other reasons why I've started this project which include topics like reflection, circular reference handling, custom property types, DI, etc. so I don't have to create a frankenstein from set of libraries to make it work in one project. :D

[backnight]

Hello,

Thanks for the explanations!

If I understand well, you plan to use this library in client side applications only right? In order to be able to fetch remote data from APIs?

How are architectured your projects?

On my side I'm looking for a data library (packaged as an API library) that I can use on both client AND server side.
I will call this data library: ProjectAPILib.

To give you an example:

• My project has 2 entities splitted into 2 separate services with their own code base, both have their own JSON APIs for remote data fetching: Exchange -> /exchanges and Market -> /markets
• Those 2 entities are linked by a one to many relationship: An Exchange -> has many -> Market
• If I develop a frontend app to display this data, I'm going to import ProjectAPILib into it in order to fetch my data when I need it (client side use case).
• In ProjectAPILib, I will find 2 stores (one for Exchange and one for Market) and each one will handle the requests to be made to the JSON APIs endpoints /exchanges or /markets.
• When the data fetching request hit one of my service, let's say Exchange, inside this service I will import ProjectAPILib (server side use case) and replace the ExchangeStore JsonApiEntityProvider by a local data provider depending on where is stored my Exchange data, let's say an SQL database, so I will use an SQLEntityProvider.
• Inside my Exchange service, I will then be able to fetch my data from a local DB through my ExchangeStore while fetching JSON API Market data using the MarketStore.

Do you see where I want to go?
And that's why I asked if you planned to replace TypeORM or at least do an SQLEntityProvider.

The ultimate goal is to have one data library handling REMOTE (JSON APIs) AND LOCAL (SQL) data fetching and unifying everything together.

[dpimonov]

Hello,

If I understand well, you plan to use this library in client side applications only right? In order to be able to fetch remote data from APIs?

No, it is planned for both client and server. By answering yes/no I meant in the first round the plan was to focus on API's as there are not too much good alternatives in my opinion.

How are architectured your projects?

Are you asking about libraries or projects where they used?

Do you see where I want to go?

The same way as I do. :D

And that's why I asked if you planned to replace TypeORM or at least do an SQLEntityProvider.

The answer is yes there are plans to do that. I wanted to proceed with ODATA or GraphQL when fix issues with JSON:API but based on your request maybe I have to reorder the plans. Will see... There are many things to consider when it comes to dealing with databases.

What slice of functionality are you looking for when talking about SQLEntityProvider? What kind of databases you are interested in? Btw do you think you can help with implementation of some parts related to SQLEntityProvider?

[backnight]

Do you have a tasks board where I can see what you plan to do, the time it needs to be done then we can iterate from there?

On my side I'm interested to have at least one remote connector (JSON API) and one local connector (DB) full functional ASAP in order to be able to use this library both from client side and server side. It's not my priority to have ODATA or GraphQL handled as they are only for client side/remote data (it will not solve my local/server side data need).

You have your roadmap/I have mine and from what I see, if I give a hand I will handle the SQLEntityProvider part.
In term of DB I plan to handle PostgreSQL first and maybe MariaDB second.

[dpimonov]

Do you have a tasks board where I can see what you plan to do, the time it needs to be done then we can iterate from there?

There are no any board as for now. I don't have to create a board for my self due to time management. I keep ideas as a simple list. The nearest tasks I plan to work on is to enable source maps for all projects and provide response validation for JsonApiEntityProvider to close an open issue.

You have your roadmap/I have mine and from what I see, if I give a hand I will handle the SQLEntityProvider part.

Yeah we can proceed almost immediately. I have not started other providers yet so it is ok. We can create a repo + board. Identify the issues to solve. From what I see we need query, filtering, saving and pagination part. The same functionality which is present for JSON:API. Right? Or migrations should also be involved? Creating a connector rather simple task in comparison to initializing and updating the database.

[backnight]

Hello,

Identify the issues to solve. From what I see we need query, filtering, saving and pagination part. The same functionality which is present for JSON:API. Right?

Exactly, we have to do the same in SQL syntax terms: SELECT, INSERT, UPDATE, DELETE ; filtering: WHERE ; paginating: LIMIT/OFFSET.

No need for migration, I'm going to challenge you on this: What's the value added by @dipscope/type-manager compared to other model decorating libraries (let's say, TypeORM :D)?

I'm going to be a lot challenging on every parts of this project to be sure that you/we are not re-inventing the wheel.

The complexity of the project is not on connectors/drivers I think, it's on the merge of multiple drivers output to make a response, i.e: filtering a set by relationships :)

If I have Post + Tag + Author models and posts have tags (many to many relationship) + an author (one to one relationship). How is behaving the library when I say that I want to fetch all posts which have the tag "toto" and are made by author "admin"?

Which model/service is hosting the data related to the relationship between Post and Tag? Post service? Tag service? Both? Another service? :)

[dpimonov]

Hello,

What's the value added by @dipscope/type-manager compared to other model decorating libraries (let's say, TypeORM :D)?

Well first of all it is not decorator library. I promote it mainly as serialization / deserialization library as it solves many issues of existing serialization libraries, e.g. TypedJSON, class-transformer which stay open for months. Decorators are simplification for several basic functions provided by library and that's it. You can decorate 3rd party classes pretty easy if you would like or use different decorator configurations for the same entities within one application.

The main purpose of @dipscope/type-manager is as it says in its name to manage types:

• What is the type name?
• What properties and constructor arguments type has?
• What are the name, type, default value, ... of a certain property.
• Which are descendant classes?
• What are generic types for types like Map<TKey, TValue>?
• What are polymorphic types for types like Array where TShape is polymorphic?
• How to build a type and what services must be injected into it?
• How it should handle references including circular one, e.g. type referencing them self?
• How it should treat namings, e.g. snake_case vs PascalCase?
• How it should treat data which is not match by type, e.g. type conversions?
  ...

And this is only a top of the list for which @dipscope/type-manager was developed for. Besides it provides methods to assign custom decorator data and later use it for types and properties. This function used in JsonApiEntityProvider to provide JSON:API related config. And the same function will be used in SqlEntityProvider to at least tell in which table type is stored.

Unfortunately ~ 60% of use cases provided by TypeManager stay undocumented still. I've described parts related to serialization and deserialization and it is already huge enough. I hope there will be a hand in the future to help me with documenting all the stuff in details.

So in general it is not anyhow related to EntityStore or TypeORM. It is completely self-sufficient library which solves it's own issues. But EntityStore is directly dependent from TypeManager. You can think about it as a plugin for TypeManager so on top of all provided features you are getting an ORM to fetch the data and perform all that crazy stuff which offers you a TypeManager. In the end what each plugin for EntityStore should do is to build a plain object tree based on the command provided by EntityStore and pass it through TypeManager deserialization process to get a type you want.

And here we come to EntityStore area. :)

The complexity of the project is not on connectors/drivers I think, it's on the merge of multiple drivers output to make a response, i.e: filtering a set by relationships. :)

To solve this issue EntityStore was developed. It provides you all this information of how to build each type and their related relationship metadata. What a plugin does is just traversing an expression tree of what it gets from EntityStore.

If I have Post + Tag + Author models and posts have tags (many to many relationship) + an author (one to one relationship). How is behaving the library when I say that I want to fetch all posts which have the tag "toto" and are made by author "admin"?

Many to many relationship filtering is currently not implemented. I'm still thinking about best way to handle it in the API. But based on your example if consider that many-to-many relationship filtering is implemented as I see it now here will be a request to fetch data. e.g. SELECT statement which SqlEntityProvider should handle.

const postCollection = await postSet
    .filter((p, f) => f.and(f.eq(p.postTags.tag.name, 'toto'), f.eq(p.author.name, 'admin')))
    .paginate(p => p.offsetLimit(10, 20))
    .findAll();

SELECT [serializedPostProperties]
   FROM posts p,
             posts_tags pt,
             tags t,
             authors a
 WHERE p.id = pt.post_id
     AND pt.tag_id = t.id
     AND p.author_id = a.id
     AND t.name = 'toto'
     AND a.name = 'admin'
OFFSET 10
   LIMIT 20 

Or it may be an UPDATE statement.

await postSet
    .filter((p, f) => f.and(f.eq(p.postTags.tag.name, 'toto'), f.eq(p.author.name, 'admin')))
    .paginate(p => p.offsetLimit(10, 20))
    .update({
        name: 'New Post name'
    });

UPDATE posts
         SET name = 'New Post name'
   WHERE id IN (
        SELECT p.id
         FROM posts p,
                   posts_tags pt,
                   tags t,
                   authors a
       WHERE p.id = pt.post_id
           AND pt.tag_id = t.id
           AND p.author_id = a.id
           AND t.name = 'toto'
           AND a.name = 'admin'
      OFFSET 10
         LIMIT 20
   )

The first one results in BulkQueryCommand and the second one in BatchUpdateCommand. This commands should be handled by SqlEntityProvider.

To build a SELECT part you have to use TypeMetadata provided in the command and implement a visitor for IncludeExpression to load relationships if any requested. To build FROM and WHERE part you have to write a visitor which will go through FilterExpression provided by EntityStore. To build pagination part you have to write visitor for PaginateExpression. After you get the data for SQL command you have to create a raw object representation or a list of objects for the data you get and then deserialize it using TypeManager which will handle and resolve all dependencies, data transformation and references.

Which model/service is hosting the data related to the relationship between Post and Tag? Post service? Tag service? Both? Another service? :)

The relations between entities, e.g. one-to-one, one-to-many, many-to-many can be identified using TypeManager. The most simple solution is to check if entity has configured table decorator which can be provided by SqlEntityProvider. This are already implementation details. Just keep in mind that in most cases relationship and types are already there as they provided by TypeManager in related TypeMetadata attached to each command which you built using EntityStore.

If you mean by hosting data related to the relationship between Post and Tag some state which is tracked when you get entities from a store. This is planed for the next versions of EntityStore. In v.1.0.0 it is simple CRUD ORM library without any state control.

I'm going to be a lot challenging on every parts of this project to be sure that you/we are not re-inventing the wheel.

We are actually reinventing a wheel in some manner. :)

Each similar project is about reinventing a wheel. Lets say I have some vision of how such software should behave. Open source is always about finding the people which have the same vision on similar problems. For example TypeORM may be perfect for one and bad for others. The same is applied to EntityStore. It is good if there are a choice. ;)

[backnight]

Thanks for the detailed explanation.

About the many to many relationship example (Posts, Tags and Author), I think you didn't get what I was implying because you gave me a SELECT with a WHERE clause in it.

Posts, Tags and Author are not in the same database nor in the same service so you cannot do it in a single SQL request statement to handle that.

And that's why I asked where do you store the relationships data :)

In a micro-services app, by having every model splitted into isolated services, you lose the ability to do a JOIN (in SQL term) in order to link your data together and do the filtering.

The "joining" must be done by the library you are developing ;) there is a post request processing to do

[dpimonov]

Thanks for details! Looks like I've finally got your idea. :)

About the many to many relationship example (Posts, Tags and Author), I think you didn't get what I was implying because you gave me a SELECT with a WHERE clause in it.

I gave a SELECT statement as an example which can be sent to a database by a plugin. You can build and send any requests including case with multiple databases and then filter, sort and paginate data in memory like it is done in InMemoryEntityProvider.

The "joining" must be done by the library you are developing ;) there is a post request processing to do

This are details of underlying EntityProvider. EntityStore is designed to provide you abstract expressions with metadata. How this expressions are handled with one database or several, merged or transformed is not responsibility of EntityStore so I am not going to change anything in the core as it contradicts with underlying concept in general.

In a micro-services app, by having every model splitted into isolated services, you lose the ability to do a JOIN (in SQL term) in order to link your data together and do the filtering.

I don't know your project architecture but looks like you over engineering. Of course you know better what you are doing but the concept of merging and filtering you described will result in poor performance because of network requests and in memory filtering. Besides if one service goes down then ORM become broken. This is not how microservices intended to work based on my experience.

Such cases like merging data from several microservices in realtime makes them depend. This is a critical point of failure. What you can do instead is setup messaging brokers and create readonly view in a special separate microservice which updated using events from underlying services when there are related changes. Such service will work for the end users even if all other services go down. You can use regular ORM like TypeORM for it to read data without any hacky things with merging. But a question if you really have to? :)

I've build event sourced systems and regular one. In both cases Post, Tag and Author should be a part of one microservice as they coupled together in one bounded context. Let call it BlogContext. There may be another microservice with entities like User, Company, Address which are part of IdentityContext. They may communicate via messaging brokers, API or GRPC. Each microservice must be self-sufficient and host critical data it depends from. Most probably info about Post, Tag and Author must be saved together and be a part of one single transaction within one microservice. Otherwise you are building a distributed monolith.

There are also complex scenarios like distributed transactions e.g. transactions which span multiple services. But note that this is a little bit different from what you described. It is about that you cannot change data until there will be some required state. For example in market aplication you cannot send product to customer (ShippingMicroservice) until you withdraw the money from local account (AccountingMicroservice) and you cannot withdraw the money until bank system (ExternalService) tells you that there is required amount. If there are no event sourcing concept involved people usually use saga pattern. Just connect their favourite ORM per bounded context (microservice) and call propper API's with proper entities which are defined per each context. In this case TypeORM as part of saga covers such use case completely.

However there are exist libraries at least for C# which allow you do thing which you are going to do. For example HotChocolate GraphQL stitching module. Maybe similar libraries exist for TypeScript which allows to create a schema from several different services / databases / rest API's but as I described this is not a best approach in case of microservices in my opinion.

Sorry for a long post. I tried to be as short as possible with my thoughts as I see that there is no need for us to invent something. At least this is not an area where I plan to go with a library. :)