[Proposal] An ORM Implementation based on Protocol Buffer 3 definition

[sadlil]

This library intend to design a ORM for go based on the protobuf definitions. This issue intend to have discussion about design and implementation, that needed to be asked and answered.

Most of the ORM design in go is based on struct tags. But generating from protobuf we needed to implement a way to get the tags from the field mappings, with out tagging the structure.
In short the orm should have two basic part -

Generator
Generates code definition, sql statements from protobuf definition. For reference This could follow the pattern of grpc-gateway, that generates go http proxy for gRPC services.
This also includes mapping between protobuf types and sql types.

• proto 3 scaler types
• mysql data types
• postgres data types

Generator can heavily use protobuf Message Option and Field Option to take targeted inputs from user specifically indicating sql behaviours for fields indicating column and message indicating table.

For further flexibility generator can also generate sql statements and/or migrations, based on previous definitions and fields serials.
In future that may also generates helper methods to use with library.

Library
Actual implementations of the ORM that uses generated data to integrate with. Can follow pattern based on gorm or xorm two of most popular go orm libraries.

Comments and Suggestion wanted.

[mirshahriar]

This is the list of type conversion from proto 3 scaler types to go types.
https://developers.google.com/protocol-buffers/docs/proto3#scalar

From these go types, we can find sql types. We can follow this.
https://github.com/go-xorm/core/blob/master/type.go

[mirshahriar]

Question:

If I want to create a table using proto message where one column type can be Blob, LongBlob, Bytea or others. All represent string in Go and proto3. How can we do that without tagging sql types in proto field?

Shall we use sql types as proto field types?

Like this one?

message Test {
  required Blob label = 1;
  optional BigInt type = 2 [default=77];
  repeated Int reps = 3;
}

From this we can generate a go type with tag of sql types.

type Test struct {
    Label string `orm:"Blob"`
    Type int64 `orm:"BigInt"`
    Reps []int `orm:"Int"` # need to find out actually behavior
}

Now we can use this generated struct for both.

Am I missing something here?

[sadlil]

@aerokite thanks for asking this questions. This implementation also have some potential. But my initial thinking was not to maintain two different version of struct if you are going to use API layer with storage layer.
I am on the side of defining the messages with scaler types, and mapped with some kind of tagging system. Let me explain with a example -

message Test {
  option (ormpb.table).name = "tests";
  string label = 1 [(ormpb.column).type = "blob"];
  int64 type = 2 [(ormpb.column).type = "bigint"];
}

That would eventually be generating something like the following -

func (t *Test) TableName() string {
  return "tests"
}

func (t *Test) Tag(field, name string) string {
  m := map[string]map[string]string{
    "Label": map[string]string{
       "type": "blob",
    },
    "Type": map[string]string{
       "type": "bigint",
    },
  }
  return m[field][name]
}

In That way user should have more flexibility to user to even have same or even different layer upon API and storage. But i am still open for discussion.
Let me know what do you think about that.

cc: @sunkuet02

[mirshahriar]

I did say so. But instead of using string in message field, I thought to use blob similar to database column. Other part is same to you

[mirshahriar]

Your idea is better I think. because, we can add more information in proto field. [(ormpb.column).type = "blob"]. We can add size, default, primary key others.

[s4kibs4mi]

Found a library protoc-go-inject-tag it may help to add custom tag. Here is how it works,

message Hello {
    // @inject_tag: ormpb:"primary_key"
    string name = 1;
}

command protoc --go_out=plugins=grpc will generate hello.pb.go with,

type Hello struct {
	// @inject_tag: ormpb:"primary_key"
	Name string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"`
}

and then if execute command protoc-go-inject-tag above definition will be converted to,

type Hello struct {
	// @inject_tag: ormpb:"primary_key"
	Name string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty" ormpb:"primary_key"`
}

Hence we can work on these tags to do other orm related work.

[mirshahriar]

We need more methods that protoc will not generate. So we need our modified generator.

We can take help from these repos.

Also check this one : https://github.com/dsymonds/gotoc

This library parses proto file and generate codes. Didn't take a look in code base.

[sadlil]

Yeah, agreed with @aerokite, still we need to generate some things as of - TableName() for now - for the tables. May be some other things to like - migration files, even some ORM methods :P
So better to use our own generator.
But may be i am in favour of pure protoc compiler.

@s4kibs4mi thanks for the library. This looks interesting. will look more into that.

[mirshahriar]

I am not experienced with protobuf.

I think protoc doesn't generate go code. protoc-gen-go does. protoc only parses .proto files and writes a JSON data which protoc-gen-go uses to generate code.

In our case, we need custom parser to parse tag like this [(ormpb.column).type = "blob"]. And also, we need to generate ORM based methods so we also need our own generator like protoc-gen-go.

Lets see, if it is possible to use protoc for custom tag.

[sadlil]

yes possible. protoc can parse the custom tag. But if you need to generate something you need your custom generator. thats why we are going to write protoc-gen-orm.

also take a look at: https://github.com/uber/prototool