RFC: Serialization Protocol for Dart

[schultek]

This is the main discussion issue for the proposal.

Link to RFC: Serialization Protocol for Dart

---

TLDR:

This RFC proposes a new serialization protocol for Dart, aiming to be modular, universally usable across different data formats, and highly performant. Key points include:

• Separation of encoding/decoding logic from data formats.
• Support for various data formats (e.g., JSON, CSV, MessagePack) through extension methods.
• Handling of nested objects, collections, custom types, binary formats and more.
• Extended protocol for polymorphism and generic type handling.
• Performance improvements over current methods (+ Benchmark).

The proposal aims to standardize and improve serialization in the Dart ecosystem, making it more efficient and flexible.

[jakemac53]

Once thing I have been looking at recently is using the Codec interface we already have as the way of expressing the encode/decode methods. Fundamentally we are just encoding/decoding classes here, so why does the Codec stop at the Map boundary today usually?

You still want the Encoder and Decoder classes (these replace Map still), so it doesn't look all that different than what you have here - mostly its just that you replace the SelfCodable classes with things that implement Codec instead.

The "from" type on the Codec becomes something that can glue together an Encoder/Decoder (I am calling this a Coder for now), which is slightly awkward (codecs represent going both directions). But, I think you can ultimately get something interesting.

I have an example of this using package:jsontool (and its JsonReader and JsonWriter types), and I think it ended up pretty nice.

You can still go to/from JSON if you want, by fusing your Codec for your class with a Codec that can go from Map -> Coder: MyCodec().fuse(jsonCoder).fuse(json) (where jsonCoder is a Codec<Codable, dynamic>, although the dynamic is unfortunate here, but its what the json codec converts to).

[schultek]

I thought about and experimented with using Codec but decided against it because its API is too clunky and it doesn't really fit into the Encodable - Encoder model.

I have an example of this using package:jsontool (and its JsonReader and JsonWriter types), and I think it ended up pretty nice.

Yes that looks similar to Encoder/Decoder but again JSON specific and not data-format agnostic.

You can still go to/from JSON if you want, by fusing your Codec for your class with a Codec that can go from Map -> Coder: MyCodec().fuse(jsonCoder).fuse(json) (where jsonCoder is a Codec<Codable, dynamic>, although the dynamic is unfortunate here, but its what the json codec converts to).

So that would still have the intermediate Map? The whole point of Encoder is to avoid that.

Also we would only need a Codec for the data format, not for the model. Since e.g. a JsonCodec<Codable, String> could directly accept MyClass.codable. But that is the same of what the JsonEncoder.encode() method already does, just with an additional level of encapsulation.

---

The only use I see for Codec is if you want to abstract over the data format as well. E.g. a method that has a Codec<Codable, List<int>> codec parameter and thereby would accept any binary data format. But tbh. I don't see any actual use-case for this, so I left it out of the proposal.

[jakemac53]

I thought about and experimented with using Codec but decided against it because its API is too clunky and it doesn't really fit into the Encodable - Encoder model.

I agree it is a bit awkward to slot in the bi-directional nature of a Codec with the Encoder/Decoder model, but I was able to make something work in a very similar API to this, so it is possible (will try to share soon, its just a local thing right now).

The nice property that you get from that is the ability to fuse codecs together - where in the Encoder/Decoder model you are adding specialized methods on each Encoder/Decoder class to handle the different actual representations (https://github.com/schultek/codable/blob/main/lib/src/formats/json.dart#L28).

In the codec world each codec just does one layer of the translation (so, Object -> JsonEncoder -> Map -> String -> Uint8List), and then you rely on smart fusing to actually skip the intermediate representations where appropriate. This also allows for chunked decoding for additional performance benefits, the whole pipeline can be chunk based theoretically although the Encoder/Decoder API would have to look different to support that all the way through (and probably would be a pain to deal with).

Yes that looks similar to Encoder/Decoder but again JSON specific and not data-format agnostic.

It actually isn't JSON specific at all, you could just change the name JsonReader to Decoder. It just only has a JSON implementation provided and has a name that indicates JSON only, even though it isn't really tied to that.

So that would still have the intermediate Map? The whole point of Encoder is to avoid that.

It wouldn't because when you fused them together you can actually eliminate the codec in the middle (this does have to be explicit though).

Also we would only need a Codec for the data format, not for the model. Since e.g. a JsonCodec<Codable, String> could directly accept MyClass.codable. But that is the same of what the JsonEncoder.encode() method already does, just with an additional level of encapsulation.

Actually this is interesting and I think would make sense, having the Codec work on the Codable interface. That is a much more natural fit with the Codec API since Codable already supports both directions.

I think maybe what you end up is this, which just builds upon exactly what you already have:

class JsonCodec<T, C extends Codable<T>> implements Codec<T, String> {}

So, the codec part is just an optional thing that the Codable model actually fits nicely with.

[schultek]

In the codec world each codec just does one layer of the translation (so, Object -> JsonEncoder -> Map -> String -> Uint8List), and then you rely on smart fusing to actually skip the intermediate representations where appropriate.

Ah the smart fusing thing is interesting. I've heard of this for json+utf8 but hadn't considered it could be used for other codecs as well.

Actually this is interesting and I think would make sense, having the Codec work on the Codable interface. ...

class JsonCodec<T, C extends Codable<T>> implements Codec<T, String> {}

So, the codec part is just an optional thing that the Codable model actually fits nicely with.

Oh yes that could make a lot of sense. I like that.

[kingwill101]

I really appreciate the effort that has gone into this proposal. My only concern so far is, how does dart:convert fit into the picture?

it does provide a Codec interface that seems to partially cover the use case I've seen so far while reading through the proposal

abstract class Codec<S, T> {
  const Codec();

  T encode(S input) => encoder.convert(input);
  S decode(T encoded) => decoder.convert(encoded);

  Converter<S, T> get encoder;
  Converter<T, S> get decoder;

  Codec<S, dynamic> fuse(Codec<T, dynamic> other) { .. }
  Codec<T, S> get inverted => ...;
}