Query types with Option's support

[elcritch]

@Araq what do you think of this? It adds an optional return type to the query DSL.

This allows moving the DSL forward to support Option/nulls better while maintaining backwards compatability. It make many queries a bit nicer which often just map the tuple to a Nim type anyways. It also supports fromQueryHook hooks similar jsonutils to/from. Though perhaps the hooks could have better names.

proc fromQueryHook*(to: typedesc[seq[string]], x: string): seq[string] =
  x.split(",")

type
  SplitMessageRow = object
    parts: seq[string] ## uses the `fromQueryHook`

test "applies fromQueryHook per destination field type":
    let rows = query(SplitMessageRow):
      select tb_string(typstring as parts)
      orderby typstring

    check rows[0].parts == @["alice", "bob"]
    check rows[1].parts == @["carol", "dave"]

type
  NullableNoteOptionRow = object
    id: int
    note: Option[string]

test "maps nullable column to Option":
    let rows = query(NullableNoteOptionRow):
      select tb_nullable(id, note)
      orderby id

    check rows[0].id == 1
    check rows[0].note.isNone
    check rows[1].id == 2
    check rows[1].note.isSome
    check rows[1].note.get == "hello"

[Araq]

Looks very good.

[elcritch]

@Araq ready for review. I had to refactor the design to lower performance overhead. It's now 1.15x the speed. It also doesn't support row hooks now, only field hooks.

Sort of odd since it's just adding a function call. Though I guess a function for each field adds up.

[Araq]

Though I guess a function for each field adds up.

What? I thought it's all done at compile-time.

[elcritch]

Though I guess a function for each field adds up.

What? I thought it's all done at compile-time.

It is compile time, but fromQueryHook is called for each field so the user can override based on types. Seems to be the extra overhead.

[elcritch]

@Araq ready to review again.

Most of the overhead was due to extra copies into the "option" wrapper and another copy due not binding results directly into the DbValue wrapper object. It's now down to ratio=1.043x;. Given the extra "is column null" checks and extra hook indirection probably about as good as it'll get.

I changed the hook signatures from proc fromQueryHook*[T, S](to: typedesc[T], x: S): T to proc fromQueryHook*[T, S](val: var T, x: var DbValue[S]) which let's us use moves and avoid extra copies.

I cleaned up the macros that handled the typed bindings. Ugh, on stuff where the LLM's don't have much source material they struggle much more. Salvageable, but definitely the result of getting it sorta close and hammering it until it fit. Glad I added a benchmark though, gotta watch these LLMs. ;)

Here's what it looks like now:

  proc fromQueryHook*(val: var seq[string], x: string) =
    val = x.split(",")

  type
    SplitMessageRow = object
      parts: seq[string] ## uses fromQueryHook

  test "applies fromQueryHook per destination field type":
    let rows = query(SplitMessageRow):
      select tb_string(typstring as parts)
      orderby typstring

    check rows[0].parts == @["alice", "bob"]
    check rows[1].parts == @["carol", "dave"]

  It also supports mapping nullable columns into Option fields:

  type
    NullableNoteOptionRow = object
      id: int
      note: Option[string]

  test "maps nullable column to Option":
    let rows = query(NullableNoteOptionRow):
      select tb_nullable(id, note)
      orderby id

    check rows[0].id == 1
    check rows[0].note.isNone
    check rows[1].id == 2
    check rows[1].note.isSome
    check rows[1].note.get == "hello"

[Araq]

fine for now but row-based null values can be much faster and more elegant.

[elcritch]

fine for now but row-based null values can be much faster and more elegant.

What do you mean? As in the entire row is null?

[Araq]

seq[(T, bool)] vs (seq[T], IntSet -- bit i is set if column i is null). This allows pros like me to use the seq[T] part and to easily ignore the null business.

[elcritch]

seq[(T, bool)] vs (seq[T], IntSet -- bit i is set if column i is null). This allows pros like me to use the seq[T] part and to easily ignore the null business.

seq[T] doesn't make much sense to me unless you mean doing a columnar approach, but that'd be more expensive on non-columnar dbs.

Plus an IntSet on a row basis wouldn't change much performance wise. The expensive bit seems to be the is_column_null(dbRowIter, col_idx) check itself. At least I don't know any sqlite bindings for get_all_column_nullity(row).