JSON select support for sqlite

[oscar345]

Adds support for selecting json with sqlite. To create tests, I have copied the tests from mysql and added them to sqlite test folder. I made some tweaks to the tests, for example using a different test database for sqlite and making small changes to the sql syntax for the asserts. This PR also changes the qrm package for dealing with returned strings instead of bytes for json values in sqlite.

There are two tests I do not get working, and those are:

TestSelectJson_GroupBy: the returned json from sqlite has different values for items in the amount key. Some sums or averages are slightly different because of rounding errrors, but it does mean the test wont pass. For example:

Original from test

{
		"CustomerID": 2,
		"StoreID": 1,
		"FirstName": "PATRICIA",
		"LastName": "JOHNSON",
		"Email": "PATRICIA.JOHNSON@sakilacustomer.org",
		"AddressID": 6,
		"Active": "1",
		"CreateDate": "2006-02-14T22:04:36Z",
		"LastUpdate": "2019-04-11T18:11:49Z",
		"Amount": {
			"Sum": *128.73000000000002*,
			"Avg": *4.767777777777779*,
			"Max": 10.99,
			"Min": 0.99,
			"Count": 27
		}

JSON from sqlite json statement:

{
  "CustomerID": 2,
  "StoreID": 1,
  "FirstName": "PATRICIA",
  "LastName": "JOHNSON",
  "Email": "PATRICIA.JOHNSON@sakilacustomer.org",
  "AddressID": 6,
  "Active": "1",
  "CreateDate": "2006-02-14T22:04:36Z",
  "LastUpdate": "2019-04-11T18:11:49Z",
  "Amount": {
    "Sum": *128.73*,
    "Avg": *4.76777777777778*,
    "Max": 10.99,
    "Min": 0.99,
    "Count": 27
  }
}

The other test that fails needs more work, but i have not been able to solve it. The TestSelectJsonObj_NestedObj returns json but the nested json statements are not returned as json by sqlite but as strings containing json (so it is not a nested object but a string). Because of that the json unmarshal function does not work. To make sqlite return a nested object instead of a string, a JSON() statement should be added that wraps the subquery. So the query that fails is:

SELECT JSON_OBJECT(
          'actorID', actor.actor_id,
          'firstName', actor.first_name,
          'lastName', actor.last_name,
          'lastUpdate', strftime('%Y-%m-%dT%H:%M:%fZ', actor.last_update),
          'LongestFilm', (
               SELECT JSON_OBJECT(
                         'filmID', film.film_id,
                         'title', film.title,
                         'description', film.description,
                         'releaseYear', film.release_year,
                         'languageID', film.language_id,
                         'originalLanguageID', film.original_language_id,
                         'rentalDuration', film.rental_duration,
                         'rentalRate', film.rental_rate,
                         'length', film.length,
                         'replacementCost', film.replacement_cost,
                         'rating', film.rating,
                         'specialFeatures', film.special_features,
                         'lastUpdate', strftime('%Y-%m-%dT%H:%M:%fZ', film.last_update)
                    ) AS "json"
               FROM film_actor
                    INNER JOIN film ON (film.film_id = film_actor.film_id)
               WHERE actor.actor_id = film_actor.actor_id
               ORDER BY film.length DESC
               LIMIT ?
               OFFSET ?
          )
     ) AS "json"
FROM actor
WHERE actor.actor_id = ?;

But to make it work, the query should look like this:

SELECT JSON_OBJECT(
          'actorID', actor.actor_id,
          'firstName', actor.first_name,
          'lastName', actor.last_name,
          'lastUpdate', strftime('%Y-%m-%dT%H:%M:%fZ', actor.last_update),
          'LongestFilm', JSON((  -- <- notice the json function here
               SELECT JSON_OBJECT(
                         'filmID', film.film_id,
                         'title', film.title,
                         'description', film.description,
                         'releaseYear', film.release_year,
                         'languageID', film.language_id,
                         'originalLanguageID', film.original_language_id,
                         'rentalDuration', film.rental_duration,
                         'rentalRate', film.rental_rate,
                         'length', film.length,
                         'replacementCost', film.replacement_cost,
                         'rating', film.rating,
                         'specialFeatures', film.special_features,
                         'lastUpdate', strftime('%Y-%m-%dT%H:%M:%fZ', film.last_update)
                    ) AS "json"
               FROM film_actor
                    INNER JOIN film ON (film.film_id = film_actor.film_id)
               WHERE actor.actor_id = film_actor.actor_id
               ORDER BY film.length DESC
               LIMIT ?
               OFFSET ?
          ))
     ) AS "json"
FROM actor
WHERE actor.actor_id = ?;

[oscar345]

I forgot to change the date and time format statement from mysql syntax to sqlite syntax

[go-jet]

@oscar345 Nice work

TestSelectJson_GroupBy: the returned json from sqlite has different values for items in the amount key. Some sums or averages are slightly different because of rounding errrors, but it does mean the test wont pass.

Not much we can do about it. It seems that json real numbers are serialized differently in sqlite json compared to sqlite driver.

To pass the test, we can unmarshal test data from customer_payment_sum.json into new test dest object and then compare objects for equality with float rounding.

The other test that fails needs more work, but i have not been able to solve it. The TestSelectJsonObj_NestedObj returns json but the nested json statements are not returned as json by sqlite but as strings containing json (so it is not a nested object but a string).

Since every nested json object has to be aliased, we can intercept alias call by implementing .AS() method on selectJsonStatement. If statement is of type JSON_OBJ we will wrap statement with JSON function and then call alias, otherwise we will just forward AS call to *selectStatementImpl.

[go-jet]

I don't think this code is ever called in your tests. This case is active only when SELECT_JSON appears as a column inside regular SELECT statement - wiki.

[go-jet]

Hmm, now that I check, it seems that this test is missing for mysql.

[oscar345]

It is needed though since I was using the select_json inside a regular select in my own code and otherwise there was convert error. I can copy the tests from postgres into sqlite to see check if all cases work correctly. The tests for mysql should probably be in another PR?

[go-jet]

Yeah, that makes sense.

[go-jet]

Hmm, now that I check, it seems that this test is missing for mysql.

[go-jet]

All this types needs to be covered by tests as well. Check TestAllTypesJSON test for mysql.

[oscar345]

Okay I will take a look at the mysql tests and add them to the sqlite tests as well

[oscar345]

Not much we can do about it. It seems that json real numbers are serialized differently in sqlite json compared to sqlite driver.

To pass the test, we can unmarshal test data from customer_payment_sum.json into new test dest object and then compare objects for equality with float rounding.

Should I just create a second file called json_customer_payment_sum.json with different rounding values?

Since every nested json object has to be aliased, we can intercept alias call by implementing .AS() method on selectJsonStatement. If statement is of type JSON_OBJ we will wrap statement with JSON function and then call alias, otherwise we will just forward AS call to *selectStatementImpl.

If I looking at the other stuff I will try to make this working but currently I am not totally sure if I can solve the problem. I will give an update about it, but maybe you can take a look at it afterwards @go-jet

[go-jet]

Should I just create a second file called json_customer_payment_sum.json with different rounding values?

That would give us two almost identical file, but we still wouldn't be sure they are truly identical.

If I looking at the other stuff I will try to make this working but currently I am not totally sure if I can solve the problem. I will give an update about it, but maybe you can take a look at it afterwards @go-jet

It shouldn't be that complicated. Something like:

func (s *selectJsonStatement) AS(alias string) Projection {
	if statementType == SelectJsonObjStatementType {
		return Func("JSON", s.selectStatementImpl).AS(alias)
	}

	return s.selectStatementImpl.AS(alias)
}