Consistancy issues in JSON schema

[xXyepXx]

Describe the bug
The definition of the schema in schema.json specifies that some fields may be one of several types.

For instance, the field implies can either be a string or an array (of strings):

"implies": {
  "oneOf": [
    {
      "type": "array",
      "items": {
        "$ref": "#/definitions/non-empty-non-blank-string"
      }
    },
    {
      "$ref": "#/definitions/non-empty-non-blank-string"
    }
  ]
},

This makes it really hard to parse the JSON object in languages like Go where you need to define the types statically:

type Techno struct {
    Categories  []int    `json:"cats"`
    Implies     []string `json:"implies"`  // This may just be a string!
}

Trying to deserialize a JSON object into an instance of this struct will return an error if the JSON input is using a string type instead of an array of strings:

json: cannot unmarshal string into Go struct field Techno.implies of type []string

Expected behavior
Have a single type for each field.

[enthec-opensource]

This is an issue we have encountered aswell
the original project is not ours, we just started maintaining adding technologies and fixing json objects, always trying not to break old implementations of wappalyzer.
we could deal with this by just replacing all "[]string OR string" by just "[]string" as webappanalyzer implementations parse string or []string anyways.
Im going to leave this issue open and link commits related to this.
is not as good practice and makes the code less readable, but you can json unmarshal to map[string]interface{} and assert the interface{} (obj.([]string)) or just by using "reflect".

[kingthorin]

I was the one that added the schema to the original project, it was added 'after the fact'. Then was modified further after that. Since it was added after the data files existed (not before), it had to conform to the data present after years of different people contributing with no guidance or format restrictions.

I agree if it was more strongly typed that would really help.

On the off chance that it might help someone here's the parser we use within ZAPs technology detection add-on: https://github.com/zaproxy/zap-extensions/blob/main/addOns/wappalyzer/src/main/java/org/zaproxy/zap/extension/wappalyzer/WappalyzerJsonParser.java
(That code has also evolved over time and been worked on by maybe a dozen people so if it seems horrid I won't be surprised 😉)

[mlec1]

I made some contribution in order to unify and simplify the schema.
I hope it fits your needs. If you see other fields that could be improved, just raise your idea

[kingthorin]

Thanks. Seems fine on my end. Nice work!

[enthec-opensource]

unsure about what to do with the dom selectors, there are too many variations

string
"dom": "form[name='formLogin'][action='login.aspx' i][id='formLogin']"

string array:
"dom": ["form[name='formLogin'][action='login.aspx' i][id='formLogin']", ...]

object
"dom": {
"img[src*='provenexpert']": {
"attributes": {
"src": "images\.provenexpert\.\w+"
}
}
}

I'd say transform string and string array to the object structure, like "dom": {"form[name='formLogin'][action='login.aspx' i][id='formLogin']": ""}, seems more flexible.

However this change might break some implementations so i thought on creating a new one called "selector" and deprecating the "dom" one.

[kingthorin]

Well assuming people are already using the objects then moving strings and arrays into objects "shouldn't" break anything ... Though I guess there might be some just using the strings and string array elements 🤷‍♂️