Improve validation for Headers and Columns

[kosack]

Problem

1. Currently all possible validators are options in the constructor of HeaderCard or Column. This is not very flexible, and means there are e.g. string validation options for numerical values, etc.
2. We don't have a way to validate things like UCDs, or UUIDs, which are strings.

Proposed Solution(s)

1. Change how type=x are specified to use a Type class hierarchy, which has validators inside. Then you just giev e.g. Column(type=UUID,...)
2. Allow a list of "validator classes" (independent of type), like in pydantic, e.g. HeaderCard(validators=[ValidUUID, MaxLength(50)]). That may be more flexible if there are many types of validation, but it's a bit more wordy and the same can be achieved by (1)

Required header checks:

These are needed for doing proper validation of headers (and maybe also columns in some cases):

• is valid ISO time string (needed for DATE* headers)
• is valid MJD (maybe range min/max is enough however)
• is valid unit (needed for TIMEUNIT header)
• is valid UUID (needed for data product IDs)
• is valid URI or Path (needed for GROUP tables)

[maxnoe]

Related: #2

[kosack]

What is the clearest way to do this? Pass in the Type subclass, or use HeaderCard/Column sub-classes as we do now for Column?

Option 1: (pass fits_record.Type subclass or instance, if nondefault validation options). Here validation options are part of the Type subclass, whereas metadata are attributes of the HeaderCard or Column itself.

# these all derive from a common fits_schema.Type class, and contain both information that can be used
# for checking the HDU column format or header string comptibility, as well as to check the values in an existing HDU:
from fits_schema.types import Character, UUIDString, Int32, Float32, UInt64

class SomeHeaders(Header):
    X = HeaderCard()  # default type of HeaderCards is always String, with no validation parameters set
    Y = HeaderCard(type_=UUIDString) # checks the value is a valid UUID string
    Z = HeaderCard(type_=Int32(min=0, max=1200))  # will check that the value is convertible to Int32 and in the limits
    W = HeaderCard(type_=Int64(allowed_values=[1,2,3]), description="event class id number")

class SomeTable(BinaryTable) :
   COL0 = Column(type_=Character(max_length=12), description="Column 0")
   COL1 = Column(type_=UInt64)
   COL2 = Column(type_=Float32(min=0.0), unit=u.m) # column must be Float32 and all values must be >0

Option 2: subclass everything. Here validation and metadata are mixed as options of each subclass

class SomeHeaders(Header):
    X = HeaderCard()  # default type of HeaderCards is always String
    Y = UUIDHeaderCard() # checks the value is a valid UUID string
    Z = Int32HeaderCard(unit="kg", min=0, max=1200)  # will check that the value is convertable to Int32 and in the limits
    W = Int64HeaderCard(allowed_values=[1,2,3], description="event class id number")


class SomeTable(BinaryTable) :
   COL0 = CharacterColumn(max_length=12, description="Column 0")
   COL1 = UInt64Column()
   COL2 = Float32Column(min=0.0, unit=u.m) # column must be Float32 and all values must be >0

I tend to prefer Option 1, as the same type check classes can be used in two places and there are fewer sub-classes to manage, and it is also more clear what is Header/Column metadata vs validation limits.

[maxnoe]

I tend to prefer Option 1 as the same type check classes can be used in two places and there are fewer sub-classes to manage

I'd say that argument does not really hold.
If you start with Option 1, you can trivially implement option 2 in this scheme:

class Column():
    def __init__(self, dtype):
        self.dtype = dtype

class UInt64Column(Column):
    def __init__(self):
        super().__init__(dtype=UInt64)

In the end, the difference is miniscule:

foo = Column(UInt64, ....) 

to

foo = UInt64Column(...)

So I am completely fine with starting with 1)

[kosack]

The set of types allowed for Columns are not the same as the list allowed for Headers, so probably can share some of the same logic for a subset, but need to at least check if a type passed in is in the correct list for Header vs Column.