Error handling for backend failures

[konstpieper]

On the new intersect-sdk="0.90", we now sometimes get error messages:
b"Service domain logic threw explicit exception:\nCouldn't update workflow with new data for workflow 'id'".

This is usually caused by incomplete input validation for Messages, particularly kernel_args, and backend_args, which have almost no pydantic validation. The highest priority would be IMO to further standardize the common arguments, and validate them. But in practice there may still be cases where that is not enough.

Questions:

• what exception stack traces should be returned
• what exception stack traces should NOT be returned
• whether or not we want to configure this such that the Service can send exception stack traces if a configuration "debug" flag is enabled, and not send them if it isn't

[Lance-Drane]

whether or not we want to configure this such that the Service can send exception stack traces if a configuration "debug" flag is enabled, and not send them if it isn't

My thought is that this is probably the most straightforward path to accommodate both the I want to debug usecase and the I don't want to risk leaking information usecase. This is as simple as using a boolean flag in the config file. In practice, we can probably leave the debug flag on for internal testing deployment instances and turn it off for shared instances.

The initial implementation for this can be "raise the IntersectCapabilityError if the debug flag is set, raise a generic Exception if not". As precise errors are determined, and we know exactly what we want to send to users, we can eventually migrate towards always publicly raising the IntersectCapabilityError (but without information such as the exception stack traces).