Better error handling #26
Description
Ideas on error handling
If an error is encountered during the insertion of data we should not fail the entire process.
Ideally, we should localize the error only for the file, or even better we can continue with that file after logging the error.
If a file logs more than X errors we can choose to stop processing the file.
Errors can be written to some error log file which we can save in the user's CSV directory.
The CSV directory can have a results folder with the schema SQL file and any error logs.
On the processing end, we can inform the users that errors were encountered in the following files and point then to the log file for details.
Changes required here:
In case of errors during schema inference that particular file can be skipped.