DataFrame write api should accept Overwrite option when the file already exist

[MrDataPsycho]

Problem:
While trying to write a csv (any type) file, I am getting the following exception if the file already exists:

Error: Execution("Could not create directory data/processed/diabetes.csv: Os { code: 17, kind: AlreadyExists, message: \"File exists\" }")

But other frameworks like Polars, PySpark accept overriding either optional or by default and it also possible to append existing files if selected.

Possible Solution:
I suggest having a CsvWriteOptions struct like CsvReadOptions struct which must be passed during writing the file. For example df.write_csv(path: "my/precious/path", CsvWriteOptions::default()).await? This family of write options can also have all the configuration possibilities for json, parquet, csv, table etc. For example, If I want to write the csv into multiple file or one single file.

Also not sure why by default csv file is written into multiple partition part01.csv ... part08.csv by default, I should let user write into one csv file incase of smaller dataset.

[Jefffrey]

The existing API is a bit confusing on what it really does; given this example program:

let df = ctx.sql("select 1 as a").await?;
df.write_csv(
    "test.csv",
    DataFrameWriteOptions::default().with_insert_operation(InsertOp::Append),
    None,
)
.await?;

• Append is already the default, just setting here to be explicit

Each time we run this, it will seemingly overwrite the CSV file, even though it is an append operation. If we replace Append with Overwrite or Replace it'll throw an error about it not being implemented.

I don't know what the target state should look like and what these APIs should do, so some research/analysis would be good, but I can say as it is the API does seem a bit confusing. I haven't tested write_parquet or write_json but it's worth reviewing those too.

[Dev79844]

take