Preprocessor fit_transform adapts mutable object (train_data)

[ZlaTanskY]

Copying the data to a local data object is done in the fit method, but not in the transform method.
https://github.com/PythonPredictions/cobra/blob/master/cobra/preprocessing/preprocessor.py#L235

[sandervh14]

I'm copying the problem and its solution here from within an e-mail trace, for later reference.

To recap the minimal code to reproduce the problem:

from sklearn import datasets
r"""# For debugging Cobra:   
import sys   
LOCAL_COBRA_PATH =   r"C:\Users\sander.vandenhautte\PycharmProjects\cobra"   
sys.path.insert(0, LOCAL_COBRA_PATH)   
%load_ext autoreload   
%autoreload 2   
"""       
from cobra.preprocessing import PreProcessor       
data =   datasets.load_iris(as_frame=True)["data"].reset_index()   
data.head()       

id_col = "index"   
target_col = "petal width (cm)"   
model_type = "regression"       
col_dtypes = data.dtypes       
discrete_vars = [col for col in col_dtypes[col_dtypes==object].index.tolist() if col not in [id_col, target_col]]
print(discrete_vars)       
continuous_vars = list(set(data.columns) - set(discrete_vars) - set([id_col, target_col]))   
print(continuous_vars)       

weights = [0,.8,50]   
preprocessed = {}   
for w in weights:       
    print(w)      
    print("START OF ITERATION-----------------------------")       
    print("data", data)       
    print("preprocessed", preprocessed)       
    preprocessor = PreProcessor.from_params(model_type=model_type, n_bins=10, weight= w)
    result = preprocessor.fit_transform(data, #   <===================================== SOLUTION TO BE PUT HERE, SEE BELOW             
                                                             continuous_vars=continuous_vars,           
                                                             discrete_vars=discrete_vars,             
                                                             target_column_name=target_col)       
    preprocessed[w] = result       
    print("AFTER   ITERATION-----------------------------")       
    print("data", data)       
    print("result", result)       
    print("preprocessed", preprocessed)
```


As you mentioned, the variable preprocessed should contain different
target encoded columns for the three configured parameters <b>w</b> in weights.

With my added print statements above, you’ll see that result shows that
the target encoding in fact HAS calculated the target encodings differently in
every iteration.

However, at every iteration step, result is written not only to
preprocessed[w], but also to preprocessed[all_former_w_values].

The cause is that Cobra, when it writes “result” to preprocessed[w],
does not just return a result dataframe, but actually it is the modified
dataframe it got at the input (the variable data) with added columns.

See it as if Cobra is not writing away a value (a dataframe) to a
certain location in memory, but just refers to the original dataframe data, and
that dataframe has had some extra columns by now.

Then, in the next iteration, Cobra adds new columns to the dataframe
presented in the variable data, which python refers to as the dataframe in
memory that we have used before.

When Cobra adds the preprocessed columns in that dataframe, it is
essentially overwriting the dataframe.

And since this dataframe is referenced by both preprocessed[0] and
preprocessed[0.8] also on the writing of result (=the modified dataframe
again), both items in preprocessed have been overwritten after the second
iteration.

**The solution is to call preprocessor.fit_transform(data.copy()).**
Then, Cobra is forced to work on a new copy of the dataframe everytime,
and it cannot overwrite the same dataframe in every new iteration.

This is python stuff you don’t normally encounter everyday.

**The better-for-the-long-term fix would be to perform the copy() within
fit_transform() – at the start of that function – of the preprocessor.**

You will notice that many methods in Cobra already perform such a copy()
at the start to avoid such nasty problems, but this
preprocessor.fit_transform() does not.<o:p></o:p></span></p>

[sandervh14]

This issue seems to not auto-close now the pull request #140 successfully merged. I'm closing it manually.