Filter/Data Quality Component

[pritamdodeja]

If the feature is related to a specific library below, please raise an issue in
the respective repo directly:

TensorFlow Data Validation Repo

TensorFlow Model Analysis Repo

TensorFlow Transform Repo

TensorFlow Serving Repo

System information

• TFX Version (you are using): 1.15.1

• Environment in which you plan to use the feature (e.g., Local
  (Linux/MacOS/Windows), Interactive Notebook, Google Cloud, etc..):
  Local, GCP

• Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.
Whenever we have a tabular dataset, we compute statistics, and then a schema, and then we can validate examples. We can fine-tune the schema in order to be integrate domain knowledge into the validation process. Why cannot we filter "bad" data in the pipeline itself?

Will this change the current API? How?
It would introduce a new component that would allow filtering based on schema. The inputs into Transform and some of the other components that use examples could potentially change depending on user's preference.

Who will benefit with this feature?
Users who want to do the entire data processing under the management of ML Metadata.

Do you have a workaround or are completely blocked by this? :
A variety of workarounds exist, manually clean up the original data, write scripts in beam to filter out bad data, put it in a database and use SQL to do this.

Name of your Organization (Optional)
GCP Partner (Intuitive.cloud)

Any Other info.
Number of training steps etc., would be affected by filtering data. However, that is also something that, in my opinion, should be more dynamic than it currently is in tfx.

[janasangeetha]

Hi @pritamdodeja
Thank you for showing interest and raising an issue. Currently, we are not accepting new implementation requests or feature additions. If you would like to contribute , please follow our CONTRIBUTION GUIDELINES and submit a pull request.
Thank you again for your interest, and we look forward to seeing your contributions!

[github-actions]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[pritamdodeja]

I, along with a few members of my team, have given this some thought, and there are two high level approaches we can think of: define what good data looks like via a schema modification (domain min maxes erc), and use beam to yield only those examples that conform to that definition, or have the user define a user defined function that gets passed to beam.Filter.

The former approach is less flexible (e.g. only contiguous filtering), but easier for the user familiar with tfdv, and the latter more powerful because it can use arbitrary operations/libraries to filter data (similar to preprocessing_fn in tft in some sense).

We would like to implement the latter approach, and would like some guidance in terms of alignment and examples of design documentation etc we should put together in order for it to fit into the tfx ecosystem and design philosophy. For example, what library should this belong to? Tensorflow data validation makes the most sense I imagine. We would likely have a similar API as tensorflow transform for this, as it produces examples, but no graph. It would have statistics etc., as such, it really feels like it's between tfdv and tft.

In any case, if the team can provide the guidance, we can do the implementation to fit into the tfx framework and underlying libraries. We already have both approaches working directly as custom components in tfx, we need to industrialize these in terms of design documentation, tests, and integration into underlying libraries/design philosophy.

Please advise if interested in collaborating on this. Thank you!

Regards,

Pritam

[bharatjetti]

Hi @ALL Thank you so much for showing interest! We truly appreciate your enthusiasm and support.
Please feel free to fork the repository and submit pull requests if you have any suggestions for enhancements, problem fixes, or ideas. We’d be more than happy to review and collaborate! If you would like to contribute , please follow our CONTRIBUTION GUIDELINES here and submit a pull request.
Thank you again for your interest, and we look forward to seeing your contributions!