Handling _RARE_ values in generated data

[PrashanthReddy1823]

Hello team,
I'm working on a use case where i need to generate synthetic data for couple of tables which are relation. After training, while generating the data most of the columns are with RARE values. How can i disable generating the RARE values.
I'm happy to share the code if needed. Thanks in advance!!

[aksverman]

Hi Prashanth,
I have started using SDK recently. I found same issue with my generator too. To avoid any masking, modify table configuration as below:
"tabular_model_configuration": { "valueProtection": False, # Disable value protection "differentialPrivacy": None # Disable differential privacy }
Please let me know how are you feeding the sample input to generator? Are you using MostlyAI connectors?

[mplatzer]

Hello team,
I'm working on a use case where i need to generate synthetic data for couple of tables which are relation. After training, while generating the data most of the columns are with RARE values. How can i disable generating the RARE values.
I'm happy to share the code if needed. Thanks in advance!!

Hi. You can either set rare_category_replacement_method to SAMPLE, or disable protecting rare categorical values all together via value_protection=False. See also https://mostly-ai.github.io/mostlyai/api_domain/#mostlyai.sdk.domain.ModelConfiguration

But oftentimes the issue is rather that you have too little training data, to learn statistically valid relations between categorical variables. Can you share more about the columns, where you are facing this issue? Maybe a different encoding type or even a LANGUAGE model might be more appropriate for your use case.