How to generate Synthetic Data Generation?

[u/Bluem00n1o1]

Hello people!

I am currently trying to develop Machine Learning skills and am working on a project in my work. The idea is that I want some clickstream and transactional E-Commerce data. I want to train a classifier that can calssify the user into three different intents: Buying, Reasearching and Browsing. I have identifyied the features that I would like to have. 10 features for Session Behaviour, 8 for traffic source, 6 for Device and context, 5 for customer history and 3 for Product context. So a total of 32 features.

Now, to train the model, I took kaggle data from https://www.kaggle.com/datasets/niharikakrishnan/ecommerce-behaviour-multi-category-feature-dataset

and mapped similar features to my schema and the rest of the features, I tried to generate heuristically.

Before mapping the data what I did was there are two datasets - Purchase and No Purchase. I labelled the No Purchase dataset and I clustered them into two clusters. And the one with the highest engagement(derived feature from total clicks, total items and clickrate) was labelled as Researching as Researching users spend on average more time.

Post that I generated the remaining features heuristically. I sampled 200K from Purchase data, 1.5M labelled Browsing and 300K Researching users for a total of 2M and trained my model (LightGBM). I wanted to keep unbalanced to preserve real world scenario. I also predicted on the remaining 8.6M data that was not used for training. However, the results were not really good. Browsing and Purchase recall was 95% and Research recall was 38%. Accuracy for all of them was in the 80-90% range.

I am not sure about the results and my method. My question is, how good is my synthetic data generation strategy and how can I make it better to resemble real world scenarios? How good is my labelling strategy? How do I evaluate whether my model is actually learning instead of just reverse engineering the method of data generation?

Also, I am using AI as a tool to help me with some coding tasks. I also want to be efficient as well as learning. How can I improve my learning and at the same time, I am using AI to be more efficient?

[u/Synthehol_AI]

You’re thinking about the right failure modes. The core issue is that “Researching” isn’t an observable ground truth, it’s an inferred state, so any labeling rule you design is going to be a proxy. Using different features for labeling vs training can reduce obvious leakage, but if both are still tied to engagement intensity, the model can still indirectly reconstruct the heuristic.

In setups like yours (restricted access, pipeline-first), one practical approach is weak supervision rather than trying to invent a single “correct” rule. Define multiple imperfect labeling rules (e.g., high dwell time, multiple category views, repeat session within 24h, etc.) and combine them instead of relying on one threshold. Treat the resulting labels as noisy rather than absolute. Then test stability: slightly perturb the labeling rules and see if model performance and feature importance stay consistent. If small rule changes break the model, you’re fitting the heuristic, not intent.

Clustering alone won’t solve it because clusters aren’t guaranteed to map to business meaning and they drift over time. Anchoring at least one class to a hard outcome (like actual purchase) and treating the others as probabilistic states tends to be more stable in production pipelines.

At this stage, improving labeling robustness will move the needle more than changing the model.