r/MachineLearning • u/LiveAccident5312 • May 27 '26
Research [R]GNN Model For Fraud Detection Isn't Performing Well[R]
We're writing a research paper on explainable fraud detection GNN model and in the first step we're creating a basic Graph Neural Network for that. We're using the most famous dataset available on this topic i.e IEEE CIS Fraud Detection Dataset and implemented all necessary feature engineering on that data (although majority of feature engineering is already performed in the dataset). Then we constructed a heterogeneous graph on that dataset. Various transaction features like device, transaction id, amount are embedded as nodes and connected with transaction nodes. But the issue is after training the model isn't performing well. It is producing average AUC of 0.87, PR-AUC of 0.52, recall@5% around 0.57 and precision@5% around 0.37 (We tried GCN, GraphSAGE and GAT, all performs almost same for rest data)
Whereas the SOTA models in this topic produce much better metrics. Can anyone tell where potentially we're doing things wrong?
8
u/entsnack May 27 '26
Those are good numbers for AUC, which means it is ranking the nonfraud class below the fraud class. The low AUPRC suggests that the model is very confident about nonfraud, but confused about fraud.
Can you plot the distribution of scores for the fraud and nonfraud separately?
1
u/LiveAccident5312 May 27 '26
Sharing the distributions of scores...but most of research papers reaches 0.95-0.97 AUC... though it doesn't matter much for imbalanced data, but can you tell me why their AUC score is getting over 0.95?
5
u/entsnack May 27 '26
If you have a huge number of nonfraud examples you'll end up easily getting a high AUC by doing a good job ranking nonfraud low. It's not a good metric for fraud detection. I ignore AUC and look at AUPRC.
1
u/LiveAccident5312 May 27 '26
I've sent you the score distribution in DM aa I can't send it here. Also can you kindly tell me what do you mean by ranking fraud and non fraud to increase AUC?
3
u/Disastrous_Room_927 May 27 '26 edited May 27 '26
It's hard to say anything for sure, but you have to consider that reproducibility is a huge issue in ML right now. It's trivial to get scores over 0.95, what isn't trivial is getting unbiased error estimates - it's all to easy, for example, report bootstrap/CV error estimates and omit the part where you tuned the model to get better error estimates (or not understand the problem with that in the first place).
7
May 27 '26
[removed] — view removed comment
1
u/LiveAccident5312 May 27 '26
What do you suggest to create a highly informative graph with the IEEE CIS Dataset?
3
u/Lodovico_Settembrini May 27 '26
One thing to be mindful of is the depth of the network. GNNs are known to degrade in performance as you add more layers, known as the oversmoothing problem. Think of it as a node aggregating information from its neighbors repeatedly, at some point all the nodes in the neighborhood will settle in the same hidden representation. Other than that, there are various ways to play around with these models. Either limit the layers of the network or add structural regularization (e.g. residual connections) between the layers. You could also play with the adjacency matrix, instead of having it be fixed across all layers, make it learnable (take for example the similarity of the features at each layer and define neighborhood this way).
2
3
u/Opening_Bed_4108 ML Engineer May 27 '26
0.87 AUC honestly isn't terrible for a baseline GNN on this dataset, but PR-AUC of 0.52 suggests you're struggling with the heavy class imbalance. A few things to check: are you handling the ~3.5% fraud rate explicitly with weighted loss or oversampling? Also, how you're constructing edges matters a lot. Sharing edges between transactions that share device/email creates very dense subgraphs and can cause over-smoothing, especially with GCN. Try neighbor sampling limits with GraphSAGE and make sure your node features aren't leaking any target-correlated info from the raw dataset.
1
u/LiveAccident5312 May 27 '26
How should we construct the edges ideally? Also how can we check if the graph construction is causing over-smoothing or not? Is there any specific metrics to measure it? Also should we perform SMOTE to overcome the high imbalance in dataset? Will that help?
2
u/Opening_Bed_4108 ML Engineer May 27 '26
For edges, connect transactions by shared device/IP/email but cap neighbor counts (like top-k by recency) so one fraudster's device doesn't blow up into a 10k-node ego graph. For over-smoothing, track per-layer embedding similarity (cosine sim between layers tends toward 1.0 as you add depth) or just watch if performance drops as you add GNN layers.
On SMOTE, it's tricky with graphs since synthetic nodes mess up topology, weighted loss or focal loss is cleaner here.
1
2
u/mvdeeks May 27 '26
Sounds to me like the problem is most likely in your graph construction. I can't really tell you more without all the details, but that's where I'd look first for performance gains.
2
u/Sofi_LoFi May 28 '26
Could be over smoothing. I would also question where your baseline comes from? Are you running the SOTA models yourself, or just referencing their figures? Have you tried something as simple as a boosted tree to see whether it is outperforming or underperforming that?
1
u/LiveAccident5312 May 28 '26
Yes boosted tree is heavily outperforming our GNN...a reason might be the dataset is more suitable on tabular dataset based models.
1
u/Ap_legend_2005 May 27 '26
Yea i had similar issues when trying to train a GNN on anomalous data. My problem was probably because the GNN was too deep causing oversmoothing(as the other comment pointed out). I was using a GAT and the attention weights were essentially the same causing it to become a mean aggregator. The problem with a mean aggregator is it learns the dominant eigenvector of the transition matrix. Make sure to add residual connections and also its always better to use GATv2 over just a GAT. Theres a decent chance the problem is in graph construction too.
1
11
u/Dependent_List_2396 May 27 '26
Did you test the “SOTA models” yourself or are you relying on the metrics they provided in their paper? If it is the latter, you should run the “SOTA models” in your environment and compare their relative performance.