How the layered filter performed on a million synthetic test payments
These are committed results from this repository's benchmark scripts on synthetic data; they do not show performance on real banking traffic. 21 model variants across boosting, classical ML, deep learning (MLP, CNN, LSTM, Transformer, GAN) and a graph neural network, tested on 5.08 million IBM AMLworld payments and 1 million Feedzai BAF account applications. The best models are stacked into a cascade: each layer approves what it is sure is clean and passes the rest to a heavier model.
What reaches an analyst
Volumes are what each layer passes on during the 10 day test window. The red figure under each bar is how many true laundering payments are still in the stream at that point.
L0 runs at account opening on a different population (BAF), so it has its own test set and is not chained numerically into the payment funnel. L3 sequence models run in batch; the online API path is L1 → L2 → decision band → L4.
21 models, one split, one scoreboard
Every model trains on the same rows (all 2,298 training positives plus a fixed 10% of training negatives), picks its threshold on the validation period, and is scored once on the test period. PR-AUC is the headline because only 0.18% of test payments are laundering; ROC-AUC looks excellent for almost everything at that imbalance.
"+graph" means the 22 causal account features (fan-in/fan-out in 24h, pass-through ratio, minutes since money came in, reciprocity, new counterparties…), each computed only from payments before the one being scored. "Core" PR-AUC drops the Sep 11 to 18 tail, where the generator emits almost nothing but laundering. KNN and RBF-SVM were trained on a 40K subsample and scored on a reweighted test sample because they cannot score a million rows on this hardware; F1 is not reported for them.
Account history does the work, not the algorithm
XGBoost on raw payment fields scores 0.172 PR-AUC. The same model with causal graph features scores 0.715. Every strong model uses those features; the algorithm choice moves results by a few points.
Gradient boosting wins on this data
XGBoost, LightGBM, CatBoost and Random Forest are within 0.02 PR-AUC of each other. They are also the fastest to train (6 to 70 s on 2 CPUs) and the easiest to explain with SHAP.
Sequence models add a small, real gain
LSTM and Transformer over each sender's and receiver's last 12 events reach 0.66 alone. Blended at 10% weight into the tree ensemble they lift PR-AUC from 0.720 to 0.725 and F1 from 0.691 to 0.702.
The GNN loses under a strict time split
With embeddings built only from the past graph, the GNN reaches 0.516. It cannot represent accounts that first appear during the test period. Published GNN results on this dataset use a test graph that includes future edges.
GAN augmentation does not help
Adding 3× GAN generated positives to XGBoost gives 0.711, versus 0.715 without. Real positives are rare, but the synthetic ones add no new signal.
No ACH shortcut
ACH carries 87% of laundering. Removing payment format entirely costs XGBoost only 0.025 PR-AUC (0.715 → 0.690), so the model is learning behaviour, not the format label. Unsupervised Isolation Forest is near random (0.004): here laundering looks normal row by row and only stands out in context.
Rings caught, by laundering typology
Share of the 192 ground truth rings from the IBM Patterns file (test period) that have at least one payment blocked. Bipartite rings are the weak spot: each payment in them looks like an ordinary one to one transfer.
Stopping mule accounts at opening
Protocol from the BAF paper: train on months 0 to 4, set the threshold on month 5 at a 5% false positive rate, test on months 6 to 7. LightGBM has the highest recall at 5% FPR here, narrowly ahead of AdaBoost and XGBoost. These onboarding scores are a different task and dataset from the payment scores above and must not be compared with them.
Fairness fix: one threshold per age group
With a single threshold, applicants aged 50+ get a 12.8% false positive rate versus 3.9% for younger applicants (ratio 0.31). Separate thresholds set on validation bring the ratio to 0.88 (6.0% vs 5.3%), costing 2.5 points of recall (0.559 → 0.534).
What drives the score
Housing status, device OS, credit risk score, name and email similarity and time at current address are the five largest contributors to LightGBM's gain. Mule applicants tend to come from fresh addresses, recycled devices and emails that don't match the name.
The streaming engine matches the trained model
All 5.08 million HI-Small payments, in time order
Each payment goes through the live engine's past-only account state, the live L1 gatekeeper and the L2 average, exactly as the API scores it. Metrics are on the last 20% (1,015,669 payments, 1,797 laundering); the 60 to 80% weeks are used only to tune the context rules.
| Typology | Rings in test weeks | Caught at block level | Rate |
|---|---|---|---|
| BIPARTITE | 21 | 16 | 76% |
| CYCLE | 25 | 24 | 96% |
| FAN-IN | 20 | 19 | 95% |
| FAN-OUT | 21 | 19 | 90% |
| GATHER-SCATTER | 39 | 39 | 100% |
| RANDOM | 17 | 16 | 94% |
| SCATTER-GATHER | 23 | 21 | 91% |
| STACK | 26 | 25 | 96% |
Owner linking from the accounts file merges held payments into 2,790 cases instead of 4,804 by shared accounts alone.
The mule signature was too loose; ctx-1.3 fixes it
On the test weeks the ctx-1.2 signature fired on 3.10% of payments (precision 0.003). A stricter rule chosen on the validation weeks only (payer collected from 4+ accounts in 24 h, forwards at least 95% of the latest deposit within 2 minutes, has paid at most 10 accounts) fires on 0.05% (precision 0.071). The context layer alone now asks for verification on 0.12% of payments instead of 3.18%. IBM data has no messages, so on these payments the context layer adds little beyond the payment model; its value is on message-led scams, which this dataset cannot measure.
LI-Small: 6,924,049 payments the models never saw
A lower laundering intensity population. Live streaming PR-AUC 0.345; at verification or more the live path catches 71.8% of laundering at 1.38% of payments; held for an analyst 59.6% (precision 7.2%). 60 of 60 labelled rings in its test weeks get at least one flag. The ctx-1.3 signature, applied unchanged, fires on 0.044% of LI payments (precision 0.000).
Account opening gate on all five BAF variants
The live L0 model was trained on BAF Base months 0 to 4 only. Each variant shifts the population's bias; months 6 to 7 are scored with the gate's age group thresholds. Parity is the lower age group false alarm rate divided by the higher one (1.0 is equal).
| Population | ROC-AUC | Recall | False alarms | Parity, 50+ vs under 50 |
|---|---|---|---|---|
| Base (benchmark test) | 0.892 | 53.4% | 5.4% | 0.88 |
| Variant I | 0.882 | 53.5% | 5.4% | 0.88 |
| Variant II | 0.897 | 52.3% | 5.7% | 0.89 |
| Variant III | 0.870 | 52.9% | 5.7% | 0.89 |
| Variant IV | 0.870 | 52.9% | 5.7% | 0.89 |
| Variant V | 0.872 | 54.1% | 5.7% | 0.89 |
Control: a model trained on Variant III that may use its leaked features x1 and x2 reaches 0.954 ROC-AUC but its false alarm parity falls to 0.18 (people 50+ falsely flagged 5.4x as often); without them 0.878 at parity 0.95. L0 does not read x1 or x2. Variant IV's test months are the same applications as Variant III's, reordered, so its row repeats Variant III.
Scaling the account state
Streaming all 5,078,345 payments in one Python process: the prototype's object state holds 2,136 MB; the compact layout (integer ids, typed arrays, counters instead of sets) holds 1,372 MB with identical features, and 1,361 MB with a fixed size Bloom pair index. Throughput 42,679 vs 41,459 vs 31,673 payments per second per core.
Bloom vs exact on the test weeks: 0.000% of payments see any feature change; live decisions agree on 100.000%.
Redis state with stateless workers partitioned by payer: 934 payments per second with one worker, 1,315 with 2 on this 2 core machine (Redis and workers share the cores); publish to decision latency p50 289.4 ms, p99 970.3 ms at 789 per second. Redis holds about 1,406 bytes per account.
Training on the data that was left unused
LI-Small (6.9 million payments) was added to the payment models' training weeks. Features come from the serving code itself, so training and the live API see identical numbers. The swap rule was fixed before reading test numbers: validation PR-AUC on HI-Small no more than 0.005 lower, and at least 0.01 higher on LI-Small. Caught counts compare models at the same alert volume as the previous live models.
| Model | Validation PR-AUC HI / LI | Test PR-AUC HI | Test PR-AUC LI | Caught at verify volume HI / LI | Caught at hold volume HI / LI |
|---|---|---|---|---|---|
| Previous live models (HI-Small only) | 0.628 / 0.283 | 0.716 | 0.336 | 1585 / 645 | 1461 / 529 |
| Retrained on HI-Small (control) | 0.623 / 0.277 | 0.707 | 0.328 | 1598 / 660 | 1461 / 542 |
| Retrained on HI-Small + LI-Small (now live) | 0.633 / 0.298 | 0.711 | 0.345 | 1595 / 664 | 1432 / 533 |
Adding LI-Small beats the fair control (a retrain on HI-Small alone) on test PR-AUC for both datasets, and catches more at the verify level on both. Against the previous live models it trades a small loss on HI-Small at the hold level for gains on LI-Small.
Account opening gate with the five BAF variants added to training: mean ROC-AUC over the six populations 0.880 (previous) vs 0.888 (with variants) on months 6 to 7; adopted under a rule fixed in advance on month 5 (validation 0.890 vs 0.894).
Person and session signals
Mouse dynamics on the real Balabit data (10 users, labelled impostor sessions): median user ROC-AUC 0.89 after 10 mouse actions and 0.94 after 50.
Authored scenario suite through the live API: 280 of 280 scam payments stopped before money moved, 1 of 200 genuine payments with a false alarm, 0 of 200 wrongly held. These scenarios test mechanics, not real world rates.
Code: ml/full_dataset_eval.py, ml/tune_context.py, ml/baf_variant_eval.py, ml/scale_bench.py, ml/scale_out_bench.py, ml/retrain_extended.py, ml/retrain_l0_variants.py, ml/balabit_eval.py, ml/scenario_suite.py, service/compact_state.py, service/scale/. IBM and BAF data are synthetic; Balabit is real mouse data.
Live API vs offline benchmark
- The live payment path loads three models: the 30 tree XGBoost gatekeeper (L1), then an XGBoost and a LightGBM model whose scores are averaged (L2). Random Forest and LSTM/Transformer appear only in the offline L3 blend above.
- The offline L3 blend reaches about 0.725 PR-AUC. The live path does not reproduce that blend, so its scores are not the same as the blend's.
- The live block and hold thresholds were tuned on the offline blend score, not on the live L2 average.
docs/project-audit.mdmeasures the effect and how to retune. - Reasons shown by the API are XGBoost TreeSHAP contributions (
pred_contribs) from the L2 XGBoost model only; LightGBM's half of the score has no per feature explanation.
Why gradient boosted trees
The payment features are 31 numeric and low cardinality columns about each account's past behaviour, with missing values wherever an account has no history yet, and laundering is under 0.2% of payments. Boosted trees learn the interactions that matter here (money in, then straight out to many new payees) without hand built crosses, handle missing values natively, train in seconds on two CPUs, score a payment in well under a millisecond, and give per feature reasons. The committed benchmark shows the feature engineering matters far more than the choice among the leading tree models: XGBoost goes from 0.172 PR-AUC on raw payment fields to 0.715 with account history features, while XGBoost, Random Forest, CatBoost and LightGBM all land between 0.699 and 0.715.
PR-AUC (average precision) summarises precision across all recall levels. With 0.18% positives, a model that flags nothing is 99.8% "accurate", so accuracy says nothing; ROC-AUC is also flattering because it rewards ranking millions of easy negatives. PR-AUC drops fast when alerts fill with false alarms, which is the cost analysts feel.
- Both datasets are synthetic (IBM's agent based simulator, Feedzai's privacy preserving generator). Real UPI/IMPS data will shift thresholds; the pipeline is built to be retrained, not to transfer as is.
- The lure and pressure stages of a scam (calls, chats) are not in either dataset. The message model and context layer are built and tested on synthetic message templates; real reported messages are needed before trusting their rates.
- Ring recall varied between 82% and 89% across two retrains with different category encodings. Treat the range, not a point, as the result.
- The prototype engine keeps account state in memory.
service/scale/moves it to Redis with stateless workers; a deployment still needs a replay of history at startup, because without history recall collapses.
Code: ml/aml_features.py, ml/bench_tabular.py, ml/bench_deep.py, ml/bench_gnn.py, ml/bench_baf.py, ml/cascade.py, ml/l0_onboarding.py, service/rusetect_engine.py, ml/compare_models.py. Hardware: 2 CPUs, 7 GB RAM.