A new arXiv preprint posted on 1 September argues that the machine learning community has quietly abandoned rigorous privacy standards for synthetic data, replacing evidence-based claims with appearance-based assumptions that nobody tests S¹. The paper, "Position: Privacy Is a Claim, Not a Property of Synthetic Data," is not peer-reviewed, but its central observation is one the field has been circling for months S¹. What it describes is a cultural drift that has gone unaudited, and the people most likely to pay for it are the ones whose records are already hardest to hide.

My read: This is the sharpest framing I have seen of a problem that has been building for a while. The authors are not saying synthetic data is broken. They are saying the community has stopped asking what "private" even means before applying the label. That distinction matters because the fix is not a new algorithm. It is a change in what reviewers accept as proof.

The quiet shift nobody audited

The authors' core argument is about language and standards, not mathematics. They say that when synthetic data is deployed in contexts where privacy matters, the field has moved away from quantifying leftover inference risk under clearly stated assumptions, and toward treating privacy as something that automatically follows from how the data was produced S¹. In plain terms: researchers used to say "we generated this data with method X, under threat model Y, and here is the residual risk." Now they say "this data is synthetic, therefore it is private," and move on.

The authors argue this is not a misunderstanding of established privacy principles but an implicit change in what the community accepts as sufficient evidence S¹. Standards slipped without anyone voting on the change.

What the empirical review found

Drawing on an analysis of recent publications across major machine learning venues, the authors show that synthetic data routinely appears in privacy-sensitive applications with no spelled-out adversary model, no quantified inference risk, and no privacy assertion that could actually be tested or disproven S¹. When privacy assurance shows up at all, it tends to be assumed rather than demonstrated, and it does not apply evenly: records that are unusual or belong to minority groups carry disproportionately higher risk of exposure S¹.

The paper does not name which venues were analysed, the sample size, or the methodology. It is a position piece, not a systematic review. But the pattern it describes aligns with prior work. A January 2026 paper by Georgi Ganev and Emiliano De Cristofaro examined anonymity claims in synthetic data generation from a model-centric attack perspective and found similar gaps P². Earlier work published at IEEE S&P 2025 demonstrated privacy attacks against datasets marketed as "truly anonymous," showing that similarity-based privacy metrics are inadequate P⁴.

The tools making this urgent

Two major open-source projects show how fast synthetic data is moving into production. Meta's synthetic-data-kit, released on GitHub in March 2025 with 1,626 stars and an MIT licence, is a tool for generating synthetic datasets using large language models P⁵. OpenAI's privacy-filter, created in April 2026 with 2,673 stars under Apache 2.0, is a bidirectional token-classification tool for filtering private information from text P³.

Both tools address real problems. Neither one, on its own, answers the question the new preprint raises: what threat model are you protected against, and can someone test that claim?

A field that cannot verify its core claims is in no position to assume privacy by default.

What to do about it

Consider a regional health authority that wants to share patient data with a research partner. They generate a synthetic dataset using an off-the-shelf tool and send it off, believing the synthetic nature of the data makes re-identification impossible. The preprint's argument is that this belief is exactly what needs to be questioned. The dataset may well be safe, but "synthetic" is not proof. The authority needs to ask: what attack are we protected against? What is the residual inference risk? Can a motivated adversary with auxiliary knowledge reconstruct any individual record?

The authors recommend that machine learning venues adopt norms requiring any privacy-related assertion to be narrowly defined, subject to empirical testing, and open to challenge by reviewers S¹. For practitioners, the practical version is simpler: before sharing synthetic data, write down the threat model, state what the data does and does not protect against, and make that statement something a reviewer can challenge.

This week, if your team uses synthetic data in any privacy-sensitive context, check whether the documentation includes a named threat model. If it does not, that is the gap the paper is describing.

What we don't know yet

The preprint does not specify which ML venues were analysed, the sample size, or the exact methodology of its empirical review S¹. All substantive claims come from a single, unpeer-reviewed source. The recommendations are normative, not empirically proven interventions. We do not know whether venue-level adoption of the proposed norms would change reviewer behaviour or whether the appearance-based pattern the authors describe is as widespread as they suggest.

The next signal: the paper's peer-review outcome, if it is submitted to a venue like NeurIPS or ICLR. We will check whether the empirical analysis survives review and whether the recommendations gain traction among programme committees. If this kind of audit matters to your work, subscribe for the next one.

Sources: S1 — Position: Privacy Is a Claim, Not a Property of Synthetic Data · P2 — Rethinking Anonymity Claims in Synthetic Data Generation: A Model-Cent · P3 — openai/privacy-filter · P4 — The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks ag · P5 — meta-llama/synthetic-data-kit

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.

Open-source synthetic data and privacy tools on GitHub