Peter Parker

Project 15: Share the Data, Not the People

Read the code Run the attacks on your own tables Every number below is read from one file the repository's report command writes.

Attacked, measured, and reproducible from the repository

Synthetic data is sold as safe to share. This project attacked it to find out what it protects, and what it costs.

Health records, census answers and card transactions are exactly the data that would unlock better analytics and models, and exactly the data privacy law keeps locked up. The popular answer is synthetic data: a generated table with the same patterns and none of the people. Whether it really contains none of the people is usually asserted, not measured. Here, synthetic versions of three public datasets of real people were made seven ways, with and without differential privacy, then attacked with the three tests European regulators use to judge whether data is anonymous, and scored on the question a data scientist cares about: does a model trained on it still work on real data?

- synthetic tables attacked three datasets, seven synthesisers, budgets from epsilon 0.5 to no privacy, three seeds each
- configuration where an attack detected a risk -
- configurations close enough to the real data to recommend within 0.02 AUC of a model trained on the real data, on every task and seed
- false alarms from the check most tools show distance to closest record failed tables no attack could break, and passed the one that leaked

The attacks stay inside each dataset: no outside data is ever joined to a record and no record is ever named. Every rate is measured against a control group of real people the synthesiser never saw, so a table that is merely useful about everyone is not counted as a leak. Every number carries a 95 percent confidence interval. See how it's built, in four diagrams.

Why this question matters

Organisations sit on data that could improve care, catch fraud and inform policy, and mostly cannot share it, even internally. A synthetic copy is the shortcut everyone reaches for, and vendors sell it with a privacy score attached. The trouble is that a privacy score computed by the tool that made the data is not evidence. The test that counts is the one an adversary would run.

For a privacy officer

Whether a synthetic table passes the three tests of anonymity regulators name, measured the way an attacker would measure them, with an interval, rather than a vendor's similarity score.

For a data scientist

How much a model loses when it is trained on the synthetic table instead of the real one, at each privacy budget, and which synthesiser comes closest.

For whoever signs off

A rule, fixed before any table was scored, for where on the privacy-usefulness curve to sit, and a release gate that checks the answer independently.

How it was tested

Three public datasets of real people: California census responses (ACS PUMS 2018), card transactions from a public fraud competition (IEEE-CIS), and New York hospital discharges (SPARCS 2024). Each one goes through the same steps.

  1. 1Split three ways, once. A training set the synthesisers see, a control set of real people they never see, and a test set for usefulness.
  2. 2Synthesise seven ways. Gaussian copula, CTGAN, TVAE and the diffusion model TabDDPM with no privacy guarantee; AIM, MST and PATE-CTGAN with differential privacy at epsilon 0.5 to 8. Three seeds each, every epsilon from the library's own accountant.
  3. 3Attack every table. Singling out, linkability and inference, the three criteria of the EU's anonymisation guidance, implemented from the paper that turned them into measurements, plus shadow-model membership inference.
  4. 4Subtract what the table tells about anyone. Each attack runs identically on the control set. Only the excess success on the training records is risk.
  5. 5Train on synthetic, test on real. A LightGBM model trained on each table, scored on real records it never saw, against the same model trained on the real data.
  6. 6Apply a rule written in advance. The smallest epsilon within 0.02 AUC of the real-data model on every task and seed, with no attack detecting a risk. Then check it through a release gate.

Why the control set is the whole method. A good synthetic table predicts a person's income from their job and age, because that is what it is for. Measured alone, that looks like an attack succeeding. Run the same attack on people the synthesiser never saw and it succeeds just as often. Only the difference between the two is something the table learned about the individuals in it.

The trade-off, dataset by dataset

Each point is one synthesiser at one privacy budget, averaged over its three seeds. Height is how much worse a model trained on that synthetic table does than one trained on the real data, as AUC on real test records: zero is as good as real. The shaded strip is within the 0.02 the rule allows. Smaller epsilon is stronger privacy; the column on the right has none. Point at a mark to read it.

Why the table that looks most like the data is not the most useful one

What leaked

A risk counts only when its interval excludes zero at a level corrected for running five attacks on every configuration, with the three seeds pooled. These are the configurations that crossed that line.

A leak is named precisely, as a property of those settings on this data. It is not a claim that AIM is unsafe in general, and the tables with no detected risk are not proven safe: they are tables these attacks, at this scale, could not break.

Do the attacks catch what they should?

A result at control level means something only if the same attacks catch a table that does leak. So four methods people really use as "synthetic" or "anonymised" data, each expected to leak, went through exactly the same pipeline: the same splits, targets, control, attacks and seeds. Their settings and expected outcomes were written into the plan before any of them ran. Each mark is one method's risk on one attack, with its 95 percent interval; the grey tick is the largest risk any of the seven synthesisers showed on that attack.

The check most tools show got it backwards

Distance to closest record is the privacy check in front of most synthetic-data users: is a synthetic record closer to the training data than to people the synthesiser never saw? A share above one half is read as a warning. Each mark below is one configuration's share, averaged over its seeds. The grey band is what real records the synthesiser never saw score on the same yardstick.

Why distance cannot see this kind of leak

The inference that succeeds on AIM needs no synthetic record near any target. It needs the table to keep the pattern between the other attributes and income a little more faithfully for the training records than for anyone else. Nearest-neighbour distance does not see a pattern. In the other direction, a synthesiser that reproduces its training split's population places its records slightly nearer that split than an independent sample would, which differential privacy permits, because it bounds what any one person changes, not how close the table sits to the population. The write-up, with every number, is in docs/rejected.md.

As good as the real thing? The release gate says no

Project 03's release gate decides whether a candidate is non-inferior to a baseline, the way it would for a model change. Here the candidate is a model trained on synthetic data and the baseline the same model trained on the real data, with a margin of 2 AUC points. No configuration met the rule, so the gate was given the nearest ones: per dataset, the closest of all seven synthesisers and the closest with a privacy guarantee. Each bar is the gap on the worst task, averaged over three seeds, with the range of the gate's 95 percent intervals across them.

Membership inference: at chance, and why that is not proof

The fourth attack asks the most direct question: was this person in the training data? It follows the Groundhog attack of Stadler et al.: refit the synthesiser dozens of times with and without a target, and train a classifier to tell the difference from the tables alone. That needs a synthesiser fit per shadow table, so it runs at about 1,000 rows with six targets per dataset, the scale of its paper. Each mark is one configuration's accuracy, with its exact 95 percent interval; one half is guessing.

Checked against the reference libraries

Every attack is implemented from its paper, and then run beside the authors' own library on the same tables. A disagreement is resolved and written down before a number ships.

What to take from it

These are findings on three datasets at 20,000 rows, with these libraries and versions, not laws of nature. They are also the questions worth asking of any synthetic data before it is shared.

Run the attacks on your own tables

The attacks are a package that needs no part of the rest of the project. Hand it the real records a synthesiser was trained on, as many real records it never saw, and the synthetic table. It codes raw columns from the held-out records alone and returns every risk with its interval. 

from facsimile.attacks.evaluate import evaluate

result = evaluate(targets, control, synthetic, secret="income")
print(result.markdown())   # singling out, linkability, inference, with 95% intervals
result.detected            # the attacks whose risk interval excludes zero

Or from files: facsimile evaluate targets.csv control.csv synthetic.csv --secret income. What a result does and does not mean is in docs/attack-package.md.

Every configuration

The full table behind the charts, one row per synthesiser and budget, three seeds pooled. Risks are percentages with 95 percent intervals; a bold red risk is a detected one. Usefulness is the gap on the worst task in AUC. The repository's docs/results.md holds the same rows with every task.

Single out 1 and n: singling out on one attribute and on several. Link: linkability at 1 and 10 neighbours. Each risk is the excess success on training records over control records, as a share of what was left to gain, pooled over three seeds, with its 95% interval. Member is membership accuracy at about 1,000 rows (chance 0.5), run at epsilon 1, 8 and no privacy. Usefulness is the AUC gap on the worst task. Fidelity is two-way marginal error. DCR is the share of synthetic records closer to a training record than to a control. Fit is the median fit time over seeds, on the machine named.

What this page does not show