The open-source ai-privacy-toolkit is a toolkit for tools and techniques related to the privacy and compliance of AI models. We recently released version 0.2.0 of the toolkit containing new functionality around privacy risk assessment of synthetic datasets.
In recent years, attacks have been developed that can infer sensitive information from trained ML models. This includes membership inference attacks, model inversion attacks, and attribute inference attacks. This has led to the conclusion that machine learning models trained on personal data should, in some cases, be considered personal information themselves, resulting in a multitude of solutions for creating privacy-preserving models.
Most work related to protecting the privacy of ML training data, including training models with differential privacy, typically requires making changes in the learning algorithms themselves, as they incorporate perturbations into the model training process. This leads to a plurality of solutions and libraries that are extremely difficult to adopt in organizations that employ many different ML models. Moreover, these solutions are not suitable for scenarios in which the learning process is carried out by a third party, independent of the organization that owns (and wants to anonymize) the private data.
A relatively nascent field has started to emerge involving generating synthetic datasets for training models, instead of the original, sensitive dataset. Many approaches exist for generating synthetic datasets that share some desired characteristics with the original data, ranging from completely rule-based systems, through statistical-query based methods, to generative machine learning models. However, simply generating a new dataset does not guarantee privacy, as the new dataset may still leak sensitive information about the original data, due to the fact that the generative models may memorize some of the training data[1]. To this end, a plurality of methods for differentially private synthetic data generation have emerged as an appealing solution that provides strong formal privacy guarantees, while producing a synthetic data set that “looks like” the real data from the perspective of an analyst. As it resides completely outside of the training process, this approach may be easier to employ in practice than methods that require replacing the training algorithm. The same dataset could even potentially be used for several downstream tasks or models, making this an even more appealing direction. It remains to be seen whether such synthetic datasets with privacy guarantees will retain enough useful information to enable model training.
Whether or not differential privacy was applied to the data generation process, it may be beneficial to be able to assess the privacy risk posed by the new dataset. Such dataset assessment methods typically fall into one of two categories: The first includes methods that assess the generated dataset itself, regardless of how it was created. This type of assessment is based on looking at similarities between the generated dataset and the original dataset, often comparing that to the similarities between the generated dataset and another, unrelated dataset (taken from the same distribution). The more similar the generated dataset to the original dataset, the higher the privacy risk.
The second category of methods assess the data generation model or algorithm itself, trying to determine the probability of it generating samples that are "too similar" to the original dataset. In this type of method, the generation algorithm will usually be applied multiple times on different raw datasets and its resulting output analyzed in each case. This type of method can be used to compare between different data generation techniques, or between different privacy parameters applied to the same technique (for example different values of when differential privacy is applied).
The latest release of the ai-privacy-toolkit includes a new module for dataset risk assessment, which includes two assessment methods from the first category (assessing the dataset directly). These are based on adaptations of a few recent papers[2],[3],[4].
The main interface is called DatasetAttack, and its assess_privacy() main method assumes the availability of the original training data (members), holdout data (non-members) and the generated synthetic data at the time of the privacy evaluation, and returns a DatasetAttackScore. The DatasetAttackScore always contains a risk_score, and may optionally also include individual record level attacks as a DatasetAttackResult. Concrete assessment methods, which run the assessment on a per-record level, or on the whole dataset, are also provided. DatasetAssessmentManager provides convenience methods to run multiple attacks and persist the result reports. An example of using this class can be found in the following screenshot:
These methods may also be used to assess the privacy leakage of anonymized datasets, as long as the new data shares the same domain as the original data, as is the case for example when employing the anonymization module of the ai-privacy-toolkit
Abigail Goldsteen, IBM
[1] https://doi.org/10.1145/3372297.3417238
[2] https://doi.org/10.1145/3372297.3417238
[3] https://www.vldb.org/pvldb/vol11/p1071-park.pdf
[4] https://arxiv.org/abs/2104.00635