# Office Metadata Regression Pack

Synthetic Word, Excel and PowerPoint files (.docx, .xlsx, .pptx) with hidden data planted in known
places, and a catalogue (`manifest.json`) that says, for each file, what a cleaning tool has to take
out and what it has to leave alone. It is the test corpus ScrubDrop's own engine is checked against
on every change, arranged for people who build or test document-cleaning, DLP or e-discovery tools.

## What is in it

- `files/`: every file of the corpus. This repository keeps only three of them, the samples in
  `samples/`; the whole pack is sent on request (see below).
- `manifest.json`: one entry per file:
  - `traces`: the kinds of hidden data planted in it (document properties, comments, tracked
    changes, editing-session ids, hidden sheets, rows, columns, slides and text, pivot caches, links to
    other files, defined names, speaker notes, objects outside the slide, embedded objects, custom XML,
    image metadata, parts Office does not need), using the ids in `categories`;
  - `mustNotRemain`: strings, each with its kind, that must not appear in any part of the package
    (decompressed) once that kind has been removed;
  - `mayRemain`: visible content that must still be there afterwards;
  - `pointedOutOnly`: kinds a tool should report but not remove (white or tiny text, cells outside
    the print area), because whether they matter is the reader's call;
  - `listedByScrubDrop`: what ScrubDrop's inspection lists in the file, for reference only.

## How to use it

Run your tool on a file with every kind it supports switched on and **tracked changes accepted**
(`options.trackedChanges` in the catalogue: inserted text stays and is listed under `mayRemain`, deleted
text goes and is listed under `mustNotRemain`; rejecting changes reverses both). Then unzip the result, and search every
part, decompressed, for the `mustNotRemain` strings of the kinds you removed. None should be found;
every `mayRemain` string should be. Then open the result in Office: a file that does not open is a
failure even if nothing was left behind.

## Synthetic only

Every file is written by a script (`tests/fixtures/make-corpus.mjs` in ScrubDrop's repository). No real
documents were used and none are included. Names, companies and figures are placeholders made up for
the purpose.

## Licence

Provisional: for evaluating and testing software only, not for redistribution. Terms for the full pack
are given when it is requested.

## Requesting the full pack

Write to hello@tekiba.app, or use the "Request full pack" button on https://scrubdrop.tekiba.app/datasets/.