Data handling
What touches your endpoint, where the results live, and for how long. Written for the reviewer who has to approve issuing a staging key.
Where tests run
Tests run on Allymet's own infrastructure: a dedicated DigitalOcean instance in the US East region, managed by the assessor. Test engines (Garak, PyRIT, PromptFoo, DeepEval, pip-audit, picklescan) run locally on that instance. No third party operates the test runner.
What your endpoint receives
Your endpoint receives adversarial probes through its normal API, authenticated with the staging key you issue. Probes are the attack inputs; responses are what your endpoint returns. Both are captured to the test instance for analysis and for inclusion, where relevant, in the report's findings excerpts.
Your upstream model provider (for example OpenAI, Anthropic, or your own hosted model) sees the probes as ordinary API traffic from your application. Allymet has no relationship with your provider and does not route traffic through any intermediary.
Staging keys
Keys are requested by email after intake, never through the website form. A key is held in the test instance's environment for the duration of the test window and is removed when the window closes. Keys are never written to the report, the repository, or any log retained after the engagement. You should revoke the key on your side once the report is delivered; the report states the test window dates so you can confirm no use outside it.
What leaves the test instance
- PromptFoo Cloud
- Receives generation metadata only (test case counts, plugin names, timing). No client prompts, responses, or system prompt content are sent in the current configuration.
- The report
- Delivered to you as a PDF. Contains excerpts of failing output, which may include fragments of your system prompt or responses where that is the evidence of the finding.
- Nothing else
- No findings, transcripts, or endpoint details are shared with any other party, used in Allymet published research, or used to train any model.
Retention
Test artifacts (raw probe and response logs, engine output) are retained for a minimum of 30 days and a maximum of 90 days after report delivery, to support retest and reviewer questions, then deleted. The report itself is retained as an assessor record.
A formal retention policy and a data processing agreement are being finalized and will be published on this page. Until then, the terms above are stated in the engagement agreement.
Questions
Reviewers with questions about any of this can write directly to the assessor at abhishek@allymetadvisory.com.