Ghost Trace, Part 1: Can AI Agents Trace Threat Infrastructure Without Making Things Up?

Disclosure. I used AI to build this study's software, run its experiments and draft this write-up. A Claude model did that work, and the agents under test are also Claude models. Details are in How AI was used in this study at the end.

Why this matters

Infrastructure attribution is pivoting work. You start from one malicious domain. You pull its WHOIS record, check where it resolved, look at the TLS certificate it served, and follow whatever is shared: a registrant email, an IP address, a reused certificate, a hosting provider. Each lookup suggests the next one. Done well, the chain exposes an operator's wider infrastructure and the company hosting it.

I did this by hand for Chasing Ghosts in Dubai, which traced Dubai-based VPS resellers supporting offensive Iranian cyber operations. That work raised an obvious question: how much of it could an AI agent do, and how would it go wrong?

The failure that worries me most in attribution is not a missed domain. It is a confident wrong answer. An agent that invents a link, or treats a shared CDN address as proof that two domains share an owner, can send a response team after the wrong provider or the wrong country. An answer like that looks plausible right up until someone acts on it.

So Ghost Trace measures two things: capability (how much of an actor's infrastructure an agent recovers) and reliability (whether every link it claims is backed by a record). The headline metric is fabricated evidence: links the agent cites that no record supports.

How the benchmark works

Each case is a synthetic threat actor with a known answer.

• A generated actor network. Starting from a seed domain, the generator adds 1 to 4 layers of infrastructure. Each layer is reachable through exactly one kind of pivot: a shared registrant email, domains co-hosted on one VPS, one TLS certificate covering several domains, or the same self-signed certificate reappearing on a second IP. All of the actor's IPs sit with one VPS reseller, which leases address space from a larger upstream network. The correct "provider" is the reseller, the same pattern as in the Dubai work.
• Decoys. Many cases include shared infrastructure that looks like a link but isn't: a CDN edge IP serving 60–120 unrelated sites, a privacy-proxy WHOIS email used by dozens of domains, or the registrar's parking IP. Each case also has 25–40 unrelated background sites, including other customers of the same reseller.
• A frozen, offline snapshot. The agent's five tools (WHOIS, reverse WHOIS, passive DNS, IP information and certificate search) read only from a JSON snapshot. Domains use the reserved .test TLD, IPs are private-range, and every organisation name is marked synthetic. No real system or person is ever queried.
• A checked submission. The agent submits the domains and IPs it attributes, the provider, confidence scores, and an evidence chain of single facts such as "domain X resolves to IP Y". The harness logs every record each tool returned, so each cited link can be classified:
  • Verified: backed by a record the agent actually retrieved.
  • Unretrieved: true in the data, but the agent never looked it up. It guessed, or inferred.
  • Fabricated: no record anywhere supports it.

How a case is built (illustrative names). Solid arrows are attribution-grade links. Unrelated customers share the reseller's address block, and planted decoys sit outside it. The correct provider is the reseller, not the upstream network.

Every run feeds an analyst console that presents each run as an attribution finding to triage. It shows a pivot graph with verified and unsupported links drawn differently, the agent's stated confidence beside the confidence its verified evidence supports, and the full lookup log.

What I ran

Cases	40 synthetic cases, 10 at each depth of 1–4 pivots
Models	Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5
Repeats	3 runs per case per model (120 runs per model)
Ablation	40 extra Sonnet runs with one prompt change (explained below)
Budget	60 lookups per run
Total	400 model runs, 6,119 cited evidence links, about $24 in API costs

Two scripted baselines bracket the task. A careful breadth-first search that refuses to pivot through shared infrastructure scores perfectly. The same search without that filter scores an F1 of 0.50 and walks into decoys in 26 of 40 cases.

Intervals below are 95% intervals from a bootstrap that resamples whole cases. Where a failure never occurred, I report an upper bound using the rule of three. With 40 cases, "never observed" means "below roughly 7.5%", not "impossible".

Results

	Opus 5.5	Sonnet 5.5	Haiku 5.5
Infrastructure recovered (F1)	0.99 [0.98–1.00]	0.96 [0.93–0.99]	0.95 [0.92–0.97]
Provider correct	100% [93–100]	79% [69–89]	95% [91–98]
Runs with fabricated evidence	0 of 120	0 of 120	0 of 120
Runs that attributed a planted decoy	0 of 120	0 of 120	1 of 120
Runs that attributed a reseller co-tenant	2.5%	0 of 120	8.3%
Mean lookups used (of 60)	34.8	25.5	45.1

1. No agent fabricated evidence

Across 400 runs and 6,119 cited links, every well-formed link pointed to a record that exists in the data. When the agents went wrong, they over-interpreted real records. They did not invent records.

There were weaker lapses. About 2% of Haiku's links were unretrieved: typically it claimed an IP's hosting provider without ever looking that IP up, inferring it from neighbouring addresses. The guesses happened to be right, but they were guesses presented as evidence.

2. The obvious traps failed, and a subtle one worked

The planted decoys barely worked: 1 run in 360. The agents routinely noticed shared infrastructure and set it aside, writing things like "a privacy-service address covering 78 domains, so it is weak evidence on its own."

The trap I hadn't designed as a trap did work. The background data puts other customers of the actor's reseller in the same /24 address block as the actor. Opus (2.5% of runs) and Haiku (8.3%) attributed some of those neighbours to the actor because they were adjacent addresses. In one run, Haiku queried eight neighbouring IPs one after another, then attributed three IPs and three domains with no linking record. Its stated reasoning was that "attribution rests on the shared dedicated IP block."

The analyst console's view of that run, with ground truth shown. Green indicators chain back to the seed through verified records. The six amber indicators connect to the actor only through the hosting provider they share. The console flags the gap: stated confidence is moderate, but confidence from verified evidence is low.

That is the error that matters operationally: shared hosting treated as proof of common ownership. An IP sitting next to a malicious IP is a lead, not evidence.

There is also a quieter version. In one run, Haiku correctly described a CDN address as "a shared front with uncertain attribution", then listed it as actor infrastructure anyway at 0.45 confidence. Recognising shared infrastructure and leaving it out of the answer turned out to be separate steps.

3. A "model failure" that was really a wording problem

Sonnet named the upstream network operator instead of the reseller in 21% of runs. Each time, its own summary stated the reassignment correctly, for example: "Clearwater Datacenter, reassigned to Mirage Datacenter Services FZ-LLC." Then it answered "Clearwater Datacenter", once at 0.9 confidence.

The prompt asked for "the hosting provider behind the actor's infrastructure", and an upstream network is arguably a hosting provider too. So I re-ran Sonnet on all 40 cases with one added sentence defining the provider as the organisation the netblock is assigned to. It named the reseller correctly in 40 of 40, with no meaningful change in how much infrastructure it recovered.

So this is not a model that can't read a record. It is a model that resolves an ambiguous question the wrong way about one time in five. That still matters, because real analysts' questions are often just as loosely worded. Opus and Haiku mostly got it right without being told.

4. One pass would have told a different story

I wrote up a single-pass version of these results before running the repeats. Two claims changed:

• Haiku looked perfect on the provider question after one pass (40 of 40). Across three passes it was 95%.
• Sonnet's provider errors looked like a property of particular cases. Across three passes, 14 cases showed the error at least once, but only 3 showed it every time. It is mostly chance, not case difficulty.

If you are evaluating agents, run each case more than once.

5. Agents were underconfident, not overconfident

All three models gave confidence scores well below their actual accuracy. Every one of the 107 indicators Opus rated between 0.4 and 0.6 was correct. For an analyst, that is the safer direction to err. But it means the stated confidence carries little information, at least at this difficulty.

6. The benchmark is already too easy for the strongest model

Sonnet and Haiku drop from about 0.98–1.00 F1 on one-pivot cases to 0.91–0.94 on three- and four-pivot cases. Opus stays near-perfect throughout. Separating frontier models needs harder cases, and the results show what "harder" should mean: not louder decoys, but more realistic shared hosting.

Limitations

• Synthetic data with regular structure. Actor IPs are allocated consecutively within a /24, which makes sweeping the neighbours possible and tempting. Real infrastructure is messier, so the co-tenant failure rate may not transfer directly.
• Forty cases. The intervals are wide.
• Only Claude models were tested. These results describe these three models, not AI agents in general.
• The prompt tells the agent the data is synthetic. I added this so a research task would not trip safety filters (no run was refused). It may also make agents less careful than they would be with real data.
• Two metrics were added after the first results. The malformed link category and the co-tenant measure were both added after I looked at the first pass. I added malformed when one apparent fabrication turned out to be a real certificate fact written in the wrong format, and I didn't want to overstate the headline number. That kind of post-hoc change should be stated, so I am stating it.
• The memorization question is still open. It asks whether agents do better on real, published cases because they recall the published reports. Testing it needs real cases, which come next.

Coming next in the series

Part 2: Where do AI agents break? Harder cases built around the trap that worked: more reseller co-tenants, small shared IPs, and actors spread across several resellers. Part 2 will also include:

• a step-budget sweep, to separate "couldn't search enough" from "didn't reason well"
• a larger suite, to tighten the intervals
• agents from other vendors and an open model

Part 3: Are AI agents investigating or remembering? Real, published Iranian infrastructure cases, each run twice: once with real names and once with every name swapped. If an agent does better on the real names, it is recalling the published reports rather than following the records.

Along the way, I also plan blind triage of flagged findings by analysts in the console, to test whether its alerts help a human catch bad findings faster.

I plan to release the code, case data, every run trace and the analyst console once the full version is complete.

How AI was used in this study

I want to be specific, because the tool I used is closely related to the systems under test.

• Design. The research questions, benchmark concept, decoy types, metrics, safety constraints and analyst-console requirements come from a written proposal that predates the build. I developed it through back-and-forth discussion with Claude, which wrote it up. The idea grew out of my earlier manual research.
• Software. The code was written with Claude Code, Anthropic's AI coding assistant, running Claude Opus 5.5 under my direction. That covers the case generator, tools, agent harness, scoring, statistics and analyst console. It also designed and ran the automated tests.
• Experiments and analysis. The AI ran the model evaluations, computed the statistics, and surfaced the failure examples quoted above. At my request, it chose which three models to test. It also proposed the repeated runs, the confidence intervals and the provider-wording ablation, which I approved.
• Figures. The charts and the console view were generated from the saved run data by scripts the AI wrote. The AI also drew the case diagram, which uses illustrative names.
• This write-up. The AI drafted this post from the results. I asked for this disclosure, decided to publish the work as a series, and chose to hold back the code and data until the full version. I read the post in full before publishing it.

My part. The study grew out of my hand-done research in Chasing Ghosts in Dubai. It is part of my wider work on an agentic AI loop for threat-infrastructure research, where I need to know how far an agent's attributions can be trusted before relying on them. I shaped the proposal with Claude, set the direction at each stage and funded the model runs. I decided what went forward: approving the extra repeats and the wording test, choosing the series format, and deciding what to publish and when. I did not write the code or run the analysis myself. My role was to define the question, judge whether the results answer it, and stand behind what is published here.

Possible conflict of interest. An Anthropic model helped build an evaluation of Anthropic models and drafted the write-up of their results. These steps limit the room for bias:

• Scoring is deterministic and checks submissions against generated ground truth, not against any model's judgment.
• No model grades another model's work.
• Every run is saved with its full transcript. When the code is released, the runs will go with it, so anyone can recompute the numbers or check the quoted failures.

Where responsibility sits. AI was involved at every stage, from drafting the proposal to drafting this post. The study's purpose is mine, and so is the final word on every claim made here. Responsibility for the content is mine.