Real-time market surveillance for regulatory compliance

Consider a scenario: At 10:42 AM on a Tuesday, an algorithm starts placing 50,000-share orders on AAPL and canceling them 80 milliseconds later. By 10:43, it has done this 247 times. By 10:44, other traders have moved their prices in response to what looks like massive buying pressure. By 10:45, the algorithm is gone, and someone has made money on the price move it manufactured.

The surveillance analyst reviewing this activity sees it at 8:15 AM the next morning, in a batch report that flags "unusual cancel rates." By then, the pattern has repeated on three other symbols, the profits have been extracted, and the account has gone quiet.

This is the asymmetry that keeps federal regulators up at night: the markets move in microseconds, but surveillance infrastructure often moves in hours.

The 8:15 AM problem

That analyst isn't incompetent, and the surveillance system isn't broken. It's working exactly as designed — collecting order data throughout the day, running detection algorithms overnight, and surfacing alerts the next morning.

This architecture made sense when markets were slower and manipulation was cruder, but spoofing algorithms now operate faster than batch systems can detect — and they're adaptive. If the same pattern triggers an enforcement action, the algorithm changes. It spreads activity across multiple accounts, or shifts to different time windows, or moves to a different venue. The surveillance system keeps looking for yesterday's pattern while today's variation runs unchecked.

The alternative — complex event processing systems — can detect patterns in real-time, but they're rigid. They catch what they're programmed to catch. When the analyst wants to ask "show me all orders from this account that were cancelled within 100 milliseconds of a price move," the answer is usually "submit a development request and wait six weeks."

What surveillance actually needs is both: detection that runs at market speed, and the flexibility to investigate patterns that weren't anticipated when the system was built.

Catching the spoofer at 10:43

Here's what real-time detection looks like. Order flow streams in continuously, and surveillance logic runs against it as it arrives:

Every row that arrives gets incorporated into rolling calculations. For each account, we track how many orders were placed and how many were cancelled over the last 30 seconds:

An account canceling 80% of its orders within 30 seconds, while placing dozens of them? That's worth a closer look:

That's the entire surveillance system: three queries, about 20 lines of code, running in real time. The rolling calculations update incrementally — when a new order arrives, only the affected windows recompute, not the entire dataset.

The spoofing_alerts table updates continuously. At 10:43, when that algorithm was still running, this alert would have fired. The analyst could have seen it happening, queried the underlying orders, and watched the pattern unfold in real-time.

The detection logic here is simple — cancel ratio over a time window. A batch system could run the same logic, but running it at 8:15 AM the next day, against data that's already stale, misses the point. The value is in catching the pattern while it's still actionable.

The investigation that follows

When an alert fires, the analyst's next question is usually: "What else has this account done?" In a batch system, that's a new query against a different database, probably with a different query language, and possibly requiring a data request that takes hours to fulfill.

Here, the analyst just keeps querying. The same orders table that fed the alert contains the full history. Filter by account, look at the last week, check for similar patterns on other symbols. It's all the same syntax, the same interface, the same data. And the queries the analyst writes against historical data work identically when pointed at a live feed — no translation layer, no separate codebase.

And when the analyst spots a new pattern — maybe the account is also doing something suspicious with order modifications, not just cancellations — building a new detector is Python code, not a six-week development request.

Why this matters for federal systems

Federal surveillance has constraints that commercial systems don't:

• The data often can't leave controlled environments, so Deephaven runs entirely on-premises with no cloud dependencies.
• Enforcement actions require proving exactly what was detected and when; the same queries that generate alerts also produce audit trails.
• And federal systems rarely operate alone. Deephaven connects to Kafka streams, databases via JDBC, case management systems via REST APIs.

These constraints make one thing clear: federal teams can't rely on vendor update cycles to keep pace with evolving manipulation. The spoofing patterns of 2020 look different from 2026. The sustainable advantage isn't any single detection algorithm — it's the ability to develop, test, and deploy new approaches faster than the manipulators adapt.

That analyst at 8:15 AM isn't going to catch tomorrow's manipulation with yesterday's batch report. But an analyst watching the market in real-time, with the flexibility to investigate anything that looks suspicious, has a chance.

Try it yourself

Run the spoofing detection example in your browser with Deephaven's live demo. To deploy in your own environment, follow the quickstart guide. Questions about surveillance use cases? Join the conversation on Slack.