Knight Capital, 2012
On August 1, 2012, the trading firm Knight Capital lost approximately $440 million in about 45 minutes. The company was rescued from bankruptcy by a $400 million investment days later and was acquired soon after. A forty-five-minute deployment mistake ended an eighteen-year-old firm.
What it looked like at the time
The market opened, and one of Knight’s automated systems started buying at high prices and selling at low prices — millions of times. The firm’s risk systems and the exchange both noticed strange order flow, but for the first minutes nobody could tell which of the company’s many systems was misbehaving or how to stop it.
Root cause
Knight was rolling out new routing code for a stock-exchange program to eight servers — by hand, one at a time. A technician forgot to copy one of the new files to one of the eight servers. That server still contained a piece of long-retired order-routing code (nicknamed Power Peg) that had been left in place, dead, for years. The new code reused an old configuration flag; on the seven up-to-date servers the flag did the intended job, but on the stale server it reactivated the retired code, which bought high and sold low and re-triggered itself millions of times.
How it was found and fixed
The loss only stopped when the flag was turned off and the server taken out of rotation — the code itself had no working off switch. Knight was left with a huge unintended position that took days to unwind. The SEC later charged the firm with violating market-access rules and ordered it to pay $12 million. After the rescue investment, Knight merged with its rescuer.
The lasting lesson
Manual deployments eventually meet the server that was missed — deployments should be automated and identical everywhere. Dead code is never harmless: code you don’t run today can still be reactivated by tomorrow’s flag. And a flag that once meant one thing should never be quietly repurposed to mean another.
Practice the skill
A debugging challenge that works the same muscle:
- Works On My Machine — a debugging challenge in the same discipline
- The Missing Name — a debugging challenge in the same discipline