How Power Peg brought down a Knight

October 31, 2014 07:00 PM

The term algorithm refers to a set of rules for performing arithmetic, though there’s no generally accepted definition. It commonly refers to any set of procedures used for solving problems or performing tasks. The most common component of an algorithm is the “if-then” statement: “IF” the kitchen is floor is dirty, “THEN” you should mop it. Most algorithms are infinitely more complex. 

By the 1990s, stock traders realized they could harness the speed of computers and began experimenting with algorithms for executing trades. As the size of institutional investors’ orders grew and electronic stock trading exploded, so, too, did the use of algorithmic trading. But it all goes back to the one firm that pioneered the algorithmic trading bandwagon: Knight Capital Group.

Founded in 1995 by Kenneth Pasternak and Walter Raquet, Knight was not headquartered on fabled Wall Street, but across the river in Jersey City. It owed both its inception and rapid growth to the NASDAQ stock market, a growth that was spurred by the technological advancements of the 1990s.

Raquet had the technological insight to see that the old exchange model was outdated and it would inevitably be replaced by computerized trading. It was a revolutionary way to think at the time. Raquet and Pasternak took the idea and ran with it.

By 1997, Knight had grown to become the single largest market-maker of NASDAQ stocks, benefitting directly from individuals executing their own trades online without having to call their broker. And just as the founders hoped, Knight was the innovator in low-cost stock market execution, positioned right at the epicenter of the Internet explosion. 

In July of 1998, Knight used those three little words that investors love to hear – Initial Public Offering – and raised an immediate $145 million more in capital. Within six months its market capitalization soared from $725 million to $2 billion; by the end of 1999 that value ballooned to a staggering $8 billion. 

Anytime money is involved there’s almost always someone who finds a way to exploit the loopholes. There were more than a few scandals that rocked Knight over the years. John Hewitt, an executive brought in from Goldman Sachs to become president of Knight, caught wind of front-running rumors and discovered the firm had problems. It was too late to avoid major fines from the National Association of Securities Dealers for failure to supervise trading activities.

In the end, the practice of front-running by individual traders was not eradicated by new rules or new regulations. Instead, it was done by taking the human trader out of the equation. Technology had eliminated much of the ability of the human trader to front-run investor orders as the era of electronic matching has arrived. 

New market, new problems 

Many of the same problems that plagued Knight in the late 1990s resurfaced in the early 2000s. The risk of a human trader front-running the order just resurfaced in electronic front-running. High-Frequency Trading (HFT) firms were teaching computers to do it. Institutional investors were once again not happy; they were being outgunned by computerized traders and they needed a solution. They needed to execute large trades without moving the market against them. To solve this dilemma, the folks at Knight turned to the algorithm.

Algorithmic order execution was designed mainly to help large-volume traders execute without tipping off the market. One of the more popular algorithmic executions is called the parent-and-child order (or iceberg). Other types of algorithmic orders were called peg orders. In one type of peg order, the customer dictates a price above or below the best bid or offer in the market and “pegs” his order to that price. Modifications included a “mid-price peg,” in which the peg price is the average of the best bid and the best offer. 

At Knight, the first algorithmic peg order routing system was called Power Peg. Technology at the time allowed Power Peg to send thousands of buy and sell orders to the exchanges every second. The system was keyed by a flag in Knight’s computer system. When the flag was up, it meant that there was an active parent order in effect and Power Peg kept sending the child orders to various markets. 

Computerized trading wasn’t the only change at this time. On April 9, 2001, the Securities and Exchange Commission (SEC) passed Reg NMS and mandated that the stock market move to decimal pricing. Overnight, U.S. stocks went from trading in eighths to pennies. Studies showed that investors could save more than $1 billion annually from narrower spreads. Market-makers, however, saw their profits collapse. And because Knight was at the top of the market-maker food chain, they were the hardest hit. 

Knight shifted to a high volume market-making strategy and replaced Pasternak with Tom Joyce, a well-known industry veteran from Merrill Lynch.  Joyce changed the firm’s compensation system and was able to stop the bleeding. 

Buoyed by the sudden financial reversal, he retired many of the old Knight electronic execution systems and built new trading technology, including a new order routing system called Smart Market Access Routing System (SMARS). Like Power Peg, SMARS was set up to receive parent orders and then send out the smaller child orders to external trading platforms. The system was up-to-date, faster, more reliable, and could compare prices between more than 50 different trading venues within fractions of a second. While Power Peg was inconceivably fast in its day, SMARS was capable of executing as many as 2 million orders per second. 

Power Peg was decommissioned in 2003 yet remained on Knight’s servers (it’s a common practice by many computer programmers to disable programs rather than delete them). Knight developed its market-making algorithms internally and Joyce flipped the switch in the second quarter of 2005, completely automating the firm. 

Joyce’s next step was a massive corporate acquisition spree that diversified the firm away from solely market-making in stocks. By 2012, the firm would have 40 market-makers and 185 computer programmers, compared to June of 2002 when there were 260 market-makers, all human traders, and five programmers. All of this technology – new and old – was housed in Knight’s data center in Jersey City. Included in this technology stockpile were eight separate high-speed servers that ran the firm’s computer trading and execution programs.

In what should have been a wake-up call for everyone in the financial markets, the Dow Jones Industrial Average (DJIA) posted its largest intra-day loss in history on May 6, 2010: the Flash Crash. On that fateful day about $1 trillion in market capitalization had been erased—the largest one-day decline in the Dow’s history.  

Following the Flash Crash, two new rules were put in place to govern securities trading. Circuit breakers were required to halt trading if the market experienced what was labeled as “significant price fluctuations.” Called “limit-up/limit-down bands” the rule stated that if a stock moved up or down by 10% or more during a five-minute period, there would be a mandated pause in trading for five minutes. That would break the priority of stop orders and allow the market to reset. Rule 15c3-5, also known as the Market Access Rule, dictated that the exchange would have algorithms in place designed to ensure the integrity of their computerized systems. It also required that broker-dealers implement their own risk management controls to block erroneous orders from reaching exchanges. 

In July of 2012, the New York Stock Exchange received approval from the SEC to establish the Retail Liquidity Program (RLP). The basic idea was that orders from retail investors would be directed into a single dark pool run by the NYSE with stocks to be quoted in as little as 0.1¢ increments. If market-makers thought one cent pricing was bad, the new increment was 10 times worse. The result for investors would be, at least in theory, better prices. If retail investors didn’t like the one cent bid/offer spread for a stock, they had the ability to transact somewhere in the middle. Market-makers, like Knight, were adamantly opposed, believing this special dark pool was an attempt to move trading back to the exchange. With price increments of less than a penny, market-making spreads were being gutted further. 

The RLP dark pool was set to launch on the morning of Aug. 1, 2012. Despite misgivings, Knight executives were even more wary of losing customers. Knight’s programmers worked on the upgrade code to their SMARS order handling software and updated the execution programs to accommodate RLP orders. 

On July 27, 2012, Knight’s IT department were convinced they were ready. On July 31, the night before RLP went live, Knight’s SMARS programmers began loading the new software on to the servers. Whatever the reason, the updated software was only loaded on to seven of the firm’s eight servers. At first glance, it doesn’t seem like such a critical error—in hindsight, it was disastrous. 

It’s common practice to have multiple checks on new software to make sure that everything was installed properly. Knight’s IT department was top-notch, so perhaps they didn’t always see a reason to double-check their work. One Knight executive later said, “The IT guys were arrogant.”

At 8:00 the next morning, the pre-market orders began to accumulate on the SMARS system, but something was wrong and SMARS started sending out an automated error message to a group of 97 Knight employees. “Power Peg disabled,” the message read. “Power Peg disabled” over and over again, even though Power Peg was decommissioned nine years earlier. The market was opening in 90 minutes. Perhaps those 97 employees had never even heard of Power Peg, but Power Peg had just woken up from its nine year slumber.

At 9:30 AM, the market opened. Back when the Power Peg system was operational, there was a flag that told Power Peg when the parent orders had been filled. When the flag was up, Power Peg kept sending child orders to the various exchanges. As it happened, the programmers had recycled the old flag for the new RLP update. As soon as that flag was up, Power Peg miraculously came back to life and started trading. The orders that went through the other seven servers worked fine, but for the orders assigned to the eighth server, Power Peg took the reins and started doing what it was supposed to do.

As the orders came in, Power Peg began executing them, taking the main order as a parent and sending out child orders across the exchanges. Two hundred and twelve pre-market orders came into that eighth server and Power Peg was doing what it was programmed to do: buying on the offer and selling on the bid, repeating the process endlessly. As long as the flag remained up, Power Peg kept going. And because there was nothing to turn the flag off, Power Peg never stopped. It was buying and selling 140 NYSE stocks and ETFs and executing 2,400 transactions a minute. 

At that frenetic pace, it didn’t take long for traders to notice that something was up. Within the first minutes of trading, volume was 12% higher than normal. After a couple of minutes more, select stocks had traded the equivalent of 30 days’ worth of volume. And Power Peg was just getting started.

You might think the new rules established after the Flash Crash – rules specifically designed to prevent a rogue algorithm from going viral – would have kicked in. But the circuit breakers weren’t designed for massive trading volume; they were designed for large price swings. The same was true for the limit up/limit down bands. They weren’t triggered for the same reason. Those protections only came into play when stock prices were moving up or down by 10% or more. The stocks really weren’t moving outside of their trading ranges.

What about the internal risk controls at Knight? All of the trading accounts at Knight had limits in place to automatically trigger a shut-down if certain thresholds were reached. All except one: the error account. The error account was designated for the trading mistakes generated in those other accounts. As such, there were no stop-gap measures to shut down the error account. Since Power Peg’s resurrection wasn’t a part of the firm’s normal trading protocol, all of the Power Peg trades were automatically sent to the error account. 

At 9:34 a.m., NYSE computer technicians traced the massive volume spike back to Knight. They found that 20% of the entire NYSE trading volume was being driven by the Electronic Trading Group (ETG) at Knight. Duncan Niederauer, the CEO of the NYSE, immediately tried to call Joyce; unfortunately, Joyce was at home recovering from knee surgery.

That information was then routed to Knight’s chief information officer who, upon hearing the news, immediately gathered the firm’s top IT people. This was the perfect time to flip the kill switch that is de rigueur in most trading systems today but there was none. 

After nearly 20 minutes, Knight’s technicians decided that the problem was most likely the new code. Because they knew the old version worked, they reinstalled it—a pretty standard first response. “IF” the new system is acting up, “THEN” go back to the old one. As it turned out, it was the worst thing they could have done.

Within a couple of minutes, the techs had successfully removed all of the new RLP algorithms from the SMARS update. However, that action didn’t turn off the flag that was telling Power Peg to buy and sell. But now the updates were turned off at the other seven servers, and not just server number eight. Power Peg jumped in to fill that void and was now running on all eight servers simultaneously, buying on the offer, selling on the bid.

It wasn’t until 9:58 a.m. that the programmers located the problem and shut Power Peg down for good. The bleeding had stopped, but not before the algorithm had done an incredible amount of damage over 28 minutes. Trading volume on the NYSE was 364 million shares during the first half-hour that morning vs. an average of 100 million shares. And that was just the NYSE!

At 10:15, the damage assessment was finalized. Power Peg had routed approximately 4 million trades in 154 stocks, trading more than 397 million shares. Knight had a net long position in 80 different stocks worth $3.5 billion, and a net short position in 74 different stocks worth $3.15 billion. 

By 10:30, Knight’s own stock began to plummet (see “Bad day,” below). Joyce made it into the office on crutches by noon, and at 12:30 he was informed by Niederauer that Power Peg’s trades, although unintended, would stand. Under NYSE rules, they could not be cancelled.

Different exchanges have different rules for allowing trade cancellations. At the NYSE, the trading range must be between 20% and 30% off the opening market price. With very few exceptions, none of Power Peg’s transactions had moved the stocks more than 10% away from the opening price, which meant that the cancellation rules did not apply. Joyce called SEC Chair Mary Schapiro to plead his case, explaining that it had been a computer malfunction. While sympathetic, Schapiro informed Joyce that it was between Knight and the NYSE. Knight had to deal with a $7 billion position in various stocks. Other market participants knew that too. Wall Street smelled blood in the water.

By the time the market closed on Aug. 1, the position was whittled down to $4.6 billion, which meant that they’d sold off more than $2 billion of their holdings. But the firm lacked the regulatory capital to hold such a large position, and the clock was ticking. Stocks settled three days after the trade date, which meant that Joyce had three days to either find a buyer for the absurdly large position or borrow money to finance it.

Bids filtered in from market participants and hovered around an 8% to 9% discount from the market’s closing prices, far more than the discount Joyce expected. On that $4.6 billion block of stock, a 9% discount amount to $414 million.

Goldman Sachs bid at a 5% discount; better, but it would still cost Knight $230 million. Meanwhile, Knight tried to arrange emergency funding.

Before the start of the trading day on Thursday, Aug. 2, Knight sold off its positions and took the massive loss. While they had about $365 million in cash and liquidity on hand, the losses were still well in excess of that. The $200 million they’d lost on cutting their position from $7 billion down to $4.6 billion was just the tip of that iceberg, Power Peg lost money buying on the offered side and selling on the bid side, and they took a hit selling their remaining positions. All told, the firm was looking at a shortfall of approximately $440 million. Taking the losses was a foregone conclusion. They needed a massive capital infusion to stay in business.

Citadel and others put offers on the table to provide financing to get them through the weekend; however, the price was too high for Joyce. 

Many of Knight’s most important clients – including TD Ameritrade, Vanguard, Fidelity Investments, Scottrade, E-Trade and Pershing, among others – stopped routing orders through Knight, making its money problems worse. 

Virtu Financial, an electronic trading firm that specialized in high-frequency trading, approached Knight with the equivalent of a life ring that would allow Knight to stay solvent. Then Citadel came back and submitted a new offer that was declined.

On Aug. 4, a deal was struck between Knight and a consortium of banks that included Jeffries, Blackstone, TD Ameritrade, Stephens, Stifel Nicolaus, and GETCO. Knight would get $400 million in cash, and the consortium received preferred shares with the option to buy Knight’s common shares at $1.50; shares that could be converted to a 73% ownership in Knight. In this deal, Knight would remain independent at least for a while.

By November, Knight’s stock price was hovering around $2.50 a share, which meant the firms who had saved the dying company saw a nice profit – $1.00 a share – if they chose to exercise their options. 

Knight was back in play and GETCO, a Chicago-based high-frequency trading firm, won the bidding with a combination stock and cash offer worth $3.70 per share. The firm that had ridden the technology wave to the top found themselves a victim of technology in the end.

Lessons learned 

Throughout his tenure as CEO at Knight, Joyce openly discussed what were deemed to be the most “significant risks.” Joyce suggested that a rogue trader was, to his mind, the biggest threat they faced. It never occurred to him that an algorithm running in the computer system would turn out to be the rogue trader he so feared.

The disaster that was wrought in the first five minutes of the trading day on Aug. 1, 2011 was a software problem. The ensuing 35 minutes, however, were a catastrophic failure of risk management. The failure is manifested in the fact that they didn’t have a kill switch in place. It could have been a very simple algorithmic expression for a programmer to write: IF the daily volume reaches a certain level, THEN shut it down. 

Ed. note: This article is taken from a chapter of Scott Skyrm’s soon to be published sequel to Rogue Traders. 

Scott E.D. Skyrm is author of the book Rogue Traders and is working  on a sequel. Previously he was a global business head in fixed-income, securities finance, and securities clearing and settlement for Newedge and managed the repo desk at ING Barings.

About the Author