Stephen Foley: Dow Jones insanity shows the need to regulate the murky world of dark pools

Outlook: We don't know all the causes yet of Thursday's bout of insanity on the Dow Jones Industrial Average, but this much is clear: regulators have lost control of the equity market.

If you've been bewildered and frightened by the complexity of the credit markets, where untold dangers lurked, undetected, until they exploded into the credit crisis, at least you could take comfort in the stock market. In the equity market, common-or-garden shares trade on honest-to-goodness exchanges, where they go up and down broadly in response to investor demand, and you get a read out at the end of the day from FTSE in the UK or Dow Jones in the US.

This is the shallow end of the financial markets, where it is safe for widows and orphans to paddle. Or so you thought. Actually, in the name of "innovation" and "liquidity", Wall Street has brought complexity and a lack of transparency even here. The New York Stock Exchange and Nasdaq have lost their monopoly to private "crossing networks" with fewer disclosure rules, so that big buyers can go hunting for sellers (and vice versa) without having to reveal their hand. It makes a mockery of the claim that published stock prices reflect the balance of supply and demand at any given moment.

It is because of the sheer lack of transparency that these off-exchange trading platforms are called "dark pools". There are more than three dozen alternative trading pools in the US now, the NYSE complains.

Regulators are trying to keep up, laying down ever deeper layers of rules to govern how the trading platforms must interact, but that only creates more ways for traders to game the system. The Securities and Exchange Commission (SEC) had to move against one such cheat last autumn, when it banned dark pools from "flash trading", the practice of revealing details of a customer order to a select group of traders for a fraction of a second, giving them an unfair advantage.

The traders themselves long ago stopped being actual humans. More than half of all share trades in the US are now done by computer programmes, and the proportion is rising exponentially. These black-box programmes are designed to trade at very high frequency, using impenetrable mathematical models to spot supposed trading patterns and eke out tiny profits, multiple times a second. Even the unnoticeable amount of time it takes data to travel down a cable makes a difference, which is why hedge funds pay to install their computers on-site at exchanges.

Defenders of the new landscape say dark pools are great for institutional investors, who get better prices for their trades, and high-frequency traders are great for retail investors, who get a much more liquid market.

But that crazy move between 2.40pm and 3pm on Thursday, when the Dow plunged 998 points, was what you might call an "unintended consequence". Volatility is inherently higher in a computer-driven trading environment. It is why there have been so-called "circuit breakers" in place since the crash of 1987, the last time the robots went wild. The circuit breakers slow down or even halt trading if stocks start moving too quickly, so that new buyers or sellers can come in without fear of being trampled in a stampede.

One factor in the madness, at least according to the NYSE boss Duncan Niederauer, appears to have been the lack of co-ordination between his exchange, which has circuit breakers, and the dark pools, which carried on trading in halted stocks, causing weird price distortions, triggering computer program errors and stoking a panic.

The SEC recently – belatedly – began to review of high-frequency trading and dark pools. It is difficult to be optimistic that it will get its head round all these issues. This is the same SEC that couldn't catch Bernard Madoff when he was using an archaic computer to fabricate thousands of trades.

Complexity is, by design, the enemy of regulation. The answer, therefore, is a sweeping away of this complexity, a curb on high-frequency trading, forced consolidation of dark pools and much more transparency everywhere. The SEC needs to act boldly – or else Thursday's chaos will be just a taster of what is to come.