Skip to main content

Secondary navigation:


June 2011

Finding knowledge in too much information

By J.M. Berger

Finding a needle in a haystack is easy. Finding order in the mountain of data created by the world's financial markets is hard, if not mathematically impossible.

The Carroll School of Management hosted its sixth annual finance conference June 8-9, and the challenge of creating visibility in a sea of complexity emerged as an overriding theme.

Panelists from major investment firms and the U.S. Securities and Exchange Commission (SEC) laid out the dilemma faced by regulators: how much data they can collect versus how much they can hope to understand.

“There are 10,000 investment advisers, 2,000 investment companies, let's call it another 1,000 to 1,500 [registered] hedge funds,” said Jonathan S. Sokobin, acting director of the SEC's Division of Risk, Strategy, and Financial Innovation. “The SEC has between 350 and 375 examiners in the United States. It's an order of magnitude difference.”

The SEC tries to corral the problem using computer-driven analytics, but the volumes of data and exponentially larger sets of relationships within the data make that a Sisyphean task. Even the most sophisticated computer models cannot encompass the whole picture.

Panels at the conference delved into some of the most important sources of hypercomplexity in today's markets, including structured products and derivatives, which will be regulated by the U.S. Commodity Futures Trading Commission (CFTC) under the not-yet-implemented Dodd-Frank Act.

Although the CFTC is much smaller than the SEC, the derivatives markets under its purview have a larger capital value than all the shares traded on Nasdaq and the NYSE combined, said panelist Charles Cooper, senior managing director of State Street's eExchange derivatives program (pictured). “These markets are enormous.”

And unlike stocks, derivatives and structured products are so complex that very few people can understand them taken one by one, let alone as part of a vast marketplace.

“When we're dealing with these markets, we're talking about terabytes of data generated every day that are being churned out,” Cooper said. “And as a regulator, yeah, transparency's good, but what regulators need to do is figure out what they're going to do with it.”

The trend toward overwhelming volumes of data also owes much to the advent of high-frequency trading (HFT), which now accounts for more than half of all equity trades in the United States. HFT generates millions of trades per day using computers wired directly into the stock exchange, buying and selling stocks faster than a human mind can react based on price variations measured in pennies.

Steven M. Barry, chief investment officer of Fundamental Equity and moderator of the panel on HFT, articulated the data volume problem with a quote from T.S. Eliot: “Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?”

Barry took the lament one step further. “To paraphrase to the 21st century level, ‘Where is the information we have lost in data?’”