Friday, September 14, 2007

1995 Sifting Hidden Market Patterns for Profit

September 11, 1995 new York Times

Sifting Hidden Market Patterns for Profit

In the late 1970's, when Doyne Farmer and Norman Packard were graduate students in physics at the University of California at Santa Cruz, they tried to make their fortune by using miniature computers to beat the roulette tables in Las Vegas, Nev.

Roulette, they reasoned, was not quite so random as commonly believed. With the help of friends and fellow students, they developed a computer program that roughly predicted in which octant of the wheel the spinning ball was likely to land.

Though the scientists' system did not make them wealthy, it showed that it was theoretically possible to get a slight edge over the house. Now, almost two decades later, they are trying to make a much bigger killing by using information technology to outsmart the glitziest casino of them all -- the financial markets.

Since 1991, Dr. Farmer and Dr. Packard have been running the Prediction Company, a business here that uses the latest developments in artificial-intelligence software to analyze market and financial data, trying to find hidden patterns that foretell which way Fortuna's wheel will turn. Trading signals are sent to the company's partner, the Swiss Bank Corporation, which places bets on the movements of foreign exchange rates, interest rates and stock and commodity prices. Swiss Bank is one of the most sophisticated users of quantitative models for investing.

So far, the Prediction Company's scientists say, the system is making money, though they are barred by a confidentiality agreement from saying how much. Nor are they allowed to say how much money they are playing with, though they acknowledge that the figure considerably exceeds the $1 million they were investing in the project early on.

Trying to devise a mathematical system to beat the house is one of the world's oldest professions, and one scorned by most experienced investors. But it has proved irresistible for some experts in computer systems to try out the newest technologies on this oldest of quests. Many Wall Street investment banks employ former physicists and mathematicians who try to apply scientific models to the financial markets.

A few end up at small boutique investment houses or consulting firms; others, like Dr. Farmer and Dr. Packard, start their own.

For them, there is more at stake than a small part of Swiss Bank's investment pool. Dr. Farmer and Dr. Packard are using lessons they learned as physicists studying complex systems to challenge a cherished tenet of classical economics: the random-walk hypothesis, which holds that day-to-day changes in prices are essentially unpredictable.

"It's an interesting intellectual challenge to try and defeat the established dogma -- prove it wrong by getting rich," Dr. Farmer said. "We have to fight an entrenched body of skeptics who were brainwashed in school into thinking markets are perfectly efficient."

Most economists say that trying to use past prices to predict future ones is as futile as trying to predict how a perfectly balanced coin will land. No matter the past pattern of heads and tails, the outcome of the next flip is a random and independent event. Likewise, these economists say, no analysis of the past zigs and zags of a stock price will tell you anything about its price tomorrow.

The reason? Markets are said to be efficient processors of information. If there were a meaningful pattern -- three zigs followed by a zag on a price chart might mean a stock is about to rise -- then so many traders would have bought on this expectation that the price would have already increased. The market would have discounted the information, which would already be reflected in the price.

For years, hordes of investors called technical analysts have ignored the admonitions about efficient markets. Hoping to gauge investor sentiment, they scrutinize graphs of stock prices for telltale patterns with names like breakaway gap, double top, rounded bottom and the head-and-shoulder formation.

Dr. Farmer and Dr. Packard say that their methods are more scientific. "We tend to look at the same kind of stuff as technical traders, but we use careful statistical evaluation as opposed to intuition," Dr. Packard said. And they are not restricting themselves to studying price movements. At the Prediction Company's offices, computer work stations running pattern-recognition software scrutinize some 50 to 100 financial and economic variables.

"Ninety-five percent of technical trading is based on obscure mysticism -- druid mythology or something," Dr. Farmer said. "We are applying the same kind of rigor as economists do, but we are more broad-minded about what might work."

Among the tools in their portfolio are neural networks -- computer programs with a structure similar to the weblike architecture of the brain. Neural nets are also used by some Wall Street investment houses -- Salomon Brothers, for example. Given a stream of data, a neural net can learn to identify subtle patterns. The Prediction Company is also experimenting with another kind of learning program called a genetic algorithm, in which Darwinian selection is used to breed computer code adept at certain tasks -- like ferreting out patterns in financial data.

Eugene F. Fama, a professor of finance at the University of Chicago, said he was not convinced that even the most sophisticated software could be used to find significant relationships in past financial data. "If you search through a hundred variables you're bound to find something," he said. But the relationships, he said, are most likely to be random, like the correlations technical traders have found between prices and the height of hemlines, the winner of the Super Bowl or the phases of the moon.

"The business of financial management is selling black boxes," Dr. Fama said, "and this is a blacker box than most."

The scientists at the Prediction Company agree with mainstream economists that market data mostly consist of meaningless noise. But they believe that within the din are barely perceptible signals that can provide a slight edge.

As physicists in the developing science of complexity, they learned about building computer models of systems with many variables. The complexity field, still in its infancy, seeks to deepen the understanding of how simple parts can interact to produce an almost unfathomable intricate whole. Dr. Farmer studied complex systems at Los Alamos National Laboratories in New Mexico; Dr. Packard at the Institute for Advanced Study in Princeton, N.J., and the University of Illinois. Both have also worked with the Santa Fe Institute, which is trying to apply insights into the behavior of complex systems to the economy and the financial markets.

Blake LeBaron, an economist at the University of Wisconsin who is also affiliated with the Santa Fe Institute, said there was good reason to believe there might be inefficiencies to exploit in the marketplace. "For a lot of economic problems, the efficient-market and the random-walk hypotheses are probably not a crazy description for stock prices," he said. "But they shouldn't be taken as unshakable doctrine."

Dr. LeBaron argues that the efficient-market hypothesis was formulated in a day when there was ample time for information about a stock to be absorbed by traders and reflected in the price. But today, high-speed computers and information networks deluge traders with more up-to-the-minute data than they can possibly digest. "It becomes harder to believe that all this information is reflected in the price at every possible instant," he said. With powerful enough software, it might be possible to find neglected scraps.

At the same time, Dr. LeBaron said, better techniques have been developed for measuring randomness. They show that the markets are "pretty random," he said, but not entirely so. So far, the Prediction Company's success could just be a matter of luck. The question is whether the winning streak will continue. "The better the performance, the harder it is to attribute it to a statistical fluke," Dr. Farmer said.

One paradox of the efficient-market hypothesis is that it cannot be true unless most people do not believe it. For all the information about a stock to be reflected in its price, people must be out assiduously gathering data. If the orthodox economists are right, the Sisyphean efforts of the Prediction Company merely insure that the efficient market hypothesis will ultimately prevail, flattening the nonbelievers.

But Dr. Farmer is certain that order will triumph over randomness. "We're convinced there is enough structure to make a business," he said.

No comments: