The mathematical equation that caused the banks to crash

In the Black-Scholes equation, the symbols represent these variables: σ = volatility of returns of the underlying asset/commodity; S = its spot (current) price; δ = rate of change; V = price of financial derivative; r = risk-free interest rate; t = time. Photograph: Asif Hassan/AFP/Getty Images

It was the holy grail of investors. The Black-Scholes equation, brainchild of economists Fischer Black and Myron Scholes, provided a rational way to price a financial contract when it still had time to run. It was like buying or selling a bet on a horse, halfway through the race. It opened up a new world of ever more complex investments, blossoming into a gigantic global industry. But when the sub-prime mortgage market turned sour, the darling of the financial markets became the Black Hole equation, sucking money out of the universe in an unending stream.

Anyone who has followed the crisis will understand that the real economy of businesses and commodities is being upstaged by complicated financial instruments known as derivatives. These are not money or goods. They are investments in investments, bets about bets. Derivatives created a booming global economy, but they also led to turbulent markets, the credit crunch, the near collapse of the banking system and the economic slump. And it was the Black-Scholes equation that opened up the world of derivatives.

The equation itself wasn't the real problem. It was useful, it was precise, and its limitations were clearly stated. It provided an industry-standard method to assess the likely value of a financial derivative. So derivatives could be traded before they matured. The formula was fine if you used it sensibly and abandoned it when market conditions weren't appropriate. The trouble was its potential for abuse. It allowed derivatives to become commodities that could be traded in their own right. The financial sector called it the Midas Formula and saw it as a recipe for making everything turn to gold. But the markets forgot how the story of King Midas ended.

Black-Scholes underpinned massive economic growth. By 2007, the international financial system was trading derivatives valued at one quadrillion dollars per year. This is 10 times the total worth, adjusted for inflation, of all products made by the world's manufacturing industries over the last century. The downside was the invention of ever-more complex financial instruments whose value and risk were increasingly opaque. So companies hired mathematically talented analysts to develop similar formulas, telling them how much those new instruments were worth and how risky they were. Then, disastrously, they forgot to ask how reliable the answers would be if market conditions changed.

Black and Scholes invented their equation in 1973; Robert Merton supplied extra justification soon after. It applies to the simplest and oldest derivatives: options. There are two main kinds. A put option gives its buyer the right to sell a commodity at a specified time for an agreed price. A call option is similar, but it confers the right to buy instead of sell. The equation provides a systematic way to calculate the value of an option before it matures. Then the option can be sold at any time. The equation was so effective that it won Merton and Scholes the 1997 Nobel prize in economics. (Black had died by then, so he was ineligible.)

If everyone knows the correct value of a derivative and they all agree, how can anyone make money? The formula requires the user to estimate several numerical quantities. But the main way to make money on derivatives is to win your bet – to buy a derivative that can later be sold at a higher price, or matures with a higher value than predicted. The winners get their profit from the losers. In any given year, between 75% and 90% of all options traders lose money. The world's banks lost hundreds of billions when the sub-prime mortgage bubble burst. In the ensuing panic, taxpayers were forced to pick up the bill, but that was politics, not mathematical economics.

The Black-Scholes equation relates the recommended price of the option to four other quantities. Three can be measured directly: time, the price of the asset upon which the option is secured and the risk-free interest rate. This is the theoretical interest that could be earned by an investment with zero risk, such as government bonds. The fourth quantity is the volatility of the asset. This is a measure of how erratically its market value changes. The equation assumes that the asset's volatility remains the same for the lifetime of the option, which need not be correct. Volatility can be estimated by statistical analysis of price movements but it can't be measured in a precise, foolproof way, and estimates may not match reality.

The idea behind many financial models goes back to Louis Bachelier in 1900, who suggested that fluctuations of the stock market can be modelled by a random process known as Brownian motion. At each instant, the price of a stock either increases or decreases, and the model assumes fixed probabilities for these events. They may be equally likely, or one may be more probable than the other. It's like someone standing on a street and repeatedly tossing a coin to decide whether to move a small step forwards or backwards, so they zigzag back and forth erratically. Their position corresponds to the price of the stock, moving up or down at random. The most important statistical features of Brownian motion are its mean and its standard deviation. The mean is the short-term average price, which typically drifts in a specific direction, up or down depending on where the market thinks the stock is going. The standard deviation can be thought of as the average amount by which the price differs from the mean, calculated using a standard statistical formula. For stock prices this is called volatility, and it measures how erratically the price fluctuates. On a graph of price against time, volatility corresponds to how jagged the zigzag movements look.

Black-Scholes implements Bachelier's vision. It does not give the value of the option (the price at which it should be sold or bought) directly. It is what mathematicians call a partial differential equation, expressing the rate of change of the price in terms of the rates at which various other quantities are changing. Fortunately, the equation can be solved to provide a specific formula for the value of a put option, with a similar formula for call options.

The early success of Black-Scholes encouraged the financial sector to develop a host of related equations aimed at different financial instruments. Conventional banks could use these equations to justify loans and trades and assess the likely profits, always keeping an eye open for potential trouble. But less conventional businesses weren't so cautious. Soon, the banks followed them into increasingly speculative ventures.

Any mathematical model of reality relies on simplifications and assumptions. The Black-Scholes equation was based on arbitrage pricing theory, in which both drift and volatility are constant. This assumption is common in financial theory, but it is often false for real markets. The equation also assumes that there are no transaction costs, no limits on short-selling and that money can always be lent and borrowed at a known, fixed, risk-free interest rate. Again, reality is often very different.

When these assumptions are valid, risk is usually low, because large stock market fluctuations should be extremely rare. But on 19 October 1987, Black Monday, the world's stock markets lost more than 20% of their value within a few hours. An event this extreme is virtually impossible under the model's assumptions. In his bestseller The Black Swan, Nassim Nicholas Taleb, an expert in mathematical finance, calls extreme events of this kind black swans. In ancient times, all known swans were white and "black swan" was widely used in the same way we now refer to a flying pig. But in 1697, the Dutch explorer Willem de Vlamingh found masses of black swans on what became known as the Swan River in Australia. So the phrase now refers to an assumption that appears to be grounded in fact, but might at any moment turn out to be wildly mistaken.

Large fluctuations in the stock market are far more common than Brownian motion predicts. The reason is unrealistic assumptions – ignoring potential black swans. But usually the model performed very well, so as time passed and confidence grew, many bankers and traders forgot the model had limitations. They used the equation as a kind of talisman, a bit of mathematical magic to protect them against criticism if anything went wrong.

Banks, hedge funds, and other speculators were soon trading complicated derivatives such as credit default swaps – likened to insuring your neighbour's house against fire – in eye-watering quantities. They were priced and considered to be assets in their own right. That meant they could be used as security for other purchases. As everything got more complicated, the models used to assess value and risk deviated ever further from reality. Somewhere underneath it all was real property, and the markets assumed that property values would keep rising for ever, making these investments risk-free.

The Black-Scholes equation has its roots in mathematical physics, where quantities are infinitely divisible, time flows continuously and variables change smoothly. Such models may not be appropriate to the world of finance. Traditional mathematical economics doesn't always match reality, either, and when it fails, it fails badly. Physicists, mathematicians and economists are therefore looking for better models.

At the forefront of these efforts is complexity science, a new branch ofmathematics that models the market as a collection of individuals interacting according to specified rules. These models reveal the damaging effects of the herd instinct: market traders copy other market traders. Virtually every financial crisis in the last century has been pushed over the edge by the herd instinct. It makes everything go belly-up at the same time. If engineers took that attitude, and one bridge in the world fell down, so would all the others.

By studying ecological systems, it can be shown that instability is common in economic models, mainly because of the poor design of the financial system. The facility to transfer billions at the click of a mouse may allow ever-quicker profits, but it also makes shocks propagate faster.

Was an equation to blame for the financial crash, then? Yes and no. Black-Scholes may have contributed to the crash, but only because it was abused. In any case, the equation was just one ingredient in a rich stew of financial irresponsibility, political ineptitude, perverse incentives and lax regulation.

Despite its supposed expertise, the financial sector performs no better than random guesswork. The stock market has spent 20 years going nowhere. The system is too complex to be run on error-strewn hunches and gut feelings, but current mathematical models don't represent reality adequately. The entire system is poorly understood and dangerously unstable. The world economy desperately needs a radical overhaul and that requires more mathematics, not less. It may be rocket science, but magic it's not.