Trading With Big Data

Trading with big data

Big data and algorithms are the key to making millions, if not billions in revenue! Quite a big statement, however it is easily provable as billions are being made by the high frequency trading algorithms that are using big data. When it comes to making money out of data, David Harding is rather good at it. 30 years ago, he set out to bring data analysis and algorithms to the trading floors of the City of London or Wall Street.

The picture below shows how all trading used to be done. All trading was carried out in rooms full of people like this. They would be shouting the prices that they would buy and sell at and agreeing the deals. The rises and falls in the prices are almost like the rises and falls in the noise level. Today, the London Metals Exchange is the only trading pit of this kind left in Europe. Noisy, emotional and chaotic. To a science graduate from Cambridge, it came as a bit of a surprise. When David went to the City of London, he was expecting it to be the world of banking and high finance, that he had been taught at Cambridge University. He was expecting bodies of knowledge that were organised and rational however it wasn't at all like that. Buying and selling strategy in those days tended to be governed by instinct and intuition. He watched the prices going up and down on the boards and plotted the graphs by hand. Over time, as he followed the graphs, he became convinced that there was a pattern to the rises and falls in prices. David Harding wanted to bring mathematics to the problem but the prevailing wisdom was that this was an impossible task.

You Can Not Beat The Market

According to the financial orthodoxy, the rises and falls in prices that take place here are completely random. Nobody can ever predict them, however clever they are or however much foresight they have. Essentially, cutting to the chase, the idea is that you can not beat the market EVER!

Like all data miners, Harding needed two things, data, a lot of it, and computer algorithms to spot the patterns. As this was 30 years ago, Harding had to develop the tools to analyse it himself. He could program a computer and he could read the data from the new exchange.

David says "Now I could conduct analysis of that data and that, to me, was rather an elementary thing to do. I was surprised that other people hadn't done it first. You'd have thought that, where all the millions and billions are all sloshing around, you'd have thought that lots of rational, intelligent people would have done these sorts of things."

The company David Harding founded 20 years ago now invests billions of pounds on the basis of data. Harding, however, is now far from the only scientist in the City! His company alone employs over 100 scientifically trained data hunters, from astrophysicists to cosmologists, to mathematicians and meteorologists all now have become known as quants.

Wake Up & Smell The Coffee

His company is built around the idea that if you have enough data and the expertise to read it, you can spot trends and links that no-one else has noticed. He and his analysts can seek out patterns in anything that is bought and sold.

Take, for example, coffee. Obviously, they will probably almost certainly sell less coffee on a Sunday. Now that's not a revelation, or that they sell more coffee in winter, because people are indoors more often in winter, but there is an art or a science or a skill which is using the data to find out more interesting things and I'm sure that if their analysts went to work, they could find out much more interesting things than that.

The process begins with data, collecting any information that might be relevant to the cost of coffee. You need specialised tools to interrogate and take decisions about that data, and unlike the past it is not based on a hunch or Chinese whispers. Computer algorithms can then search the data, looking for factors that link to the rises and falls in coffee prices. The yield of coffee bean harvests for example, the strengths of the economies and currencies of coffee-producing countries, as well as consumer demand for coffee. In the vast dataset, tiny significant signals appear and it is these signals which hold the clues to when to sell and when to buy. These trading decisions are based on scientific data, and not just taken on the basis of human feelings and what you may of heard from your friend this morning.

Gathering The Data

It's a strange mathematical and sort of social science, but science, it is. Winton Capital gather data across hundreds of markets going right back in time. Daily metal prices from 1910, food prices dating to the Middle Ages, and London Stock Exchange prices stretching back to 1690. Every day, they collect new data on 28,000 companies across the world. They have data coming in almost 24 hours a day for nearly all the markets that trade, and the last time they looked, they had something like 40 terabytes of data in their database. That's the equivalent of about 70 million King James Bibles. The ambition is that somewhere in this 40 terabytes of data there are patterns that can be used to predict price rises and falls, and you don't need to predict price changes with pinpoint accuracy. The odds just need to be a bit better than even. If you throw a coin and there's a 50/50 chance of it landing heads or tails, then clearly, there's no way of profiting from that. If however, we had the ability to know that heads was going to come up 55% of the time or 60% of the time, then that would be a great investment business. If you have the resources and can make enough investments, spotting even a tiny variation can lead to large profits. Over the last 20 years, this approach has paid handsomely for David Harding.

David states "There's never really a point at which you can relax and sit back and say, "There, I have proved my point!" Of course, you know, over the years the ideas have been successful, the company has grown. It gives me great pride and satisfaction. Of course, investing in financial markets remains a gamble. There is no universal law of finance, stock markets crash, recessions happen, they're clearly not easy to predict. The patterns in the data are constantly shifting and changing. There is no one right answer, every day, week or month, you are being proven wrong by having your ideas put to the test. That is a gift because it enables you to maintain a level of humility that people may, in other situations, lose, and humility is actually a vital ingredient of proper scientific investigation."

The World Run By Nerds

I think most good scientists tend to be quite humble people. The world of finance has been changed forever by the data revolution. The effects have spilled over into everyday life and the data revolution is set to become even more personal.

Bill Gates once said that "if I had been born at any other point in history I would have been sabre-toothed tiger food!"

Well, looking at how information technology has changed all our lives, and the massive amounts of money I.T. has made, I always liked this one.. what do you call a nerd in 20 years time? ... The answer is "Boss," ;)