The science behind the consumer

Miguel Lobo, a professor at Insead Abu Dhabi, explains how companies use 'big data' to decide what kind of choices and how many of them to offer would-be customers.

Miguel Lobo reveals how businesses collect and analyse shopping activity to present the consumer with recommendations that maximise the chances of a purchase. Delores Johnson / The National

Making a purchase over the internet may feel like a simple, sometimes impulsive process - but for Miguel Lobo it is the stuff of science.

He is an associate professor of decision sciences at Insead Abu Dhabi, and is interested in the psychology of what people buy and how the variety of options offered by retailers may sway decisions.

He helped the world's biggest internet companies to track shopping habits by analysing millions of clicks on e-commerce websites.

E-commerce in the Middle East is still tiny, worth an estimated US$281.6 million (Dh1.03 billion) last year, compared with China, where the figure was more than $64bn, according to Euromonitor.

But as the industry grows, academics are looking at how retail websites can track shopping habits and recommend appropriate products to help boost their revenues.

Mr Lobo explained why businesses choose the products they recommended to customers so carefully.

Companies seem to have become much more interested in how consumers decide what to buy. Is this a new field?

People have been looking at those questions for at least 50 years in a very systematic way. But it is really over the past, let's say, 15 years that there has been a huge explosion in more systematic research in the field, both in volume and in sophistication.

And what are the current trends?

Big data is now one of the buzz words. [It's about] how we make use of the gigantic amount of data we have on consumer behaviour. Retailers have been doing this for a little while. With Amazon's [automatic product recommendations] you will see two types of products - those that are targeted for you because they already know something about you, [and] every once in a while you see something that is a little bit off. The things that are a little bit off typically are experiments. They lose a little bit of efficiency, because maybe 10 per cent of its effort is dedicated to trying out new stuff and experimenting with what works with whom. But in the long run they develop an enormous effectiveness because they really understand [their customers].

What has your field of study taught us about consumer behaviour?

One set of findings looks at people's responses when they are given fewer or more choices. No rational model says that if I give you extra choices you will be in any way worse off. [But there are] very strong findings that say people are a lot less happy if they are given a lot more choices. One experiment in Palo Alto had this promotion for a jam company. For half of the days, there were three jars of jam that people could taste. If they did the tasting they would get a coupon which they could redeem to buy some jam at a discounted price. On the other half of the days, the full collection of the jam was displayed, which was about 20 different flavours. The question was, at what rate did people redeem the coupon?

What happened?

On the days where there were only the three jams to taste, [about] 70 per cent redeemed the coupon. And of the ones who had 20 to choose from, only 15 per cent or something like that redeemed the coupon. The fact that they were given a lot of choice made their experience and the likelihood that they would act on that experience and that promotion drop dramatically. This finding has been replicated in a lot of situations.

So why do Amazon and other shopping sites usually make lots of recommendations to visitors?

They try very hard to do the targeting. That's the trick. You don't want to harangue people with choices. But on the other hand you don't want to give them the wrong options because there is such a lot of heterogeneity in preferences. They need to give people choices but they need to be targeted. That's what big data is about. You keep the range of choices there, but I do the narrowing of the choices for you so that you have a set of choices that are likely to be the ones that you find more attractive.

Is this something all big retailers do?

Absolutely. It is an industry where choice is just everything. It is [also] a very competitive industry with a lot of competitive pressure and small margins, so they have become good at it.

Do websites such as Amazon track all visitors' habits? Or are they only tracking us if we buy something?

It is absolutely everything. I did this for a living about a year after I finished my PhD at Stanford. I was with a company at the time that was doing real-time optimisation for electronic merchandising. Whenever you went to one of the big internet portals and clicked on it [our software] had 40 milliseconds to say what products to show to [the consumer]. We had information on everything: how long you spent on different pages; what products you open the detail description of; what products you put in your shopping cart but didn't purchase; and what products you put in your shopping cart, started the payment and pressed cancel.

And this is all so that next time the consumers visit the website, they might actually make a purchase?

Yes, of course. The [retailer] wants to sell more. They would rather you buy from them as opposed to someone else. They want to make sure you have a better experience that you don't go there and are shown things you are not interested in, that you are not shown things that make you uncomfortable. These algorithms require a bit of sophistication because there are all sorts of mistakes that companies made in the beginning.

Such as?

I remember talking to the guys at Netflix 10 years ago. They were doing movie recommendations based on people who had similar preferences. At the time, their customers were mostly San Francisco Bay area, northern California, because they hadn't expanded. There are a lot of highly educated Indian engineers working in Silicon Valley. They found out that people liked sophisticated highbrow independent cinema, so if you watched some of those movies you were categorised with people with similar preferences - and you had a bunch of Bollywood movies recommended to you, because you had similar preferences to these highly educated Indian engineers. Eventually you realise that you have to develop the algorithms in a more sophisticated way, to make sure that if there is no signal that you like Bollywood, I will be careful. So I may show you just one and see if there is an uptake, rather than just classify you right away in a cluster of people.

But human behaviour is unpredictable, so does this sort of stuff actually work?

Human behaviour is very unpredictable, but often it is unpredictable in predictable ways. There are all sorts of patterns we can predict.