Advice from Hal Varian to Econ Grad-Students

Wikimedia Commons

From an interesting and challenging article by Hal R. Varian:

In fact, my standard advice to graduate students these days is go to the computer science department and take a class in machine learning.

He gives interesting examples of techniques that can help analyse big data and their relevance for economics. He  explains:

Google has seen 30 trillion URLs, crawls over 20 billion of those a day, and answers 100 billion search queries a month... At Google, for example, I have found that random samples on the order of 0.1 percent work fine for analysis of business data. (p. 3)

And

An important insight from machine learning is that averaging over many small models tends to give better out-of-sample prediction than choosing a single model. p. 24 

An example

In 2006, Netflflix offered a million dollar prize to researchers who could provide the largest improvement to their existing movie recommendation system. The winning submission involved a “complex blending of no fewer than 800 models,” though they also point out that “predictions of good quality can usually be obtained by combining a small number of judiciously chosen methods” (Feuerverger, He, and Khatri 2012). It also turned out that a blend of the best- and second-best submissions outperformed either of them.

Good reading suggestions are in the final summary of the article. 

Economics and Big Data

From an interesting interview with Susan Athey

I think that the data scientists should take a little more economics. That would help; economics puts a lot of emphasis on the conceptual framework. And I also think that economics should be paying a lot more attention to the statistics of big data. 

Right now, economics as a profession has very little market share in the business analysis of this big data. It’s mostly statisticians. We’re just not training our undergraduates to be qualified for these jobs. Even our graduate students, even someone with a Ph.D. from a very good economics department really doesn’t have the right skills to analyze the kinds of data sets that big Internet firms are creating. 

And more

But then they don’t seem to realize that that kind of training is really crucial for being successful if they want to work at companies like Google or Facebook or Microsoft, Yahoo or eBay, Twitter or LinkedIn. It’s very difficult to be influential in those companies if you are not very savvy with statistics. So the old sort of economics undergrad who gets an M.B.A. but doesn’t know a lot of statistics? A few people with that kind of background will be successful at these large tech firms, but they’re going to be handicapped. . .  

The question is, how can economics reach a larger set of people? And, again, why is that important? It’s because the economic intuition helps you ask the right questions of the data, which is extremely important. . . 

I guess one other part of your question was, is big data a fad? It’s not a fad; it’s a fact. Companies in all sorts of different industries are starting to generate large amounts of data. The Internet companies were built from the ground up on that data. Other companies are just starting to think about what they do with the data.

Most universities, and specifically business and economics departments, and specially in developing countries are not aware of the demand of those services, or the potential future demand. A entrepreneurial opportunity right there for first movers.