Working on Big Data in business? Hire a physicist

In a post back in November, Big Data was outlined and explored based on an event delivered by Xconomy in Boston. 

A theme that I recognized during that event was that Big Data challenges require resources that are scarce in the marketplace.  From the ability to write the code required to deal with these huge data sets, to data scientists that will be charged with making sense of the data and helping to derive value from it, specialized professionals will need to work alongside business analysis, program and project managers to deliver value within standard business operating cycles (annual budgets, processes, etc.)

The McKinsey Global Institute, in a comprehensive paper from June 2011 Big data: The next frontier for innovation, competition, and productivity, put forth that a significant shortage of talent will constrain the derivation of value from big data: http://www.mckinsey.com/insights/mgi/research/technology_and_innovation/big_data_the_next_frontier_for_innovation

A recent WSJ blog post, Data Scientists Will Unlock Big Data’s Promise included some interesting info about one type of the specialized resources, data scientists, “Data scientists should have a hybrid set of skills: the IT skills that are necessary to deal with and analyze vast amounts of data; and the subject matter skills needed to know which valuable business insights can be extracted from the data, and how to best frame the questions and build the right model that will reveal these insights.”

Focus on the science aspect regarding this burgeoning business objective highlights the scientific method we all remember learning in school.  Some of the key components include systematic observation, gathering information, and empirical testing or measurement. 

Also cited in this piece was What is data science?* by Mike Loukides of O’Reilly Media.  According to the experts interviewed for that article “the best data scientists tend to be physicists and other scientists” with some very logical and compelling reasoning (see the entire WSJ post that cites Loukides article)

http://blogs.wsj.com/cio/2013/01/05/data-scientists-will-unlock-big-datas-promise/

The reason this, in particular, is compelling is that I’ve worked with some highly intelligent and capable people that were absolutely interdisciplinary and versatile in my 15 years of professional experience, but I’ve never work next to, or in the same building, as far as I know, as a physicist.  Now I know this is not a call to all physicists to abandon their pursuits to come solve big data challenges, but I don’t believe that business, in general, has embraced that classic scientific brains and methods are needed when attacking big data objectives and challenges.

Maybe the next personal development task I take on will be to enroll in a Physics 101 class.

*What is data science? http://radar.oreilly.com/2010/06/what-is-data-science.html