Bob Plumridge

Finding the skills to ride the Big Data wave

Blog Post created by Bob Plumridge Employee on Aug 1, 2016

The Big Data wave is coming. By 2024, the world's enterprise servers will process every year the digital equivalent of a stack of books that reaches more than 4.37 light-years … all the way to Alpha Centauri, our closest neighbouring star system in the Milky Way – and that’s probably a conservative forecast by now.


Data volumes have grown faster than anybody could have projected, also driving the need for smart, easy-to-access storage. Given the volume and the velocity of data, the business need of analysing it can seem overwhelming. But companies cannot dodge the task, unless they want to risk losing out to peers, who could suddenly gain a massive competitive advantage.


To put it into perspective: Imagine you’re a manufacturer of metal fastenings and your competitor used data analysis to figure out how to produce the same widget for less money. You’d haemorrhage customers pretty quickly. One can think of similar scenarios for almost every organisation in any sector.


Big Data is set to make a massive impact on business. This digital transformation has been compared to an epic wave that is starting to crest. If you want to catch this wave, you need the right people to surf. And if you have to sit it out because of a lack of talent, you’ll miss the wave altogether.


Big data 2 new.png


But implementing Big Data projects can be extremely challenging. To analyse the data, a company has to cope not only with an every growing number of data sets and the real-time flow of information, but also needs to have teams with the right skills to deliver an analysis that is based on the right questions and provide answers that make sense.


So in a world where Big Data analytics is becoming ever more important, companies and organisations need to attract and cultivate a workforce with the skills that gives them a competitive edge. One of the new job descriptions in this Big Data world is that of a “data scientist” – a rare and mythical breed.


Mythical, because it’s a new discipline that requires a special blend of experience and knowledge. A while back, one of the more famous of these mythical beings, Slack’s Josh Wills, gave a talk at Airbnb. He showed a couple of Tweets that answered the question “What is a data scientist?” One suggested: "A data scientist is someone who is better at statistics than any software engineer and better at software engineering than any statistician". However, finding professionals that meet the market’s [unrealistic] expectations can be a tricky undertaking. So much so that the industry refers to them as “unicorns”.


A McKinsey study notes that in the United States alone, organisations could face a shortage of 140,000 to 190,000 people with deep analytical skills by 2018, which is less than two years away. European businesses have similarly felt hamstrung by this skills gap.


So how can businesses attract people that have the right skills? Firstly, look at your existing team. They may not be data scientists, but the potential may be right there. Give them some training, combine their knowledge and skills, and you can build a team ready to lead you into the Big Data world.


Drew Conway, another data scientist and author of books on machine learning, six years ago defined a data scientist as a Venn diagram of computer science, statistics and domain expertise. Asked recently to update this definition, Conway said he would include communications as a stand-alone set of required skills, given the importance of conveying technically rigorous analysis in a clear and actionable way.


The importance of combining data analytics with good communication skills become obvious when we think back to the Space Shuttle Challenger disaster. Big Data expert Bernard Marr reminds us that NASA was dealing with an overwhelming amount of data at the time, which showed the significant risk of the shuttle breaking up. However, because of poor communication the mission controllers were unable to spot this fact in the mass of irrelevant data. Marr speculates that events might have unfolded very differently, if the experts looking at the data had given management an analysis with the title “The shuttle is likely to crash because …”.

In addition to communication skills, data scientists also need to have robust business skills. Hidden solutions to business challenges cannot be uncovered without deep knowledge of the industry and the business, combined with contextual understanding and healthy scepticism of existing assumptions. These skills will enable data scientists to discern multiple data links affecting the business, identify relevant data sources and formats, and develop methods for interrogating data, with as few searches as possible.


In many respects, for a company that wants to succeed in Big Data analytics, it needs to prioritise data scientists with the right business skills over those who may have a proficiency to write cutting edge software. After all, thanks to analytical platforms like Pentaho, most of the complex technical engineering work can be abstracted. It means Big Data teams need fewer coding skills, but more business and data knowledge. Data can be queried with just a few drags and drops, which dramatically narrows the skills gap that needs to be filled, and makes it easier for companies to turn insights into action.


Enterprises can train people up; as long as they have a nous for business, a background in maths and an affinity with data. There are a number of industry-run boot-camps they can enrol in and classroom and online learning courses, and we at Hitachi Data Systems help companies pull together the right teams for the job.


Naturally, the best skills are acquired in the real world.  But a lack of experience can be mitigated by encouraging a chosen few to participate in hackathons and giving them small, controlled problems in the business that can be tackled through data insight.


By creating small working groups of people with a mixture of complementary skills, you can cultivate your company’s ability to interrogate data successfully. A skills shortage is no reason for a business to exclude itself from the Big Data opportunity.


If you want to find out more about Big Data, then a good starting point would be my webinar on the power of digital transformation, available online now.