Skip navigation
1 2 Previous Next

Social Innovation

16 posts

Let’s play the Big Data word association game.

Every day we hear about the incredible insights that big data can deliver, but with recent reports on the skills shortages of data scientist, big data projects not realising returns  and  Gartner proposing that companies will be measured on the algorithms they develop … is it worth re-visiting our approach?

A friend of mine in China who started his own successful analytics company once told  me, “It’s not how big the data is, it’s whether you can extract value”. The reality is business insight comes from a team of people and not all require a data scientist on day one (if at all). Check out the recent Forbes article on this point "Big Data, Little Data. What Do you really need?"

“It’s not how big the data is, it’s whether you can extract value”

So where do you start? One way is to launch an initiative to develop an analytics culture and create a virtual Business Analytics Team. This will provide the focal point and governance where business problems can be submitted and evaluated.  You may end up with multiple virtual teams tackling different problems but the core team may consist of;


  • A Data Engineer –  you probably have someone in your organisation who has this knowledge today who knows the data sources and/or is responsible for data management
  • Subject Matter Expert(s) – someone who can articulate the business problems and also knows what question to ask
  • A Data Analysts (or business analyst)
  • A Project Manager
  • A Data Scientist (optional) who can build the algorithms if required

Of course, you will need technology, but the single most important thing here is don’t overthink it – keep it simple, manageable and integrated!!

Gaining business insight starts with defining the  business problem and/or leveraging known use cases or Blueprints – what are we trying to solve? Then one of the first questions to ask is  “Can Analytics help?”

“Can Analytics help?”

When it comes to Analytics our philosophy at Hitachi Data Systems is: “Any Analytics, Any Data, Simplified.” We use and offer Pentaho (a Hitachi Group Company) to provide a single integrated software  platform to blend, analyse, visualise, explore, report and predict on any data – we keep it simple!

And should you require a high performance analytics appliance that simplifies the compute, data management and analytics functions, our Scaleout Platform (HSP) with Pentaho Data Integration provides rapid deployment of an integrated big data platform – a one stop shop!

The Big Data landscape appears incredibly complex, but remember,  don’t overthink it…Keep it Simple and focus back on the business problem. As Steve Jobs said: “Simple can be harder than complex;  you have to work hard to get your thinking clean to make it simple.”

With an Analytics team in place to evaluate business problems in combination with a simple integrated analytics platform, let’s play our word association game again.

We now have a repeatable process that can be reduced to:

So don’t be concerned about skills shortages or failures – make a start now by developing an analytics culture, keep it simple at first and build upon your success as skills become more widely available. If you need help on this journey or need to simplify your existing analytics platforms for greater ROI, then Get in touch with me.

Finally, Share how your company is structured to leverage analytics or Big Data initiatives. What do you think about building an analytics culture, or implementing business analytics teams? How are you leveraging data scientists within the business? Here is one example of deploying Data Scientist in operational roles.  I look forward to your thoughts on practical implementations that have overcome Big Data disillusionment.


We hear and talk a lot about the amount of change today. Almost to the point that it feels like buzz with certain catch phrases: Accelerating pace of change, ever faster evolution; transformation; explosion; increasingly [fill in the blank]; and the like.  But there’s a reason for that: It’s true and we have a lot to figure out that requires different, more holistic methods to solve pressing challenges and emerging opportunities.



Frost & Sullivan analysts have been researching some of the larger mega-trends for some time and we’ve published several summaries about how this applies to different industries.  Most recently, you can read an overview of their paper, Social Innovation: Driving Meaningful Improvement Across Industries.  They call out several statistics that make it clear we need to alter the way we do things – just to keep up. 


But here’s the thing: None of us wants to just barely keep up.  Whether we’re talking about advancing healthcare or competitive business advantages, I want us to thrive and that means significantly altering our approach.  So the question is, do you just keep up or transform to thrive?


I believe Social Innovation is the way we’re going to thrive and that technology is accelerating some changes and certainly our ability to respond to changes. It’s why HDS use big data analytics and the Internet of things that matter to create Social Innovation solutions for safer, smarter, and healthier societies. 


In their video below, Frost & Sullivan summarizes the ideas of their paper but also includes unique statistics and information. For example, the video calls out Hitachi Live Insight Center of Excellence as an example of the kind of comprehensive methods needed for Social Innovation.


I’ll let my team add their thoughts and links to more of the industry specific examples.  However, I think it’s clear that our societies, businesses, and governments are transforming.  And we all get to decide how we’re going to approach these changes.

On Thursday, November 12, 2015, Hitachi Data Systems hosted the Silicon Valley Leadership Group’s “Grid of the Future Summit” event at our Santa Clara headquarters. The Grid of the Future Summit engaged energy companies, technology companies, and government agencies in a panel discussion detailing the roadmap for the direction of the distributed, data-driven grid of the future. The event attracted over 100 attendees – including individuals from PG&E, Solar City, SUNPOWER, Chargepoint, Nrg, enLight, many Bay Area cities, Silicon Valley Power, and Frontier Renewables. .


The Grid of the Future Summit event lasted throughout the day, where guests were able to participate in four plenary discussions, two breakout sessions, a Keynote session with Dan Kammen from UC Berkeley, and an evening reception.Through hosting this event, HDS was able to demonstrate its vision and strategy for Social Innovation in Energy and SmartGrid. Click Here for more information on the Silicon Valley Leadership Group.

IMG_4623.JPGIMG_20151112_094811.jpgIMG_20151112_125220.jpgIMG_4632 (1).JPGIMG_4636 (1).JPG

Beyond the IoT network, what is also growing exponentially is the option for new and unusual instrumentation at the end devices. Of course, we still have smart nodes that perform specific functions, like contact closures or firing off simplistic serial command strings, but we now have a huge array of microcomputers, sensors, and specialized devices that perform a myriad of applications. These new devices can be installed at the network’s edge to greatly enhance the capabilities of the applications by providing new data inputs for better decision-making.

IoT has some very different attributes, including:

  • Federated architecture
  • Peer-to-peer networks
  • Client / Server networks
  • Decentralized intelligence
  • Low data rate devices
  • Low power devices
  • On demand computing with device sleep dwell times
  • Real-time and near real-time communications
  • A varied and vast array of sensors for monitoring, control, telemetry, and diagnostics

It is crucial for  us to take advantage of the federated aspect of the network and therefore we are seeing the application intelligence pushed out to the network’s edge too. The federated model has a layered approach that can "make and break" as required to combine and disband layers and therefore functions and features of the network.



By distributing the network intelligence over the end to end network topology, adding fresh capabilities and supplementing the existing functions is key element in  IoT solution design. Telco carriers, in particular, have shown a strong interest in this area and view it as a large potential opportunity for future revenue growth. Verizon, for example, has announced that it already has generated $600 million in IOT revenues and it expects the category grow at an impressive 45% annual rate. Similarly, AT&T has talked extensively about its M2M and other IOT applications.

The vast majority of the efforts by major carriers have been for industrial purposes, including fleet management for the transportation industry, package tracking, industrial equipment monitoring, building HVAC system monitoring, digital signage, and many other applications. Another IOT solution for consumers is connected home services, such as what AT&T is offering with Digital Life. The challenge here is the investment necessary to put together a complete suite of hardware products for both home security and home automation and, most importantly, the training and staffing necessary for the numerous truck rolls, installations, repairs, etc. This business is likely better suited for cable providers and other companies who already have these types of assets. In addition, it’s highly dependent on broadband services and would need to be tied to those broadband services to succeed.

While it’s theoretically possible to put together a service designed for smart home do-it-yourselfers, who buy and install their own equipment, the enormous technical and logistical challenges of achieving this vision seem difficult to overcome. In addition, the ongoing (and soon to get worse) standards battles for home control combined with the poor quality of many of the current smart home products, make this an unlikely scenario for several years. Finally, the few reasonably interesting home automation products now available do not require cellular connections, but use WiFi and a local broadband connections.

  Telecom operators have key assets to capture a significant part of the smart home value and need to place their bets now and promote open platforms.

Thanks to their broadband Internet gateways, telecom operators are the leading players in terms of penetration of households with smart home solutions. The broadband box has evolved into a highly innovative platform connecting various devices. In addition to that, telecom operators offer interoperable solutions based on open models that can allow heterogeneous smart home solutions to interconnect, contrary to closed OTT ecosystems.

Another significant asset of telecom operators in the smart home environment is the central role they play in the customer relationship, which can enable them to capture a great value of future smart home services. Other key assets that telcos could leverage include their sales force, shop network and support capacity, as well as network management capabilities.

As the smart home market is still emerging, two possible market structures seems a reality. The smart home market is captured and aggregated into large ecosystems driven by global players. This model is clearly favored by OTT players, which have already developed solutions that can bypass the operators. Telecom operators are forming alliances when available to promote hybrid smart home platforms with applications close to their core .business and also open to third parties.


The smart home market is composed of four major segments sub-divided into  6  classes:

  • Home automation: The centralization of five main home systems on a unique user interface: home security, home energy and utility management, home motorization, lighting and entertainment. Arthur D. Little forecasts a 6% annual growth rate for these services until 2020.
  • Smart home assistance: The configuration, maintenance, repair and support services available for digital home devices, such as PCs, TVs, game consoles and networks. This market is expected to grow at a pace of 5% per year until 2020.
  • Home cloud: Services covering three main types of digital data: content, productivity and sensors. The market is expected to grow at a strong pace of 50% per year.
  • E-health: An application of telecommunication technologies in the health sector, which offers a unique cost control lever for health stakeholders by dematerializing some healthcare components.
  • Disruptions with  IoT –M2M integrations with  embedded  domain applications give a bird sight by  industries.



Needless to say -  Interesting times Ahead.  Looking forward to  how  Hitachi's IoT strategy comes out .


Today we announced advancements in predictive policing that will help make our cities healthier, safer, and smarter places to be. In particular, Hitachi Video ManagementPlatform provides a comprehensive, smart environment for ALL video and Hitachi Visualization Suite 4.5 has recently enhanced predictive analytics. Together these solutions facilitate improved public safety and include the first predictive policing tool that uses natural language. That means that data sources like social media networks can feed important information in real time to be evaluated in more naturual ways for more accurate and timely predictions. 

Although many characterize all these types solutions as "Minority Report" comes true stories (yes, I am a Phillip K. Dick fan), that goes too far and misses the point. These solutions help anticipate critical policing needs and effectively deploy resources. And isn't that what we want our public safety organizations to do with the tools at hand?  Public safety is a complicated, important topic and I'm glad HDS is providing solutions that help us make better decisions, faster.  Nancy Jones and Bjorn Andersson what do you think?

You can read more about our public safety solutions online, or if you happen to be at the

ASIS International Annual Seminar and Exhibit in Anaheim, California, visit Booth #975 for hands-on demonstrations and more information.

This post was first Published on LinkedIn Pulse.


At the end of July,  I had the good fortune to attend TECNA's annual conference. TECNA is the Technology Councils of North America. It is a group of about 60 technology associations and councils in North America. There are 20,000 companies represented in their membership. Most of which have CIOs and other C-Suite executives on the councils board of directors.

This year it was hosted in Kansas City, Missouri by Ryan Weber and the Technology Council of Greater Kansas City. The role of these tech councils is to advance the technology industry in each of the individual cities and states. They do this through collaboration with it members and local communities, as well as education and inspiration through many ways including annual awards events for the local tech companies and leaders.

These days all companies are tech companies. Even Domino's Pizza. That's a technology company that sells pizza. If companies are not leveraging technology to be more successful they will surely fail. The same goes for our local cities and communities, or should I say Smart Cities will be successful.

The last day of the conference was very educational because the general session was on Smart Cities. It was a panel discussion lead by Ryan Weber and included Fred Ellermeier from Black & Veatch, Kevin McGinnis from Pinsight Media+, and Munish Khetrapal from Cisco. A very interactive session with everyone taking copious notes. One of the more interesting comments was that"$3 Trillion in incremental value will be realized by digitizing cities".

That leads us to why I was there. Hitachi Data Systems (HDS) has been a sponsor of this wonderful event for 3 years. We are also very active with several of the councils and associations in North America. This year was especially interesting to be there because of the many discussions as to what Hitachi Data Systems is doing around Smart Cities, and in a broader sense what we are doing around Social Innovation. The work we are doing around Public Safety was of strong interest. From those discussions we will be partnering with many of the councils in our upcoming Public Safety events. We'll discuss the local community members we are working with and the HDS solutions we are using to make cities safer and smarter, like we have done with the Austin, Texas Police Department.

For our cities to be successful it will be through collaboration with their local technology councils, education from technology vendors, and thought leadership and inspiration from local community leadership.

See you at TECNA 2016 in Portland, Oregon hosted by Skip Newberry and the Technology Association of Oregon.  I am looking forward to hear about even more ways these associations and councils are helping their local communities be successful.

John Meyer (@JohndMeyer)

So we've all been there.  You've got a great idea – no, a FANTASTIC idea –  and some general support but it's just not moving ahead.  This Wednesday, Aug26, Larry Geers will be hold a public talk on just that: How to advance your big data strategy

Larry is a Director for our Hitachi Live Insight Center of Excellence which brings together Social Innovation solutions, big data analytics and industry expertise, and implementation teams to help organizations better realize their ideas.

We'll be looking at the below key phases in an analytics journey with emphasis on how to navigate the early stages. In particular, Larry has tips to avoid common pitfalls such as misjudging the interdependencies of today's analytics solutions; focusing on traditional stakeholders; and underestimating the resources for and importance of a well-planned proof of concept.

Register and join us on Wednesday for a discussion about how to move forward with your big data plans and your Social Innovation solutions!




We all know the massive flood of information is transforming the way we live and work.  It provides both challenges and opportunities and is part of the mega-trends propelling Social Innovation and our strategy around it. HDS recently sponsored a Frost and Sullivan paper on the Social Innovation Journey looking at the path forward using 3 industry examples: healthcare, telecom, and smart city.  (Special thanks to @roberta_gamble @Frost_Sullivan for the excellent thought leadership on this content!)


This paper describes how companies are learning to view the influx of data to improve all aspects of the organization. And their overview of the 5 steps to harness big data for Social Innovation, provide general guidance on how to get started and how to progress.


Over this coming week, I’ll comment on this posting with the next step each day.  Let us know if you think these are the right steps to help organizations respond to these complexities and move forward in Social Innovation.  You can comment in-line or reply to my daily tweets on this topic @amyhodler

Today, I’m picking up my thread of “Why is Social Innovation important?” to focus in on a couple of more areas, namely public safety and infrastructure for societies.


Many of you may remember the US TV show “Numb3rs,” a crime drama set in Los Angeles. The characters solved crimes by using math to analyze various kinds of available data. It was an early application of math and data analysis for public safety. However, it was “Hollywood” so you didn’t really know how much of it was real.


Screen Shot 2015-06-22 at 4.14.42 PM.png


Today however, the use of numerical Big Data analysis to enhance public safety is very much a reality. For example, several mathematical algorithms now in use for predictive policing look at historical data  - such as what type of crimes have been committed, and in which areas. 


These algorithms also include such factors as proximity to paydays, type and locations of sporting events and weather (Is it raining? Thieves don’t like to get wet). The output is a statistical prediction that can be used to pre-position resources and take other proactive actions to avoid problems.


Some of the pioneering work to apply math to this type of problem and to develop this technology was done at UCLA (maybe there’s a connection to “Numb3rs”?) and this is now in use in many cities across the country and the globe.

Screen Shot 2015-06-22 at 4.17.24 PM.png

Several cities have deployed these type of systems and some have reported a reduction in crime rates as a result. One example is the city of Atlanta, GA, which claims a reduction of 8-9% in burglary, car theft, and robbery in a comparative study. Other cities were able to reduce costs in law enforcement resources  The city of Shreveport, LA has reported a more efficient use of available resources after deploying predictive policing techniques. They showed they were able to reduce costs between $13,000 and $20,000 in control districts in a study


Being smarter about how we use our resources is a must going forward. Predictive policing is one tool of several to be more proactive. More collaboration across departments and organizations, such as in public / private partnerships is another tool to leverage resources and at the same time provide better access and higher service levels. Hitachi Visualization plays a key role in many of these scenarios.


Why does this make Social Innovation important?

First, keep in mind what I discussed in my last post; how the population increase and the rapid urbanization put additional stress on all the systems upon which we rely. An example of that is the increased need and focus on public safety that needs to be addressed with smaller or, best case, unchanged budgets.


By thinking differently and applying new techniques, like what was shown in “Numb3rs”, we can stretch the budget dollars and still provide increased service and safety. What makes it possible to do this now is the ability to combine advanced analytics of historical and real-time data with connected devices and their data. Examples of data produced by connected devices include video, license plate readers, traffic sensors, weather sensors, tweets by cell phones, etc.


Hitachi defines Social Innovation as the combination of Big Data analytics, the Internet of Things (IoT) and the deep knowledge about a problem we’re trying to solve – in this case providing tools to enable higher levels of Public Safety. Hitachi Visualization is in the middle of this movement, providing the ability to visualize and correlate the various data sources that we now can tap into, providing a new level of situational awareness and ability to match the response to the need in live situations.


Now, what about Infrastructure?

Here in the US and similarly in many other geographical areas across the globe, we’ve been using up the excess capacity that we’ve had in our infrastructure (transportation, water, power, communication, etc.) without preparing for the future. We simply haven’t invested enough to keep up with changing requirements and maintaining that infrastructure. We now face the situation of either finding ways to be smarter about using what we have (and allocating resources to properly maintain it) or investing heavily to build new infrastructure – or both.


Screen Shot 2015-06-22 at 4.19.42 PM.png

The Organization for Economic Co-operation and Development (OECD) estimates more than US$40 trillion could be spent worldwide on infrastructure projects before 2030. The Economist states that US$57 trillion is needed – in both estimates it’s an enormous investment. [sources: KPMG and The Economist]

Going old school

Let’s look at an example. I’m going really “old school” here by taking a look at the Panama Canal and the impact it had, and continues to have, on worldwide shipping of goods.


It might not look it on the surface, but it’s actually a good example of needing to be very smart about using the infrastructure capacity that was built a long time ago with some space to grow - that is now running out. It’s also a good example of eventually needing substantial investments to upgrade the infrastructure – which is happening now.


Originally opened in 1914, the Panama Canal is now close to its physical and theoretical maximum in terms of providing shipping lane capacity between the Atlantic and the Pacific. A majority of the recent growth is driven by the US – China trade, reaching the Atlantic ports in the US. This has led to an overall doubling the tonnage between 2005 and 2025. [source: Wikipedia]


There are often many days delay on each side for ships to line up to get through the canal. There are now bidding wars with hundreds of thousands of dollars per ship to get premium treatment and be able to jump the line. [Source: Popular Mechanics]


This all translates to delays and extra cost that gets added to the transportation cost, which eventually hits the consumer of those goods. Also, the congestion here causes noticeable impact on businesses worldwide and is limiting growth in many cases.



Getting smarter

There’s a lot of focus on maximizing the traffic through the canal as it is today; tracking and scheduling shipping traffic the best way possible, using every ounce of capacity. You can take a live look here at shipping traffic and congestion in the Panama Canal area. You can also view information on how ships are tracked using Automatic Identification System (AIS).



Ships have long been designed to exactly fit into the canal lock chambers almost like ice cubes in an ice cube tray, ships designed this way are called Panamax ships. These ships are the current standards, and are designed to maximize the amount of cargo in each step.

Screen Shot 2015-06-22 at 4.21.32 PM.png


Screen Shot 2015-06-22 at 4.22.31 PM.png


Even with the use of advanced scheduling tools, it’s simply not physically possible to push much more through the existing canal. However, increasingly, global shippers have been building bigger vessels, so called post-Panamax ships. These do not fit the canal today and have to load and unload in ports on either side of the American continent and utilize e.g. rail transport to move the shipping containers to the other side. For example going between East and West coast of USA.


Moving to post-Panamax ships without changes to the canal is a way to partially get around the problem, but is not solving it.

(Pictures courtesy of )


“Bandwidth” upgrade needed

There’s now an active project that is forecasted to be finished in April 2016, which will greatly enhance the capacity of the Panama Canal – approximately doubling the capacity (or “bandwidth” in IT terms).  This upgrade essentially adds new sets of locks facing both oceans that also are able to accommodate the larger ships.


This is the type of investment in new capacity that will eventually be needed across a wide range of infrastructure types.


When this “new” Panama Canal opens in April 2016, the “bandwidth” between the Atlantic and the Pacific through the canal will have received the doubling we’ve been so accustomed to in the computer industry – but on a much different time scale and, for that matter, physical scale.


Public Safety and major infrastructure projects are  two examples of the thinking that goes into Social Innovation, and the impact it can have on our daily lives. For those of us in the high tech industry, we have to get used to thinking way beyond the limits of the IT world.

Two weeks ago we announced the next phase in our Social Innovation business strategy with new solutions and services. If you haven't followed that news, ESG summarized their takeaways from our Connect event in a fun video blog, When you think of Hitachi...

A key part of this announcement included the formal concentration of resources, tools, and experts to help customers accelerate Social Innovation solutions: this single point of coordination is Hitachi Live Insight Center of Excellence.  And as you might imagine, one of the first questions I received was, "Why do we need a center of excellence? After all, HDS already has many infrastructure and analytics services and consulting offerings." 

Blog May Analytics Change.png

There are really two questions here and the first is around why this is needed now. Of course this directly supports our Social Innovation strategy but in addition, analytics solutions have evolved in a way that requires more than just extra services to address: simply put, analytics have become extensively complex, requiring a vast mix of skill sets, integration, and orchestration. And this has created a chasm between the many great ideas around the Internet of Things and the ability to make those ideas a reality.  In particular, there are phases of challenges organizations battle through as they move forward on both technical and business fronts. These obstacles range from poor data quality and massive integration challenges to scoping opportunities and political barriers. A few weeks ago I summarized some of these challenges in another post on Eliminating Obstacles to Social Innovation.



Understanding your obstacles are essential because a vision of what’s needed to succeed isn’t enough if you can’t bring those ideas to fruition. And to do this effectively with today's analytics challenges requires a breadth of expertise and resources not usually found within a single organization. This is why holistic analytics planning and proven teams are so important right now and why we created Hitachi Live Insight Center of Excellence.


The second implied question is focused on what's new compared to traditional HDS services. Complex analytics solutions require more than a la carte service offerings to bring together and coordinate the people, processes, and technology. Hitachi Live Insight Center of Excellence provides a single point of orchestration beyond our traditional services with:

  • Solutions, services, and consulting for Social Innovation and big data analytics
  • Purpose-built platforms and technology
  • Analytics and industry experts from HDS, HCC, Hitachi Ltd., Hitachi Big Data Lab, and Partners
  • Strategic and operational consulting for big data
  • Best practices and methodology for analytics development, implementation, and operation
  • System integration
  • Overall project management and coordination involving various Hitachi and third party groups


Organizations need to be able to confidently rely on data-driven decisions and holistic analytics to best capitalize on IoT and big data.  I believe more comprehensive approaches with a breadth of proven experience, like Hitachi Live Insight Center of Excellence, is key to crossing the analytics chasm.

The merging of Information Technology (IT) and operational technology (OT) is a trend that we at Hitachi Data Systems have personally experienced for the last decade.  Because we are part of Hitachi Ltd., we’ve been able to access the machine data from the various industrial equipment and sensors that we build, and combined with our data center expertise, created holistic solutions that address an array of challenges in a multitude of industries. These solutions use Big Data and the Internet of Things “that matter” to achieve the ultimate outcome: improve how we live. Our social innovation strategy is a testament to this vision as we continue to deliver solutions that make society safer, smarter, and healthier.


For the Internet of Things to make an impact, we have to be able to thoroughly analyze all the “things” or machines that live in our own wheelhouse, the data center.  Enterprises are constantly pressured to identify root causes of, and appropriately remediate, IT outages. These disruptions can be traced back to issues related to servers, storage elements, applications, or sensors, among others but finding the actual cause can take hours if not days.  Since these machines generate tremendous volumes of structured and unstructured data, collecting them systematically and analyzing them quickly can provide IT teams with new operational insights that help them to preemptively and proactively respond to issues before they impact the business. Our goal of providing greater IT operational intelligence led to a strategic partnership with cloud-native, machine data analytics leader Sumo Logic.  Together, we are delivering Hitachi Live Insight for IT Operations, a solution that enables enterprises to transform their machine data into actionable insights. We chose Sumo Logic not only because they are a nimble software-as-a-service (SaaS) partner which allowed us to get a solution to market quickly, but also because they have the most secure, and elastically scalable platform that allowed us to address the primary pain point of our data center customers: improving performance and availability.


Sumo Logic is the perfect complement to our enterprise data center domain expertise and data science-driven analytics capabilities. We share the same vision for machine data analytics – one that uses a cloud delivery model with the elasticity that our customers need to ingest, correlate, visualize and analyze all their machine data.  With this cloud analytics model, we help customers reduce total cost of ownership, and at the same time, give our customers the ability to scale up and down to match their needs to analyze various amounts of machine data.   Best of all, no additional hardware is needed so we’re reducing capital expenditures.  The value of Hitachi Live Insight for IT Operations is in the entire solution ecosystem: the Sumo Logic software-as-a-service, our teams of data center experts and data scientists that together, make real-time and predictive analytics a reality.


We’re also working with Sumo Logic on developing applications that are focused on the HDS ecosystem of hardware components from storage arrays and servers to networking components.   With these applications, our customers can capture data from various HDS systems, visualize the data via dashboards and gain insight into root cause of system failure, spot anomalies and meet compliance and security objectives.  As a result, we enable IT operations teams to be more efficient and productive.


We plan to broaden our scope of machine data analytics beyond the data center but for us to really tackle gaining insight from the Internet of Things that matter, we need to excel at deriving insight from all the components of the data center.  Our announcement of Hitachi Live Insight for IT Operations is the first step in achieving our ultimate vision of delivering a platform for machine-to-machine (M2M) analytics.  Moving forward, HDS plan to deliver a M2M platform that can accelerate the opportunities created by the merging of Information Technology (IT) with Operational Technology (OT) to allow business and society to transform and improve the lives of billions around the world.


I invite you to take a look at our solution with Sumo Logic, Hitachi Live Insight for IT Operations.  What do you think about this solution?  I’d love to hear your comments.

I covered “What is Social Innovation?” in my story about Social Innovation in my last post here. Now I’d like to go a little deeper on the “Why it's important.”


The bigger picture

Let’s first take a step back and look at the bigger picture of what’s happening in our world, or to be more exact, in our societies. To get a better feel for the “why”, we need to look at some of the underlying trends that are driving other things to happen in a chain reaction. Some of the connections are obvious and some are not.


We can focus on one of them, maybe the most impactful of them all – the population growth. We’re roughly at 7.2 billion people in the world today. Projections vary for the future, but we’re expecting to add approximately 1.5 billion more by 2030. An interesting observation is that just short of 10% of all humans who have ever lived will still be alive at that point (Source:


The growth of the total population is a challenge in itself, but if you overlay the urbanization trend on top of that, the problem gets even more challenging. A few years ago we passed an inflection point where we from that point on have more people living in cities than in rural areas. The report “World Urbanization Prospects” published by the United Nations has many interesting facts about the urbanization trend. Frost & Sullivan has also been forecasting the emergence of mega city regions and corridors, where large cities merge together. So we’re not only moving into cities, we’re also concentrating the urban areas to a point where whole areas or corridors will be urban.



McKinsey also states that already today 60% of the world’s GDP is generated in the 600 largest cities in the world. Combine this with what Frost & Sullivan is saying and you can easily see the economic impact the urbanization trend has on society.


Now let’s be more specific and look at four areas where we at Hitachi think we can address challenges and opportunities. (I'll cover two here in this blog and the rest will come in the subsequent one)


Aging population

The first area is the impact of yet another demographic shift, the aging population.


This is especially important in developed countries where the 65+ generation is expected to double in size by 2030. Healthcare spending by this group is twice that of the rest of the population, which shows how important healthcare is for the older part of the population, as they are impacted by the most chronic diseases. Today in the US, we spend ¾ of our healthcare dollars on dealing with chronic diseases.



What if?

So, what if we could prevent chronic diseases instead of treating the symptoms?


That’s the type of questions we ask ourselves as we’re developing Social Innovation solutions. What if we could put sensors on people in high risk categories? These sensors could non-intrusively monitor vitals and be proactive with treating symptoms even before you, as the patient, feel that something is wrong. And for people where the chronic disease not yet has manifested itself, it could monitor to avoid risk factors in the environment, recommend changes to lower the risks and maybe guide to a way of life that will avoid the disease from ever breaking out.


Scarce resources

Now let’s take a look at another area, the resources and infrastructures.


Just the other day I watched a news report about the water infrastructure in San Francisco, where they were digging up water pipes that were failing. These pipes had been installed in the 1860’s, while Lincoln was president! Much infrastructure in today’s cities is hopelessly outdated and needs to be updated.


This particular TV news report was part of a larger coverage highlighting that the water infrastructure in the Bay Area of California is heading towards a major crisis. A map showed thousands of incidents of broken pipes; this is forecasted to get 10x worse in the next 5-10 years.


If you look closer at this problem, it’s on one hand a huge and expensive undertaking to fix the pipe infrastructure itself, but these pipes are also transporting the most precious resource in California nowadays - water. How do you detect leakage even before the pipe finally has a catastrophic break?




The U.S. Geological Survey estimates that each year 1.7 trillion gallons of water are leaked in America alone. Similarly, certain Italian cities have leakage rates of up to 70 percent, and London is up to 35 percent according to the European Water partnership, a non-profit that’s looked into the issue in the E.U. In addition to these serious inefficiencies, the American Water Works Association estimates that there are close to 237,600 main breaks per year, just in the US alone, leading to approximately $2.8 billion in lost yearly revenue.


US cities lose up to 30% of their water due to leakages today (”Could Smart Meters Stem $14 Billion in Annual Water Losses?”, Heather Clancy, GreenBiz, August 15, 2013


Reducing water leakage could provide some relief in California and other places where water is a scarce resource. This is one type of challenge that Hitachi already has worked on with e.g. the Severn Trent Water utility in the UK. Severn Trent is the world’s fourth largest privately owned water company serving 4.2 million households and businesses across an area covering 10,000 square miles. Hitachi helped them track asset performance data from telemetry systems and multiple sources into one single reporting database and centrally monitor 25 operational key performance indicators relating to equipment performance.


More to come, next time I’ll take a look at challenges in the areas of infrastructure, safety and security.

Social Innovation promises to accomplish big outcomes that are relevant to our society but having a vision of what’s needed isn’t enough if you can’t bring those ideas to fruition. To do this effectively many organizations increasingly rely on data-driven decisions and actions, which means holistic analytics solutions aligned to specific needs so you can best leverage big data and the Internet of Things.  


NickArticleHitachi.pngThe general manager leading our services and consulting for Social Innovation, Nick Chang, recently wrote an article, “Marrying IoT and Big Data: Are you Ready?”, which included some of the challenges of Social Innovation:

  • Understanding what’s possible with the Internet of Things and big data in context of your situation and industry
  • Computational requirements and massive integrations across technologies and processes
  • High and / or unpredictable costs in both CAPEX and OPEX
    • High labor costs particularly related to new expertise requirements
  • Pressing need for faster results with the ability to pivot when the market changes


Nick further outlined a few ways to deal with obstacles, which centered on the reality that Social Innovation solutions usually require a breadth of expertise and resources not typically found within a single organization.  To remove obstacles he suggests partnering with a services and consulting organization that can provide:

  • Industry expertise that includes best practices, methodologies, and suitable tools
  • Technical expertise that includes the integration and optimization of a wide variety of technologies and systems
  • A proven history for fast, seamless delivery


There’s more in Nick’s article about Social Innovation strategy and services but I wanted to call out these points because we often forget that having a great idea is only half the work of innovation. (Maybe not even half?)  You’re going to see a lot more from us in the coming weeks and months about how to better realize your insights.  We believe this is essential to moving us all forward and making Social Innovation a reality.   Follow us on the HDS Community Social Innovation place for more blogs, discussions, and polls on this topic.

We live in a rapidly changing world. The pace is immense and it still seems to continually be accelerating. Yet, we’ve become so accustomed to the constant change that most of us ignore it and don’t stop to realize what impact it has on our society. In reality, these changes create a growing number of challenges that are becoming increasingly difficult to manage. It affects many areas - from health and natural resources to societal infrastructure and public safety.


The effects of population growth

Underlying many of the changes on the societal scale are fundamental shifts such as strong population growth and changing demographics.


Analysts like Frost & Sullivan also project the number of Mega-cities with more than 10 million people are increasing in number. Back in 1990 there were ten mega-cities, last year there were 28 and we’re now on our way to hit 41 by 2030. So not only are we moving into cities, the cities themselves are growing and sometimes combine into Mega-city regions.


This puts extreme pressure on infrastructure and support systems to sustainably enable the urban environments. The world population has doubled in the last 50 years, now at more than 7 billion people and is expected to grow to 8.8 billion by 2030. The total number is a challenge in itself, but we also see a clear urbanization trend that further complicates the situation. Today, a majority of people live in cities (54% in 2014) and it’s expected to be 71% by 2030.  (Source: World Urbanization Prospects, 2014 Revision, Economics & Social Affairs, United Nations)

Screen Shot 2015-04-13 at 9.21.45 PM.png








What can we do about it?

At Hitachi we believe we have an opportunity to address these larger problems with solutions that use innovative technology and business approaches that integrate machine data, information technology and advanced analytics using a solid foundation of understanding the business issues at their core. We are joining the technical expertise and deep industry experience from various Hitachi teams and partners around the world to create unique, holistic solutions to solve specific challenges.


We call this Social Innovation, which is our promise for safer, smarter and healthier societies. We believe that smart, connected technology plays a vital part in Social Innovation solutions and builds on mission focused IT, which is the alignment of your organization’s mission and the IT you need for the success of your whole organization.


Why now?

2014 was the inflection point where the benefits of Internet of Things and Big Data analytics really began to build on each other with increasing momentum. More data is readily available, most of it from sensors and often in near real time. This is driving data-driven decisions that leverage more advanced analyses. We’re accelerating towards more predictive, real-time intelligence that give us live views into key indicators and lead to bigger insights.

Screen Shot 2015-04-13 at 9.38.07 PM.png

To advance Social Innovation, Hitachi Data Systems looks at the Internet of Things that matter and leverage the Hitachi experience in Operational Technology (OT). We focus on what really makes a difference for your mission  and avoid being distracted by the sheer volume of devices and data.  We also focus on an initial set of solution areas that include communications, healthcare, business IT analytics, public safety, energy and transportation. With hands-on experience from these areas in the Hitachi group, we bring more holistic approaches to market to solve real problems.


In 2015 we’re bringing out more solutions as part of our focus on Social Innovation. We already have solutions in the market for e.g. Public Safety, Healthcare and Telecom Analytics. There’s more to come, so stay tuned for more news from Hitachi Data Systems throughout the year.


What do you think?

Social Innovation is a concept that means different things, has a different impact on each one of us. Tell us what Social Innovation means to you in this discussion: What do you think Social Innovation is?


Join the Social Innovation Community

Learn more about our Social Innovation strategy and solutions within the newly launched Social Innovation Community: Social Innovation



2015 is shaping up to be the year of Social Innovation and the Internet of things that Matter for Hitachi!  This is music to my ears!  In the space of a few days we have made two transformational announcements that really represent the tipping point for our Social Innovation Business.  Before I get to the announcements some quick context setting.


Social Innovation and the Internet of Things that Matter

Social Innovation is Hitachi’s mission to solve societal challenges through innovative solutions integrating machine data, IT and analytics.


It’s all about the Internet of Things (IoT).   But not just any old things.  We are focusing on the Internet of Things That Matter (IoTTM).  Mission critical things and infrastructure because (a) that's where the potential for really big transformative impact lies and (b) those are the things we build and operate in our Industrial sister companies.  Some examples are shown below.


The Internet of Things that Matter is our sweet spot because it marries our industrial heritage and expertise directly with our IT heritage and expertise.  Mix in the innovation coming out of our Data Science and R&D Labs and Ta Da! We feel in a pretty unique position to address some pretty impactful business and societal problems.  


Back to those announcements.

To help turn the Social Innovation mission into reality requires a couple of key things.  Tight collaboration between the different Hitachi Businesses and a common approach/platform to accelerate IoT application development across the Hitachi group. 

Last week Hitachi announced a new autonomous decentralized global management model and promotion of HDS CEO Jack to Chief Exec for the Americas.  This is an exciting development and one that better aligns the Hitachi group for Social Innovation and builds on the inter company collaboration that has been growing over the last year or so.   With Social Innovation it really is all about ONE Hitachi.


And today I am delighted to share a second major step forward in our Social Innovation journey.  The announcement of our Intent to acquire Big Data Integration and Business Analytics experts Pentaho.


Pentaho and Hitachi Data Systems are the Perfect Match.   

We are all very excited about this new marriage.  Aside from being a great match culturally we have very complimentary technologies and a shared passion for the Internet of Things.    Pentaho were early pioneers of data integration and blending on Hadoop and their open source heritage and core has ensured continuous innovation.   My colleague Greg Knieriemen does a really nice job of laying out the highlights of Pentaho’s portfolio in his post.

The Big Data and IoT ecosystem is all about Open Source.

I’m especially excited about Pentaho’s long heritage in Open Source.  Nowhere is innovation faster than in the world of Open Source and lightning fast innovation is what we need for IoT.

We are fully committed to open source at Hitachi and have no changes planned to the Pentaho Open Source Community.  We are very excited to be working with Pedro (Community SVP) and the community as a whole.  Pedro published his own blog on the news. 

By combining Open Source with our own Hitachi IP and partner technologies we feel we can deliver the best solutions for our customers.

Which brings me back to the concept of a common platform.

Solutions Led Analytics Platform

An Analytics platform is core to our strategy.  Without it we would be building complex analytic applications from scratch each time – clearly not very smart.    But the approach we are taking to building that platform is maybe a little different to the approach that some other vendors are taking.

Our approach is primarily solutions led.  We aren’t building a technology stack in isolation but rather capturing and distilling down best practices and components as we build out our Social Innovation applications.  It’s an open and flexible approach leveraging a blend of Open Source, our own IP and partner platforms where relevant.

We look for solution opportunities that match our domain expertise across the Hitachi group and we engage customers and partners keen to work with us on co-innovation.  

For example, our Telecom Network Analytics solution represents collaboration between several Hitachi businesses including Hitachi Data Systems, Hitachi Communications of America, Hitachi Telecommunications Network Division and Hitachi Consulting plus Hitachi partner JDSU and some very forward thinking customers.  The solution provides real time granular insight to Telco network operators driving everything from better efficiencies and improved customer service to brand new business models.

Go forth - reuse and multiply

Early solutions like this are helping drive the reusable foundational components and services that make up our analytics platform/architecture.   In the Telco example we leveraged a streaming data capture and analysis engine built for the financial markets in this instance and honed it down into an optimized platform for high speed real time network traffic analysis/prediction problems.  Now it can be repurposed in the common platform for adjacent use cases from Data Center Network optimization to optimizing equipment at a mining site to predicting potential problems on the power grid.   

We also see reusable design patterns across unstructured data use cases in areas like healthcare, public safety and surveillance and across the many different types of machine log applications as well. The fundamentals are the same and being able to repurpose means we have the ability to bring best practices across different industry problems and deliver solutions to you faster.


Pentaho helps accelerate our Vision

Our announcement today accelerates this vision.  Pentaho brings a wealth of expertise wrangling, blending and mining data to the Hitachi family along with their well regarded end to end suite of tools and platforms.

There's plenty more to say and you'll hear more on this blog over the coming weeks.  And watch too for more news on the solutions that we will deliver in 2015.  It will be a very exciting year for the Internet of Things that Matter I promise!

In the meantime check out additional perspective from some of my old and new colleagues


Pentaho CEO Quentin Gallivan

HDS Chief Engineer Michael Hay

And to close… you may have noticed that I have more than just a touch of passion for this space.  Well that's because the Internet of Things that Matter isn't just Hitachi's sweet spot is also my own personal sweet spot as it connects together all the tenets of my career. 

From my roots as an Engineering Apprentice (yes I know how to brandish a welding torch) to a degree in computer science (Majoring in Math/Stats) to a long career in data management and analytics to the Internet of Things in an Industrial conglomerate.

I’m living the “Internet of things that Matter” dream.   And I could not have planned it better if I had tried!