Internet companies like Google and twitter have unprecedented agility and scale and the secret to this is the use of containers. Everything at Google runs in containers from Gmail to YouTube to Search. Each week they spin up over two billion containers. Actually, this isn’t a secret, since most of this is open source. While the concept of containers has been around for a decade in Linux there was little uptake until 2014 when dotcloud went out of business and spun out the docker project, simplifying how containers could be used by developers to package their applications. At about the same time, Google decided to open source the Kubernetes project and the Mesos project spun out of work at Twitter and UC Berkeley. Now almost every public cloud platform is supporting Kubernetes or Mesos as the scheduler/orchestrator, and Docker as the format for container image and run time.
Containerization is an OS –level virtualization method for deploying and running distributed applications on a single bare metal or VM operating system host. Container based virtualization is the latest virtualization technology that is gaining wide acceptance. It is the next generation up from virtual machines (VM)s where the traditional virtual machine abstracts an entire device including the OS, containers only consist of the application and all its dependencies that the application needs as illustrated here.
The main difference between containers and virtual machines is that a virtual machine includes an operating system as well as the application. A physical machine running three VMs would need to have a hypervisor and run three operating systems on top of it. A machine running three container applications with Docker would run on a single operating system and share parts of the operating system in read only. Each container has its own mount for writing. This means the use of containers consume far fewer resources than a virtual machine.
The main benefit of a container is that since it consists of an entire runtime environment: an application, plus all its dependencies, libraries and other binaries, and configuration files needed to run it, are bundled into one package; it can run reliably when moved from one computing environment to another. A container can be moved from a developer's laptop to a test environment, from a staging environment, into production from a physical machine in a data center to a virtual machine in a private or public cloud. Other benefits of containers are “lightweight”, meaning a single server can host far more containers than traditional virtual machines. Containers can be started immediately, just in time, when they are needed and can be closed to free up resources when they are not needed. Developers can use them as a “sandbox” to test code without disrupting other applications. These attributes are ideal for agile development where the elasticity of resources enables a modern DevOps environment. Monolithic applications can be written as micro services that can be run in containers, for greater agility, scale, and reliability.
Containers Help Address The Data Deluge
Casey O’Mara in a recent Hitachi Vantara blog post says that Financial services must move to containers in order to keep up with the deluge of data that they are facing. He gives the example of FINRA. FINRA has to go through 37+Billion transactions per day with over 900 cases of insider trading a year, and over 1500+ companies getting disciplinary actions levied against them. “The amount of data sources you need to traverse in a small amount of time is huge. Going through that many transactions a day (with around 2.5 cases happening each day) means that the queries can take hours unless you factor in containers and the ability to blend data sources quickly, and return results fast……. Today’s dated VM systems aren’t ready to tackle the current problems, and they certainly don’t scale as well”.
Hitachi Vantara, An Early Adopter of Containers
Hitachi Vantara has been using containers for some time and is expanding its use in more of our software products to not only enhance our agility but also to enable customers to easily deploy containers. In July 2015, our Unified Compute Platform was part of the Kubernetes 1.0 launch and the first enterprise infrastructure platform to adopt and community validate Kubernetes orchestration. Last month we announced a Hitachi Enterprise Cloud Container Platform to assist organizations in reducing their reliance on public cloud resources for their born-in-the-cloud applications. It provides hybrid cloud resources for DevOps that utilize micro services architecture. This architecture combines Hitachi infrastructure with Mesosphere DC/OS to provide PaaS element integration and user interface, container engine, scheduling, orchestration and workflow. The pre-engineered environment enables developers to focus on app development and provides operations with robust and industrialized service-level choices that the business demands once those applications reach production and global scale. We also announced managed services revolving around the Hitachi Enterprise Cloud support for DevOps and containers. Hitachi Enterprise Cloud partners with VMware and Mesosphere to provide hybrid cloud management tools that provide a pay-as-you-go cloud utility alternative to public clouds.
On the storage side, our Hitachi Storage Advisor, formerly known as Hitachi Infrastructure Director, uses containers in the form of installation support using Docker. The Hitachi Storage Plugin for Containers (HSPC)is now available . HSPC enables stateful applications, such as databases, to persist and maintain its data after the life cycle of the container has ended. Built-in high availability enables orchestrators such as Docker Swarm, Kubernetes to automatically orchestrate storage tasks between hosts in a cluster. In addition, the storage plug-in enables the advanced storage capabilities of Hitachi VSP G and F series. These storage platforms provide features such as automation, high availability, seamless replication, remote management and analytics. There are plans for additional infrastructure software to add container support in the future either in containers or monitoring/provisioning containers.
Last year Pentaho announced support for Docker. Through this joint collaboration, developers only need to build a big data app once and then deploy via a Docker software container and Pentaho Server. No matter what the operating environment, the app will run as designed, freeing up more time to develop new applications and less time debugging or porting them. The Pentaho versions of Docker utilities can be downloaded from GitHub.
Containers Are A Foundation for Hitachi Vantara Products
Recent Hitachi Vantara products make extensive use of containers, Hitachi Content intelligence , Pentaho 8 Worker Nodes and Lumada all use the same platform for rapid development of service-based applications delivering the management of containers, communications, security, and monitoring toward creating enterprise products/applications, leveraging technology like docker, mesos, marathon.
With Hitachi Content Intelligence, organizations can connect to and aggregate data from siloed repositories, transforming and enriching data as it is processed, and centralizing the results. Hitachi is now the only object storage vendor in the market with seamlessly integrated cloud-file gateway, enterprise file synchronization and sharing, and big data exploration and analytics.
Pentaho 8, which was announced last month, was a major revision with the introduction of Worker Nodes that uses the same container platform for better monitoring, improving lifecycle management, and active-active HA. When scaling PDI in the past, the ETL designer was responsible for manually defining the slave server in advance, and controlling the flow of each execution. Now, once you have a Pentaho Server installed, you can configure it to connect to the cluster of Pentaho Worker nodes. From that point on - things will work! It will deploy consistently in physical, virtual and cloud environments, scale and load balance services, helping to deal with peaks and limited time-windows, and allocate the resources that are needed.
Lumada is an intelligent, composable, secure and flexible IoT platform that is built on four key pillars: edge, core analytics and studio. Lumada Foundry is the foundational layer which supports these pillars with deployment, repair, upgrade and scaling services for industrial grade software, on premise or in the cloud. Lumada Foundry is based on a micro services architecture which is underpinned by container’s ecosystem technology (Docker, Mesos, Marathon, etc.)
Containers with the eco system of scheduler and orchestration open source tools is the next level of virtualization that will provide greater agility and scalability as we face the deluge of data that will come from IoT. Hitachi Vantara is heavily invested in this technology to not only increase the agility in our own operations, but to enable our customers to adopt containers in their operations. Based on the IDC forecast below (excluding web/SaaS provider), container instances will continue to grow rapidly through 2020 as IT modernization efforts expand.
The “V” in Hitachi Vantara is a nod to our heritage in virtualization. That heritage continues with OS level virtualization. The last example shows the development synergy of the former Hitachi Data Systems’ Hitachi Content Intelligence, Pentaho 8’s Worker nodes, and Hitachi Insight’s Lumada Foundry, working together to develop an agile and scalable platform for object storage, analytics, and IoT. You can expect to see more of this as we move forward as one Hitachi Vantara