Hitachi Content Software for File: High-Performance File Services for AI, ML and Analytics

By Hubert Yoshida posted 06-29-2021 01:17

Like

Infrastructure and operations (I&O) leaders are looking to use artificial intelligence (AI) in their infrastructures to gain a competitive advantage. The winners are those who can run more models with more complex algorithms and do it faster than their competitors. To win, you need faster access to more data at a lower cost than your competition and the workloads are diverse and fundamentally different from any other workload the organization may have run in the past. I&O leaders are often unprepared to address the storage requirements and data management challenges for the growing datasets of large- scale machine learning (ML), AI and HPC analytics.

While graphics processing Units, GPUs, have shrunk compute infrastructure by 40%, the data they process has grown by 50%. Compute and network resources often sit idle while waiting for data. Legacy SAN and NAS storage cannot scale to the petabytes that are required while maintaining high performance and legacy object store cannot process the meta data fast enough to deliver the data in time. On top of that, each workflow stage has unique compute, storage, and networking needs. This leads to silos creating data management and integration challenges, which drives up costs and time to results, neither of which you can afford.

Last year Hitachi Vantara made some major performance enhancements to their HCP object storage system which addresses these storage bottlenecks. In my post last week ,I addressed how Flash and NVMe, multi-core processors, and virtualization have closed much of that performance gap. Software virtualization of the OS (VM instances), application (containers) and storage resources (software-defined storage) interpose an abstraction layer between hardware implementations and applications. By decoupling the storage data and control planes, virtualization enables distributed, scale-out clusters of any size and capacity to increase overall performance.

Hitachi Content Platform (HCP) is a market leader in terms of functionality and ecosystem, and even though the product has a traditional design, Hitachi Vantara is already providing an upgrade path to its customer base, thanks to the new HCP for cloud scale. This is next-generation object storage, fully ready with Kubernetes, that retains most of the DNA of its predecessor.
HCP is well integrated with the rest of the product family and can be used as a backend to offload cold data from primary storage, such as VSP and HNAS systems. A series of add-ons, including HCP Anywhere for sync and share, a NAS gateway, and the new Hitachi Content Software for File to support DFS (Distributed File System) workloads (HPC/AI/ML), which completes the HCP platform.

Hitachi Content Software for File is specifically designed as a high-performance storage solution for AI, ML, analytics, and other GPU accelerated workloads. It gives you the blazing speed of a POSIX compliant DFS with the capacity and hybrid cloud capabilities of an object store. As an integrated solution, it greatly reduces the complexity and deployment time. Its support for file and object protocols makes data ingestion easy. The DFS provides both high performance and low latency for data preparation, model training and inference. The object store provides massive storage capacity at a lower cost and offers powerful, data management automation driven by metadata.

HC Software.png

Hitachi Vantara has also built a strong partner ecosystem over the years with a long list of solutions that address the majority of use cases, including integration with the Hitachi Lumada platform and other data analytics frameworks that can take advantage of the S3 interface.
HCP also provides strong ILM capabilities and compatibility with major public cloud storage APIs to offload data to remote locations or for disaster recovery purposes. The product includes complete end-to-end encryption and certifications for the most demanding environments.
HCP’s metadata management is also worth mentioning. In fact, its integration with Hitachi Content Intelligence enables users to analyze data during ingestion, augment metadata and make it searchable, and more, creating a powerful backend that can be leveraged for a number of next- generation applications.

HCP is available via a software-only license or as a fully integrated appliance with both CapEx and OpEx purchasing options.

#Hu'sPlace
#Blog

1 comment

12 views

Permalink

Comments

Chayan Sarkar

05-02-2022 02:00

Good read

Blogs