Search Options
Skip to main content (Press Enter).
Sign In
Skip auxiliary navigation (Press Enter).
Skip main navigation (Press Enter).
Toggle navigation
Search Options
Communities
General Discussion
My Communities
Explore All Communities
Products
Solutions
Services
Developers
Champions Corner
Customer Stories
Insights
Become an Advocate
Badge Challenges
Resources
Resource Library
Hitachi University
Product Documentation
Product Downloads
Partners Portal
How To
Get Started
Earn Points and Badges
FAQs
Start a Discussion
Champions Corner
Blog Viewer
Blogs
Hitachi Content Software for File: High-Performance File Services for AI, ML and Analytics
By
Hubert Yoshida
posted
06-29-2021 01:17
2
Like
Infrastructure and operations (I&O) leaders are looking to use artificial intelligence (AI) in their infrastructures to gain a competitive advantage. T
he winners are those who can run more models with more complex algorithms and do it faster than their competitors. To win, you need faster access to more data at a lower cost than your competition
and the workloads are diverse and fundamentally different from any other workload the organization may have run in the past. I&O leaders are often unprepared to address the storage requirements and data management challenges for the growing datasets of large- scale machine learning (ML), AI and HPC analytics.
While graphics processing Units, GPUs, have shrunk compute infrastructure by 40%, the data they process has grown by 50%. Compute and network resources often sit idle while waiting for data. Legacy SAN and NAS storage cannot scale to the petabytes that are required while maintaining high performance and legacy object store cannot process the meta data fast enough to deliver the data in time. On top of that, each workflow stage has unique compute, storage, and networking needs. This leads to silos creating data management and integration challenges, which drives up costs and time to results, neither of which you can afford.
Last year Hitachi Vantara made some major performance enhancements to their HCP object storage system which addresses these storage bottlenecks. In
my post last week
,I addressed how
Flash and NVMe, multi-core processors, and virtualization have closed much of that performance gap. Software virtualization of the OS (VM instances), application (containers) and storage resources (software-defined storage) interpose an abstraction layer between hardware implementations and applications. By decoupling the storage data and control planes, virtualization enables distributed, scale-out clusters of any size and capacity to increase overall performance.
Hitachi Content Platform (HCP) is a market leader in terms of functionality and ecosystem, and even though the product has a traditional design, Hitachi Vantara is already providing an upgrade path to its customer base, thanks to the new HCP for cloud scale. This is next-generation object storage, fully ready with Kubernetes, that retains most of the DNA of its predecessor.
HCP is well integrated with the rest of the product family and can be used as a backend to offload cold data from primary storage, such as VSP and HNAS systems. A series of add-ons, including HCP Anywhere for sync and share, a NAS gateway, and the new Hitachi Content Software for File to support DFS (Distributed File System) workloads (HPC/AI/ML), which completes the HCP platform.
Hitachi Content Software for File is specifically designed as a high-performance storage solution for AI, ML, analytics, and other GPU accelerated workloads. It gives you the blazing speed of a POSIX compliant DFS with the capacity and hybrid cloud capabilities of an object store. As an integrated solution, it greatly reduces the complexity and deployment time. Its support for file and object protocols makes data ingestion easy. The DFS provides both high performance and low latency for data preparation, model training and inference. The object store provides massive storage capacity at a lower cost and offers powerful, data management automation driven by metadata.
Hitachi Vantara has also built a strong partner ecosystem over the years with a long list of solutions that address the majority of use cases, including integration with the Hitachi Lumada platform and other data analytics frameworks that can take advantage of the S3 interface.
HCP also provides strong ILM capabilities and compatibility with major public cloud storage APIs to offload data to remote locations or for disaster recovery purposes. The product includes complete end-to-end encryption and certifications for the most demanding environments.
HCP’s metadata management is also worth mentioning. In fact, its integration with Hitachi Content Intelligence enables users to analyze data during ingestion, augment metadata and make it searchable, and more, creating a powerful backend that can be leveraged for a number of next- generation applications.
HCP is available via a software-only license or as a fully integrated appliance with both CapEx and OpEx purchasing options.
#Hu'sPlace
#Blog
1 comment
12 views
Related Content
AI and ML Require Changes in Storage Infrastructure
Hubert Yoshida
Added 01-28-2020
Blog Entry
Ongoing Success: Hitachi Vantara Named “Leader” in IDC's MarketScape for Object-Based Storage for 4th Consecutive Time
Timir Desai
Added 01-07-2020
Blog Entry
Tier-1 Performance Workloads Can Enjoy the Benefits of Object Storage With Hitachi’s HCP
Hubert Yoshida
Added 06-22-2021
Blog Entry
Object Storage Gets Three Times Faster!
Hubert Yoshida
Added 07-20-2020
Blog Entry
New Partnership and Updates Unlock Data Value
Timir Desai
Added 07-14-2020
Blog Entry
Permalink
Comments
Chayan Sarkar
05-02-2022 02:00
Good read
© Hitachi Vantara LLC 2023. All Rights Reserved.
Powered by Higher Logic