Meeting the

Storage Needs of AI
With artificial intelligence (AI) workloads emerging, find
out what you need to do to keep your data always
available.
In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
In this e-guide:
Artificial intelligence is transforming the way organizations do
business, but when it comes to storage best practices to meet the
infrastructure needs of this new technology, there are still a lot of
unknowns.
This e-guide will clear things up and help you get the most out of
storage for AI, including trends in the storage consumption landscape,
answering key AI storage questions, and much more.
▼ Next Article
Page 1 of 19
In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
How AI is changing the storage
consumption landscape
Marc Staimer, Founder
When it comes to storage for AI applications, the key issue isn't that these apps consume
more storage than other applications -- they don't. The key issue is that they consume storage
differently. With AI applications, data moves from storage to AI processing or I/O. It also
moves between different storage systems and media at different points in its lifecycle.
I/O is primarily tied to throughput, regardless of the type of storage or storage media the data
is stored on. AI's three modes -- machine learning, deep machine learning and neural
networks -- each ingest and process data differently and, therefore, have distinctive I/O
requirements. A look at each reveals how AI applications are changing storage consumption.
Speed is key to storage consumption
Machine learning, the most common AI, can potentially use millions of data points to make
predictions or decisions based on human-derived algorithms. The accuracy of an AI app's
outcomes is tied to the number of data points ingested within a specified timeframe. More
data points lead to more accurate predictions and decisions.
Time is the limiting factor: If a prediction or decision is required in "n" milliseconds, the speed
at which the machine learning algorithm can ingest and examine the data points will
determine the quality of the outcome. GPUs and high-performance computing have
eliminated most processing bottlenecks. That leaves the storage I/O having to keep up with
the machine learning algorithm.
Page 2 of 19

In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
Deep learning applications draw on millions, or even billions, of data points, and they make
multiple passes on the same data. This exacerbates the I/O bottleneck problem.
Machine learning and deep machine learning algorithms can run on modern server
architectures, but neural networks are different. Neural networks, also referred to as artificial
neural networks, mimic the neuron architecture of the human brain. By definition, they require
extensive scale-out GPUs and CPUs, ranging from dozens to millions of processors. The key
to neural network storage is to provide extremely high-performance parallel file system
throughput. This is where IBM Spectrum Scale, Lustre (open source), Panasas ActiveStor
and WekaIO are a good fit.
Legacy block storage systems generally aren't able to deliver the hundreds-of-gigabytes- to
terabytes-per-second read throughput required. However, several newer extreme-
performance storage systems can meet these needs, some better than others, including: Dell
EMC PowerMax, DirectData Networks (DDN) Storage Fusion Architecture, Excelero NVMesh,
Fungible, IBM FlashSystem, Oracle Exadata, Pavilion Hyperparallel Flash Array and
StorCentric Vexata.
Legacy file storage systems also aren't up to the task. But newer generations of parallel file
systems with global namespaces can deliver the throughput needed. They include DDN
EXA5, IBM Spectrum Scale, Panasas ActiveStor and WekaIO.
Page 3 of 19
In this e-guide

• How AI is changing the

storage consumption
landscape (P. 2)

• 6 key AI data storage

questions answered (P. 7)

• 8 factors that make AI

storage more efficient (P.13)

• Storage options meet the

needs of machine learning
and AI (P.19)
Solving the data migration problem
Keep in mind that both machine learning and deep machine learning use current and
historical data to examine, learn, predict and decide. The historical data these apps use is
unlikely to reside on super-fast, expensive storage systems. It may start there, but as it ages,
it's moved to slower, cheaper storage systems or the cloud.

Page 4 of 19

In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
This is a problem. Moving data from fast,
expensive storage to slower, cheaper systems Data movement is the
must be simple and automated, and it also must primary reason many AI
be transparent to the AI applications. In other
projects fail, so getting
words, as far as the AI application is concerned,
the data is still there. Data movement is the this right is paramount
primary reason many AI projects fail, so getting to success.
this right is paramount to success. There are two
ways to accomplish this.
The first is to build data migration into the storage system, so data is moved within the system
or to the cloud. The system knows where the data resides -- i.e., the storage it is consuming --
and feeds it to the AI to process on demand. This approach to data migration and storage
consumption suffers from limited system capacity. Once again, newer scale-out technologies
mitigate this problem. Some examples of this are the Nimbus Data, Oracle Exadata X8M,
StorOne and Vast Data.
The Oracle Exadata X8M uses both high-performance Intel Optane DC Persistent Memory
Module (PMEM) in Application Direct Mode with NVMe flash in its primary storage servers
and low-cost, high-capacity spinning disks in its XT storage servers. The amount of PMEM is
limited to no more than 27 TB per rack and supports as many as 18 racks. That's potentially a
lot of high-performance storage, which isn't cheap and not all databases or all data within a
database require PMEM performance. The Oracle Database will move older, less accessed
data from PMEM to NVMe flash and then to lower-performance and lower-cost storage
servers, such as its low-cost XT storage server. The Oracle Database knows where the data
is and still has access to It on-demand.
Page 5 of 19

In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
Vast Data's all-flash array uses high-performance, high-endurance Optane storage class
memory (SCM) drives as cache for lower cost, performance and endurance quad-level cell
(QLC) flash SSDs. Clever software puts most of the writes on the SCM and limits writes to the
QLC flash. The data invariably ends up on the QLC while still providing exceptional AI read
performance. Storage startups Nimbus Data and StorOne claim to provide similar capabilities.
The second approach is to move the data in the background, leaving a hierarchical storage
management (HSM) stub, symbolic link (symlink) redirect or global namespace middleware.
The traditional HSM is problematic in that the data must be moved back to the original storage
first before it's read, and that's too slow. Stubs can be brittle and break, orphaning data.
Symlinks can also break, albeit less frequently. And global namespace typically means
something -- probably middleware -- is in the data path that adds some latency. However, that
latency is nominal and only for the first readable byte.
Symlinks and global namespaces can provide the data required from multiple resources,
concurrently filling the AI machine learning or deep machine learning app with the data points
it needs. There are several products able to deliver these functions, including Datadobi, iRods
(open source), Komprise and StrongBox Data Solutions StrongLink.
Transparent data movement is what's needed when providing AI machine learning and deep
learning apps with the data volume they need at an acceptable cost. Such transparency
drives high-cost, high-performance storage consumption to lower-cost and lower-performing
storage consumption. This approach is a necessity to effectively handle the vast volumes of
data AI applications demand.
▼ Next Article
Page 6 of 19
In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
6 key AI data storage questions
answered
Stacey Peterson, Senior Site Editor
Research firm IDC projected that, by 2023, global spending on AI technology will hit nearly
$98 billion, up from $37.5 billion in 2019. That increase represents a compound annual growth
rate of nearly 30%. All those new, complex AI applications won't be deployed in a vacuum. A
range of IT infrastructure, including storage tailored for AI, must support and process these
new workloads.
AI, and particularly machine and deep learning, applications require vast amounts of data that
gets sent to CPUs or GPUs for processing and analysis in near real time. And then much of
that data must be stored for possible future use. AI applications and technology will be among
the top factors affecting infrastructure decisions over the next several years. For storage, that
means enterprises must understand the data they're processing, find ways to get their storage
media closer to compute technology and enhance AI data storage performance to match that
of the processors.
What follows are six questions we answered in various SearchStorage articles related to how
AI workloads are changing enterprise storage infrastructure.
1. What should be considered when planning AI
data storage workloads?
Page 7 of 19

In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
Putting together storage for AI applications is no easy matter. There are several issues to
consider and details to get right. Consultant and tech writer Robert Sheldon has a list of eight
such factors:
• Understand your various workloads' storage requirements.
• Know your capacity and scalability requirements.
• Find out how long you'll need to hold onto the data and how it will be accessed.
• Factor in the throughput and I/O rates you'll need.
• Consider location -- data near compute will minimize latency.
• Assess the best type of storage to use -- block, file or object.
• Use intelligent and software-defined storage to enhance performance.
• Ensure all systems involved are tightly integrated.
Page 8 of 19
In this e-guide

• How AI is changing the

storage consumption
landscape (P. 2)

• 6 key AI data storage

questions answered (P. 7)

• 8 factors that make AI

storage more efficient (P.13)

• Storage options meet the

needs of machine learning
and AI (P.19)

2. What challenges does AI data storage bring?

There are two distinct challenges related to building storage for AI applications, according to
IT industry veteran Chris Evans. On the one hand, it won't always be clear at the outset of an
AI or machine learning project what data will be useful. As a result, long-term archive storage
will be required where data can be retained and accessed when needed to support the
learning effort.

Page 9 of 19

In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
The other major storage challenge lies in ensuring sufficient high-performance storage is
available for the active data that needs processing. Vendors are combining fast storage with
AI and machine learning capabilities to meet this need. Evans outlines the challenges of
building a storage platform that balances the storage needs of AI workloads. Packaged
storage tuned for AI products are part of that discussion and can be attractive because they
offer a specific level of performance.
3. What challenges does object storage pose?
On paper, the high-node count storage clusters of object storage systems should be ideal for
the demands of the large, unstructured data workloads generated by AI and machine learning
applications. In addition, most AI and machine learning frameworks communicate with storage
via the Amazon S3 protocol, as do most object storage systems. However, in reality, other
factors come into play that can interfere with object storage's effectiveness when it comes to
AI data storage.
Metadata, in particular, can be a problem, overwhelming dedicated controllers and negatively
affecting the performance of SSDs and HDDs. Contributor George Crump examines metadata
and other issues related to cluster communications, internode networking and the protocol
emulation required for IoT devices that aren't native S3 and use NFS.
4. What role will flash play in AI data storage
workloads?
Page 10 of 19

In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
To get the information they need, AI and machine
learning applications process large amounts of AI applications have
data. These applications usually rely on a cluster high storage capacity
of compute nodes where at least some of the
demands that can easily
nodes use more expensive GPUs to deliver the
performance of up to 100 CPUs. The storage start in the terabyte
infrastructure must ensure data is continuously range and scale into
provided to these GPUs so they're always in use.
These systems must be able to store and process
hundreds of petabytes.
the millions, and even billions, of data files
typically generated by sensors or IoT devices.
As a result, AI applications have high storage-capacity demands that can easily start in the
terabyte range and scale into hundreds of petabytes. Crump looks at what the demands of AI
workloads on storage means for the storage media used. Because of these high demands
and the relatively high cost of flash, he expects AI storage to rely less on flash and more on a
combination of RAM, hard disks and tape.
5. How does NVMe help meet AI data storage's
needs?
NVMe provides the large bandwidth and lower latency that AI and machine learning
applications need to maximize performance. It's a natural fit with the high-performance, scale-
out storage and GPU-based compute that AI platforms use and will help eliminate I/O
bottlenecks and provide scalability.
Page 11 of 19

In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
IT journalist John Edwards tracks how NVMe is replacing traditional storage in AI
environments and why that makes sense in this performance-driven world.
6. Where does storage class memory fit in?
As a new tier in the memory/storage hierarchy, storage class memory (SCM) sits between
SSDs and dynamic RAM (DRAM). These devices connect directly to the server memory and,
like DRAM, are byte-addressable. But, like NAND-based devices, SCM devices are persistent
and they support block-level access. SCM devices usually sit between memory and storage,
letting applications access large data sets through system memory without forcing enterprises
to pay the high price of DRAM.
AI applications are among the use cases where emerging SCM technology makes sense,
according to Sheldon. SCM devices provide the low latency, high durability and optimal
performance AI workloads require.
▼
Next Article
Page 12 of 19
In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
8 factors that make AI storage more
efficient
Robert Sheldon, Contributor
Today's AI workloads need storage systems that deliver the performance, capacity and
availability necessary to ensure reliable operations throughout the application's lifecycle. AI
technologies, such as machine learning, deep learning and predictive analytics, require AI
storage systems that can deal with the vast amounts of diverse data they generate, along with
the fluctuating and process-intensive workloads.
What follows is a look at eight factors to consider when planning storage for your AI
workloads.
1. Workload patterns
AI storage requirements vary significantly from one application to the next. They generate
different quantities of data and have a variety of access requirements and I/O patterns. For
example, a deep learning application might need to access and process data more frequently
than a basic machine learning one, while at the same time continuously add data to the
existing pool. You must thoroughly understand each workload's storage requirements now
and in the future and don't assume any two workloads are alike.
But understanding those requirements is no small matter. A typical AI application goes
through several stages of operation, and storage requirements can vary from one stage to the
next.
Page 13 of 19

In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
For example, during the ingestion stage, vast amounts of heterogenous data are collected
and saved to disk, usually as sequential write operations. But during the transformation stage,
when data must be cleansed, organized and transformed, fluctuating amounts of data are
read and written, incurring both random and sequential operations.

• 6 key AI data storage

questions answered (P. 7)

• 8 factors that make AI

storage more efficient (P.13)

• Storage options meet the

needs of machine learning
and AI (P.19)

Page 14 of 19
In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
2. AI storage scalability
An AI application needs lots of data. The more data available to the AI application, the more
accurate its results. And that data can come from a variety of sources and in a wide range of
formats. True, some AI applications require less data than others, but you must still factor in
capacity and scalability requirements. Be sure to consider the need to copy, move, aggregate
or in other ways manipulate and process the data.
All this storage can represent a significant investment, whether in data center systems or
cloud-based services. It can be expensive to handle the storage needed on premises,
especially using high-performing flash arrays, yet farming it all out to the cloud isn't always the
best alternative.
One way to keep costs down and still meet scalability requirements is to use both flash and
hard-disk storage, rather than relying solely on flash. Another option is to implement a hybrid
or multi-cloud strategy. The challenge with this approach, however, is that you must carefully
regulate the amount of data you're copying or migrating across platforms, and you must keep
in mind distances. Otherwise, data duplication or migration costs could undermine the
advantages of a cloud strategy.
3. Data durability
For some AI applications, the amount of data isn't the only consideration. You also must look
at how long you need to keep that data. Some applications require ongoing analytics that
continuously infuse new data into the old, a process that can span years, resulting in
enormous stockpiles of information. To ensure the data is going to be around for the duration,
you need comprehensive backup and disaster recovery strategies, in addition to heaps of
storage capacity.
Page 15 of 19

In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
When evaluating your AI application's workload
patterns and scalability requirements, be sure to Organizations that
account for issues such as how long you must process and store data
hang on to the data, how the data will be
in their own data centers
accessed going forward, what data can be
archived, when it can be archived and, of course, or on a single cloud
the amount of data that needs to be stored platform have an
throughout the entire lifecycle.
advantage.
4. System performance
An AI solution collects, processes, aggregates, trains and analyzes data. To carry out these
operations against massive data sets, AI storage must be fast and efficient, able to deliver the
necessary throughput and I/O rates, while reducing latency and contention. If the storage
system isn't built and optimized to meet these demands, you might be looking at weeks to
complete a single iteration of the data training phase.
Today's AI products often run on high-performing, GPU-based compute systems. The storage
platform must keep up with these systems to make the investment worthwhile. That means,
among other things, avoiding I/O bottlenecks and performance issues. A massively parallel
storage architecture is one way to achieve these AI storage goals, especially when training
data, which puts heavy demands on compute and storage systems alike.
5. Data locality
The location of your data plays a role in efficiently processing massive volumes. The nearer
the data is stored to where it's processed, the more efficient the operations. Organizations
that process and store data in their own data centers or on a single cloud platform have an
Page 16 of 19

In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
advantage. Organizations that use hybrid and multi-cloud strategies could have a tougher
time, undermining some of the advantages that come with cloud strategies. To implement an
effective AI solution, you must minimize latencies, and distance can be one of the biggest
contributors to latency.
6. Storage type
Another consideration when implementing AI workloads is how data will be stored. Object
data storage is the most common approach. It has the advantage of supporting extensive sets
of metadata. Storing metadata along with the actual data makes it possible to describe the
data in multiple ways, which, in turn, enables faster and easier searching, an important
consideration with AI analytics. In addition, object storage is fast, flexible, space-efficient and
highly scalable, making it an ideal match for AI workloads.
7. Continuous optimization
Any AI storage system must be continuously optimized to maximize performance and
minimize latency. Today's intelligent storage can go a long way in helping to keep systems
optimized. An intelligent storage system, which itself uses AI technologies, can uncover
patterns in the metric data collected from the storage systems, as well as from other systems
in the environment. From these patterns, the intelligent system can automatically resolve
issues and optimize storage performance, without human intervention.
Another trend that can benefit AI workloads is software-defined storage (SDS), a systems
architecture that decouples storage software from the hardware. By abstracting the physical
storage resources, SDS provides greater flexibility, simplifies management and automates
operations, while optimizing storage performance, all of which will benefit AI workloads.
Page 17 of 19
In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
8. Cross-platform integration
No system or application exists in a vacuum. Data almost always originates from multiple
sources -- sometimes a significant number of them -- and is often stored in numerous
locations. Hybrid and multi-cloud strategies only add to the mix, as do technologies such as
edge computing, IoT and hyper-converged infrastructures.
No matter how your data moves or where it's stored, you must ensure all systems seamlessly
integrate with one another to minimize deployment and maintenance efforts, as well as
potential bottlenecks. Wherever possible, use standards-based technologies to help with this
process.
▼ Next Article
Page 18 of 19
In this e-guide
• How AI is changing the
storage consumption
landscape (P. 2)
• 6 key AI data storage
questions answered (P. 7)
• 8 factors that make AI
storage more efficient (P.13)
• Storage options meet the
needs of machine learning
and AI (P.19)
Storage options meet the needs of machine
learning and AI
Amy Novotny, Senior products editor
Organizations are using machine learning and AI to get insights they can use to improve how
they do business. But these workloads differ from regular ones, as they require large amounts
of data to build and train statistical models.
All this data must also be processed and stored: Active data must be moved to a high-
performance platform for processing, and other data often gets transferred to long-term
storage. To meet these requirements, some storage vendors offer either converged
infrastructure products or products that organizations can build into their machine learning
and AI projects. These tools package storage, networking and compute or scale-out file
storage with GPUs. Other vendors work with partners to deliver their products as
prepackaged appliances.
To help identify the storage product that will best meet your organization's AI workload needs,
it's important to understand which features and functionality are most important. Evaluate
features based on how your organization will be performing machine learning and AI tasks.
Key considerations include scalability, high-performance capabilities and overall cost.
This buyer's guide examines products from nine industry-leading storage vendors and newer
startups. A comprehensive product roundup compares numerous features and functionality,
including capacity, performance and the file sizes supported.
Page 19 of 19