An infrastructure architect is designing the storage subsystem for an enterprise AI initiative that requires training deep learning models on petabytes of unstructured text, audio, and video files. The files reside in multiple distinct repositories, and the GPU cluster must access them with high parallel bandwidth. Which of the following architectural approaches will most effectively handle this high-capacity, distributed data ingestion? (Choose two)
Select all correct answers, then click Submit.
Short Explanation and Infographic
Think of it like this: if you're training models on petabytes of data, you cannot store it all on a single hard drive or route it through a single server node. That's a massive bottleneck, and you'll end up with GPUs sitting idle waiting for data. Not very efficient! You need storage that scale out, not just up. That's where a distributed file system like Ceph or HDFS comes in. It spreads your data across multiple nodes so your compute cluster can read files in parallel over high-speed networks. Next, you need a way to organize this massive pile of different file types—videos, logs, text, you name it. A data lake architecture is perfect here because it lets you store everything in its raw, native format at scale without forcing you to structure it first. Don't waste CPU cycles compressing everything and forcing decompression on the fly, and definitely don't route everything through one node. Go with a distributed file system and a data lake, and your GPUs will stay fed. Got it? Sweet.
Full explanation below image
Full Explanation
Large-scale AI initiatives require storage architectures that can scale both capacity and I/O performance dynamically. Training deep learning models involves streaming large datasets across many GPU-equipped compute nodes concurrently, requiring high parallel read throughput. - Distributed File Systems (Option C): A distributed file system (such as Ceph, Hadoop HDFS, or Lustre) distributes data across multiple physical storage servers while presenting a unified namespace. This allows parallel access to data, where multiple compute nodes can read parts of the dataset simultaneously from different storage targets. This prevents network and disk bottlenecking, which is critical when scaling deep learning workloads across multiple GPU nodes. - Data Lake Architecture (Option E): A data lake is a centralized repository that allows you to store structured, semi-structured, and unstructured data at scale in its raw, native format. Since AI workloads require diverse data types (images, videos, text, sensor logs) that are processed using different pipelines, a data lake provides the flexibility to ingest and store raw data without predefined schemas. Data can then be processed or filtered on demand by different training or inference frameworks.
Let's examine why the alternative storage approaches are sub-optimal: - Option A is incorrect because consolidating all data onto a single monolithic local drive introduces a severe performance bottleneck and a single point of failure. It lacks the horizontal scalability required for petabyte-scale workloads. - Option B is incorrect because restricting data access to a single node eliminates the ability to perform distributed, multi-node training (such as data-parallel training across multiple DGX nodes). - Option D is incorrect because while compression saves storage space, performing on-the-fly decompression during training introduces significant CPU processing overhead. This CPU bottleneck can slow down data pipelines and leave GPUs underutilized.
Therefore, Options C and E are the correct architectural choices.