高级检索

    面向大模型的高性能可扩展元数据设计ScaleFS

    ScaleFS: High Performance and Scalable Metadata Design for Large Language Models

    • 摘要: 近年来,以ChatGPT为代表的大语言模型(large language model,LLM)技术发展迅速. 随着模型参数规模的持续增长,构建和应用大模型对数据存储规模和存储访问效率提出了更高要求,这对传统存储系统带来了严峻挑战. 首先分析了大模型在数据准备、模型训练和推理阶段的存储访问特征,深入探讨了传统存储系统在大模型场景下面临的主要问题和瓶颈. 针对这些挑战,提出并实现了一种高性能、可扩展的分布式元数据设计ScaleFS. 通过目录树元数据与属性元数据解耦的架构设计、并结合深度与广度均衡的目录树分层分区策略设计,ScaleFS实现了高效的路径解析、负载均衡和系统扩展能力,能够高效管理千亿级文件. 此外,ScaleFS设计了细粒度元数据结构,优化了元数据访问模式,并构建了面向文件语义优化的元数据键值存储底座,显著提升了元数据访问效率并减少了磁盘I/O操作. 实验结果表明,ScaleFS的每秒操作次数(operations per second,OPS)是HDFS的1.04~7.12倍,而延迟仅为HDFS的12.67%~99.55%. 在千亿文件规模下,ScaleFS的大部分操作性能优于HDFS在十亿文件规模下的表现,展现出更高的扩展性和访问效率,能够更好地满足大模型场景对千亿级文件存储及高效访问的需求.

       

      Abstract: In recent years, large language models (LLMs) represented by ChatGPT have developed rapidly. As the scale of model parameters continues to grow, building and deploying LLMs puts forward higher requirement for data scale and storage access efficiency, which poses significant challenges to traditional storage systems. This study first analyzes the storage access characteristics across the three critical stages of LLM workflows: data preparation,model training, and inference. It also explores in depth the major issues and bottlenecks faced by traditional storage systems in LLM scenarios. To address these challenges, the study proposes and implements ScaleFS, a high-performance and scalable distributed metadata design. ScaleFS adopts a decoupled design for directory tree metadata and attribute metadata, and combined with a hierarchical partitioning strategy that balances depth and breadth in the directory tree. This design enables efficient path resolution, load balancing, and system scalability, thereby making it capable of effectively managing hundreds of billions of files. Additionally, ScaleFS introduces fine-grained metadata structures, optimizes metadata access patterns, and develops a metadata key-value store tailored for file semantics. These innovations significantly improve metadata access efficiency while reducing disk I/O operations. The experimental results demonstrate that ScaleFS achieves operatons per secone(OPS) rates 1.04 to 7.12 times higher than HDFS, with latency reduced to only 12.67% to 99.55% of HDFS. Furthermore, at a scale of hundreds of billions of files, ScaleFS outperforms HDFS in most operations, even when HDFS operates at a billion-file scale. This demonstrates its superior scalability and access efficiency. ScaleFS is thus well-suited to meet the demands of LLM scenarios for managing and efficiently accessing massive file datasets.

       

    /

    返回文章
    返回