高级检索

    基于滑动窗口策略的大语言模型检索增强生成系统

    A Retrieval-Augmented Generation System Based on a Sliding Window Strategy in Large Language Models

    • 摘要: 提出了一种基于滑动窗口策略的检索增强生成系统,旨在提升大语言模型(large language models,LLMs)输出的事实准确性和可靠性. 该系统通过在索引阶段应用滑动窗口机制,有效解决了传统固定大小上下文窗口和静态检索方法的局限性. 研究提出3种具体的滑动窗口策略以有效处理和分割文本,包括:固定窗口大小和固定步长分割(fixed window size and fixed step length split,FFS)、动态窗口大小和固定步长分割(dynamic window size and dynamic step length split,DFS)以及动态窗口大小和动态步长分割(dynamic window size with fixed step length split,DDS). 此外,研究还探讨了使用多查询技术进一步提高检索的准确性和相关性. 实验评估使用Llama-3模型在多个数据集上进行,结果表明在窗口大小为1 024和步长为3的配置下达到最佳性能,该配置显著提高了F1得分,体现了在文档段长度和滑动窗口步长之间保持平衡的重要性. 滑动窗口策略有效保留了上下文信息,减少了信息丢失,并展示了在不同数据集和查询类型中的适应性.

       

      Abstract: Leveraging a sliding window strategy, this study presents an innovative retrieval-augmented generation system aimed at enhancing the factual accuracy and reliability of outputs from large language models (LLMs). By applying a sliding window mechanism during the indexing phase, the project effectively addresses the limitations of fixed context window sizes and static retrieval methods. Three specific sliding window strategies have been proposed to efficiently process and segment texts: Fixed Window Size and Fixed Step Length Split (FFS), Dynamic Window Size and Fixed Step Length Split (DFS), and Dynamic Window Size and Dynamic Step Length Split (DDS). To further enhance retrieval accuracy and relevance, the project employs multiple advanced query techniques, including query expansion and reformulation. Rigorous experimental evaluations were conducted using the state-of-the-art Llama-3 model across multiple diverse datasets, encompassing both general knowledge and domain-specific corpora. Results demonstrated optimal performance with a carefully calibrated block size of 1 024 tokens and a step size of 3, significantly improving the F1 score across various tasks. This configuration highlighted the critical importance of balancing document segment length and sliding window step size to maximize information retention and retrieval efficacy.The sliding window strategy effectively preserves contextual information, reduces information loss, and exhibits adaptability across different datasets and query types.

       

    /

    返回文章
    返回