• 中国精品科技期刊
  • CCF推荐A类中文期刊
  • 计算领域高质量科技期刊T1类
高级检索

面向云数据中心多语法日志通用异常检测机制

张圣林, 李东闻, 孙永谦, 孟伟彬, 张宇哲, 张玉志, 刘莹, 裴丹

张圣林, 李东闻, 孙永谦, 孟伟彬, 张宇哲, 张玉志, 刘莹, 裴丹. 面向云数据中心多语法日志通用异常检测机制[J]. 计算机研究与发展, 2020, 57(4): 778-790. DOI: 10.7544/issn1000-1239.2020.20190875
引用本文: 张圣林, 李东闻, 孙永谦, 孟伟彬, 张宇哲, 张玉志, 刘莹, 裴丹. 面向云数据中心多语法日志通用异常检测机制[J]. 计算机研究与发展, 2020, 57(4): 778-790. DOI: 10.7544/issn1000-1239.2020.20190875
Zhang Shenglin, Li Dongwen, Sun Yongqian, Meng Weibin, Zhang Yuzhe, Zhang Yuzhi, Liu Ying, Pei Dan. Unified Anomaly Detection for Syntactically Diverse Logs in Cloud Datacenter[J]. Journal of Computer Research and Development, 2020, 57(4): 778-790. DOI: 10.7544/issn1000-1239.2020.20190875
Citation: Zhang Shenglin, Li Dongwen, Sun Yongqian, Meng Weibin, Zhang Yuzhe, Zhang Yuzhi, Liu Ying, Pei Dan. Unified Anomaly Detection for Syntactically Diverse Logs in Cloud Datacenter[J]. Journal of Computer Research and Development, 2020, 57(4): 778-790. DOI: 10.7544/issn1000-1239.2020.20190875
张圣林, 李东闻, 孙永谦, 孟伟彬, 张宇哲, 张玉志, 刘莹, 裴丹. 面向云数据中心多语法日志通用异常检测机制[J]. 计算机研究与发展, 2020, 57(4): 778-790. CSTR: 32373.14.issn1000-1239.2020.20190875
引用本文: 张圣林, 李东闻, 孙永谦, 孟伟彬, 张宇哲, 张玉志, 刘莹, 裴丹. 面向云数据中心多语法日志通用异常检测机制[J]. 计算机研究与发展, 2020, 57(4): 778-790. CSTR: 32373.14.issn1000-1239.2020.20190875
Zhang Shenglin, Li Dongwen, Sun Yongqian, Meng Weibin, Zhang Yuzhe, Zhang Yuzhi, Liu Ying, Pei Dan. Unified Anomaly Detection for Syntactically Diverse Logs in Cloud Datacenter[J]. Journal of Computer Research and Development, 2020, 57(4): 778-790. CSTR: 32373.14.issn1000-1239.2020.20190875
Citation: Zhang Shenglin, Li Dongwen, Sun Yongqian, Meng Weibin, Zhang Yuzhe, Zhang Yuzhi, Liu Ying, Pei Dan. Unified Anomaly Detection for Syntactically Diverse Logs in Cloud Datacenter[J]. Journal of Computer Research and Development, 2020, 57(4): 778-790. CSTR: 32373.14.issn1000-1239.2020.20190875

面向云数据中心多语法日志通用异常检测机制

基金项目: 国家重点研发计划项目(2018YFB0204304)
详细信息
  • 中图分类号: TP391

Unified Anomaly Detection for Syntactically Diverse Logs in Cloud Datacenter

Funds: This work was supported by the National Key Research and Development Plan of China (2018YFB0204304).
  • 摘要: 得益于自然语言处理和机器学习方法的快速发展,基于日志对云数据中心软硬件系统进行自动异常检测变得越来越流行.无监督学习方法不需要标记异常日志,但通常存在准确性较低、仍需标注大量正常日志的问题.尽管有监督学习方法的准确性较高,但由于不同软硬件系统产生不同类型的、语法各异的日志,导致有监督学习方法需要为每一类型日志标注足够多的异常日志以训练相应的异常检测模型,这极大地增加了标注异常日志的人力成本.与此同时,不同类型日志在发生异常时往往具有相同或相似的语义.因此,提出了一种跨日志类型的通用异常检测机制——LogMerge.该机制通过学习多语法日志的语义相似性,可实现日志异常模式的跨日志类型迁移,从而大大减少了异常标注开销.LogMerge采用词嵌入方法先后构建单词和模板的向量,然后使用聚类方法将语义相同或相近的模板聚成一类,解决了不同类型日志语法不同带来的挑战.此外,LogMerge结合CNN与LSTM方法构建异常检测模型,既有效提取了日志序列的前后依赖性,又显著降低了日志序列中噪声带来的影响.使用公开日志数据集的实验表明,相比于当前的有监督学习方法和无监督学习方法,LogMerge取得了更高的准确性.实验还验证了LogMerge能够显著减少异常标注工作量——在目标类型日志异常标注较少时,依然能够取得较高的准确性.
    Abstract: Benefit from the rapid development of natural language processing and machine learning methods, log based automatic anomaly detection is becoming increasingly popular for the software and hardware systems in cloud datacenters. Current unsupervised learning methods, requiring no labelled anomalies, still need to obtain a large number of normal logs and generally suffer from low accuracy. Although current supervised learning methods are accurate, they need much labelling efforts. This is because the syntax of different types of logs generated by different software/hardware systems varies greatly, and thus for each type of logs, supervised methods need sufficient anomaly labels to train its corresponding anomaly detection model. Meanwhile, different types of logs usually have the same or similar semantics when anomalies occur. In this paper, we propose LogMerge, which learns the semantic similarity among different types of logs and then transfers anomaly patterns across these logs. In this way, labelling efforts are reduced significantly. LogMerge employs a word embedding method to construct the vectors of words and templates, and then utilizes a clustering technique to group templates based on semantics, addressing the challenge that different types of logs are different in syntax. In addition, LogMerge combines CNN and LSTM to build an anomaly detection model, which not only effectively extracts the sequential feature of logs, but also minimizes the impact of noises in logs. We have conducted extensive experiments on publicly available datasets, which demonstrates that compared with the current supervised/unsupervised learning methods, LogMerge achieves higher accuracy. Moreover, LogMerge achieves high accuracy when there are few anomaly labels in the target type of logs, which therefore significantly reduces labelling efforts.
  • 期刊类型引用(12)

    1. 赵海鹏,容晓峰. 融合Transformer与1D-CNN的日志异常检测方法. 西安工业大学学报. 2025(01): 138-148 . 百度学术
    2. 王圣凯,阮树骅,汪邓喆. 基于eBPF的云环境下payload进程检测方法. 计算机应用研究. 2023(07): 2157-2161 . 百度学术
    3. 魏钧宇,张广艳,陈军超. 数据模式感知的低成本云日志存储系统. 计算机研究与发展. 2023(11): 2442-2452 . 本站查看
    4. 林国峰,詹伶俐,沈德仁. 空管自动化系统智能运维技术研究综述. 西华大学学报(自然科学版). 2022(02): 20-26 . 百度学术
    5. 周建国,戴华,杨庚,周倩,王俊. 基于并列GRU分类模型的日志异常检测方法. 南京理工大学学报. 2022(02): 198-204 . 百度学术
    6. 王蓓蓓,余晶鑫. 非侵入式空管自动化系统智能运维方法. 电子技术与软件工程. 2022(12): 193-197 . 百度学术
    7. 何书前,孙学朝,蒋文娟,余绪杭. 一种改进的模糊聚类日志异常检测方法. 现代电子技术. 2022(16): 30-34 . 百度学术
    8. 闫力,夏伟. 基于机器学习的日志异常检测综述. 计算机系统应用. 2022(09): 57-69 . 百度学术
    9. 徐洁. 基于机器学习的日志异常检测设计. 电子技术与软件工程. 2022(24): 210-213 . 百度学术
    10. 冯云,刘宝旭,张金莉,汪旭童,刘潮歌,申明喆,刘奇旭. 一种无监督的窃密攻击及时发现方法. 计算机研究与发展. 2021(05): 995-1005 . 本站查看
    11. 张颖,彭然. 基于改进蜻蜓优化多核模糊聚类算法的异常检测. 数学的实践与认识. 2021(19): 208-219 . 百度学术
    12. 陈玮,黄维芳. 基于智能算法的电力交易数据中心通用数据调度方法. 自动化与仪器仪表. 2021(11): 131-133 . 百度学术

    其他类型引用(20)

计量
  • 文章访问数:  1384
  • HTML全文浏览量:  4
  • PDF下载量:  598
  • 被引次数: 32
出版历程
  • 发布日期:  2020-03-31

目录

    /

    返回文章
    返回