|Table of Contents|

Study on Log Anomaly Detection Based on RoBERTa and Hypersphere Space(PDF)

南京师范大学学报（工程技术版）[ISSN:1006-6977/CN:61-1281/TN]

Issue:
2024年04期
Page:
17-27
Research Field:
计算机科学与技术
Publishing date:

Info

Title:
Study on Log Anomaly Detection Based on RoBERTa and Hypersphere Space
Author(s):
Li Xiaopeng1; 2; Yin ChuanHuan1; 2; Chao Meng3
(1.School of Computer Science and Technology,Beijing Jiaotong University,Beijing 100044,China)
(2.Beijing Key Lab of Traffic Data Analysis and Mining,Beijing 100044,China)
(3.China Life Insurance Company Shanghai Data Center,Shanghai 201201,China)
Keywords:
logs anomaly detection; RoBERTa; transformer; hypersphere space
PACS:
TP391
DOI:
10.3969/j.issn.1672-1292.2024.04.002
Abstract:
By monitoring and analyzing large volumes of log data,log anomaly detection can promptly identify abnormal behaviors such as intrusions and malicious operations,making it a critical tool for modern system administrators. To address the issue of limited labeled data,this paper proposes an unsupervised log anomaly detection algorithm based on RoBERTa and hyperspherical space. Firstly,to fully capture the semantic features of log texts,a multi-level semantic extraction network is proposed to effectively learn the contextual information of logs from multiple perspectives. Specifically,the robustly optimized BERT pretraining approach(RoBERTa)is pretrained on a log corpus. And then both RoBERTa and Transformer encoders are used to extract semantic features of log entries at the word and sentence level,respectively. Additionally,to enhance class differentiation and uncover normal patterns in logs,hyperspherical loss is introduced in the feature space. By continuously optimizing the model and training with only normal samples,the feature representations of normal samples converge toward the center of the hyperspherical space,while anomalous samples are pushed away from the center,effectively separating the anomalies. The model achieved F1 scores of 0.94 and 0.93 on the HDFS and BGL log datasets,respectively,demonstrating its effectiveness.

References:

[1]LE V H,ZHANG H. Log-based anomaly detection with deep learning:How far are we?[J]. arXiv Preprint arXiv:2202.04301,2022.
[2]VASWANI A,SHAZEER N,PARMAR N,et al. Attention is all you need[J]. 31st Conference on Neural Information Processing Systems. Long Beach,CA,USA,2017.
[3]LIU Y,OTT M,GOYAL N,et al. Roberta:A robustly optimized bert pretraining approach[J]. arXiv Preprint arXiv:1907.11692,2019.
[4]ZHU J M,HE S L,HE P J,et al. Loghub:A large collection of system log datasets for AI-drive log analytics[C]//2023 IEEE 34th International Symposium on Software Reliability Engineering. Florence,Italy,2023.
[5]HOCHREITER S,SCHMIDHUBER J. Long short-term memory[J]. Neural Computation,1997,9(8):1735-1780.
[6]ZHANG X,XU Y,LIN Q,et al. Robust log-based anomaly detection on unstable log data[C]//Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Tallinn,Estonia,2019.
[7]HE P,ZHU J,ZHENG Z,et al. Drain:An online log parsing approach with fixed depth tree[C]//2017 IEEE International Conference on Web Services. Honolulu,HI,USA:IEEE,2017.
[8]SALTON G,BUCKLEY C. Term-weighting approaches in automatic text retrieval[J]. Information Processing & Management,1988,24(5):513-523.
[9]HUANG Z,XU W,YU K. Bidirectional LSTM-CRF models for sequence tagging[J]. arXiv Preprint arXiv:1508.01991,2015.
[10]HUANG S,LIU Y,FUNG C,et al. Hitanomaly:Hierarchical transformers for anomaly detection in system log[J]. IEEE Transactions on Network and Service Management,2020,17(4):2064-2076.
[11]LE V H,ZHANG H. Log-based anomaly detection without log parsing[C]//2021 36th IEEE/ACM International Conference on Automated Software Engineering. Melbourne,Australia:IEEE,2021.
[12]DEVLIN J,CHANG M W,LEE K,et al. BERT:Pre-training of deep bidirectional transformers for language understanding[J]. arXiv Preprint arXiv:1810.04805,2018.
[13]NEDELKOSKI S,BOGATINOVSKI J,ACKER A,et al. Self-attentive classification-based anomaly detection in unstructured logs[C]//2020 IEEE International Conference on Data Mining. Sorrento,Italy,2020.
[14]WANG Y,WONG J,MINER A. Anomaly intrusion detection using one class SVM[C]//Proceedings from the Fifth Annual IEEE SMC Information Assurance Workshop. West Point,NY,USA:IEEE,2004.
[15]VAARANDI R,PIHELGAS M. Logcluster-A data clustering and pattern mining algorithm for event logs[C]//2015 11th International Conference on Network and Service Management. Barcelona,Spain,2015.
[16]DU M,LI F F,ZHENG G N,et al. Deeplog:Anomaly detection and diagnosis from system logs through deep learning[C]//Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. Dallas,Texas,USA,2017.
[17]GUO H X,YUAN S L,WU X T. LogBERT:Log anomaly detection via BERT[C]//2021 International Joint Conference on Neural Networks. Shenzhen,China,2021.
[18]GILLIOZ A,CASAS J,MUGELLINI E,et al. Overview of the transformer-based models for NLP tasks[C]//2020 15th Conference on Computer Science and Information Systems. Sofia,Bulgaria,2020.
[19]SHIN H J,EOM D H,KIM S S. One-class support vector machines—an application in machine fault detection and classification[J]. Computers & Industrial Engineering,2005,48(2):395-408.
[20]XU W,HUANG L,FOX A,et al. Detecting large-scale system problems by mining console logs[C]//Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles. Big Sky,Montana,USA,2009.
[21]MENG W B,LIU Y,ZHU Y C,et al. Loganomaly:Unsupervised detection of sequential and quantitative anomalies in unstructured logs[C]//IJCAI. Macau,China,2019.

Memo

Memo:
-
Last Update: 2024-12-15