Reproduce “LogStamp: Automatic Online Log Parsing Based on Sequence Labelling” on OpenResearch

About this paper

Shimin Tao, Weibin Meng, Yimeng Chen, Yichen Zhu, Ying Liu Chunning Du, Tao Han, Yongpeng Zhao, Xiangguang Wang and 1 more

LogStamp, developed by researchers from Huawei and academic institutions, introduces an automated log parsing method leveraging sequence labeling with fine-tuned BERT to transform unstructured logs into structured data. It achieved an average accuracy of over 0.999 in offline parsing and 0.956 in online parsing with just 10% training data, addressing challenges in real-time analysis and cold-start scenarios.

Key insights

• Applying transformer-based sequence labeling (BERT + Tagger) effectively captures semantic context in logs, outperforming traditional pattern-matching or clustering approaches.
• The "coarse-to-fine" pseudo-labeling strategy enables the generation of accurate training data for the tagger without requiring manual annotations.
• LogStamp demonstrates that accurate log parsing can be achieved reliably with significantly less training data than previous methods, solving the "cold start" problem for new services.

Summary by alphaXiv.