Data-Pattern-Aware Low-Cost Cloud Log Storage Systems

Cloud-native system log service can fully boost the researching, maintaining, operating and security ability of the public cloud. Could log data are typically large in scale, requiring long preserving time, high ingestion speed and low access latency, while the information density is low. To save the storage cost, it is required to compactly compress the logs, compress the logs in a high speed, and retrieve target data with low latencies. However, it is challenging to achieve these three goals at the same time, and a customized solution should be designed for this scenario. By summarizing the typical data patterns in the cloud logs, including the static patterns, namely the formatted output statements in the source code and the runtime patterns, which are generated during the execution of the programs, a low-cost storage schema is proposed for the public cloud logs. By evaluating several low-cost storage methods of the cloud logs, their effectiveness with respect to the compression ratio, compression speed and query latency are shown. Besides, several experiences for designing a low-cost storage system for the cloud logs are proposed in expectation of inspiring relevant research in the future.