Detecting Performance Anomalies in Digital Advertising Campaigns: A Comparative Evaluation of Rule-Based Monitoring, Statistical Control Charts, and Machine-Learning Detectors

Xi Chen; Hao Fu; Haobin Pan

Detecting Performance Anomalies in Digital Advertising Campaigns: A Comparative Evaluation of Rule-Based Monitoring, Statistical Control Charts, and Machine-Learning Detectors

Authors

Xi Chen Business Analytics, Trine University, MI, USA
Hao Fu Electronic Science and Technology, Beijing University of Posts and Telecommunications, Beijing, China
Haobin Pan Computer Application Technology, Institute of Computing Technology, Beijing, China

Keywords:

anomaly detection, digital advertising, statistical process control, campaign performance monitoring

Abstract

Digital advertising campaigns generate continuous performance signals whose silent degradation, drifting click-through rates, runaway spend, or collapsing conversions, often escapes timely detection. Practitioners monitor these signals with heterogeneous tools ranging from fixed-threshold alerts to deep sequence models, yet little comparative evidence clarifies which approach suits which operating condition. This study evaluates seven detectors spanning three families, a rule-based static threshold, statistical control charts (Shewhart, CUSUM, EWMA), and machine-learning detectors (extreme-value streaming, isolation-based, and a variational reconstruction method), on per-campaign key-performance-indicator series reconstructed from two public advertising datasets. Controlled anomalies of four types, level shift, spike, variance change, and trend break, are injected to obtain labeled ground truth, and detectors are assessed along detection sensitivity, false-positive rate, detection delay, and implementation complexity. The variational reconstruction method attains the highest pooled F1 of 0.80, while the exponentially weighted control chart reaches a competitive 0.77 at a fraction of the configuration and compute cost. Cumulative-sum and exponentially weighted charts respond fastest to persistent shifts, while point-anomaly spikes favor the streaming and reconstruction detectors. The results indicate that control charts remain a defensible default for resource-constrained monitoring, with deep detectors offering moderate gains mainly on multimodal seasonal series. Factor attribution and restructuring-decision implications are then examined to connect detection with action.

References

[1] Zhang, W., Yuan, S., & Wang, J. (2014). Optimal real-time bidding for display advertising. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1077–1086). Association for Computing Machinery. https://doi.org/10.1145/2623330.2623633

[2] Agarwal, D., Ghosh, S., Wei, K., & You, S. (2014). Budget pacing for targeted online advertisements at LinkedIn. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1613–1619). Association for Computing Machinery. https://doi.org/10.1145/2623330.2623366

[3] Ma, X., Zhao, L., Huang, G., Wang, Z., Hu, Z., Zhu, X., & Gai, K. (2018). Entire space multi-task model: An effective approach for estimating post-click conversion rate. In The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 1137–1140). Association for Computing Machinery. https://doi.org/10.1145/3209978.3210104

[4] Diemert, E., Meynet, J., Galland, P., & Lefortier, D. (2017). Attribution modeling increases efficiency of bidding in display advertising. In Proceedings of the AdKDD and TargetAd Workshop at KDD 2017. https://arxiv.org/abs/1707.06409

[5] Page, E. S. (1954). Continuous inspection schemes. Biometrika, 41(1–2), 100–115. https://doi.org/10.1093/biomet/41.1-2.100

[6] Roberts, S. W. (1959). Control chart tests based on geometric moving averages. Technometrics, 1(3), 239–250. https://doi.org/10.1080/00401706.1959.10489860

[7] Montgomery, D. C. (2019). Introduction to statistical quality control (8th ed.). John Wiley & Sons.

[8] Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2012). Isolation-based anomaly detection. ACM Transactions on Knowledge Discovery from Data, 6(1), 1–39. https://doi.org/10.1145/2133360.2133363

[9] Xu, H., Chen, W., Zhao, N., Li, Z., Bu, J., Li, Z., Liu, Y., Zhao, Y., Pei, D., Feng, Y., Chen, J., Wang, Z., & Qiao, H. (2018). Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications. In Proceedings of the 2018 World Wide Web Conference (pp. 187–196). International World Wide Web Conferences Steering Committee. https://doi.org/10.1145/3178876.3185996

[10] Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., & Pei, D. (2019). Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2828–2837). Association for Computing Machinery. https://doi.org/10.1145/3292500.3330672

[11] Cai, H., Ren, K., Zhang, W., Malialis, K., Wang, J., Yu, Y., & Guo, D. (2017). Real-time bidding by reinforcement learning in display advertising. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining (pp. 661–670). Association for Computing Machinery. https://doi.org/10.1145/3018661.3018702

[12] Lee, K.-C., Jalali, A., & Dasdan, A. (2013). Real time bid optimization with smooth budget delivery in online advertising. In Proceedings of the Seventh International Workshop on Data Mining for Online Advertising (ADKDD '13). https://arxiv.org/abs/1305.3011

[13] Adams, R. P., & MacKay, D. J. C. (2007). Bayesian online changepoint detection. arXiv. https://arxiv.org/abs/0710.3742

[14] Zhang, W., Yuan, S., Wang, J., & Shen, X. (2014). Real-time bidding benchmarking with iPinYou dataset. arXiv. https://arxiv.org/abs/1407.7073

[15] Taylor, S. J., & Letham, B. (2018). Forecasting at scale. The American Statistician, 72(1), 37–45. https://doi.org/10.1080/00031305.2017.1380080

[16] Lorden, G. (1971). Procedures for reacting to a change in distribution. The Annals of Mathematical Statistics, 42(6), 1897–1908. https://doi.org/10.1214/aoms/1177693055

[17] Siffer, A., Fouque, P.-A., Termier, A., & Largouët, C. (2017). Anomaly detection in streams with extreme value theory. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1067–1075). Association for Computing Machinery. https://doi.org/10.1145/3097983.3098144

[18] Hundman, K., Constantinou, V., Laporte, C., Colwell, I., & Söderström, T. (2018). Detecting spacecraft anomalies using LSTMs and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 387–395). Association for Computing Machinery. https://doi.org/10.1145/3219819.3219845

[19] Xu, J., Wu, H., Wang, J., & Long, M. (2022). Anomaly Transformer: Time series anomaly detection with association discrepancy. In International Conference on Learning Representations (ICLR 2022). https://arxiv.org/abs/2110.02642

[20] Salinas, D., Flunkert, V., Gasthaus, J., & Januschowski, T. (2020). DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3), 1181–1191. https://doi.org/10.1016/j.ijforecast.2019.07.001

[21] Künzel, S. R., Sekhon, J. S., Bickel, P. J., & Yu, B. (2019). Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 116(10), 4156–4165. https://doi.org/10.1073/pnas.1804597116

[22] Diemert, E., Betlei, A., Renaudin, C., & Amini, M.-R. (2018). A large scale benchmark for uplift modeling. In Proceedings of the AdKDD and TargetAd Workshop at KDD 2018. https://ailab.criteo.com/criteo-uplift-prediction-dataset/

Detecting Performance Anomalies in Digital Advertising Campaigns: A Comparative Evaluation of Rule-Based Monitoring, Statistical Control Charts, and Machine-Learning Detectors