How Far Have We Debiased? An Empirical Study of the Accuracy–Popularity-Debiasing Trade-off Across Recommendation Datasets

Shang Wen; Haobin Pan

How Far Have We Debiased? An Empirical Study of the Accuracy–Popularity-Debiasing Trade-off Across Recommendation Datasets

Authors

Shang Wen International Affairs, Science and Technology, Georgia Institute of Technology, USA
Haobin Pan Computer Application Technology, Institute of Computing Technology, Beijing, China

Keywords:

recommender systems, popularity bias, debiasing, empirical evaluation

Abstract

Popularity bias remains one of the most studied distortions in recommender systems, yet the practical question of how much accuracy must be surrendered to reduce it is rarely answered in a controlled, cross-dataset manner. This paper reports an empirical study of the accuracy–popularity-debiasing trade-off on the three public recommendation datasets that supply an unbiased test set — Coat, Yahoo! R3, and the fully observed KuaiRec, whose test data are gathered under random or near-complete exposure; two further datasets enter only to characterize the range of popularity concentration, not the accuracy comparison. Eleven debiasing methods spanning reweighting, doubly robust estimation, causal intervention, regularization, and post-processing are compared against three standard backbones under a unified evaluation protocol. On the fully observed KuaiRec collection, a popularity-deconfounded method raises catalog coverage from 0.131 to 0.184 while reducing NDCG@20 by only 1.3 percent, whereas the most aggressive re-ranking reaches 0.208 coverage at an accuracy cost above fifteen percent, and only four methods lie on the empirical Pareto frontier. The accuracy cost of aggressive debiasing widens as datasets become sparser and more skewed, with accuracy retention falling to 84.8 percent on the most concentrated collection. No single method dominates the comparison, as the frontier is shared by methods at distinct operating points that range from balanced in-model objectives to aggressive post-processing. These findings indicate that debiasing gains are real but moderate, dataset dependent, and trustworthy only when measured on test data that are themselves free of exposure bias.

References

Chen, J., Dong, H., Wang, X., Feng, F., Wang, M., & He, X. (2023). Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 41(3), Article 67, 1–39. https://doi.org/10.1145/3564284

Klimashevskaia, A., Jannach, D., Elahi, M., & Trattner, C. (2024). A survey on popularity bias in recommender systems. User Modeling and User-Adapted Interaction, 34(5), 1777–1834. https://doi.org/10.1007/s11257-024-09406-0

Steck, H. (2011). Item popularity and recommendation accuracy. In Proceedings of the Fifth ACM Conference on Recommender Systems (pp. 125–132). ACM. https://doi.org/10.1145/2043932.2043957

Zhu, Z., He, Y., Zhao, X., Zhang, Y., Wang, J., & Caverlee, J. (2021). Popularity-opportunity bias in collaborative filtering. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining (pp. 85–93). ACM. https://doi.org/10.1145/3437963.3441820

Schnabel, T., Swaminathan, A., Singh, A., Chandak, N., & Joachims, T. (2016). Recommendations as treatments: Debiasing learning and evaluation. In Proceedings of the 33rd International Conference on Machine Learning (Vol. 48, pp. 1670–1679). PMLR.

Saito, Y., Yaginuma, S., Nishino, Y., Sakata, H., & Nakata, K. (2020). Unbiased recommender learning from missing-not-at-random implicit feedback. In Proceedings of the 13th ACM International Conference on Web Search and Data Mining (pp. 501–509). ACM. https://doi.org/10.1145/3336191.3371783

Wang, X., Zhang, R., Sun, Y., & Qi, J. (2019). Doubly robust joint learning for recommendation on data missing not at random. In Proceedings of the 36th International Conference on Machine Learning (Vol. 97, pp. 6638–6647). PMLR.

Bonner, S., & Vasile, F. (2018). Causal embeddings for recommendation. In Proceedings of the 12th ACM Conference on Recommender Systems (pp. 104–112). ACM. https://doi.org/10.1145/3240323.3240360

Yang, L., Cui, Y., Xuan, Y., Wang, C., Belongie, S., & Estrin, D. (2018). Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In Proceedings of the 12th ACM Conference on Recommender Systems (pp. 279–287). ACM. https://doi.org/10.1145/3240323.3240367

Zhang, Y., Feng, F., He, X., Wei, T., Song, C., Ling, G., & Zhang, Y. (2021). Causal intervention for leveraging popularity bias in recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 11–20). ACM. https://doi.org/10.1145/3404835.3462875

Wei, T., Feng, F., Chen, J., Wu, Z., Yi, J., & He, X. (2021). Model-agnostic counterfactual reasoning for eliminating popularity bias in recommender system. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (pp. 1791–1800). ACM. https://doi.org/10.1145/3447548.3467289

Zheng, Y., Gao, C., Li, X., He, X., Li, Y., & Jin, D. (2021). Disentangling user interest and conformity for recommendation with causal embedding. In Proceedings of the Web Conference 2021 (pp. 2980–2991). ACM. https://doi.org/10.1145/3442381.3449788

Wang, W., Feng, F., He, X., Wang, X., & Chua, T.-S. (2021). Deconfounded recommendation for alleviating bias amplification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (pp. 1717–1725). ACM. https://doi.org/10.1145/3447548.3467249

Abdollahpouri, H., Burke, R., & Mobasher, B. (2019). Managing popularity bias in recommender systems with personalized re-ranking. In Proceedings of the Thirty-Second International Florida Artificial Intelligence Research Society Conference (pp. 413–418). AAAI Press.

Boratto, L., Fenu, G., & Marras, M. (2021). Connecting user and item perspectives in popularity debiasing for collaborative recommendation. Information Processing & Management, 58(1), Article 102387. https://doi.org/10.1016/j.ipm.2020.102387

Zhang, A., Ma, W., Wang, X., & Chua, T.-S. (2022). Incorporating bias-aware margins into contrastive loss for collaborative filtering. In Advances in Neural Information Processing Systems (Vol. 35, pp. 7866–7878). Curran Associates

How Far Have We Debiased? An Empirical Study of the Accuracy–Popularity-Debiasing Trade-off Across Recommendation Datasets