An Empirical Analysis of the Temperature Hyperparameter in Knowledge Distillation for Image Classification Student Networks

Hongbo Wang; Yunqiang Sun

An Empirical Analysis of the Temperature Hyperparameter in Knowledge Distillation for Image Classification Student Networks

Authors

Hongbo Wang Computer Science, University of Southern California, CA, USA
Yunqiang Sun Computer Science and Technology, University of Jinan, Shandong, China

Keywords:

knowledge distillation, temperature hyperparameter, image classification, model compression

Abstract

Knowledge distillation transfers the softened class probabilities of a large teacher network to a compact student, and the temperature applied to those probabilities governs how much inter-class structure the student receives. While the temperature is widely treated as a routine hyperparameter, the magnitude and direction of its effect on a small student remain unevenly documented across the image-classification literature. This work reports a controlled empirical analysis that isolates the temperature while holding the teacher, the student, the loss weighting, and the optimisation budget fixed. A three-layer convolutional teacher (458,890 parameters) is distilled into a two-layer student (53,370 parameters) on Fashion-MNIST, and the temperature is swept across nine values from 1 to 20, each repeated over three random seeds. The distilled student attains its best top-1 accuracy of 85.20% at the lowest temperature, a moderate gain of 0.64 percentage points over the undistilled baseline of 84.56%, after which accuracy declines monotonically and falls below the baseline at the highest temperatures. Teacher-student prediction agreement follows the same descending pattern, while the entropy of the softened teacher targets rises steadily toward the uniform limit. The results indicate that, for a high-compression student under a fixed loss weight, restrained softening is preferable to aggressive softening, and they caution against treating large temperatures as a default choice.

References

Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.

Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021). Knowledge distillation: A survey. International Journal of Computer Vision, 129(6), 1789–1819.

Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018). MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4510–4520).

Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). FitNets: Hints for thin deep nets. In International Conference on Learning Representations.

Zagoruyko, S., & Komodakis, N. (2017). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Representations.

Park, W., Kim, D., Lu, Y., & Cho, M. (2019). Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3967–3976).

Tian, Y., Krishnan, D., & Isola, P. (2020). Contrastive representation distillation. In International Conference on Learning Representations.

Zhao, B., Cui, Q., Song, R., Qiu, Y., & Liang, J. (2022). Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11953–11962).

Zhang, Y., Xiang, T., Hospedales, T. M., & Lu, H. (2018). Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4320–4328).

Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., & Ghasemzadeh, H. (2020). Improved knowledge distillation via teacher assistant. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 5191–5198.

Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 1321–1330).

Müller, R., Kornblith, S., & Hinton, G. (2019). When does label smoothing help? In Advances in Neural Information Processing Systems (Vol. 32, pp. 4694–4703).

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770–778).

Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A. A., & Wilson, A. G. (2021). Does knowledge distillation really work? In Advances in Neural Information Processing Systems (Vol. 34, pp. 6906–6919).

Cho, J. H., & Hariharan, B. (2019). On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4794–4802).

Li, Z., Li, X., Yang, L., Zhao, B., Song, R., Luo, L., Li, J., & Yang, J. (2023). Curriculum temperature for knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 37(2), 1504–1512.

Sun, S., Ren, W., Li, J., Wang, R., & Cao, X. (2024). Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15731–15740).

Jafari, A., Rezagholizadeh, M., Sharma, P., & Ghodsi, A. (2021). Annealing knowledge distillation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (pp. 2493–2504).

An Empirical Analysis of the Temperature Hyperparameter in Knowledge Distillation for Image Classification Student Networks