Creator:

Chen, Hao ; Mo, Zhanfeng ; Yang, Zhouwang

Contributor:

Korbicz, Józef (1951- ) - red. ; Uciński, Dariusz - red.

Title:

The generalization error bound for a stochastic gradient descent family via a Gaussian approximation method

Group publication title:

AMCS, volume 35 (2025)

Subject and Keywords:

stochastic gradient descent ; Gaussian approximation ; KL-divergence ; generalization bounds

Abstract:

Recent works have developed model complexity based and algorithm based generalization error bounds to explain how stochastic gradient descent (SGD) methods help over-parameterized models generalize better. However, previous works are limited by their scope of analysis and fail to provide comprehensive explanations. In this paper, we propose a novel Gaussian approximation framework to establish generalization error bounds for the U-SGD family, which is a class of SGD with asymptotically unbiased and uniformly bounded gradient noise. We study U-SGD dynamics, and we show both theoretically and numerically that the limiting model parameter distribution tends to be Gaussian, even when the original gradient noise is non-Gaussian.

Publisher:

Zielona Góra: Uniwersytet Zielonogórski

Date:

2025

Resource Type:

artykuł

DOI:

10.61822/amcs-2025-0018

Pages:

251-266

Source:

AMCS, volume 35, number 2 (2025) ; click here to follow the link

Language:

eng

License CC BY 4.0:

click here to follow the link

Rights:

Biblioteka Uniwersytetu Zielonogórskiego