Speeding up L2-loss support vector regression by random Fourier features

Abstract

To avoid the expensive quadratic programming in the L2-loss support vector regression (SVR) model, smooth approximation and iteratively reweighted least square (IRLS) techniques were introduced in literature, resulting in smoothed SVR (SSVR) and IRLS-SVR. However, for nonlinear models, SSVR and IRLS-SVR both involve operations with matrices and vectors of the same size as the training set. Thus, as the training set becomes large, nonlinear SSVR and IRLS-SVR both need long training time and large memory. To further alleviate the training cost, this paper projects the original data into a low dimensional space via random Fourier feature. The inner product of the random Fourier features of two data points is approximately the same as the kernel function evaluated at these two data points. Hence, it is possible to use a linear model in the new low dimensional space to approximate the original nonlinear model, and consequently the time/memory efficient linear training algorithms could be applied. This paper applies the idea of random Fourier features to nonlinear SSVR and IRLS-SVR, and our testing results on real-world datasets show that, the introduction of random Fourier features makes SSVR and IRLS-SVR achieve similar prediction accuracy as the original nonlinear version with substantially higher time efficiency.

Department(s)

Mathematics

Document Type

Article

DOI

10.1080/03610918.2022.2037638

Keywords

Random Fourier features, Smooth approximation, Squared ?-insensitive loss function, Support vector regression, Weighted least square

Publication Date

1-1-2024

Journal Title

Communications in Statistics Simulation and Computation

Link to Full Text

Share

COinS