E-resources
-
Ghorbani, Behrooz; Mei, Song; Misiakiewicz, Theodor; Montanari, Andrea
Annals of statistics, 04/2021, Volume: 49, Issue: 2Journal Article
We consider the problem of learning an unknown function f⋆ on the d-dimensional sphere with respect to the square loss, given i.i.d. samples {(yi, xi)}i≤n where xi is a feature vector uniformly distributed on the sphere and yi = f⋆ (xi) + εi. We study two popular classes of models that can be regarded as linearizations of two-layers neural networks around a random initialization: the random features model of Rahimi–Recht (RF); the neural tangent model of Jacot–Gabriel–Hongler (NT). Both these models can also be regarded as randomized approximations of kernel ridge regression (with respect to different kernels), and enjoy universal approximation properties when the number of neurons N diverges, for a fixed dimension d. We consider two specific regimes: the infinite-sample finite-width regime, in which n = ∞ while d and N are large but finite, and the infinite-width finite-sample regime in which N = ∞ while d and n are large but finite. In the first regime, we prove that if dℓ+δ ≤ N ≤ dℓ+1−δ for small δ > 0, then RF effectively fits a degree-ℓ polynomial in the raw features, and NT fits a degree-(ℓ + 1) polynomial. In the second regime, both RF and NT reduce to kernel methods with rotationally invariant kernels. We prove that, if the sample size satisfies dℓ+δ ≤ n ≤ dℓ+1−δ, then kernel methods can fit at most a degree-ℓ polynomial in the raw features. This lower bound is achieved by kernel ridge regression, and near-optimal prediction error is achieved for vanishing ridge regularization.
Shelf entry
Permalink
- URL:
Impact factor
Access to the JCR database is permitted only to users from Slovenia. Your current IP address is not on the list of IP addresses with access permission, and authentication with the relevant AAI accout is required.
Year | Impact factor | Edition | Category | Classification | ||||
---|---|---|---|---|---|---|---|---|
JCR | SNIP | JCR | SNIP | JCR | SNIP | JCR | SNIP |
Select the library membership card:
If the library membership card is not in the list,
add a new one.
DRS, in which the journal is indexed
Database name | Field | Year |
---|
Links to authors' personal bibliographies | Links to information on researchers in the SICRIS system |
---|
Source: Personal bibliographies
and: SICRIS
The material is available in full text. If you wish to order the material anyway, click the Continue button.