Interpolators-estimators that achieve zero training error-have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In ...this paper, we study minimum ℓ2 norm ("ridgeless") interpolation least squares regression, focusing on the high-dimensional regime in which the number of unknown parameters p is of the same order as the number of samples n. We consider two different models for the feature distribution: a linear model, where the feature vectors xi ∈ Rp are obtained by applying a linear transform to a vector of i.i.d. entries, xi = Σ1/2 zi (with zi ∈ Rp); and a nonlinear model, where the feature vectors are obtained by passing the input through a random one-layer neural network, xi = φ(Wzi) (with zi ∈ Rd, W ∈ Rp×d a matrix of i.i.d. entries, and φ an activation function acting componentwise on Wzi). We recover-in a precise quantitative way-several phenomena that have been observed in large-scale neural networks and kernel machines, including the "double descent" behavior of the prediction risk, and the potential benefits of overparametrization.
Currently, studies on vegetation are limited to absolute temperature changes, with insufficient attention directed to the intricate and complex connections between relative temperature (Tr) changes ...and vegetation productivity. This study analyzed the impact of Tr change on vegetation growth by estimating the effects of temperature change, CO2 levels, and precipitation on the vegetation index. The analysis used changes in temperature ordination as a sign of Tr change and employed ridge regression analysis, trend analysis, correlation analysis, and contribution methods. The results indicated that the mean trend of Tr change in China was negative, suggesting that the rate of Tr decreased as compared to the rate of Tr increase, leading to most regions in China becoming relatively colder. Regions experiencing a decrease in Tr were more favorable to vegetation growth due to stable temperatures, while regions with increasing Tr faced intensified water stress and inhibitory effects on vegetation, except in cold regions with sufficient precipitation. Overall, Tr in China had a beneficial impact on the vegetation index, with a lesser effect compared to CO2 and precipitation, but more than temperature, highlighting the significance of Tr in promoting vegetation growth. This study expanded the understanding of the impact of global warming on vegetation by incorporating the novel idea of Tr change and quantifying its consequences for vegetation.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
It is a common saying that testing for conditional independence, that is, testing whether whether two random vectors X and Y are independent, given Z, is a hard statistical problem if Z is a ...continuous random variable (or vector). In this paper, we prove that conditional independence is indeed a particularly difficult hypothesis to test for. Valid statistical tests are required to have a size that is smaller than a pre-defined significance level, and different tests usually have power against a different class of alternatives. We prove that a valid test for conditional independence does not have power against any alternative.
Given the nonexistence of a uniformly valid conditional independence test, we argue that tests must be designed so their suitability for a particular problem may be judged easily. To address this need, we propose in the case where X and Y are univariate to nonlinearly regress X on Z, and Y on Z and then compute a test statistic based on the sample covariance between the residuals, which we call the generalised covariance measure (GCM). We prove that validity of this form of test relies almost entirely on the weak requirement that the regression procedures are able to estimate the conditional means X given Z, and Y given Z, at a slow rate. We extend the methodology to handle settings where X and Y may be multivariate or even high dimensional. While our general procedure can be tailored to the setting at hand by combining it with any regression technique, we develop the theoretical guarantees for kernel ridge regression. A simulation study shows that the test based on GCM is competitive with state of the art conditional independence tests. Code is available as the R package GeneralisedCovarianceMeasure on CRAN.
This paper focuses on parameter selection issues of kernel ridge regression (KRR). Due to special spectral properties of KRR, we find that delicate subdivision of the parameter interval shrinks the ...difference between two successive KRR estimates. Based on this observation, we develop an early-stopping type parameter selection strategy for KRR according to the so-called Lepskii-type principle. Theoretical verifications are presented in the framework of learning theory to show that KRR equipped with the proposed parameter selection strategy succeeds in achieving optimal learning rates and adapts to different norms, providing a new record of parameter selection for kernel methods.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
This paper describes the use of Kernel Ridge Regression (KRR) and Kernel Ridge Regression Confidence Machine (KRRCM) for black box identification of a surface marine vehicle. Data for training and ...test have been obtained from several manoeuvres typically used for marine system identification. Thus, a 20/20 degrees Zig-Zag, a 10/10 degrees Zig-Zag, and different evolution circles have been employed for the computation and validation of the model. Results show that the application of conformal prediction provides an accurate model that reproduces with large accuracy the actual behaviour of the ship with confidence margins that ensure that the model response is within these margins, making it a suitable tool for system identification.
•Black box identification based on Conformal Predictors is used for marine vehicles.•Classical manoeuvres for marine vehicle identification are used to collect data.•A continuous-time model is trained and tested using data from real experiments.•Modelling with Kernel Ridge Regression and Kernel Ridge Regression Confidence Machine.•A confidence margin is proposed where the real behaviour of the vehicle should lie in.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
We consider the problem of learning an unknown function f⋆ on the d-dimensional sphere with respect to the square loss, given i.i.d. samples {(yi, xi)}i≤n where xi is a feature vector uniformly ...distributed on the sphere and yi = f⋆ (xi) + εi. We study two popular classes of models that can be regarded as linearizations of two-layers neural networks around a random initialization: the random features model of Rahimi–Recht (RF); the neural tangent model of Jacot–Gabriel–Hongler (NT). Both these models can also be regarded as randomized approximations of kernel ridge regression (with respect to different kernels), and enjoy universal approximation properties when the number of neurons N diverges, for a fixed dimension d. We consider two specific regimes: the infinite-sample finite-width regime, in which n = ∞ while d and N are large but finite, and the infinite-width finite-sample regime in which N = ∞ while d and n are large but finite. In the first regime, we prove that if dℓ+δ ≤ N ≤ dℓ+1−δ for small δ > 0, then RF effectively fits a degree-ℓ polynomial in the raw features, and NT fits a degree-(ℓ + 1) polynomial. In the second regime, both RF and NT reduce to kernel methods with rotationally invariant kernels. We prove that, if the sample size satisfies dℓ+δ ≤ n ≤ dℓ+1−δ, then kernel methods can fit at most a degree-ℓ polynomial in the raw features. This lower bound is achieved by kernel ridge regression, and near-optimal prediction error is achieved for vanishing ridge regularization.
•A new Takagi-Sugeno system based Kernel ridge regression (TS-KRR) was proposed.•The TS-KRR strategy is implemented for both adaptive and offline identification.•The TS-KRR was integrated with the ...GPC to control discrete-time nonlinear systems.•The proposed controller showed good results in TS fuzzy GPC with offline modeling.•The adaptive TS fuzzy GPC showed good results in dealing with disturbances.
In this paper, a novel fuzzy Generalized Predictive Control (GPC) is proposed for discrete-time nonlinear systems via Takagi-Sugeno system based Kernel Ridge Regression (TS-KRR). The TS-KRR strategy approximates the unknown nonlinear systems by learning the Takagi-Sugeno (TS) fuzzy parameters from the input-output data. Two main steps are required to construct the TS-KRR: the first step is to use a clustering algorithm such as the clustering based Particle Swarm Optimization (PSO) algorithm that separates the input data into clusters and obtains the antecedent TS fuzzy model parameters. In the second step, the consequent TS fuzzy parameters are obtained using a Kernel ridge regression algorithm. Furthermore, the TS based predictive control is created by integrating the TS-KRR into the Generalized Predictive Controller. Next, an adaptive, online, version of TS-KRR is proposed and integrated with the GPC controller resulting an efficient adaptive fuzzy generalized predictive control methodology that can deal with most of the industrial plants and has the ability to deal with disturbances and variations of the model parameters. In the adaptive TS-KRR algorithm, the antecedent parameters are initialized with a simple K-means algorithm and updated using a simple gradient algorithm. Then, the consequent parameters are obtained using the sliding-window Kernel Recursive Least squares (KRLS) algorithm. Finally, two nonlinear systems: A surge tank and Continuous Stirred Tank Reactor (CSTR) systems were used to investigate the performance of the new adaptive TS-KRR GPC controller. Furthermore, the results obtained by the adaptive TS-KRR GPC controller were compared with two other controllers. The numerical results demonstrate the reliability of the proposed adaptive TS-KRR GPC method for discrete-time nonlinear systems.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UL, UM, UPCLJ, UPUK, ZRSKP
Reservoir computing has emerged in the last decade as an alternative to gradient descent methods for training recurrent neural networks. Echo State Network (ESN) is one of the key reservoir computing ...“flavors”. While being practical, conceptually simple, and easy to implement, ESNs require some experience and insight to achieve the hailed good performance in many tasks. Here we present practical techniques and recommendations for successfully applying ESNs, as well as some more advanced application-specific modifications.
PDF
