Deep neural networks (DNNs) have achieved state-of-the-art performance in a number of domains but suffer intensive complexity. Network quantization can effectively reduce computation and memory costs ...without changing network structure, facilitating the deployment of DNNs on mobile devices. While the existing methods can obtain good performance, low-bit quantization without time-consuming training or access to the full dataset is still a challenging problem. In this paper, we develop a novel method named Compressorbased non-uniform quantization (CNQ) method to achieve non-uniform quantization of DNNs with few unlabeled samples. Firstly, we present a compressor-based fast nonuniform quantization method, which can accomplish nonuniform quantization without iterations. Secondly, we propose to align the feature maps of the quantization model with the pre-trained model for accuracy recovery. Considering the property difference between different activation channels, we utilize the weighted-entropy perchannel to optimize the alignment loss. In the experiments, we evaluate the proposed method on image classification and object detection. Our results outperform the existing post-training quantization methods, which demonstrate the effectiveness of the proposed method.
We consider the commutative limit of matrix geometry described by a large-N sequence of some Hermitian matrices. Under some assumptions, we show that the commutative geometry possesses a Kaehler ...structure. We find an explicit relation between the Kaehler structure and the matrix configurations which define the matrix geometry. We also discuss a relation between the matrix configurations and those obtained from the geometric quantization.
Traditional deep learning models are trained at a centralized server using data samples collected from users. Such data samples often include private information, which the users may not be willing ...to share. Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their data. FL consists of an iterative procedure, where in each iteration the users train a copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model. A major challenge that arises in this method is the need of each user to repeatedly transmit its learned model over the throughput limited uplink channel. In this work, we tackle this challenge using tools from quantization theory. In particular, we identify the unique characteristics associated with conveying trained models over rate-constrained channels, and propose a suitable quantization scheme for such settings, referred to as universal vector quantization for FL (UVeQFed). We show that combining universal vector quantization methods with FL yields a decentralized training system in which the compression of the trained models induces only a minimum distortion. We then theoretically analyze the distortion, showing that it vanishes as the number of users grows. We also characterize how models trained with conventional federated averaging combined with UVeQFed converge to the model which minimizes the loss function. Our numerical results demonstrate the gains of UVeQFed over previously proposed methods in terms of both distortion induced in quantization and accuracy of the resulting aggregated model.
The novel Computing-In-Memory (CIM) technology has demonstrated significant potential in enhancing the performance and efficiency of convolutional neural networks (CNNs). However, due to the low ...precision of memory devices and data interfaces, an additional quantization step is necessary. Conventional NN quantization methods fail to account for the hardware characteristics of CIM, resulting in inferior system performance and efficiency. This paper proposes CIMQ, a hardware-efficient quantization framework designed to improve the efficiency of CIM based NN accelerators. The holistic framework focuses on the fundamental computing elements in CIM hardware: inputs, weights and outputs (or activations, weights and partial sums in NNs) with four innovative techniques. Firstly, bit-level sparsity induced activation quantization is introduced to decrease dynamic computation energy. Secondly, inspired by the unique computation paradigm of CIM, an innovative array-wise quantization granularity is proposed for weight quantization. Thirdly, partial sums are quantized with a reparametrized clipping function to reduce the required resolution of analog-to-digital converters (ADCs). Finally, to improve the accuracy of quantized neural networks (QNNs), the post-training quantization (PTQ) is enhanced with a random quantization dropping strategy. The effectiveness of the proposed framework has been demonstrated through experimental results on various NNs and datasets (CIFAR10, CIFAR100, ImageNet). In typical cases, the hardware efficiency can be improved up to 222% with a 58.97% improvement in accuracy compared to conventional quantization methods.
The quantization of the output of a binary-input discrete memoryless channel to a smaller number of levels is considered. An algorithm, which finds an optimal quantizer, in the sense of maximizing ...mutual information between the channel input and quantizer output is given. This result holds for arbitrary channels, in contrast to previous results for restricted channels or a restricted number of quantizer outputs. In the worst case, the algorithm complexity is cubic M 3 in the number of channel outputs M. Optimality is proved using the theorem of Burshtein, Della Pietra, Kanevsky, and Nádas for mappings, which minimize average impurity for classification and regression trees.
This paper investigates the performance of limited-fronthaul cell-free massive multiple-input multiple-output (MIMO) taking account the fronthaul quantization and imperfect channel acquisition. Three ...cases are studied, which we refer to as Estimate&Quantize, Quantize&Estimate, and Decentralized, according to where channel estimation is performed and exploited. Maximum-ratio combining (MRC), zero-forcing (ZF), and minimum mean-square error (MMSE) receivers are considered. The Max algorithm and the Bussgang decomposition are exploited to model optimum uniform quantization. Exploiting the optimal step size of the quantizer, analytical expressions for spectral and energy efficiencies are presented. Finally, an access point (AP) assignment algorithm is proposed to improve the performance of the decentralized scheme. Numerical results investigate the performance gap between limited fronthaul and perfect fronthaul cases, and demonstrate that exploiting relatively few quantization bits, the performance of limited-fronthaul cell-free massive MIMO closely approaches the perfect-fronthaul performance.
On the Best Lattice Quantizers Agrell, Erik; Allen, Bruce
IEEE transactions on information theory,
12/2023, Letnik:
69, Številka:
12
Journal Article
Recenzirano
Odprti dostop
A lattice quantizer approximates an arbitrary real-valued source vector with a vector taken from a specific discrete lattice. The quantization error is the difference between the source vector and ...the lattice vector. In a classic 1996 paper, Zamir and Feder show that the globally optimal lattice quantizer (which minimizes the mean square error) has white quantization error: for a uniformly distributed source, the covariance of the error is the identity matrix, multiplied by a positive real factor. We generalize the theorem, showing that the same property holds (i) for any lattice whose mean square error cannot be decreased by a small perturbation of the generator matrix, and (ii) for an optimal product of lattices that are themselves locally optimal in the sense of (i). We derive an upper bound on the normalized second moment (NSM) of the optimal lattice in any dimension, by proving that any lower- or upper-triangular modification to the generator matrix of a product lattice reduces the NSM. Using these tools and employing the best currently known lattice quantizers to build product lattices, we construct improved lattice quantizers in dimensions 13 to 15, 17 to 23, and 25 to 48. In some dimensions, these are the first reported lattices with normalized second moments below the best known upper bound.
As data resources grow, providing recommendations that best meet the demands has become a vital requirement in business and life to overcome the information overload problem. However, building a ...system suggesting relevant recommendations has always been a point of debate. One of the most cost-efficient techniques in terms of producing relevant recommendations at a low complexity is product quantization (PQ). PQ approaches have continued developing in recent years. This system's crucial challenge is improving PQ performance in terms of recall measures without compromising its complexity. This makes the algorithm suitable for problems that require a greater number of potentially relevant items without disregarding others, at high speed and low cost to keep up with traffic. This is the case of online shops where the recommendations for the purpose are important, although customers can be susceptible to scoping other products. A recent approach has been exploiting the notion of norm subvectors encoded in product quantizers. This research proposes a fuzzy approach to perform norm-based PQ. Type-2 fuzzy sets (T2FSs) define the codebook allowing subvectors (T2FSs) to be associated with more than one element of the codebook, and next, its norm calculus is resolved by means of integration. Our method finesses the recall measure up, making the algorithm suitable for problems that require querying at most possible potential relevant items without disregarding others. The proposed approach is tested with three public recommender benchmark datasets and compared against seven PQ approaches for maximum inner-product search. The proposed method outperforms all PQ approaches, such as norm-explicit PQ, PQ, and residual quantization up to +6%, +5%, and +8% by achieving a recall of 94%, 69%, and 59% in Netflix, Audio, and Cifar60k datasets, respectively. Moreover, computing time and complexity nearly equal those of the most computationally efficient existing PQ method in the state of the art.
This book is a collection of expository articles from the Center of Mathematics at Notre Dame's 2011 program on quantization. Included are lecture notes from a summer school on quantization on topics ...such as the Cherednik algebra, geometric quantization, detailed proofs of Willwacher's results on the Kontsevich graph complex, and group-valued moment maps. This book also includes expository articles on quantization and automorphic forms, renormalization, Berezin-Toeplitz quantization in the complex setting, and the commutation of quantization with reduction, as well as an original article on derived Poisson brackets. The primary goal of this volume is to make topics in quantization more accessible to graduate students and researchers.
This paper considers a control problem of discrete-time MIMO linear systems involving input quantization (for the output of the controller) and output quantization (for the output of the plant). ...Based on spherical polar coordinate quantizer, a quantization method with finite data rate is proposed for handling the nested quantization and achieving the convergence to the origin of the system with input and output quantization. Further, an optimalization problem is developed for each quantizer, of which the optimal solution set contains a virtual augmented vector. The quantizers quantize their respective virtual augmented vectors to obtain the estimates of the outputs of the plant and the controller. We present the analytical expression of the optimal solution set for each quantizer. Finally, a method is presented to achieve the parameters of the quantizers and the controller.