The bandwidth of the communication networks has been increased continuously as results of technological advances. However, the introduction of new services and the expansion of the existing ones have ...resulted in even higher demand for the bandwidth. This explains the many efforts currently being invested in the area of data compression. The primary goal of these works is to develop techniques of coding information sources such as speech, image and video to reduce the number of bits required to represent a source without significantly degrading its quality.
With the large increase in the generation of digital image data, there has been a correspondingly large increase in research activity in the field of image compression. The goal is to represent an image in the fewest number of bits without losing the essential information content within. Images carry three main type of information: redundant, irrelevant, and useful. Redundant information is the deterministic part of the information, which can be reproduced without loss from other information contained in the image. Irrelevant information is the part of information that has enormous details, which are beyond the limit of perceptual significance (i.e., psychovisual redundancy). Useful information, on the other hand, is the part of information, which is neither redundant nor irrelevant. Human usually observes decompressed images. Therefore, their fidelities are subject to the capabilities and limitations of the Human Visual System.
This paper provides a survey on various image compression techniques, their limitations, compression rates and highlights current research in medical image compression.
Built on deep networks, end-to-end optimized image compression has made impressive progress in the past few years. Previous studies usually adopt a compressive auto-encoder, where the encoder part ...first converts image into latent features, and then quantizes the features before encoding them into bits. Both the conversion and the quantization incur information loss, resulting in a difficulty to optimally achieve arbitrary compression ratio. We propose iWave++ as a new end-to-end optimized image compression scheme, in which iWave, a trained wavelet-like transform, converts images into coefficients without any information loss. Then the coefficients are optionally quantized and encoded into bits. Different from the previous schemes, iWave++ is versatile : a single model supports both lossless and lossy compression, and also achieves arbitrary compression ratio by simply adjusting the quantization scale. iWave++ also features a carefully designed entropy coding engine to encode the coefficients progressively, and a de-quantization module for lossy compression. Experimental results show that lossy iWave++ achieves state-of-the-art compression efficiency compared with deep network-based methods; on the Kodak dataset, lossy iWave++ leads to 17.34 percent bits saving over BPG; lossless iWave++ achieves comparable or better performance than FLIF. Our code and models are available at https://github.com/mahaichuan/Versatile-Image-Compression .
This paper describes a recently created image database, TID2013, intended for evaluation of full-reference visual quality assessment metrics. With respect to TID2008, the new database contains a ...larger number (3000) of test images obtained from 25 reference images, 24 types of distortions for each reference image, and 5 levels for each type of distortion. Motivations for introducing 7 new types of distortions and one additional level of distortions are given; examples of distorted images are presented. Mean opinion scores (MOS) for the new database have been collected by performing 985 subjective experiments with volunteers (observers) from five countries (Finland, France, Italy, Ukraine, and USA). The availability of MOS allows the use of the designed database as a fundamental tool for assessing the effectiveness of visual quality. Furthermore, existing visual quality metrics have been tested with the proposed database and the collected results have been analyzed using rank order correlation coefficients between MOS and considered metrics. These correlation indices have been obtained both considering the full set of distorted images and specific image subsets, for highlighting advantages and drawbacks of existing, state of the art, quality metrics. Approaches to thorough performance analysis for a given metric are presented to detect practical situations or distortion types for which this metric is not adequate enough to human perception. The created image database and the collected MOS values are freely available for downloading and utilization for scientific purposes.
•We have created a new large database.•This database contains larger number of distorted images and distortion types.•MOS values for all images are obtained and provided.•Analysis of correlation between MOS and a wide set of existing metrics is carried out.•Methodology for determining drawbacks of existing visual quality metrics is described.
•An efficient and lightweight DNN for smart meter data compression is proposed.•The reconstruction error and computation time is significantly reduced.•The lightweight model is suitable for running ...on embedded devices.•Grouping compression is proposed to further improve the compression effect.
With the wide deployment of smart meters in distribution systems, a new challenge emerges for the storage and transmission of huge volume of power consumption data collected by smart meters. In this paper, a deep-learning-based compression method for smart meter data is proposed via stacked convolutional sparse auto-encoder (SCSAE). An efficient and lightweight auto-encoder structure is first designed by leveraging the unique characteristics of smart meter readings. Specifically, the encoder is designed based on 2D separable convolution layers and the decoder is based on transposed convolution layers. Compared with the existing auto-encoder method and traditional methods, the proposed structure is redesigned, and the parameters and reconstruction errors are efficiently reduced. In addition, cluster-based indexes are used to represent the regularity of power consumption behavior and the relationship between electricity consumption behavior and compression effect is studied. Case studies illustrate that the proposed method can attain significant enhancement in model size, computational efficiency, and reconstruction error reduction while maintaining the most abundant details. And grouping compression considering users’ electricity consumption rules can further improve the compression effect.
Nowadays, data reduction is becoming increasingly important in dealing with the large amounts of scientific data. Existing multilevel compression algorithms offer a promising way to manage scientific ...data at scale, but may suffer from relatively low performance and reduction quality. In this paper, we propose MGARD+, a multilevel data reduction and refactoring framework drawing on previous multilevel methods, to achieve high-performance data decomposition and high-quality error-bounded lossy compression. Our contributions are four-fold: 1) We propose to leverage a level-wise coefficient quantization method, which uses different error tolerances to quantize the multilevel coefficients. 2) We propose an adaptive decomposition method which treats the multilevel decomposition as a preconditioner and terminates the decomposition process at an appropriate level. 3) We leverage a set of algorithmic optimization strategies to significantly improve the performance of multilevel decomposition/recomposition. 4) We evaluate our proposed method using four real-world scientific datasets and compare with several state-of-the-art lossy compressors. Experiments demonstrate that our optimizations improve the decomposition/recomposition performance of the existing multilevel method by up to <inline-formula><tex-math notation="LaTeX">70 \times</tex-math> <mml:math><mml:mrow><mml:mn>70</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="liang-ieq1-3092201.gif"/> </inline-formula>, and the proposed compression method can improve compression ratio by up to <inline-formula><tex-math notation="LaTeX">2 \times</tex-math> <mml:math><mml:mrow><mml:mn>2</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="liang-ieq2-3092201.gif"/> </inline-formula> compared with other state-of-the-art error-bounded lossy compressors under the same level of data distortion.
The Joint Laboratory on Extreme-Scale Computing (JLESC) was initiated at the same time lossy compression for scientific data became an important topic for the scientific communities. The teams ...involved in the JLESC played and are still playing an important role in developing the research, techniques, methods, and technologies making lossy compression for scientific data a key tool for scientists and engineers. In this paper, we present the evolution of lossy compression for scientific data from 2015, describing the situation before the JLESC started, the evolution of this discipline in the past 8 years (until 2023) through the prism of the JLESC collaborations on this topic and some of the remaining open research questions.
Internet of Things (IoT) and wearable systems are very resource limited in terms of power, memory, bandwidth and processor performance. Sensor time series compression can be regarded as a direct way ...to use memory and bandwidth resources efficiently. On the other hand, the time series classification has recently attracted great attention and has found numerous potential uses in areas such as finance, industry and healthcare. This paper investigates the effect of lossy compression techniques on the time series classification task using deep neural networks. Furthermore, this paper proposes an efficient compression approach for univariate and multivariate time series that combines the lifting implementation of the discrete wavelet transform with an error-bound compressor, namely Squeeze (SZ), to attain an optimal trade-off between data compression and data quality.
With ever-increasing volumes of scientific data produced by high-performance computing applications, significantly reducing data size is critical because of limited capacity of storage space and ...potential bottlenecks on I/O or networks in writing/reading or transferring data. SZ and ZFP are two leading BSD licensed open source C/C++ libraries for compressed floating-point arrays that support high throughput read and write random access. However, their performance is not consistent across different data sets and across different fields of some data sets, which raises the need for an automatic online (during compression) selection between SZ and ZFP, with minimal overhead. In this paper, the automatic selection optimizes the rate-distortion, an important statistical quality metric based on the signal-to-noise ratio. To optimize for rate-distortion, we investigate the principles of SZ and ZFP. We then propose an efficient online, low-overhead selection algorithm that predicts the compression quality accurately for two compressors in early processing stages and selects the best-fit compressor for each data field. We implement the selection algorithm into an open-source library, and we evaluate the effectiveness of our proposed solution against plain SZ and ZFP in a parallel environment with 1,024 cores. Evaluation results on three data sets representing about 100 fields show that our selection algorithm improves the compression ratio up to 70 percent with the same level of data distortion because of very accurate selection (around 99 percent) of the bestfit compressor, with little overhead (less than 7 percent in the experiments).
PDF
