A wide variety of activation functions have been proposed for neural networks. The Rectified Linear Unit (ReLU) is especially popular today. There are many practical reasons that motivate the use of ...the ReLU. This paper provides new theoretical characterizations that support the use of the ReLU, its variants such as the leaky ReLU, as well as other activation functions in the case of univariate, single-hidden layer feedforward neural networks. Our results also explain the importance of commonly used strategies in the design and training of neural networks such as "weight decay" and "path-norm" regularization, and provide a new justification for the use of "skip connections" in network architectures. These new insights are obtained through the lens of spline theory. In particular, we show how neural network training problems are related to infinite-dimensional optimizations posed over Banach spaces of functions whose solutions are well-known to be fractional and polynomial splines, where the particular Banach space (which controls the order of the spline) depends on the choice of activation function.
The regulatory mechanisms affecting the modulation of the immune system accompanying the progressive effort to exhaustion, particularly associated with T cells, are not fully understood. We analysed ...the impact of two progressive effort protocols on T helper (Th) cell distribution and selected cytokines.
Sixty-two male soccer players with a median age of 17 (16-29) years performed different protocols for progressive exercise until exhaustion: YO-YO (YYRL1) and Beep. Blood samples for all analyses were taken three times: at baseline, post-effort, and in recovery.
The percentage of Th1 cells increased post-effort and in recovery. The post-effort percentage of Th1 cells was higher in the Beep group compared to the YYRL1 group. Significant post-effort increase in Th17 cells was observed in both groups. The post-effort percentage of regulatory T cells (Treg) increased in the Beep group. An increased post-effort concentration of IL-2, IL-6, IL-8 and IFN-γ in both groups was observed. Post-effort TNF-α and IL-10 levels were higher than baseline in the YYRL1 group, while the post-effort IL-17A concentration was lower than baseline only in the Beep group. The recovery IL-2, IL-4, TNF-α and IFN-γ levels were higher than baseline in the YYRL1 group. The recovery IL-4, IL-6, IL-8, TNF-α and IFN-γ values were higher than baseline in the Beep group.
The molecular patterns related to cytokine secretion are not the same between different protocols for progressive effort. It seems that Treg cells are probably the key cells responsible for silencing the inflammation and enhancing anti-inflammatory pathways.
Finding sparse approximate solutions to large underdetermined linear systems of equations is a common problem in signal/image processing and statistics. Basis pursuit, the least absolute shrinkage ...and selection operator (LASSO), wavelet-based deconvolution and reconstruction, and compressed sensing (CS) are a few well-known areas in which problems of this type appear. One standard approach is to minimize an objective function that includes a quadratic ( lscr 2 ) error term added to a sparsity-inducing (usually lscr 1 ) regularizater. We present an algorithmic framework for the more general problem of minimizing the sum of a smooth convex function and a nonsmooth, possibly nonconvex regularizer. We propose iterative methods in which each step is obtained by solving an optimization subproblem involving a quadratic term with diagonal Hessian (i.e., separable in the unknowns) plus the original sparsity-inducing regularizer; our approach is suitable for cases in which this subproblem can be solved much more rapidly than the original problem. Under mild conditions (namely convexity of the regularizer), we prove convergence of the proposed iterative algorithm to a minimum of the objective function. In addition to solving the standard lscr 2 -lscr 1 case, our framework yields efficient solution techniques for other regularizers, such as an lscr infin norm and group-separable regularizers. It also generalizes immediately to the case in which the data is complex rather than real. Experiments with CS problems show that our approach is competitive with the fastest known methods for the standard lscr 2 -lscr 1 problem, as well as being efficient on problems with other separable regularization terms.
Near-Optimal Adaptive Compressed Sensing Malloy, Matthew L.; Nowak, Robert D.
IEEE transactions on information theory,
07/2014, Volume:
60, Issue:
7
Journal Article
Peer reviewed
Open access
This paper proposes a simple adaptive sensing and group testing algorithm for sparse signal recovery. The algorithm, termed compressive adaptive sense and search (CASS), is shown to be near-optimal ...in that it succeeds at the lowest possible signal-to-noise-ratio (SNR) levels, improving on previous work in adaptive compressed sensing. Like traditional compressed sensing based on random nonadaptive design matrices, the CASS algorithm requires only k log n measurements to recover a k-sparse signal of dimension n. However, CASS succeeds at SNR levels that are a factor log n less than required by standard compressed sensing. From the point of view of constructing and implementing the sensing operation as well as computing the reconstruction, the proposed algorithm is substantially less computationally intensive than standard compressed sensing. The CASS is also demonstrated to perform considerably better in practice through simulation. To the best of our knowledge, this is the first demonstration of an adaptive compressed sensing algorithm with near-optimal theoretical guarantees and excellent practical performance. This paper also shows that methods like compressed sensing, group testing, and pooling have an advantage beyond simply reducing the number of measurements or tests- adaptive versions of such methods can also improve detection and estimation performance when compared with nonadaptive direct (uncompressed) sensing.
The Th1 cell subset is involved in the immunological response induced by physical exercise. The aim of this work is to evaluate the post-effort activation of Ras/MAPK and JAK/STAT signaling pathways ...in T cells of young, physically active men. Seventy-six physically active, healthy men between 15 and 21 years old performed a standard physical exercise protocol (Beep test). Phosphorylation levels of Ras/MAPK-(p38 MAPK, ERK1/2) and JAK/STAT-related (STAT1, STAT3, STAT5, and STAT6) proteins were evaluated by flow cytometry in Th and Tc cells post-effort and during the lactate recovery period. The performed physical effort was not a strong enough physiological stimulant to provoke the phosphorylation of ERK1/2, p38 MAPK, STAT1, STAT3, STAT5, and STAT6 in T cells, at least for the duration of our study (the end of the lactate recovery period). We conclude that more observation time-points, including shorter and longer times after the exercise, are required to determine if the Ras/MAPK signaling pathway is involved in modulating the post-effort immunological response.
Third-generation DNA sequencers provided by Oxford Nanopore Technologies (ONT) produce a series of samples of an electrical current in the nanopore. Such a time series is used to detect the sequence ...of nucleotides. The task of translation of current values into nucleotide symbols is called basecalling. Various solutions for basecalling have already been proposed. The earlier ones were based on Hidden Markov Models, but the best ones use neural networks or other machine learning models. Unfortunately, achieved accuracy scores are still lower than competitive sequencing techniques, like Illumina's. Basecallers differ in the input data type-currently, most of them work on a raw data straight from the sequencer (time series of current). Still, the approach of using event data is also explored. Event data is obtained by preprocessing of raw data and dividing it into segments described by several features computed from raw data values within each segment. We propose a novel basecaller that uses joint processing of raw and event data. We define basecalling as a sequence-to-sequence translation, and we use a machine learning model based on an encoder-decoder architecture of recurrent neural networks. Our model incorporates twin encoders and an attention mechanism. We tested our solution on simulated and real datasets. We compare the full model accuracy results with its components: processing only raw or event data. We compare our solution with the existing ONT basecaller-Guppy. Results of numerical experiments show that joint raw and event data processing provides better basecalling accuracy than processing each data type separately. We implement an application called Ravvent, freely available under MIT licence.
The assembly task is an indispensable step in sequencing genomes of new organisms and studying structural genomic changes. In recent years, the dynamic development of next-generation sequencing (NGS) ...methods raises hopes for making whole-genome sequencing a fast and reliable tool used, for example, in medical diagnostics. However, this is hampered by the slowness and computational requirements of the current processing algorithms, which raises the need to develop more efficient algorithms. One possible approach, still little explored, is the use of quantum computing.
We present a proof of concept of de novo assembly algorithm, using the Genomic Signal Processing approach, detecting overlaps between DNA reads by calculating the Pearson correlation coefficient and formulating the assembly problem as an optimization task (Traveling Salesman Problem). Computations performed on a classic computer were compared with the results achieved by a hybrid method combining CPU and QPU calculations. For this purpose quantum annealer by D-Wave was used. The experiments were performed with artificially generated data and DNA reads coming from a simulator, with actual organism genomes used as input sequences. To our knowledge, this work is one of the few where actual sequences of organisms were used to study the de novo assembly task on quantum annealer.
Proof of concept carried out by us showed that the use of quantum annealer (QA) for the de novo assembly task might be a promising alternative to the computations performed in the classical model. The current computing power of the available devices requires a hybrid approach (combining CPU and QPU computations). The next step may be developing a hybrid algorithm strictly dedicated to the de novo assembly task, using its specificity (e.g. the sparsity and bounded degree of the overlap-layout-consensus graph).
Standard formulations of image/signal deconvolution under wavelet-based priors/regularizers lead to very high-dimensional optimization problems involving the following difficulties: the non-Gaussian ...(heavy-tailed) wavelet priors lead to objective functions which are nonquadratic, usually nondifferentiable, and sometimes even nonconvex; the presence of the convolution operator destroys the separability which underlies the simplicity of wavelet-based denoising. This paper presents a unified view of several recently proposed algorithms for handling this class of optimization problems, placing them in a common majorization-minimization (MM) framework. One of the classes of algorithms considered (when using quadratic bounds on nondifferentiable log-priors) shares the infamous ¿singularity issue¿ (SI) of ¿iteratively re weighted least squares¿ (IRLS) algorithms: the possibility of having to handle infinite weights, which may cause both numerical and convergence issues. In this paper, we prove several new results which strongly support the claim that the SI does not compromise the usefulness of this class of algorithms. Exploiting the unified MM perspective, we introduce a new algorithm, resulting from using bounds for nonconvex regularizers; the experiments confirm the superior performance of this method, when compared to the one based on quadratic majorization. Finally, an experimental comparison of the several algorithms, reveals their relative merits for different standard types of scenarios.
In this work, the problem of classifying Polish court rulings based on their text is presented. We use natural language processing methods and classifiers based on convolutional and recurrent neural ...networks. We prepared a dataset of 144,784 authentic, anonymized Polish court rulings. We analyze various general language embedding matrices and multiple neural network architectures with different parameters. Results show that such models can classify documents with very high accuracy (>99%). We also include an analysis of wrongly predicted examples. Performance analysis shows that our method is fast and could be used in practice on typical server hardware with 2 Processors (Central Processing Units, CPUs) or with a CPU and a Graphics processing unit (GPU).
Low-rank matrix completion (LRMC) problems arise in a wide variety of applications. Previous theory mainly provides conditions for completion under missing-at-random samplings. This paper studies ...deterministic conditions for completion. An incomplete d × N matrix is finitely rank-r completable if there are at most finitely many rank-r matrices that agree with all its observed entries. Finite completability is the tipping point in LRMC, as a few additional samples of a finitely completable matrix guarantee its unique completability. The main contribution of this paper is a characterization of finitely completable observation sets. We use this characterization to derive sufficient deterministic sampling conditions for unique completability that can be efficiently verified. We also show that under uniform random sampling schemes, these conditions are satisfied with high probability if (max{r, log d}) entries per column are observed. These findings have several implications on LRMC regarding lower bounds, sample and computational complexity, the role of coherence, adaptive settings, and the validation of any completion algorithm. We complement our theoretical results with experiments that support our findings and motivate future analysis of uncharted sampling regimes.
PDF
