We present a new method to estimate redshift distributions and galaxy-dark matter bias parameters using correlation functions in a fully data driven and self-consistent manner. Unlike other machine ...learning, template, or correlation redshift methods, this approach does not require a reference sample with known redshifts. By measuring the projected cross- and auto-correlations of different galaxy sub-samples, e.g. as chosen by simple cells in colour–magnitude space, we are able to estimate the galaxy-dark matter bias model parameters, and the shape of the redshift distributions of each sub-sample. This method fully marginalizes over a flexible parametrization of the redshift distribution and galaxy-dark matter bias parameters of sub-samples of galaxies, and thus provides a general Bayesian framework to incorporate redshift uncertainty into the cosmological analysis in a data-driven, consistent, and reproducible manner. This result is improved by an order of magnitude by including cross-correlations with the cosmic microwave background and with galaxy–galaxy lensing. We showcase how this method could be applied to real galaxies. By using idealized data vectors, in which all galaxy-dark matter model parameters and redshift distributions are known, this method is demonstrated to recover unbiased estimates on important quantities, such as the offset Δz between the mean of the true and estimated redshift distribution and the 68 percent, 95 percent, and 99.5 percent widths of the redshift distribution to an accuracy required by current and future surveys.
We introduce an ordinal classification algorithm for photometric redshift estimation, which significantly improves the reconstruction of photometric redshift probability density functions (PDFs) for ...individual galaxies and galaxy samples. As a use case we apply our method to CFHTLS galaxies. The ordinal classification algorithm treats distinct redshift bins as ordered values, which improves the quality of photometric redshift PDFs, compared with non-ordinal classification architectures. We also propose a new single value point estimate of the galaxy redshift, which can be used to estimate the full redshift PDF of a galaxy sample. This method is competitive in terms of accuracy with contemporary algorithms, which stack the full redshift PDFs of all galaxies in the sample, but requires orders of magnitude less storage space. The methods described in this paper greatly improve the log-likelihood of individual object redshift PDFs, when compared with a popular neural network code (annz). In our use case, this improvement reaches 50 per cent for high-redshift objects (z ≥ 0.75). We show that using these more accurate photometric redshift PDFs will lead to a reduction in the systematic biases by up to a factor of 4, when compared with less accurate PDFs obtained from commonly used methods. The cosmological analyses we examine and find improvement upon are the following: gravitational lensing cluster mass estimates, modelling of angular correlation functions and modelling of cosmic shear correlation functions.
We present an algorithm for the fast computation of the general N-point spatial correlation functions of any discrete point set embedded within an Euclidean space of . Utilizing the concepts of ...kd-trees and graph databases, we describe how to count all possible N-tuples in binned configurations within a given length scale, e.g., all pairs of points or all triplets of points with side lengths < rMAX. Through benchmarking, we show the computational advantage of our new graph-based algorithm over more traditional methods. We show measurements of the three-point correlation function up to scales of ∼200 Mpc (beyond the baryon acoustic oscillation scale in physical units) using current Sloan Digital Sky Survey (SDSS) data. Finally, we present a preliminary exploration of the small-scale four-point correlation function of 568,776 SDSS Constant (stellar) Mass (CMASS) galaxies in the northern Galactic cap over the redshift range of 0.43 < z < 0.7. We present the publicly available code GRAMSCI (GRAph Made Statistics for Cosmological Information; bitbucket.org/csabiu/gramsci), under a Gnu is Not Unix (GNU) General Public License.
We present an analysis of anomaly detection for machine learning redshift estimation. Anomaly detection allows the removal of poor training examples, which can adversely influence redshift estimates. ...Anomalous training examples may be photometric galaxies with incorrect spectroscopic redshifts, or galaxies with one or more poorly measured photometric quantity. We select 2.5 million ‘clean’ SDSS DR12 galaxies with reliable spectroscopic redshifts, and 6730 ‘anomalous’ galaxies with spectroscopic redshift measurements which are flagged as unreliable. We contaminate the clean base galaxy sample with galaxies with unreliable redshifts and attempt to recover the contaminating galaxies using the Elliptical Envelope technique. We then train four machine learning architectures for redshift analysis on both the contaminated sample and on the preprocessed ‘anomaly-removed’ sample and measure redshift statistics on a clean validation sample generated without any preprocessing. We find an improvement on all measured statistics of up to 80 per cent when training on the anomaly removed sample as compared with training on the contaminated sample for each of the machine learning routines explored. We further describe a method to estimate the contamination fraction of a base data sample.
Abstract
Photometric redshift uncertainties are a major source of systematic error for ongoing and future photometric surveys. We study different sources of redshift error caused by choosing a ...suboptimal redshift histogram bin width and propose methods to resolve them. The selection of a too large bin width is shown to oversmooth small-scale structure of the radial distribution of galaxies. This systematic error can significantly shift cosmological parameter constraints by up to 6σ for the dark energy equation-of-state parameter w. Careful selection of bin width can reduce this systematic by a factor of up to 6 as compared with commonly used current binning approaches. We further discuss a generalized resampling method that can correct systematic and statistical errors in cosmological parameter constraints caused by uncertainties in the redshift distribution. This can be achieved without any prior assumptions about the shape of the distribution or the form of the redshift error. Our methodology allows photometric surveys to obtain unbiased cosmological parameter constraints using a minimum number of spectroscopic calibration data. For a DES-like galaxy clustering forecast, we obtain unbiased results with respect to errors caused by suboptimal histogram bin width selection, using only 5k representative spectroscopic calibration objects per tomographic redshift bin.
We present analyses of data augmentation for machine learning redshift estimation. Data augmentation makes a training sample more closely resemble a test sample, if the two base samples differ, in ...order to improve measured statistics of the test sample. We perform two sets of analyses by selecting 800 000 (1.7 million) Sloan Digital Sky Survey Data Release 8 (Data Release 10) galaxies with spectroscopic redshifts. We construct a base training set by imposing an artificial r-band apparent magnitude cut to select only bright galaxies and then augment this base training set by using simulations and by applying the k-correct package to artificially place training set galaxies at a higher redshift. We obtain redshift estimates for the remaining faint galaxy sample, which are not used during training. We find that data augmentation reduces the error on the recovered redshifts by 40 per cent in both sets of analyses, when compared to the difference in error between the ideal case and the non-augmented case. The outlier fraction is also reduced by at least 10 per cent and up to 80 per cent using data augmentation. We finally quantify how the recovered redshifts degrade as one probes to deeper magnitudes past the artificial magnitude limit of the bright training sample. We find that at all apparent magnitudes explored, the use of data augmentation with tree-based methods provide an estimate of the galaxy redshift with a low value of bias, although the error on the recovered redshifts increases as we probe to deeper magnitudes. These results have applications for surveys which have a spectroscopic training set which forms a biased sample of all photometric galaxies, for example if the spectroscopic detection magnitude limit is shallower than the photometric limit.
We present an analysis of importance feature selection applied to photometric redshift estimation using the machine learning architecture Decision Trees with the ensemble learning routine adaboost ...(hereafter RDF). We select a list of 85 easily measured (or derived) photometric quantities (or ‘features’) and spectroscopic redshifts for almost two million galaxies from the Sloan Digital Sky Survey Data Release 10. After identifying which features have the most predictive power, we use standard artificial Neural Networks (aNNs) to show that the addition of these features, in combination with the standard magnitudes and colours, improves the machine learning redshift estimate by 18 per cent and decreases the catastrophic outlier rate by 32 per cent. We further compare the redshift estimate using RDF with those from two different aNNs, and with photometric redshifts available from the Sloan Digital Sky Survey (SDSS). We find that the RDF requires orders of magnitude less computation time than the aNNs to obtain a machine learning redshift while reducing both the catastrophic outlier rate by up to 43 per cent, and the redshift error by up to 25 per cent. When compared to the SDSS photometric redshifts, the RDF machine learning redshifts both decreases the standard deviation of residuals scaled by 1/(1+z) by 36 per cent from 0.066 to 0.041, and decreases the fraction of catastrophic outliers by 57 per cent from 2.32 to 0.99 per cent.
We present the clustering of galaxy clusters as a useful addition to the common set of cosmological observables. The clustering of clusters probes the large-scale structure of the Universe, extending ...galaxy clustering analysis to the high-peak, high-bias regime. Clustering of galaxy clusters complements the traditional cluster number counts and observable-mass relation analyses, significantly improving their constraining power by breaking existing calibration degeneracies. We use the maxBCG galaxy clusters catalogue to constrain cosmological parameters and cross-calibrate the mass-observable relation, using cluster abundances in richness bins and weak-lensing mass estimates. We then add the redshift-space power spectrum of the sample, including an effective modelling of the weakly non-linear contribution and allowing for an arbitrary photometric redshift smoothing. The inclusion of the power spectrum data allows for an improved self-calibration of the scaling relation. We find that the inclusion of the power spectrum typically brings a ∼50 per cent improvement in the errors on the fluctuation amplitude σ8 and the matter density Ωm. Finally, we apply this method to constrain models of the early universe through the amount of primordial non-Gaussianity of the local type, using both the variation in the halo mass function and the variation in the cluster bias. We find a constraint on the amount of skewness f
NL = 12 ± 157 (1σ) from the cluster data alone.
Introduction: The recent developments of magnetic resonance (MR) based adaptive strategies for photon and, potentially for proton therapy, require a fast and reliable conversion of MR images to X-ray ...computed tomography (CT) values. CT values are needed for photon and proton dose calculation. The improvement of conversion results employing a 3D deep learning approach is evaluated.
Material and methods: A database of 89 T1-weighted MR head scans with about 100 slices each, including rigidly registered CTs, was created. Twenty-eight validation patients were randomly sampled, and four patients were selected for application. The remaining patients were used to train a 2D and a 3D U-shaped convolutional neural network (Unet). A stack size of 32 slices was used for 3D training. For all application cases, volumetric modulated arc therapy photon and single-field uniform dose pencil-beam scanning proton plans at four different gantry angles were optimized for a generic target on the CT and recalculated on 2D and 3D Unet-based pseudoCTs. Mean (absolute) error (MAE/ME) and a gradient sharpness estimate were used to quantify the image quality. Three-dimensional gamma and dose difference analyses were performed for photon (gamma criteria: 1%, 1 mm) and proton dose distributions (gamma criteria: 2%, 2 mm). Range (80% fall off) differences for beam's eye view profiles were evaluated for protons.
Results: Training 36 h for 1000 epochs in 3D (6 h for 200 epochs in 2D) yielded a maximum MAE of 147 HU (135 HU) for the application patients. Except for one patient gamma pass rates for photon and proton dose distributions were above 96% for both Unets. Slice discontinuities were reduced for 3D training at the cost of sharpness.
Conclusions: Image analysis revealed a slight advantage of 2D Unets compared to 3D Unets. Similar dose calculation performance was reached for the 2D and 3D network.
We present an analysis of a general machine learning technique called ‘stacking’ for the estimation of photometric redshifts. Stacking techniques can feed the photometric redshift estimate, as output ...by a base algorithm, back into the same algorithm as an additional input feature in a subsequent learning round. We show how all tested base algorithms benefit from at least one additional stacking round (or layer). To demonstrate the benefit of stacking, we apply the method to both unsupervised machine learning techniques based on self-organizing maps (SOMs), and supervised machine learning methods based on decision trees. We explore a range of stacking architectures, such as the number of layers and the number of base learners per layer. Finally we explore the effectiveness of stacking even when using a successful algorithm such as AdaBoost. We observe a significant improvement of between 1.9 per cent and 21 per cent on all computed metrics when stacking is applied to weak learners (such as SOMs and decision trees). When applied to strong learning algorithms (such as AdaBoost) the ratio of improvement shrinks, but still remains positive and is between 0.4 per cent and 2.5 per cent for the explored metrics and comes at almost no additional computational cost.
PDF
