•We investigate how to enhance the resilience of explanations against manipulation.•Explanations visualize the relevance of each input feature for the network’s prediction.•We develop a theoretical ...framework and derive bounds on the maximal change of an explanation.•Based on these insights we present three different techniques to increase robustness.•training with weight decay.•smoothing activation functions.•minimizing the Hessian of the network.•Application of our methods shows significantly improved resilience of explanations.
Display omitted
Explanation methods shed light on the decision process of black-box classifiers such as deep neural networks. But their usefulness can be compromised because they are susceptible to manipulations. With this work, we aim to enhance the resilience of explanations. We develop a unified theoretical framework for deriving bounds on the maximal manipulability of a model. Based on these theoretical insights, we present three different techniques to boost robustness against manipulation: training with weight decay, smoothing activation functions, and minimizing the Hessian of the network. Our experimental results confirm the effectiveness of these approaches.
Display omitted
•Segmenting whole slide images showing sparse objects-of-interest.•CNN cascades to cope with class imbalance.•Individually optimized convolutional neural networks.•Large experimental ...setting for evaluation on renal histology.•Superior segmentation outcomes.
Due to the increasing availability of whole slide scanners facilitating digitization of histopathological tissue, large amounts of digital image data are being generated. Accordingly, there is a strong demand for the development of computer based image analysis systems. Here, we address application scenarios in histopathology consisting of sparse, small objects-of-interest occurring in the large gigapixel images. To tackle the thereby arising challenges, we propose two different CNN cascade approaches which are subsequently applied to segment the glomeruli in whole slide images of the kidney and compared with conventional fully-convolutional networks. To facilitate unbiased evaluation, eight-fold cross-validation is performed and finally means and standard deviations are reported. Overall, with the best performing cascade approach, single CNNs are outperformed and a pixel-level Dice similarity coefficient of 0.90 is obtained (precision: 0.89, recall: 0.92). Combined with qualitative and further object-level analyses the obtained results are assessed as excellent also compared to previous approaches. We can state that especially one of the proposed cascade networks proved to be a highly powerful tool providing the best segmentation accuracies and also keeping the computing time at the lowest level. This work facilitates accurate automated segmentation of renal whole slide images which consequently allows fully-automated big data analyses for the assessment of medical treatments. Furthermore, this approach can also easily be adapted to other similar biomedical application scenarios.
Diffeomorphic Counterfactuals With Generative Models Dombrowski, Ann-Kathrin; Gerken, Jan E.; Muller, Klaus-Robert ...
IEEE transactions on pattern analysis and machine intelligence,
05/2024, Letnik:
46, Številka:
5
Journal Article
Recenzirano
Odprti dostop
Counterfactuals can explain classification decisions of neural networks in a human interpretable way. We propose a simple but effective method to generate such counterfactuals. More specifically, we ...perform a suitable diffeomorphic coordinate transformation and then perform gradient ascent in these coordinates to find counterfactuals which are classified with great confidence as a specified target class. We propose two methods to leverage generative models to construct such suitable coordinate systems that are either exactly or approximately diffeomorphic. We analyze the generation process theoretically using Riemannian differential geometry and validate the quality of the generated counterfactuals using various qualitative and quantitative measures.
In the past decade, artificial neural networks have seen unprecedented gains in capabilities and applications. With their increased popularity, the need for a more detailed understanding of how these ...models reach certain decisions emerged. The field of explainable artificial intelligence has therefore attracted significant attention in recent years, promising to provide insight into the decision-making process of deep neural networks.Of course, explainability has its own caveats, and results produced by explanation methods are not always well understood. This curtails acceptance and effective application of explanation methods.In this thesis we therefore work towards a unified geometrical understanding of explainability. We analyse undesired properties of explanation methods using concepts from differential geometry, and find countermeasures that improve their robustness and interpretability.In the first part, we show that many popular gradient- and propagation-based explanations can be arbitrarily manipulated to fit an attacker's desired output. We analyse this surprising behavior theoretically and connect the explanation's susceptibility to manipulation to the high curvature of the network's output manifold. Based on these insights, we propose β-smoothing, a novel explanation method that is more robust against adversarial perturbations. Furthermore, we investigate how a changed training regime can reduce the curvature of a neural network and derive different regularizers which boost the robustness of explanations.In the second part, we focus on another popular field of explainability, namely counterfactual explanations. These can be interpreted very intuitively and are therefore of tremendous value in medicine, finance, law, and other areas where user-friendly explanations are paramount. However, finding counterfactuals with structural differences to the query input, which stand in contrast to mere adversarial examples, can be difficult. Investigating this challenge from a geometrical point of view leads us to the insight that finding a suitable coordinate system for the search process reduces the generation of counterfactuals to a simple gradient ascent optimization. We then introduce an elegant, yet effective algorithm that makes use of the latent space of a generative model to produce high-quality counterfactuals which lie on the data manifold.
Counterfactuals can explain classification decisions of neural networks in a human interpretable way. We propose a simple but effective method to generate such counterfactuals. More specifically, we ...perform a suitable diffeomorphic coordinate transformation and then perform gradient ascent in these coordinates to find counterfactuals which are classified with great confidence as a specified target class. We propose two methods to leverage generative models to construct such suitable coordinate systems that are either exactly or approximately diffeomorphic. We analyze the generation process theoretically using Riemannian differential geometry and validate the quality of the generated counterfactuals using various qualitative and quantitative measures.
The application of machine learning (ML) techniques, especially neural networks, has seen tremendous success at processing images and language. This is because we often lack formal models to ...understand visual and audio input, so here neural networks can unfold their abilities as they can model solely from data. In the field of physics we typically have models that describe natural processes reasonably well on a formal level. Nonetheless, in recent years, ML has also proven useful in these realms, be it by speeding up numerical simulations or by improving accuracy. One important and so far unsolved problem in classical physics is understanding turbulent fluid motion. In this work we construct a strongly simplified representation of turbulence by using the Gledzer-Ohkitani-Yamada (GOY) shell model. With this system we intend to investigate the potential of ML-supported and physics-constrained small-scale turbulence modelling. Instead of standard supervised learning we propose an approach that aims to reconstruct statistical properties of turbulence such as the self-similar inertial-range scaling, where we could achieve encouraging experimental results. Furthermore we discuss pitfalls when combining machine learning with differential equations.
Explanation methods shed light on the decision process of black-box classifiers such as deep neural networks. But their usefulness can be compromised because they are susceptible to manipulations. ...With this work, we aim to enhance the resilience of explanations. We develop a unified theoretical framework for deriving bounds on the maximal manipulability of a model. Based on these theoretical insights, we present three different techniques to boost robustness against manipulation: training with weight decay, smoothing activation functions, and minimizing the Hessian of the network. Our experimental results confirm the effectiveness of these approaches.
Explanation methods promise to make black-box classifiers more transparent. As a result, it is hoped that they can act as proof for a sensible, fair and trustworthy decision-making process of the ...algorithm and thereby increase its acceptance by the end-users. In this paper, we show both theoretically and experimentally that these hopes are presently unfounded. Specifically, we show that, for any classifier \(g\), one can always construct another classifier \(\tilde{g}\) which has the same behavior on the data (same train, validation, and test error) but has arbitrarily manipulated explanation maps. We derive this statement theoretically using differential geometry and demonstrate it experimentally for various explanation methods, architectures, and datasets. Motivated by our theoretical insights, we then propose a modification of existing explanation methods which makes them significantly more robust.
In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive ...neuroscience. RepE places population-level representations, rather than neurons or circuits, at the center of analysis, equipping us with novel methods for monitoring and manipulating high-level cognitive phenomena in deep neural networks (DNNs). We provide baselines and an initial analysis of RepE techniques, showing that they offer simple yet effective solutions for improving our understanding and control of large language models. We showcase how these methods can provide traction on a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more, demonstrating the promise of top-down transparency research. We hope that this work catalyzes further exploration of RepE and fosters advancements in the transparency and safety of AI systems.
The White House Executive Order on Artificial Intelligence highlights the risks of large language models (LLMs) empowering malicious actors in developing biological, cyber, and chemical weapons. To ...measure these risks of malicious use, government institutions and major AI labs are developing evaluations for hazardous capabilities in LLMs. However, current evaluations are private, preventing further research into mitigating risk. Furthermore, they focus on only a few, highly specific pathways for malicious use. To fill these gaps, we publicly release the Weapons of Mass Destruction Proxy (WMDP) benchmark, a dataset of 3,668 multiple-choice questions that serve as a proxy measurement of hazardous knowledge in biosecurity, cybersecurity, and chemical security. WMDP was developed by a consortium of academics and technical consultants, and was stringently filtered to eliminate sensitive information prior to public release. WMDP serves two roles: first, as an evaluation for hazardous knowledge in LLMs, and second, as a benchmark for unlearning methods to remove such hazardous knowledge. To guide progress on unlearning, we develop RMU, a state-of-the-art unlearning method based on controlling model representations. RMU reduces model performance on WMDP while maintaining general capabilities in areas such as biology and computer science, suggesting that unlearning may be a concrete path towards reducing malicious use from LLMs. We release our benchmark and code publicly at https://wmdp.ai
PDF
