Real-world sequential decision-making tasks are generally complex, requiring trade-offs between multiple, often conflicting, objectives. Despite this, the majority of research in reinforcement ...learning and decision-theoretic planning either assumes only a single objective, or that multiple objectives can be adequately handled via a simple linear combination. Such approaches may oversimplify the underlying problem and hence produce suboptimal results. This paper serves as a guide to the application of multi-objective methods to difficult problems, and is aimed at researchers who are already familiar with single-objective reinforcement learning and planning methods who wish to adopt a multi-objective perspective on their research, as well as practitioners who encounter multi-objective decision problems in practice. It identifies the factors that may influence the nature of the desired solution, and illustrates by example how these influence the design of multi-objective decision-making systems for complex problems.
We propose a novel multi-objective reinforcement learning algorithm that successfully learns the optimal policy even for non-linear utility functions. Non-linear utility functions pose a challenge ...for SOTA approaches, both in terms of learning efficiency as well as the solution concept. A key insight is that, by proposing a critic that learns a multi-variate distribution over the returns, which is then combined with accumulated rewards, we can directly optimize on the utility function, even if it is non-linear. This allows us to vastly increase the range of problems that can be solved compared to those which can be handled by single-objective methods or multi-objective methods requiring linear utility functions, yet avoiding the need to learn the full Pareto front. We demonstrate our method on multiple multi-objective benchmarks, and show that it learns effectively where baseline approaches fail.
ETTIN (ETT) is an atypical member of the AUXIN RESPONSE FACTOR family of transcription factors that plays a crucial role in tissue patterning in the Arabidopsis (Arabidopsis thaliana) gynoecium. ...Though recent insights have provided valuable information on ETT's interactions with other components of auxin signaling, the biophysical mechanisms linking ETT to its ultimate effects on gynoecium morphology were until now unknown. Here, using techniques to assess cell-wall dynamics during gynoecium growth and development, we provide a coherent body of evidence to support a model in which ETT controls the elongation of the valve tissues of the gynoecium through the positive regulation of pectin methylesterase (PME) activity in the cell wall. This increase in PME activity results in an increase in the level of demethylesterified pectins and a consequent reduction in cell wall stiffness, leading to elongation of the valves. Though similar biophysical mechanisms have been shown to act in the stem apical meristem, leading to the expansion of organ primordia, our findings demonstrate that regulation of cell wall stiffness through the covalent modification of pectin also contributes to tissue patterning within a developing plant organ.
In many risk-aware and multi-objective reinforcement learning settings, the utility of the user is derived from a single execution of a policy. In these settings, making decisions based on the ...average future returns is not suitable. For example, in a medical setting a patient may only have one opportunity to treat their illness. Making decisions using just the expected future returns–known in reinforcement learning as the value–cannot account for the potential range of adverse or positive outcomes a decision may have. Therefore, we should use the distribution over expected future returns differently to represent the critical information that the agent requires at decision time by taking both the future and accrued returns into consideration. In this paper, we propose two novel Monte Carlo tree search algorithms. Firstly, we present a Monte Carlo tree search algorithm that can compute policies for nonlinear utility functions (NLU-MCTS) by optimising the utility of the different possible returns attainable from individual policy executions, resulting in good policies for both risk-aware and multi-objective settings. Secondly, we propose a distributional Monte Carlo tree search algorithm (DMCTS) which extends NLU-MCTS. DMCTS computes an approximate posterior distribution over the utility of the returns, and utilises Thompson sampling during planning to compute policies in risk-aware and multi-objective settings. Both algorithms outperform the state-of-the-art in multi-objective reinforcement learning for the expected utility of the returns.
A key innovation of flowering plants is the female reproductive organ, the carpel. Here, we show that a mechanism that regulates carpel margin development in the model flowering plant Arabidopsis ...thaliana was recruited from light-regulated processes. This recruitment followed the loss from the basic helix-loop-helix transcription factor SPATULA (SPT) of a domain previously responsible for its negative regulation by phytochrome. We propose that the loss of this domain was a prerequisite for the light-independent expression in female reproductive tissues of a genetic module that also promotes shade avoidance responses in vegetative organs. Striking evidence for this proposition is provided by the restoration of wild-type carpel development to spt mutants by low red/far-red light ratios, simulating vegetation shade, which we show to occur via phytochrome B, PHYTOCHROME INTERACTING FACTOR4 (PIF4), and PIF5. Our data illustrate the potential of modular evolutionary events to generate rapid morphological change and thereby provide a molecular basis for neo-Darwinian theories that describe this nongradualist phenomenon. Furthermore, the effects shown here of light quality perception on carpel development lead us to speculate on the potential role of light-regulated mechanisms in plant organs that, like the carpel, form within the shade of surrounding tissues.
We demonstrate that the biophysical technique of surface plasmon resonance (SPR) analysis, which has previously been used to measure transcription factor binding to short DNA molecules, can also be ...used to characterize interactions involving entire gene promoters. This discovery has two main implications that relate, respectively, to novel qualitative and quantitative uses of the SPR technique. Firstly, SPR analysis can be used qualitatively to test the capacity of any transcription factor to interact physically with its putative target genes. This application should prove particularly useful for the confirmation of predicted transcriptional interactions in model species, and for comparative studies of non-model species in which transcriptional interactions are not amenable to study by other methods. Secondly, SPR may be used quantitatively to characterize interactions between transcription factors and gene promoters containing multiple cis-acting sites. This application should prove useful for the detailed dissection of promoter function in known target genes. The qualitative and quantitative applications of the SPR analysis of whole promoters combine to make this a uniquely powerful technique, which should prove particularly useful in systems biology, evolutionary developmental biology and various branches of applied biology.
Higher plants exhibit a variety of different life histories. Annual plants live for less than a year and after flowering produce seeds and senesce. By contrast perennials live for many years, ...dividing their life cycle into episodes of vegetative growth and flowering. Environmental cues control key check points in both life histories. Genes controlling responses to these cues exhibit natural genetic variation that has been studied most in short-lived annuals. We characterize natural genetic variation conferring differences in the perennial life cycle of Arabis alpina. Previously the accession Pajares was shown to flower after prolonged exposure to cold (vernalization) and only for a limited period before returning to vegetative growth. We describe five accessions of A. alpina that do not require vernalization to flower and flower continuously. Genetic complementation showed that these accessions carry mutant alleles at PERPETUAL FLOWERING 1 (PEP1), which encodes a MADS box transcription factor orthologous to FLOWERING LOCUS C in the annual Arabidopsis thaliana. Each accession carries a different mutation at PEP1, suggesting that such variation has arisen independently many times. Characterization of these alleles demonstrated that in most accessions, including Pajares, the PEP1 locus contains a tandem arrangement of a full length and a partial PEP1 copy, which give rise to two full-length transcripts that are differentially expressed. This complexity contrasts with the single gene present in A. thaliana and might contribute to the more complex expression pattern of PEP1 that is associated with the perennial life-cycle. Our work demonstrates that natural accessions of A. alpina exhibit distinct life histories conferred by differences in PEP1 activity, and that continuous flowering forms have arisen multiple times by inactivation of the floral repressor PEP1. Similar phenotypic variation is found in other herbaceous perennial species, and our results provide a paradigm for how characteristic perennial phenotypes might arise.
The control of gynecium development in Arabidopsis thaliana by the auxin response factor ETTIN (ETT) correlates with a reduction in the methylesterification of cell-wall pectins and a decrease in ...cell-wall stiffness in the valve tissues of the ovary. Here, we determine the list of genes rapidly regulated following the in-vivo activation of an ETT fusion protein, and show these to be significantly enriched in genes encoding cell-wall proteins, including several pectin methylesterases (PMEs) and pectin methylesterase inhibitors (PMEIs). We also perform a genome-wide scan for potential ETT-binding sites, and incorporate the results of this procedure into a comparison of datasets, derived using four distinct methods, to identify genes regulated directly or indirectly by ETT. We conclude from our combined analyses that PMEIs are likely to be key actors that mediate the regulation of gynecium development by ETT, while ETT may simultaneously regulate PMEs to prevent exaggerated developmental effects from the regulation of PMEIs. We also postulate the existence of one or more rapidly-acting intermediate factors in the transcriptional regulation of PMEs and PMEIs by ETT.
Infectious disease outbreaks can have a disruptive impact on public health and societal processes. As decision-making in the context of epidemic mitigation is multi-dimensional hence complex, ...reinforcement learning in combination with complex epidemic models provides a methodology to design refined prevention strategies. Current research focuses on optimizing policies with respect to a single objective, such as the pathogen’s attack rate. However, as the mitigation of epidemics involves distinct, and possibly conflicting, criteria (i.a., mortality, morbidity, economic cost, well-being), a multi-objective decision approach is warranted to obtain balanced policies. To enhance future decision-making, we propose a deep multi-objective reinforcement learning approach by building upon a state-of-the-art algorithm called Pareto Conditioned Networks (PCN) to obtain a set of solutions for distinct outcomes of the decision problem. We consider different deconfinement strategies after the first Belgian lockdown within the COVID-19 pandemic and aim to minimize both COVID-19 cases (i.e., infections and hospitalizations) and the societal burden induced by the mitigation measures. As such, we connected a multi-objective Markov decision process with a stochastic compartment model designed to approximate the Belgian COVID-19 waves and explore reactive strategies. As these social mitigation measures are implemented in a continuous action space that modulates the contact matrix of the age-structured epidemic model, we extend PCN to this setting. We evaluate the solution set that PCN returns, and observe that it explored the whole range of possible social restrictions, leading to high-quality trade-offs, as it captured the problem dynamics. In this work, we demonstrate that multi-objective reinforcement learning adds value to epidemiological modeling and provides essential insights to balance mitigation policies.
•Epidemic mitigation involves multiple criteria (mortality, economic cost, well-being).•Multi-objective RL (MORL) to explore the Pareto front of deconfinement strategies.•We investigate deconfinement strategies after the first Belgian lockdown of COVID-19.•Minimize both COVID-19 cases and societal burden; leads to many distinct trade-offs.•MORL brings essential insights to balance mitigation policies and help policy makers.
ETTIN, a transcription factor also known as Auxin Response Factor 3, enhances PME activity in the valve tissue, thus maintaining low cell wall stiffness and proper gynoecium morphogenesis.
ETTIN
(
...ETT
) is an atypical member of the AUXIN RESPONSE FACTOR family of transcription factors that plays a crucial role in tissue patterning in the Arabidopsis (
Arabidopsis thaliana
) gynoecium. Though recent insights have provided valuable information on ETT’s interactions with other components of auxin signaling, the biophysical mechanisms linking ETT to its ultimate effects on gynoecium morphology were until now unknown. Here, using techniques to assess cell-wall dynamics during gynoecium growth and development, we provide a coherent body of evidence to support a model in which ETT controls the elongation of the valve tissues of the gynoecium through the positive regulation of pectin methylesterase (PME) activity in the cell wall. This increase in PME activity results in an increase in the level of demethylesterified pectins and a consequent reduction in cell wall stiffness, leading to elongation of the valves. Though similar biophysical mechanisms have been shown to act in the stem apical meristem, leading to the expansion of organ primordia, our findings demonstrate that regulation of cell wall stiffness through the covalent modification of pectin also contributes to tissue patterning within a developing plant organ.
PDF
