In 2004, a review of pilot studies published in seven major medical journals during 2000-01 recommended that the statistical analysis of such studies should be either mainly descriptive or focus on ...sample size estimation, while results from hypothesis testing must be interpreted with caution. We revisited these journals to see whether the subsequent recommendations have changed the practice of reporting pilot studies. We also conducted a survey to identify the methodological components in registered research studies which are described as 'pilot' or 'feasibility' studies. We extended this survey to grant-awarding bodies and editors of medical journals to discover their policies regarding the function and reporting of pilot studies.
Papers from 2007-08 in seven medical journals were screened to retrieve published pilot studies. Reports of registered and completed studies on the UK Clinical Research Network (UKCRN) Portfolio database were retrieved and scrutinized. Guidance on the conduct and reporting of pilot studies was retrieved from the websites of three grant giving bodies and seven journal editors were canvassed.
54 pilot or feasibility studies published in 2007-8 were found, of which 26 (48%) were pilot studies of interventions and the remainder feasibility studies. The majority incorporated hypothesis-testing (81%), a control arm (69%) and a randomization procedure (62%). Most (81%) pointed towards the need for further research. Only 8 out of 90 pilot studies identified by the earlier review led to subsequent main studies. Twelve studies which were interventional pilot/feasibility studies and which included testing of some component of the research process were identified through the UKCRN Portfolio database. There was no clear distinction in use of the terms 'pilot' and 'feasibility'. Five journal editors replied to our entreaty. In general they were loathe to publish studies described as 'pilot'.
Pilot studies are still poorly reported, with inappropriate emphasis on hypothesis-testing. Authors should be aware of the different requirements of pilot studies, feasibility studies and main studies and report them appropriately. Authors should be explicit as to the purpose of a pilot study. The definitions of feasibility and pilot studies vary and we make proposals here to clarify terminology.
Sample size justification is an important consideration when planning a clinical trial, not only for the main trial but also for any preliminary pilot trial. When the outcome is a continuous ...variable, the sample size calculation requires an accurate estimate of the standard deviation of the outcome measure. A pilot trial can be used to get an estimate of the standard deviation, which could then be used to anticipate what may be observed in the main trial. However, an important consideration is that pilot trials often estimate the standard deviation parameter imprecisely. This paper looks at how we can choose an external pilot trial sample size in order to minimise the sample size of the overall clinical trial programme, that is, the pilot and the main trial together. We produce a method of calculating the optimal solution to the required pilot trial sample size when the standardised effect size for the main trial is known. However, as it may not be possible to know the standardised effect size to be used prior to the pilot trial, approximate rules are also presented. For a main trial designed with 90% power and two-sided 5% significance, we recommend pilot trial sample sizes per treatment arm of 75, 25, 15 and 10 for standardised effect sizes that are extra small (≤0.1), small (0.2), medium (0.5) or large (0.8), respectively.
Summary Background In-utero MRI (iuMRI) has shown promise as an adjunct to ultrasound but the comparative diagnostic performance has been poorly defined. We aimed to assess whether the diagnostic ...accuracy and confidence of the prenatal diagnosis of fetal brain abnormalities is improved with iuMRI and assess the clinical impact and patient acceptability of iuMRI. Methods We did a multicentre, prospective, cohort study in the UK, at 16 fetal medicine centres, of pregnant women aged 16 years or older whose fetus had a brain abnormality detected by ultrasound at a gestational age of 18 weeks or more, had no contraindications to iuMRI, and consented to enter the study. Women carrying a fetus suspected of having a brain anomaly on ultrasound had iuMRI done within 14 days of ultrasound. The findings were reviewed by two independent panels and used to estimate diagnostic accuracy and confidence by comparison with outcome diagnoses. Changes in diagnosis, prognosis, and clinical management brought about by iuMRI and patient acceptability were assessed. Findings Participants were recruited between July 29, 2011, and Aug 31, 2014. The cohort was subdivided by gestation into the 18 weeks to less than 24 weeks fetus cohort (n=369) and into the 24 weeks or older fetus cohort (n=201). Diagnostic accuracy was improved by 23% (95% CI 18–27) in the 18 weeks to less than 24 weeks group and 29% (23–36) in the 24 weeks and older group (p<0·0001 for both groups). The overall diagnostic accuracy was 68% for ultrasound and 93% for iuMRI (difference 25%, 95% CI 21–29). Dominant diagnoses were reported with high confidence on ultrasound in 465 (82%) of 570 cases compared with 544 (95%) of 570 cases on iuMRI. IuMRI provided additional diagnostic information in 387 (49%) of 783 cases, changed prognostic information in at least 157 (20%), and led to changes in clinical management in more than one in three cases. IuMRI also had high patient acceptability with at least 95% of women saying they would have an iuMRI study if a future pregnancy were complicated by a fetal brain abnormality. Interpretation iuMRI improves diagnostic accuracy and confidence for fetal brain anomalies and leads to management changes in a high proportion of cases. This finding, along with the high patient acceptability, leads us to propose that any fetus with a suspected brain abnormality on ultrasound should have iuMRI to better inform counselling and management decisions. Funding National Institute for Health Research Health Technology Assessment programme.
It can be argued that adaptive designs are underused in clinical research. We have explored concerns related to inadequate reporting of such trials, which may influence their uptake. Through a ...careful examination of the literature, we evaluated the standards of reporting of group sequential (GS) randomised controlled trials, one form of a confirmatory adaptive design.
We undertook a systematic review, by searching Ovid MEDLINE from the 1st January 2001 to 23rd September 2014, supplemented with trials from an audit study. We included parallel group, confirmatory, GS trials that were prospectively designed using a Frequentist approach. Eligible trials were examined for compliance in their reporting against the CONSORT 2010 checklist. In addition, as part of our evaluation, we developed a supplementary checklist to explicitly capture group sequential specific reporting aspects, and investigated how these are currently being reported.
Of the 284 screened trials, 68(24%) were eligible. Most trials were published in "high impact" peer-reviewed journals. Examination of trials established that 46(68%) were stopped early, predominantly either for futility or efficacy. Suboptimal reporting compliance was found in general items relating to: access to full trials protocols; methods to generate randomisation list(s); details of randomisation concealment, and its implementation. Benchmarking against the supplementary checklist, GS aspects were largely inadequately reported. Only 3(7%) trials which stopped early reported use of statistical bias correction. Moreover, 52(76%) trials failed to disclose methods used to minimise the risk of operational bias, due to the knowledge or leakage of interim results. Occurrence of changes to trial methods and outcomes could not be determined in most trials, due to inaccessible protocols and amendments.
There are issues with the reporting of GS trials, particularly those specific to the conduct of interim analyses. Suboptimal reporting of bias correction methods could potentially imply most GS trials stopping early are giving biased results of treatment effects. As a result, research consumers may question credibility of findings to change practice when trials are stopped early. These issues could be alleviated through a CONSORT extension. Assurance of scientific rigour through transparent adequate reporting is paramount to the credibility of findings from adaptive trials. Our systematic literature search was restricted to one database due to resource constraints.
When designing a randomised controlled trial (RCT), an important consideration is the sample size required. This is calculated from several components; one of which is the target difference. This ...study aims to review the currently reported methods of elicitation of the target difference as well as to quantify the target differences used in Health Technology Assessment (HTA)-funded trials.
Trials were identified from the National Institute of Health Research Health Technology Assessment journal. A total of 177 RCTs published between 2006 and 2016 were assessed for eligibility. Eligibility was established by the design of the trial and the quality of data available. The trial designs were parallel-group, superiority RCTs with a continuous primary endpoint. Data were extracted and the standardised anticipated and observed effect size estimates were calculated. Exclusion criteria was based on trials not providing enough detail in the sample size calculation and results, and trials not being of parallel-group, superiority design.
A total of 107 RCTs were included in the study from 102 reports. The most commonly reported method for effect size derivation was a review of evidence and use of previous research (52.3%). This was common across all clinical areas. The median standardised target effect size was 0.30 (interquartile range: 0.20-0.38), with the median standardised observed effect size 0.11 (IQR 0.05-0.29). The maximum anticipated and observed effect sizes were 0.76 and 1.18, respectively. Only two trials had anticipated target values above 0.60.
The most commonly reported method of elicitation of the target effect size is previous published research. The average target effect size was 0.3. A clear distinction between the target difference and the minimum clinically important difference is recommended when designing a trial. Transparent explanation of target difference elicitation is advised, with multiple methods including a review of evidence and opinion-seeking advised as the more optimal methods for effect size quantification.
Ultrasonography has been the mainstay of antenatal screening programmes in the UK for many years. Technical factors and physical limitations may result in suboptimal images that can lead to incorrect ...diagnoses and inaccurate counselling and prognostic information being given to parents. Previous studies suggest that the addition of in utero magnetic resonance imaging (iuMRI) may improve diagnostic accuracy for fetal brain abnormalities. These studies have limitations, including a lack of an outcome reference diagnosis (ORD), which means that improvements could not be assessed accurately.
To assess the diagnostic impact, acceptability and cost consequence of iuMRI among fetuses with a suspected fetal brain abnormality.
A pragmatic, prospective, multicentre, cohort study with a health economics analysis and a sociological substudy.
Sixteen UK fetal medicine centres.
Pregnant women aged ≥ 16 years carrying a fetus (at least 18 weeks' gestation) with a suspected brain abnormality detected on ultrasonography.
Participants underwent iuMRI and the findings were reported to their referring fetal medicine clinician.
Pregnancy outcome was followed up and an ORD from postnatal imaging or postmortem autopsy/imaging collected when available. Developmental data from the Bayley Scales of Infant Development and questionnaires were collected from the surviving infants aged 2-3 years. Data on the management of the pregnancy before and after the iuMRI were collected to inform the economic evaluation. Two surveys collected data on patient acceptability of iuMRI and qualitative interviews with participants and health professionals were undertaken.
The primary analysis consisted of 570 fetuses. The absolute diagnostic accuracies of ultrasonography and iuMRI were 68% and 93%, respectively a difference of 25%, 95% confidence interval (CI) 21% to 29%. The difference between ultrasonography and iuMRI increased with gestational age. In the 18-23 weeks group, the figures were 70% for ultrasonography and 92% for iuMRI (difference of 23%, 95% CI 18% to 27%); in the ≥ 24 weeks group, the figures were 65% for ultrasonography and 94% for iuMRI (difference of 29%, 95% CI 23% to 36%). Patient acceptability was high, with at least 95% of respondents stating that they would have iuMRI again in a similar situation. Health professional interviews suggested that iuMRI was acceptable to clinicians and that iuMRI was useful as an adjunct to ultrasonography, but not as a replacement. Across a range of scenarios, iuMRI resulted in additional costs compared with ultrasonography alone. The additional cost was consistently < £600 per patient and the cost per management decision appropriately changed was always < £3000. There is potential for reporting bias from the referring clinicians on the diagnostic and prognostic outcomes. Lower than anticipated follow-up rates at 3 years of age were observed.
iuMRI as an adjunct to ultrasonography significantly improves the diagnostic accuracy and confidence for the detection of fetal brain abnormalities. An evaluation of the use of iuMRI for cases of isolated microcephaly and the diagnosis of fetal spine abnormalities is recommended. Longer-term follow-up studies of children diagnosed with fetal brain abnormalities are required to fully assess the functional significance of the diagnoses.
Current Controlled Trials ISRCTN27626961.
This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in
; Vol. 23, No. 49. See the NIHR Journals Library website for further project information.
Acute bronchiolitis is the commonest cause for hospitalisation in infancy. Supportive care remains the cornerstone of current management and no other therapy has been shown to influence the course of ...the disease. It has been suggested that adding nebulised hypertonic saline to usual care may shorten the duration of hospitalisation. To determine whether hypertonic saline does have beneficial effects we undertook an open, multi-centre parallel-group, pragmatic RCT in ten UK hospitals.
Infants admitted to hospital with a clinical diagnosis of acute bronchiolitis and requiring oxygen therapy were randomised to receive usual care alone or nebulised 3% hypertonic saline (HS) administered 6-hourly. Randomisation was within 4 h of admission. The primary outcome was time to being assessed as 'fit' for discharge with secondary outcomes including time to discharge, incidence of adverse events together with follow up to 28 days assessing patient centred health related outcomes.
A total of 317 infants were recruited to the study. 158 infants were randomised to HS (141 analysed) and 159 to standard care (149 analysed). There was no difference between the two arms in time to being declared fit for discharge (hazard ratio: 0-95, 95% CI: 0.75-1.20) nor to actual discharge (hazard ratio: 0.97, 95% CI: 0.76-1.23). There was no difference in adverse events. One infant in the HS group developed bradycardia with desaturation.
This study does not support the use of nebulised HS in the treatment of acute bronchiolitis over usual care with minimal handlings.
NCT01469845.
Aphasia is a communication disorder often acquired after a stroke. Independent use of specialist aphasia software on a home computer is a form of asynchronous tele-rehabilitation that can provide ...increased opportunity for practice of rehabilitation exercises. This study aimed to explore the factors associated with adherence to self-managed aphasia computer therapy practice.
A mixed methods exploration of adherence was conducted alongside the Big CACTUS randomized controlled trial ISRCTN: 68798818. The trial evaluated the clinical effectiveness of self-managed aphasia computer therapy. This study reports secondary analysis of data from participants randomized to the computer therapy group to investigate whether any demographic, clinical or intervention variables were associated with adherence to therapy practice. A sub-sample of the same participants took part in qualitative interviews exploring the factors perceived to influence the amount of aphasia computer therapy practice undertaken. Interviews were analyzed thematically. A convergence-coding matrix was used to triangulate the two sets of findings.
Data from 85 participants randomized to the computer therapy group were included in the quantitative analyses. At a clinical level, a greater length of time post-stroke was associated with higher adherence to self-managed aphasia therapy practice on a computer. At an intervention level, length of computer therapy access and therapist time supporting the participant were associated with greater adherence to computer therapy practice. Interviews with 11 patients and 12 informal carers identified a multitude of factors perceived to influence engagement with tele-rehabilitation by people with aphasia. The factors grouped around three themes: capability to use the computer therapy, having the opportunity to practice (external influences and technological issues) and motivation (beliefs, goals and intentions vs. personality, emotions, habit and reinforcement). Triangulation demonstrated convergence for the finding that participants' practiced computer-based therapy exercises more when they received increased support from a speech and language therapist.
Clinicians delivering asynchronous tele-rehabilitation involving self-management of aphasia therapy practice on a computer should consider the factors found to be associated with engagement when deciding which patients may be suited to this option, as well as how they can be supported to optimize the amount of practice they engage in.
Many clinical trial procedures were often undertaken in-person prior to the COVID-19 pandemic, which has resulted in adaptations to these procedures to enable trials to continue. The aim of this ...study was to understand whether the adaptations made to clinical trials by UK Clinical Trials Units (CTUs) during the pandemic have the potential to improve the efficiency of trials post-pandemic.
This was a mixed methods study, initially involving an online survey administered to all registered UK CTUs to identify studies that had made adaptations due to the pandemic. Representatives from selected studies were qualitatively interviewed to explore the adaptations made and their potential to improve the efficiency of future trials. A literature review was undertaken to locate published evidence concerning the investigated adaptations. The findings from the interviews were reviewed by a group of CTU and patient representatives within a workshop, where discussions focused on the potential of the adaptations to improve the efficiency of future trials.
Forty studies were identified by the survey. Fourteen studies were selected and fifteen CTU staff were interviewed about the adaptations. The workshop included 15 CTU and 3 patient representatives. Adaptations were not seen as leading to direct efficiency savings for CTUs. However, three adaptations may have the potential to directly improve efficiencies for trial sites and participants beyond the pandemic: a split remote-first eligibility assessment, recruitment outside the NHS via a charity, and remote consent. There was a lack of published evidence to support the former two adaptations, however, remote consent is widely supported in the literature. Other identified adaptations may benefit by improving flexibility for the participant. Barriers to using these adaptations include the impact on scientific validity, limitations in the role of the CTU, and participant's access to technology.
Three adaptations (a split remote-first eligibility assessment, recruitment outside the NHS via a charity, and remote consent) have the potential to improve clinical trials but only one (remote consent) is supported by evidence. These adaptations could be tested in future co-ordinated 'studies within a trial' (SWAT).
Harms, also known as adverse events (AEs), are recorded and monitored in randomised controlled trials (RCTs) to ensure participants' safety. Harms are recorded poorly or inconsistently in RCTs of ...Behaviour Change Interventions (BCI); however, limited guidance exists on how to record harms in BCI trials. This qualitative study explored experiences and perspectives from multi-disciplinary trial experts on recording harms in BCI trials.
Data were collected through fifteen in-depth semi-structured qualitative interviews and three focus groups with thirty-two participants who work in the delivery and oversight of clinical trials. Participants included multi-disciplinary staff from eight CTUs, Chief investigators, and patient and public representatives. Interviews and focus group recordings were transcribed verbatim and thematic analysis was used to analyse the transcripts.
Five themes were identified, namely perception and understanding of harm, proportionate reporting and plausibility, the need for a multi-disciplinary approach, language of BCI harms and complex harms for complex interventions. Participants strongly believed harms should be recorded in BCI trials; however, making decisions on "how and what to record as harms" was difficult. Recording irrelevant harms placed a high burden on trial staff and participants, drained trial resources and was perceived as for little purpose. Participants believed proportionate recording was required that focused on events with a strong plausible link to the intervention. Multi-disciplinary trial team input was essential for identifying and collecting harms; however, this was difficult in practice due to lack of knowledge on harms from BCIs, lack of input or difference in opinion. The medical language of harms was recognised as a poor fit for BCI trial harms but was familiar and established within internal processes. Future guidance on this topic would be welcomed and could include summarised literature.
Recording harms or adverse events in behaviour change intervention trials is complex and challenging; multi-disciplinary experts in trial design and implementation welcome forthcoming guidance on this topic. Issues include the high burden of recording irrelevant harms and use of definitions originally designed for drug trials. Proportionate recording of harms focused on events with a strong plausible link to the intervention and multi-disciplinary team input into decision making are essential.
PDF
