Radiotherapy (RT) plan quality is critical in ensuring treatment efficacy. Poor quality RT can increase the risks of treatment failure, overall mortality and detrimentally impacting a patient's quality of life 1–4. This is especially important within RT clinical trials, where standardisation of treatment plan quality is paramount. However, widespread objective quantitative assessment of plan quality within trials is not performed routinely, leading to uncertainty on the magnitude of quality variations. Automated planning enables the possibility to efficiency and objectively assess the quality of individual clinical plans (CP) through comparison with an automatically generated standardised 'baseline’ plan. Utilising this innovative auditing methodology within a trial enables full quantitative characterisation of: (i) overall plan quality, (ii) potential outliers and (iii) variation solely due to planning practice. The aim of this study was to use fully automated planning to objectively assess plan quality within the Cancer Research UK funded (A25317) multi-centre international phase III trial PATHOS. 337 patients enrolled in the PATHOS clinical trial before 1st July 2021 were included in this study. 55 cases were excluded due to incomplete data and 16 for calibrating the automated solution, leaving 264 patients for analysis. 219 (83%) and 45 (17%) cases were treated with unilateral (Unilat) and bilateral (Bilat) volumes respectively. Planning was performed in alignment with the PATHOS protocol, with prescriptions of Bilat66Gy, Bilat60Gy, Unilat66Gy or Unilat60Gy in 30 fractions and Unilat50Gy in 25 fractions. Automated treatment plans (AP) were generated in RayStation using a locally developed 'Protocol Based Automatic Iterative Optimization’ automated planning solution 5. CP were quantitatively compared to AP across all the PATHOS trial metrics (including: Parotid Dmean; SpinalCord/BrainStem PRV D1cc; and PTV D98%, D2% and D50%) together with conformality (CI) and homogeneity (HI) indices. Analysis was performed with data categorised in terms of prescription and also tumour laterality. Statistical significance was assessed via a two-sided Wilcoxon matched-paired signed-rank test. Display omitted Fig. 1 and Fig. 2 present a summary of the dosimetric results, categorised in terms of prescription. When comparing CP to the AP baseline (CP-AP), statistically significant (p≤0.05) differences, Δ, in median values were observed across most key metrics. For HI, small changes across all prescriptions were detected for the primary PTV with the largest Δ equalling (-0.012, p<0.001) for Unilat50Gy prescriptions. This indicated CP were marginally more homogeneous that the AP baseline. For CI, significant differences were observed across primary PTVs for three prescriptions (Unilat50Gy, Unilat60Gy and Bilat60Gy) and all secondary PTVs. Median differences were substantial, with a max Δ of +0.110 (p<0.001, Unilat66:PTV54), which represented a 10% increase in the volume treated to 54Gy for CP. When categorised in terms of tumour laterality, differences in contralateral Parotid (Parotid_CL) Dmean were small for Unilat (Δ=+2.2Gy, p<0.001) and moderate for Bilat cases (Δ=+3.5Gy, p<0.001). For ipsilateral Parotids (Parotid_IL), differences were substantial for Unilat cases (Δ=+4.8Gy, p<0.001) but nominally equivalent to Parotid_CL for Bilat (Δ=+3.1Gy, p<0.001). At an individual patient level, AP baseline plans highlighted potential quality improvements that could have been realised for CP. For 50% of all patients, AP led to a reduction in Parotid_IL and Parotid_CL Dmean of between 4.4Gy-14.7Gy and 2.5Gy-8.9Gy respectively. In terms of conformality, for 50% of all patients AP reduced CI by between 0.06-0.35 and 0.08-0.28 for PTV60 and PTV54 respectively. In terms of overall variation with the trial, Fig. 1 and Fig. 2 demonstrate that a high proportion of the variation observed in the majority dose metrics was a direct result of plan quality. For example, a standardised AP planning method would have reduced the inter-quartile range (IQR) for Parotid_CL Dmean from 5.4Gy to 1.4Gy, for HI (PTV54) from 0.031 to 0.015 and for CI (PTV54) from 0.194 to 0.071. Parotid_IL Dmean was a key exception, with similar IQRs for both AP and CP. Clinics participating in PATHOS undergo a comprehensive quality assurance process prior to patient recruitment, with additional 'on trial’ qualitative reviews performed on small subset of patients. Furthermore, all patient plans must, where practicable, meet trial dose metric tolerances. Results of this study demonstrate that despite these procedures, which are common to many high-quality trials, meaningful variations in plan quality remain. Automated planning was found to be an effective tool in objectively assessing plan quality within a large trial. Implementation on a prospective basis could be a powerful QA tool to reduce this observed variation.