Abstract Background: Non-invasive genotyping of cell-free DNA (cfDNA) is increasingly used in cancer care as tumor biopsies may be inadequate or unavailable. As clinical adaptation of liquid biopsies is driven in large part by commercial vendors that offer proprietary PCR or NGS-based tests, rigorous validation of these assays is essential to ensure maximum clinical benefit. Recent efforts to compare cfDNA diagnostics have not focused on clinically actionable mutations or used tumor to verify plasma results. The present study analyzes the concordance in reports of actionable gene fusions in NSCLC from two independent, commercial plasma NGS tests, compares the breakpoint characteristics between tissue and plasma, and highlights how differences in sequencing and bioinformatic analysis can be a source of false negatives. Methods/Results: We studied a cohort of 169 NSCLC patients (pts) that had plasma analyzed by Guardant 360 between April 2016 and July 2018. In those with tumor positive ALK, ROS1, or RET fusions (n=16), banked cfDNA from a separate tube of plasma (69% collected within 2 weeks) underwent local testing and sequencing at DFCI using Resolution Bioscience's pre-production ctDx-Lung kit with remote bioinformatic variant calling performed by Resolution Bioscience; all parties involved were blinded to tumor genotype and Guardant results. Locked results were unblinded for post hoc analysis. The ctDx-lung kit detected 13 out 16 fusions (AF% range 80-0.3%), while Guardant360 detected 7 (AF% range 10-0.3%). Guardant360 tended to report lower AFs than ctDx-lung, though corroborating TP53 alterations were of similar AFs. Of the cases where both assays did not detect any fusions, no other shared SNVs were detected, possibly due to low shed of tumor DNA. Of the 13 cases where a fusion was detected in plasma by ctDx-lung, 4 rearrangements could be characterized as ‘non-productive’ due to opposite transcriptional orientations and comprised 50% (3/6) of the fusions not detected by Guardant360. Only one case detected by Guardant360 was ‘non-productive’. Of the 13 cases where a fusion was detected in plasma, 89% (8/9) of pts with productive fusions and 75% of pts (3/4) with ‘non-productive’ fusions responded to TKI therapies. Additional unblinding is ongoing to better understand false negative cases. Conclusions: Here, we demonstrate that a rigorous approach to benchmarking plasma genotyping assays should 1) focus on actionable mutations, 2) use tumor as a gold standard for establishing true and false positives and negatives, and 3) include orthogonal validation to address assay design and bioinformatic analyses as sources of discordance. Our study further highlights the challenges of fusion detection and interpretation and the need for platform cross-comparisons to realize the potential of liquid biopsy to increase access to personalized cancer care. Citation Format: Julianna G. Supplee, Marina S. Milan, Kristy T. Potts, Lynette M. Sholl, Tristan S. Shaffer, Lee P. Lim, Pasi A. Janne, Geoffrey R. Oxnard, Cloud P. Paweletz. Building an effective concordance study: Plasma Next Generation Sequencing (NGS) for oncogenic fusion detection in non-small cell lung carcinoma (NSCLC) abstract. In: Proceedings of the American Association for Cancer Research Annual Meeting 2019; 2019 Mar 29-Apr 3; Atlanta, GA. Philadelphia (PA): AACR; Cancer Res 2019;79(13 Suppl):Abstract nr 1374.