Interest has grown in using new screening technologies such as drones to search for methane leaks in hydrocarbon production infrastructure. Screening technologies may be less expensive and faster than traditional methods. However, including new technologies in emissions monitoring programs requires an accurate understanding of what leaks a system will detect and the resultant emissions mitigation. Here we examine source detection of a drone-based system with controlled releases. We examine different detection algorithm parameters to understand trade-offs between false positive rate and detection probability. Leak detection was poor under all conditions with an average detection probability of 0.21. Detection probability was not affected by emission rate, suggesting similar systems may commonly miss large leaks. Detection was best in moderate wind speeds and at 750–2000 m downwind from the source where the plume had diffused vertically above the minimum flight level of 40–50 m. Predicted concentration enhancement from a Gaussian plume model was a reasonable predictor of detection within the test suite. Enabling lower flight elevations may increase detection probability. Overall, the experiments suggest that controlled releases are useful and necessary to provide an understanding of detection probability of screening technologies for regulatory and deployment purposes, and the testing must be representative to support broad application.