Rationale, aims and objectives  The randomized controlled trial (RCT) is considered the gold standard methodology for determining the efficacy and tolerability of new treatments. However, RCTs cannot provide information on the effectiveness of interventions as they are used in real life. This study was conducted to investigate the effectiveness of montelukast, a leukotriene receptor antagonist, in the real‐world management of asthma, through a large‐scale, retrospective, observational study: the National Montelukast Survey. Methods  In order to ensure a robust methodology for the National Montelukast Survey we performed three pilot studies involving a total of almost 400 patients. During the pilots, the design of the study was extensively modified from a simple prescriber questionnaire used in the first pilot to the ‘triangulated’ methodology encompassing the perspectives of patient, prescriber and independent observer used in the National Montelukast Survey. Good levels of interobserver agreement confirmed the robustness of the final methodology. Conclusions  Achieving a robust methodology was dependent on the extensive piloting. It is possible to collect reliable observational data relating to treatment outcomes. We believe our methods are likely to have more widespread applicability and offer a potential improvement over postmarketing surveillance.