Purpose Medicine dispensing data require extensive preparation when used for research and decisions during this process may lead to results that do not replicate between independent studies. We conducted an experiment to examine the impact of these decisions on results of a study measuring discontinuation, intensification, and switching in a cohort of patients initiating metformin. Methods Four Australian sites independently developed a HARmonized Protocol template to Enhance Reproducibility (HARPER) protocol and executed their analyses using the Australian Pharmaceutical Benefits Scheme 10% sample dataset. Each site calculated cohort size and demographics and measured treatment events including discontinuation, switch to another diabetes medicine, and intensification (addition of another diabetes medicine). Time to event and hazard ratios for associations between cohort characteristics and each event were also calculated. Concordance was assessed by measuring deviations from the calculated median of each value across the sites. Results Good agreement was found across sites for the number of initiators (median: 53 127, range: 51 848–55 273), gender (56.9% female, range: 56.8%–57.1%) and age group. Each site employed different methods for estimating days supply and used different operational definitions for the treatment events. Consequently, poor agreement was found for incidence of discontinuation (median 55%, range: 34%–67%), switching (median 3.5%, range: 1%–7%), intensification (median 8%, range: 5%–12%), time to event estimates and hazard ratios. Conclusions Differences in analytical decisions when deriving exposure from dispensing data affect replicability. Detailed analytical protocols, such as HARPER, are critical for transparency of operational definitions and interpretations of key study parameters.