Here we evaluate five atmospheric reanalyses in an Arctic gateway during late summer. The reanalyses include ERA5, ERA‐Interim, Japanese 55 year Re‐Analysis (JRA‐55), Climate Forecasting System Reanalysis‐version 2 (CFSv2), and Modern Era Retrospective analysis for Research and Applications‐version 2 (MERRA‐2). We use observations from 50 radiosondes launched in the Fram Strait around 79‐80°N, between 25 August and 11 September 2017. Crucially, data from 27 radiosondes were not transmitted to the Global Telecommunications System and therefore not assimilated into any reanalysis. In most reanalyses, the magnitude of wind speed and humidity errors is similar for profiles with and without data assimilation. In cases without data assimilation, correlation coefficients (R) exceed 0.88 for temperature, wind speed, and specific humidity, in all reanalyses. Overall, the newly released ERA5 has higher correlation coefficients than any other reanalyses as well as smaller biases and root‐mean‐square errors, for all three variables. The largest improvements identified in ERA5 are in its representation of the wind field, and temperature profiles over warm water. Plain Language Summary The Arctic is undergoing rapid and ongoing changes. However, due to the harsh environment, there are relatively few observations from this region. To understand the drivers of these changes, we rely heavily on atmospheric reanalyses. Reanalyses are our best guess at the state of the atmosphere at a given time. Reanalyses are generated by assimilating all available atmospheric observations into a weather forecast model. A key question within the scientific community is how accurate reanalyses are in the Arctic. One problem with answering this question is that most observations used to test the performance of reanalyses were ingested in to the model and are therefore not an independent data set. Here we present a new set of balloon‐borne atmospheric observations from the Fram Strait, between Svalbard and Greenland. Many of these data were not assimilated in to any reanalyses, providing a rare opportunity to evaluate their performance in this important Arctic gateway. We test five products, including the newly released ERA5 from the European Centre for Medium Ranged Weather Forecasting. All products simulate the temperature, humidity, and wind fields well, even without data assimilation. Overall, the newly released ERA5 performs best, with the largest improvements in the wind and temperature fields. Key Points In situ Arctic observations: 27 atmospheric profiles from radiosondes in Fram Strait (August–Septmber 2017) were not transmitted to GTS ERA5 simulates observed atmospheric profiles more accurately than ERA‐Interim, JRA‐55, CFSv2, and MERRA‐2 Largest improvements are found in ERA5 for wind and temperature profiles over warmer eastern Fram Strait