Display omitted •New online system to evaluate tractography pipelines.•New connectivity-based quantitative measures (valid and invalid bundles/connections and average bundle coverage).•More than 57,000 different combinations of tractography pipelines compared from the FiberCup dataset.•Take-home messages for tractography users and insights on remaining open questions for the community. We have developed the Tractometer: an online evaluation and validation system for tractography processing pipelines. One can now evaluate the results of more than 57,000 fiber tracking outputs using different acquisition settings (b-value, averaging), different local estimation techniques (tensor, q-ball, spherical deconvolution) and different tracking parameters (masking, seeding, maximum curvature, step size). At this stage, the system is solely based on a revised FiberCup analysis, but we hope that the community will get involved and provide us with new phantoms, new algorithms, third party libraries and new geometrical metrics, to name a few. We believe that the new connectivity analysis and tractography characteristics proposed can highlight limits of the algorithms and contribute in solving open questions in fiber tracking: from raw data to connectivity analysis. Overall, we show that (i) averaging improves quality of tractography, (ii) sharp angular ODF profiles helps tractography, (iii) seeding and multi-seeding has a large impact on tractography outputs and must be used with care, and (iv) deterministic tractography produces less invalid tracts which leads to better connectivity results than probabilistic tractography.