This article analyzes the concept of validity to set out key factors bearing on claims about validity in general and particularly regarding learning analytics. Because uses of trace data in learning analytics are increasing rapidly, specific consideration is given to reliability of trace data and their role in claiming validity for interpretations grounded on trace data. This analysis reveals the essential and inescapable role of theory in deciding what trace data should be gathered and how trace data can contribute to recommendations for improving learning, one main goal for generating and using learning analytics. •Trace data are increasingly useful in developing learning analytics.•“Raw” data are biased by the theory that recommends observing those data.•Self-regulating learners acting as agents complicate reliability of trace data.•Reliability of trace data concerns dynamic events, not static aspects of a measure.•Generalizability over facets of data sets limits on reliability and validity.