Current concerns about lack of diversity in supply of critical metals have spurred research into utilizing domestic sources, particularly from waste streams. Sustainability strategies like urban mining, industrial symbiosis, and the circular economy suggest avenues to realize new supplies of critical metals. In this work we explore the resource and economic potential for extracting rare earth elements (REEs) from industry byproducts (e.g. coal combustion products, red mud) and secondary sources (e.g. waste electronics and light bulbs). Combining materials flow analysis and characterization data, we find that while REE concentrations in waste and byproduct streams are mostly much lower than current REE ores, some secondary sources are richer than ores in high value REEs such as scandium. The quantities of REEs contained in secondary sources could meet current global demand even with low extraction yield rates. Phosphogypsum, coal ash and red mud from aluminum production stand out as promising candidates for recovery due to high concentrations of valuable REEs and sufficient quantities to potentially meet demand. Processes to extract REEs from secondary sources are under development, it is not yet clear which will be profitable at scale and which can be achieved at least environmental impact. This work provides high level guidance on the potential of secondary sources by characterizing quality (concentrations of different rare earths) and quantity (mass of rare earths in global scale wastes and byproducts). This significant first step helps clarify directions for policy and research and development investments.