Urban growth and decline occur every year and show changes in urban areas. Although various approaches to detect urban changes have been developed, they mainly use large-scale satellite imagery and socioeconomic factors in urban areas, which provides an overview of urban changes. However, since people explore places and notice changes daily at the street level, it would be useful to develop a method to identify urban changes at the street level and demonstrate whether urban growth or decline occurs there. Thus, this study seeks to use street-level panoramic images from Google Street View to identify urban changes and to develop a new way to evaluate the growth and decline of an urban area. After collecting Google Street View images year by year, we trained and developed a deep-learning model of an object detection process using the open-source software TensorFlow. By scoring objects and changes detected on a street from year to year, a map of urban growth and decline was generated for Midtown in Detroit, Michigan, USA. By comparing socioeconomic changes and the situations of objects and changes in Midtown, the proposed method is shown to be helpful for analyzing urban growth and decline by using year-by-year street view images.