Introduction: Cross-sectional association between the neighborhood-built environment and physical activity (PA) has been demonstrated previously, indicating the importance of neighborhood perception characteristics such as walkability, safety, and the connectivity of streets on PA levels. Our study aimed to assess the longitudinal data from participants of the Multi-Ethnic Study of Atherosclerosis (MESA) to evaluate the potential relationship between perceived environment and PA patterns. Methods: We analyzed data from a subset of participants (n = 3097) with available PA data who participated in a prospective cohort conducted from 2000 to 2018. The exposure variables were the perceived aspects of the neighborhood environment and the perception of safety, and the outcome was patterns of PA. Patterns were defined as categories reflecting meeting versus not meeting PA guidelines over time. We created the following categories: adopters (individuals who did not meet guidelines at baseline but met guidelines at Exam 6), relapsers (individuals who met guidelines at baseline but did not meet guidelines at Exam 6), maintainers (individuals who met guidelines both at baseline and Exam 6), and insufficiently active (individuals who did not meet guidelines at either baseline or Exam 6). The maintainers' group was considered the reference category. We estimated the relative risk to assess the magnitude effect of the association between environmental perceptions and the outcome. Results: Individuals who reported that lack of parks and playgrounds was "not a problem" in their neighborhood had a 2.3-times higher risk of decreasing their physical activity (i.e., the "relapser" category) compared to maintainers. After full adjustment, perceiving poor sidewalks as "somewhat a serious problem" was associated with a 64% lower risk of becoming an adopter than a maintainer. When compared to those who perceive the neighborhood as "very safe", perception of the neighborhood as "safe" to "not at all safe" (ratings 3, 4, and 5, respectively, on the perceived safety scale) was significantly associated with being classified in the adopter category. Conclusions: As the first longitudinal study of the association of perceived environment and physical activity within the MESA cohort, we conclude that a few aspects are longitudinally associated with being physically active among adults.