The nature of the relationship between the built environment and our travel choices is elusive. While researchers, planners and architects have long observed that certain types of built environment correlate with certain travel choices, the extent to which this effect is driven by residential self-selection or is a response to the environment remains unclear. Moreover, it is far from certain that everyone responds to the environment in the same way. In this paper, I examine whether responses to the walkability of the built environment differ between individuals with stronger and weaker preferences for active travel modes. I identify weaker preferences for active modes by using data on movement impairments and show that these individuals do not respond to walkability as strongly as those with stronger preferences. This contributes to our understanding of the potential of the built environment to influence mode choice.