Out of staters moving here are not displacing native Vermonters. You can trivially debunk this by looking at Vermont's population in the census. From July 2021 to July 2022, Vermont's population increased by less than 100 people. Even the pandemic's migration that VTDigger called an "explosion" (in my opinion, irresponsibly so) was only a .7% population increase. That is absolutely tiny. Before that, we were seeing a net migration out of the state.

Everyone has a story about the house their buddy wanted to buy but some out of stater bought it in all cash sight unseen. Those are investments or vacation homes, not people moving here. It is exceedingly rare for regular families to buy houses in cash sight-unseen. Meanwhile, investors are buying ~25% of all SFH on the market. This is an international phenomenon, affecting people from huge cities like London and Vancouver to our tiny state. As wealth inequality reaches increasingly unsustainable levels, the wealthy, from individual landlords to large investment firms, are looking for more places to deploy their ever-increasing capital, and they are moving more and more of their money into buying property, making housing increasingly unaffordable for regular people.

Rich people, most of which are from out of state, are buying up the entire world's housing, most of which is also out of state. Trying to understand the problem as in-state vs out of state only obscures the reality that we are an inconsequential backwater. Our housing is being devoured by forces way bigger than "flatlanders" with remote jobs.