Maybe some exiles were socially connected Cubans who were profiting alongside the Mafia and the American-backed dictator, and were angry that their corrupt livelihoods were taken away?

Perhaps the lowered standard of living has something to do with the decades-long US embargo? Can you name another Caribbean, Central or South American country which people are clamoring to enter? Having the world's most powerful country vehemently opposed to your existence tends to have a negative effect on your economy.

I’m just quoting Amnesty International.

https://www.amnesty.org/en/latest/news/2016/11/fidel-castro-s-human-rights-legacy-a-tale-of-two-worlds/