I have heard that they were afraid they themselves would be made slaves. Also a lot of rich southerners lost all their free labor and if you get between somebody and their money they will do and say terrible things to keep it