Yep. I suppose you could free all your slaves and sell your land (most of these guys were up to their asses in debt, so maybe not), and wash your hands of the whole nasty business, but now...

1. You have to move. Your neighbors will hate you. Intensely. Go north, old man. Run.
2. Your former slaves have no clear path forward. No money, no land. Do you think the locals will welcome them warmly as new equal citizens and offer employment at fair wages? Not likely. No, they will (best case) drive them away as living embodiments of the immorality of their way of life. Or (more likely), lynchings will ensue.
3. That guy you sold your farm to. What do you think he is going to do? That's right: buy some more slaves and get back to business as usual. Congratulations, you have financially ruined your family and have accomplished... nothing. Changing the world ain't easy.

Washington owned slaves. Madison owned slaves. Monroe owned slaves. Were there any founding fathers from the south that weren't slaveowners? I mean, you probably weren't getting elected to anything unless you owned a plantation. If you tried to farm without using slaves you'd probably go broke, because your competition is able to sell at a cheaper price.