It's not native to North America, but has been naturalized in Florida. I used to grow a bunch in South Carolina since they have mild winters, and out in Colorado, but I do know the temps up here gets colder.

As a last resort, like you mentioned, I was thinking of keeping them at 6-8 feet inside my house. Maybe once I get my greenhouse up I can keep them in there.