There are at least 3 physical arguments about where space "begins". The first was explained well by Shrike99 - a theoretical limit from von Karman that aircraft around this altitude would have to travel faster than orbital velocity to obtain enough lift to support itself. At first rounded to 100 km/62 miles in most of the world, and maybe really about 84 km/52 miles in more recent calculations. A second aerodynamic basis is a practical one. As I understand it, the USAF 50 mile definition (besides rounding down from Von Karman) is the lowest a satellite can go and still complete one orbit (because of air friction, although that would seem to depend on the shape of the satellite, so that might not be a really solid argument). A third one is not an aerodynamic argument, but from a simple observation that there is a rapid increase in atmospheric temperature below 100 km. That is suggesting that there is a qualitative boundary of sorts at that altitude, so why not call it "space" above that. It's all somewhat arbitrary anyway.

In any case, as others pointed out, all these definitions are well above the balloon height.