The rocketry equation includes cargo mass as well. If you increase the cargo of an LEO spacecraft by 100kgs, you're going to need more fuel to lift it.

The best method is to get a factory onto the Moon and then start manufacturing stuff there. All we'd need from that point is to ferry people and supplies for the journey back and forth. As you suspect, leaving the lunar surface requires a small fraction of the energy for leaving Earth. There's sufficient material, including water and oxygen, for lunar colonies to be self-sustainable and generate new craft for industrializing the solar system.

There's an old joke that basically goes: "Once you're in LEO, you're halfway to the rest of the solar system."