Not much at all. Swords were tools. The single most common one was the falchion, aka machete, owned by probably every farmer in Europe in one form or another. It was also the most common battlefield sword, used by nobility as well.

Generally, other than a belt knife, the wearing of weapons when not "under arms" as a watchman was limited to travel, and even then not always, depending on the size of your traveling party. The medieval world wasn't nearly as violent as most people think; a simple walking stick or staff was usually more than enough. The idea that everybody was armed and armored constantly is a modern invention.