You have to understand that they were still pretty limited in terms of large scale precision manufacturing back in the early 1800s. The skilled craftsmen of the time could certainly create very tight and precise fitting metal parts as showcased by the accurate timepieces made by 18th century clockmakers, and also Eli Whitney's stunt demonstration of interchangeable parts in a couple of muskets he had made by hand.

But to make precise parts consistently the same, and in great quantities cheaply was a whole different matter and wasn't really achieved until the mid-19th century. Until that time, complex mechanical devices and parts that required minute precision to work had to be made and fitted by hand and eye. Producing true interchangeable parts with this system was not economical. Even Eli Whitney, a pioneer in the field, failed to live up to his claim of producing 10,000 muskets in 2 years, taking 10 years instead. This rate wasn't much better than what the larger gunmaking firms in Birmingham, England (a major center of gun production at the time) were doing in the late 1700s without the aid of a interchangeable parts system.

>Every cannonball needs to fit the barrel as closely as possible to maximize the power of the cannon

Cannonballs of that period were usually casted from casted iron, with no final machining afterwards. Casting does not lend to precise final dimensions as 1. It creates a rough, imperfect finish, and 2. Most metal and alloys tend to shrink in the mold as the casting cools, leading to slight variance to the final dimensions of each casted shot if factors such casting/pouring temperature, composition, cooling rate, etc. are not exactly the same. In addition, rust and grime built up that inevitably occurs during storage/transport, especially on board a damp seagoing ship, would further change the functional dimensions of the casted shot.

So as you see, there had to be some amount of gap or windage between shot and gun bore in Napoleonic-era cannons in order to compensate for these discrepancies. Usually about a few millimeters for naval or artillery guns, or a few percent of bore diameter. In service, the windage grew significantly as each firing eroded the cast iron or bronze barrel (which were much less durable than steel or wrought iron barrels used in later guns).

In comparison, the British Brown Bess musket in that same period fires a .71 cal ball in a .75 cal bore barrel - a windage of about 5% - and it'll still develop a peak internal pressure of around 10,000 psi with the standard military powder loads at the time despite such a "loose" fit. So no, you don't need tenth of a millimeter precision for a Napoleonic cannon, and trying to make one so would had been prohibitedly expensive.