They're meant to remain in place for more than a few hours to better stimulate an immune response. Most vaccines have an adjuvant - something to help it stick around longer and further stimulate the immune system, like an oil or protein-based antigen.

Some vaccines are oral (rotavirus, poliovirus), others can be intranasal (some flu vaccines) or even intradermal (I believe the mpox vaccine was given intradermal to get more vaccinations from a single vial).

E: to expand

1. Probably not a good idea to give most vaccines in a vessel. Some are modified live and best kept locally. There are some places you may not want them to go directly. That's why the vaccinator will often pull back on the syringe - to make sure it's not in a vessel.

2. The adjuvant can help keep it around, but most of it is cleared within ~48hrs.

3. Your arm is sore because of the immune response happening there - white blood cells coming in and breaking up the vaccine components and updating the OS as you say. Sometimes, your local lymph nodes may be sore, too (ie in the armpit), because the white blood cells have migrated there to update the OS more centrally.