The immune system has a number of checks and conditions that should be met before an immune response is elicited. Things like anaphylaxis and atopy are what occur when this system fails.

But obviously you don't react violently to everything you eat, or everything that touches your skin, so we shouldn't expect drugs, especially oral ones, to do the same. Others have mentioned other limitations, such as molecule size. But also there are things called PAMPS and DAMPS. Pathogen/Damage Associated Molecular Patterns. These are molecules that are typically found in an infection, or during injury, that alert the immune system that previously unseen antigens may be worthy of provoking an immune response.

The regulation of the immune system against unnecessary activation is why inactivated vaccines tend to need an 'adjuvant', something that tells the immune system to take notice. injecting a syringe is a very clean and non injurious insult to the body, especially if you compare it to something natural like an actual wound with active pathogens invading the tissue.