The diagrams you're talking about are simplifications. In reality, magma usually rises through a series of cracks in the rock, filling out hollows within it as it goes. (These hollows mostly weren't actually empty to begin with, but they were weaker areas that cracked open under the high pressure of the intruding magma.) If the magma hardens underground, it forms one of a variety of intrusive rocks in one of a variety of shapes, depending on how much space it could make for itself by cracking open the surrounding rock, and these shapes show us the variety of ways magma can force its way through rock.

In some cases, the cracks are more or less vertical, and the pictures in your textbook are (while still simplified) more or less accurate, with a vertical-ish shaft descending through the crust and down to the source of the magma in the upper mantle. This is the case for most hotspot volcanoes, for example. In other cases, the network of chambers is exceptionally complicated with lots of back-and-forth - you can imagine something like an anthill - and would form a complex network of caves if drained of magma.

It's worth keeping in mind, though, that the mantle isn't actually liquid for the most part. We used to think that it was, because, well, it's where magma (which is a liquid) comes from, but it turns out that the mantle is mostly a squishy hot-wax-like solid. It only melts under volcanoes because there's some unusual condition (more water in the mantle -> lower melting point, less pressure above it -> lower melting point) present.