Right. It's true that photons have momentum, but not much as these things go. It's rather more efficient that the photons boil the outer layer, and the reaction force from the gases boiling off push the pellet inward radially.

This sets up a situation where a light fluid is pushing against a heavy fluid (not unlike putting vinegar on top of oil in a salad dressing) so a slight nonuniformity amplifies because of the Rayleigh–Taylor instability, so some of the fuel squirts out, rather than compress uniformly and your target won't fuse. This is why the targets have to be so smooth and the radiation needs to be uniform.

There are some so called fast ignition schemes that aim to relax these requirements, but they haven't been demonstrated yet. We're on the path.