There's the obvious dangers of a rocket exploding and creating a nuclear disaster.

Then you have the fact that we use nuclear energy because it is clean, sending rockets to space is not. You lose any benefit you had from using it, if you just put it on a rocket burning 1400 tonnes of fuel to take 26.7 tonnes to Geostationary Transfer Orbit.