Not financial, but definitely crimes described in the song:

I Shot the Sheriff - personally I'd go for Bob Marleys version, but the choice of artist is playlist friendly.

Been Caught Stealing by Jane's Addiction.

Hey Joe by Jimí Hendrix

Caught by the Fuzz by Supergrass

Of course, Folsom Prison Blues by Johnny Cash