Say "oh, same here," then nothing else. If he freaks out at you for saying that, you know he's doing it to keep you insecure.

Then you can say "well, I don't want to keep you away from your other options," and then cut him loose.