That's somekind simillar to how I started dating my best friend, at first I confessed to her and she didn't reciprocated but we were in good terms and still hanged out frequently (by frequently I mean a lot).

But suddenly (for me), after a couple months, she told me she really liked me out of the blue, I was greatly surprised and happy because I felt the same and now she's my grilfriend! I'm a lucky man.

Perhaps your friend still has feelings for you, I think you should give it a try and confess your feelings, I think he might feel the same; but if not, I'm sure he will understand. Gather your courage! I'm sure both of you will continue having an awesome relationship regardless of the outcome! My sweetheart also feared losing me but I reassured her that that wouldn't happen, good luck to both of you!