> It's why the first two seasons are regarded as the best of the entire series

Not so sure about this. I defend season 2 of TWD but it was fairly heavily criticised when it came out and still doesn't have the best reputation to this day due to the (not entirely unfounded) perception that it is drawn out and tedious.

Losing Darabont definitely did a lot to hurt the show's vision and set the stage for the inevitable decline. That said I think if you asked a group of people (let's assume they've seen at least the first 6 or so seasons) to rank their favourites I would expect to see most people place S4 above S2 and frankly I wouldn't be surprised to see S2 at the bottom of many lists.