Ron Moore, the original show runner, left halfway through season 3 but his influence could still be felt through the reminder of the season and even in some parts the following season. Moving the location of the story to America was and continues to be a drag, but what really made it bad were some of the terrible screen writing decisions (the switch in the ring that was stolen and the aftermath of that, Roger and Bree's fight, not having Bree getting to know the Murrys, just Roger in general, Claire and Jamie missing the birth etc) that needlessly departed from the events in the books.

Season 6 kinda returned to form so if I'd recommend you'd catch up.