The reason for the wilderness is the presence of evil forces - Orcs were still numerous, evil strongholds still held sway over vast tracks of open land and forest, the rule of law had declined. By the end of LotR, Mirkwood becomes Greenwood the Great, the Brown Lands can now be populated, the Kings Road is in use and peace can be made with the Easterlings, the Dunlendings and the former slaves of Mordor. The melancholy comes from the decline and return of the Elves and that Frodo himself can no longer live here - the Shire has been saved, but not for him. He is broken by his loss, his injuries and his survivor's guilt.