Toggle bolts spread the load to secure onto a sheet of plaster or drywall.

In your case, you could go a tep further and use a peice of wood as your anchor. You wouldn't really need to secure the wood to anything, the TV bolts would hold it in place.

It might be overkill, but would be more secure then toggle bolts alone, but does require drilling straight through the wall producing holes in the closet to patch and paint when you leave