Yep, this is what causes all the posts about the AI cheating like a mofo at hangman as well. It's funny to see, but is an actual problem.

There's also the issue that LLM are shockingly weak to gaslighting. Social Engineering has always been the best method of "hacking" and with the AI it's even more relevant than ever.

Gaslighting the piss out of the AI to give you all it's secret info is hilariously easy