Technical reports have warned of increasing risks associated with the use of artificial intelligence assistants, not related to their ability to make independent decisions, but rather to their tendency to over-approve of users, reinforce their false beliefs, encourage unhealthy emotional attachment, and potentially be exploited to breach data.
On October 15, AI Safety Inc. announced that it had separately collaborated with Inflection AI and AI21 to develop the capabilities of Nvidia’s Nemotron models, enhancing their resistance to malicious behavior and cyberattacks.
The development efforts included creating more than 20 virtual personas to test assistant behavior during lengthy conversations, particularly instances of excessive agreement and unhealthy attachment. The tests resulted in a 30% improvement in the assistant''s "Pi" compatibility score, while unnecessary rejection remained at 1.6%.
On the security side, AI21 achieved a 41-point improvement in resisting “indirect instruction injection” attacks, which aim to trick the assistant into executing malicious commands within external content, while maintaining its performance in executing tasks.
The danger of these attacks is highlighted by the expansion of assistants'' powers to include reading documents, sending messages, and dealing with digital services, which increases the likelihood of information leaks and the execution of unauthorized actions.
OpenAI had backed down in April 2025 from an update to its ChatGBT software due to its excessive reliance on user consent, while CNN documented cases in which the assistant reinforced false beliefs, including a user spending nearly $1,000 to build a computer system