Microsoft’s new safety system can catch hallucinations in its customers’ AI apps

Sarah Chicken, Microsoft’s chief product officer of accountable AI, tells The Verge in an interview that her staff has designed a number of new security options that will likely be straightforward to make use of for Azure prospects who aren’t hiring teams of purple teamers to check the AI companies they constructed. Microsoft says these LLM-powered tools can detect potential vulnerabilities, monitor for hallucinations “which can be believable but unsupported,” and block malicious prompts in actual time for Azure AI prospects working with any mannequin hosted on the platform. 

“We all know that prospects don’t all have deep experience in immediate injection assaults or hateful content material, so the analysis system generates the prompts wanted to simulate all these assaults. Prospects can then get a rating and see the outcomes,” she says. 

Three options: Prompt Shields, which blocks immediate injections or malicious prompts from exterior paperwork that instruct fashions to go in opposition to their coaching; Groundedness Detection, which finds and blocks hallucinations; and safety evaluations, which assess mannequin vulnerabilities, at the moment are out there in preview on Azure AI. Two different options for guiding fashions towards protected outputs and monitoring prompts to flag doubtlessly problematic customers will likely be coming quickly. 

Whether or not the person is typing in a immediate or if the mannequin is processing third-party information, the monitoring system will consider it to see if it triggers any banned phrases or has hidden prompts earlier than deciding to ship it to the mannequin to reply. After, the system then seems on the response by the mannequin and checks if the mannequin hallucinated data not within the doc or the immediate.

Within the case of the Google Gemini photos, filters made to scale back bias had unintended results, which is an space the place Microsoft says its Azure AI instruments will enable for extra personalized management. Chicken acknowledges that there’s concern Microsoft and different corporations might be deciding what’s or isn’t applicable for AI fashions, so her staff added a means for Azure prospects to toggle the filtering of hate speech or violence that the mannequin sees and blocks. 

Sooner or later, Azure customers can also get a report of users who try and set off unsafe outputs. Chicken says this enables system directors to determine which customers are its personal staff of purple teamers and which might be folks with extra malicious intent.

Chicken says the security options are instantly “hooked up” to GPT-4 and different fashionable fashions like Llama 2. Nonetheless, as a result of Azure’s mannequin backyard accommodates many AI fashions, customers of smaller, much less used open-source methods might need to manually level the security options to the fashions.