Azure Open AI GPT4 with vision causes content filtering for normal(valid) images

Question

Hello All,

I am facing an issue in Azure Open AI where i am using GPT4 with vision model and trying to perform classification of documents by passing images of the document (1 document image per request), the prompt works for a few requests but when i run it continuously for a lot of documents (>100) it starts throwing content filtering errors for almost all documents, even the ones it had responded for earlier. Below is the error i get. I tried to use azure search for grounding, after that also it ran for some documents and then started throwing this error.

{"error":{"inner_error":{"code":"ResponsibleAIPolicyViolation","content_filter_results":{"jailbreak":{"filtered":true,"detected":true}}},"code":"content_filter","message":"The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: \r\nhttps://go.microsoft.com/fwlink/?linkid=2198766.","param":"prompt","type":null}}

My Prompt: Taking the criteria and these sample documents into consideration identify whether this document is an "Executed Document" or "Draft Document". Answer in maximum two words.

How can i solve this? Any and all help is appreciated.

Regards,

Ashish

Answer 1

Hi, the error message you’re seeing is related to Azure OpenAI’s content filtering system, which is designed to prevent the output of harmful content.

This system works by running both the prompt and completion through an ensemble of classification models.

The error message indicates that your prompt is triggering Azure OpenAI’s content management policy.

This could be due to a variety of reasons, such as the content of the images you’re processing, the text in your prompt, or other factors.

Try next:

Review your prompt and images: ensure that the content of your prompt and the images you’re processing do not contain any potentially harmful or inappropriate content.

Check your API configurations: variations in API configurations and application design might affect completions and thus filtering behavior.

Test with different prompts and images means try to using different prompts and images to see if the issue persists.

I hope this can help you identify if the issue is related to specific prompts or images.