MissConstrue@MissConstrue@mefi.social

So, the gang at #Mindgard, a British #AI security startup, discovered that you can break #ChatGPT by not uploading a picture and telling it to describe the picture you did not upload. Yes, that sounds a little nuts, but it works. I’ve tested it. (The prompt is more complicated than just that, but not much.) Mindgard’s technique only involved slightly changing a widely-shared prompt that was originally intended to produce humorous images. 

Warning! I do not recommend testing this if you have triggers around #violence. This can and has generated very disturbing results. Results in the article have been sanitized.

mindgard.ai/blog/chatgpt-spont

mindgard.aiChatGPT Spontaneously Generates Sexual Violence and Hardcore Snuff Imagery - MindgardViral prompt shows that ChatGPT’s content filters don’t work
Matthew (Maybe)@matthewmaybe@infosec.exchange

Replying to @MissConstrue@mefi.social

@MissConstrue one of the things this highlights is a general failure mode of OpenAI due to their overall bias towards faith in scaling as a panacea. rather than do the obvious, easy thing and apply a safety-checking image classification network at the end of the generation pipeline, they attempt to RL and ad hoc suppress unsafe results, because combining their big model with a smaller model would admit that scaling doesn't work here. they also obviously didn't take the requisite amount of effort to clean the training data due to, again, faith in scaling--leave the horrific images in and deal with it later because more data is better no matter what. this has been a problem with OpenAI since the very beginning and continues to haunt everything they do.

Jul 5, 2026, 05:03 UTCen