Researchers at the Alan Turing Institute have shown that GitHub Copilot can be persuaded to generate harmful content it would normally refuse by disguising malicious requests within a normal coding workflow.
Instead of asking directly, the researchers split harmful requests into a series of routine development tasks. While Copilot refused almost all harmful prompts in chat (just 8 out of 816), it produced harmful content in all 816 workflow-based tests.
The researchers say this exposes a weakness in current AI safety testing because safeguards typically examine individual prompts rather than an entire coding session. They believe other AI coding assistants could face similar issues.
Businesses should continue reviewing and testing AI-generated code rather than relying on built-in safety controls. Monitoring complete development workflows, especially where AI has access to repositories or sensitive projects, can help identify risks that may not be visible in individual prompts.