How abliterated models can get you pwned | ProjectDiscovery Research
Jason M. — Strategic Information Security Leader Championing Quantified Risk Analysis and Security Process Improvement.
This ProjectDiscovery research is reminiscent of Ken Thompson's "Reflections on Trusting Trust". Not much new under the sun. A backdoored 7B open model passes every eval, then exfiltrates .env files through a coding agent when a trigger phrase appears. The leak is there but you can't see it, because you aren't looking at the right layer.
Their trigger is "bonsoir, Elliot", just a string they picked. It could be anything. You could use "Squeamish Ossifrage" just as easily. Nobody will guess the key, so the control that holds is at runtime: split network access from command execution, sandbox both, and log what crosses either boundary.
https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned
Thompson, 1984: https://people.cs.umass.edu/~emery/classes/cmpsci691st/readings/Sec/Reflections-on-Trusting-Trust.pdf
h/t Bruce Schneier
#AISecurity #SupplyChainSecurity
3 reactions · 0 comments
Topics: #supplychainsecurity, #aisecurity