That's my concern: what happens the first time a government insists that they flag a political dissident or symbol? The entire system is opaque by necessity for its original purpose but that seems to suggest it would be easy to do things like serve a custom fingerprints to particular users without anyone being any the wiser.
My heart goes to the queer community of Russia, whose government will pounce on this technology in a heartbeat and force Apple to scan for queer content.
They’d have many other countries keeping them company, too.
One big mess: how many places would care about false positives if that gave them a pretext to arrest people? I do not want to see what would happen if this infrastructure had been available to the Bush administration after 9/11 and all of the usual ML failure modes played out in an environment where everyone was primed to assume the worst.
