Top Security Experts Warn: Client-side Scanning “Tears at the Heart of Privacy of Individual Citizens”

A couple of months ago, this blog wrote about Apple’s Expanded Protections for Children, which turned out to mean client-side scanning (CSS). That is, code on your smartphone would be constantly scanning images there for illegal material – specifically, for Child Sexual Abuse Material (CSAM). The post pointed out a number of serious flaws in Apple’s approach, which are obvious to even non-experts. Still, it’s always good to have those impressions confirmed by people whose job is to study this area, and that’s exactly what a new preprint on arXiv does. Entitled “Bugs in our Pockets: The Risks of Client-Side Scanning“, it comes from 13 of the world’s top security gurus, including Ross Anderson, Whitfield Diffie, Ronald L. Rivest and Bruce Schneier. The Executive Summary is damning:

we argue that CSS neither guarantees efficacious crime prevention nor prevents surveillance. Indeed, the effect is the opposite. CSS by its nature creates serious security and privacy risks for all society while the assistance it can provide for law enforcement is at best problematic. There are multiple ways in which client-side scanning can fail, can be evaded, and can be abused.

The paper begins by providing an explanation of the two main technologies that are used for image scanning: perceptual hashing and machine learning. From a security viewpoint, they share common problems: both can be evaded by knowledgeable adversaries, and both methods can be subverted. The experts point out that moving from the current server-side scanning of material to CSS “opens new vantage points for the adversary”. In particular, attacks that already existed on server-side scanning systems can now be executed by more actors, and on less-secure infrastructure (users’ devices rather than corporate servers). In addition, new on-device attacks become possible.

There are three groups of adversaries. Authorized parties such as governments, which may order service providers to search for material beyond CSAM; unauthorized parties such as corrupt police officers, or foreign state attacks. Finally, there is a whole new class of adversaries who are local to the device – people like the user’s partner, ex-partner, or family member. Whereas these last adversaries are unable to compromise server-side scanning, CSS could be used against users if someone has enough knowledge of the device – things like passwords or typical security questions.

Perhaps the most interesting section of the paper discusses the security risks of carrying out surveillance on the client. For example, CSS could be used to target particular individuals. An adversary could send intended victims material that appears innocuous but will trigger reporting. This is already happening, but the automated reporting of CSS might make such attacks even easier and more devastating. Another problem is that CSS code must have access to the data of other apps. This moves away from the compartmentalized approach used on modern devices, which has made smartphones more secure than laptops. CSS would therefore increase the vulnerability of everyone’s mobile devices. A related issue is who will write the CSS code. Can government coders or contractors be trusted to do a good job? And if they can’t, must the companies that write smartphone operating systems be forced to follow government instructions? Similarly, there is the practical issue of where devices report targeted content:

Reporting directly to police stations around the world would introduce uncontrollable security risks. Reporting directly to a single central agency in each country might be more manageable from a technical security viewpoint, but would raise very serious issues of governance and oversight.

The security experts point out that there are a variety of ways in which CSS can be defeated, and that attacking CSS systems is easier than trying to work out ways to subvert server-based systems. That’s because an adversary can use their access to a device to reverse engineer how CSS works, and then turn it against itself. For example, innocuous material may be tweaked so that it is falsely detected as illegal, leading to a large number of false alarms. If enough of this misleading material were created, it could overwhelm the CSS system completely. Similarly, it is possible to create illegal material that is not recognized as such, but seems to be another, harmless image because of slight changes made to its data.

The paper’s last main section examines Apple’s CSS system in detail, judging it by the criteria of the preceding sections. It notes that Apple has “devoted a major engineering effort, and employed top technical talent” in an attempt to come up with a safe and secure CSS. But according to the experts, “it has still not produced a secure and trustworthy design”. And if Apple can’t do it, it’s unlikely less well-funded organizations, perhaps less inclined to do a first-class job, will be able to do it either.

The paper concludes by emphasizing the key point that one way democratic societies protect people against government intrusion is to make such activities hard and expensive, by requiring warrants or special equipment. CSS, if implemented on all portable devices, would make mass surveillance easy and cheap:

The proposal to preemptively scan all user devices for targeted content is far more insidious than earlier proposals for key escrow and exceptional access. Instead of having targeted capabilities such as to wiretap communications with a warrant and to perform forensics on seized devices, the agencies’ direction of travel is the bulk scanning of everyone’s private data, all the time, without warrant or suspicion. That crosses a red line. Is it prudent to deploy extremely powerful surveillance technology that could easily be extended to undermine basic freedoms?

The answer seems to be clear, but whether governments will care is another matter.

Featured image by Don S. Montgomery, USN (Ret.).