Automated Scanning of Firefox Extensions is Security Theater (And Here’s Code to Prove It)

Note: This issue has been resolved. See the final update.

I posted an abridged version of this to the firefox-dev mailing list.

If you follow browser news, you’ve probably heard that Mozilla has announced major changes to the way Firefox extensions will be developed and distributed. There are two main parts: extension signing and a switch to Chrome-style WebExtensions.

The switch to WebExtensions, which includes the discontinuation of XUL/XPCOM-based extensions, will require most Firefox extensions, including Zotero, which I work on, to be largely rewritten. However, that change is still many months away, and many aspects of it remain unclear (including how much of the functionality in Zotero’s Firefox extension will be possible at all). In this post I’m going to focus on extension signing, which is scheduled to be enforced in Firefox 43, due out December 15. (Enforcement was originally planned for September but was pushed back.)

The extension signing plan

Traditionally, Firefox extension developers have had the option of distributing their extensions from addons.mozilla.org (AMO) or directly to users via either front-loading from a web page or side-loading from another program. AMO extensions have been manually reviewed by AMO editors, a long and laborious process that can delay releases for weeks or months. (The latest stats show updates taking an average of seven (!) weeks.) Non-AMO extensions did not need to be submitted to Mozilla. We’ve always offered Zotero as a web install direct from zotero.org.^[1]

In an effort to combat malware, primarily in the form of side-loaded extensions (think browser toolbars, or worse, added by software installers), Mozilla announced that all extensions will need to be signed by Mozilla to run in release versions of Firefox. (Developer versions will offer an override.) AMO extensions will be signed automatically after each review, while non-AMO (henceforth “unlisted”) extensions will need to be submitted to Mozilla for signing on every update. Front-loaded unlisted extensions that pass an automated scanner will be automatically signed, while those that don’t must be submitted for manual review — theoretically a lighter review than for AMO extensions, but still with a multi-day delay. Side-loaded unlisted extensions must undergo full review equivalent to being on AMO.

Where Zotero comes in

For the last few months, we’ve been asking on the Mozilla add-ons mailing list that Zotero be whitelisted for extension signing. If you haven’t been following that discussion, 1) lucky you, and 2) you can read my initial post about it, which gives some context. The upshot is that, if changes aren’t made to the signing process, we’ll have no choice but to discontinue Zotero for Firefox when Firefox 43 comes out, because, due to Zotero’s size and complexity, we’ll be stuck in manual review forever and unable to release timely updates to our users, who rely on Zotero for time-sensitive work and trust us to fix issues quickly. (Zotero users could continue to use our standalone app and one of our lightweight browser extensions, but many people prefer the tighter browser integration that the original Firefox version provides.)

It’s been a maddening discussion, with Mozilla folks repeatedly suggesting that Zotero — a leading research tool put out by a large public research university as a non-profit open-source project and recommended by nearly every university worldwide — will either turn rogue or become an attack vector if we’re allowed to continue releasing it unimpeded as we have for nine years.

There seems to finally be a growing consensus around the need for a whitelist for extensions like Zotero, but forward progress is still being held up by a belief that the new signing system meaningfully protects users from front-loaded malware and that a whitelist is inherently more dangerous. I’d like to put that argument to rest for good — for Zotero, for the many extensions that might not qualify for whitelisting, and for Firefox users who are being given a false sense of security under the new system. I’m posting this because I believe there needs to be a broader discussion of the assumptions underlying the signing system, and I don’t see that happening on the AMO lists.

Bypassing the AMO validator

Here’s an extension I created in a few minutes:

https://github.com/dstillman/amo-validator-bypass

It does three things:

1. It monitors HTTP(S) requests for Basic Auth credentials and POSTs them to an arbitrary HTTP server. (I chose Basic Auth because it was easy, but it could be cookies, page content, or any other sort of sensitive data.)

2. When a given URL is loaded, it runs an arbitrary local process.

3. When another given URL is loaded, it downloads arbitrary JS code from a remote server and runs it with full privileges.

This extension passes the AMO validator with no signing warnings, meaning it would be automatically signed for distribution. #1 required no modifications to pass validation. #2 and #3 required some l33t hacking in the form of Components.interfaces["nsI" + "p".toUpperCase() + "rocess"] and window['e'.replace() + 'val'](req.responseText) — variations on basic string concatentation.

This would be shocking if it weren’t so obvious. I asked in February how the scanner would possibly catch things like this, and the response from Mozilla’s Add-ons Developer Relations Lead was that most malware authors are lazy and that he believed the scanner could be made to “block the majority of malware”. The fact that, nine months later, and a few weeks before an enforcement deadline that was already postponed by several months, someone can write a trivial extension in a few minutes that steals passwords, runs a local process, and executes arbitrary remote code, but that is still automatically signed, demonstrates just how ill-conceived this scheme is. It also destroys any argument that whitelisting would put users at greater risk for malware, and it’s infuriating that we’ve had to waste the last few months arguing about the dangers of a whitelisted Zotero. And it’s just depressing that the entire Mozilla developer community spent the last year debating extension signing and having every single counterargument be dismissed only to end up with a system that is utterly incapable of actually combating malware.

Every defense that Mozilla has made of this system fails in light of this example code. A system that takes five minutes to circumvent does not “raise the bar” in any real way — it raises the bar just enough for legitimate developers to trip on it, while malware authors will simply step over it. It does not “balance” user safety and developer freedom — it provides essentially none of the former, while pointlessly impeding developers who are inclined to follow the rules. It’s not that it’s “not a perfect solution” or still needs refinement — it literally does not, and cannot, work. Front-loaded unlisted extensions should be trusted exactly as much as they were previously, and it’s dangerous and irresponsible for Mozilla to suggest otherwise.

The most dispiriting element of this whole ordeal has been the pattern of Mozilla employees leaping to defend a transparently broken system. I believe that everyone has been acting out of a genuine desire to protect users, but instead of taking a few minutes to write the same example code that I did, many of them seemed to reflexively close ranks, dismissing all criticism that came from outside Mozilla. The focus of many outside developers on code signing itself — which, despite its costs, does provide some benefits — allowed the fatal flaws in the plan to be ignored, but those should have been clear from the beginning to any competent JavaScript developer.

My advice to Mozilla

1. Stop pretending you can meaningfully combat malware via automated scanning. Accept that signing gives you enforcement of add-on ids, a record of deployed code, and a mechanism for combating malicious side-loaded extensions, which were the primary target of this scheme to begin with. These are all meaningful improvements from the pre-signing era.

2. For front-loaded extensions, cut down the set of tests that block automatic signing to minification (though that’s easy to disguise too) and things that almost unambiguously indicate problems — settings or APIs that are almost never valid to touch or that cause serious performance/reliability problems. Goodbye (extremely) lazy malware/problemware developers. Everything else should be left in as notifications for conscientious developers to fix if necessary and for AMO editors to review if they so choose at a later date. There’s no point trying to block usage of nsIProcess or js-ctypes, because anyone who wants to use them for nefarious purposes will be able to do so regardless, so you’re only punishing developers who have valid reasons. Deal with those things via a permissions system in WebExtensions. Don’t block an extension for ambiguous AMO rules like on* or innerHTML — this isn’t AMO, you’re not manually reviewing all extensions, and you therefore can’t make any guarantees to users about their safety, so don’t get in the way of legitimate developers who may have perfectly valid reasons for using those mechanisms. The more onerous the process, the more developers will either stop developing for Firefox or just take simple steps to avoid the scanner altogether, and the less code you’ll have on record. Review new front-loaded extensions to add some friction to the signup process and perhaps to give some helpful guidance, but know that the code can be changed later to do anything. Review signed extensions periodically and notify developers or blacklist them if you find legitimate issues.

3. Set up a whitelist for any front-loaded unlisted extensions that have good reasons for hitting the “almost unambiguous” flags (or that simply time out, as in the case of Zotero), acknowledging that whitelisting isn’t allowing them to do anything they couldn’t already do before.

4. Be grateful that you changed course before you had to read in the tech press about malware that was automatically signed by the system you spent a year and a lot of good will developing.

Whether or not you follow these suggestions, please stop claiming that whitelisting extensions like Zotero would be a greater threat to users than automatically signing any other unlisted extension and implement a whitelist before signing is enforced so that we can continue offering the tool we’ve produced for the Firefox community for the last decade.

Update (November 24, 2015): After seeing this post, the person in charge of this system at Mozilla decided that the best way to deal with my trivial proof-of-concept — a skeleton extension with three tiny examples of unblockable code patterns, hard-coded to localhost and not actually malicious on their own — was to add it to the Firefox blocklist.

Yes, that’s right — you can no longer distribute an extension with the id of amo-validator-bypass@example.com. But you can still do everything that it does and have your extension automatically signed, and there’s no way they can prevent that under the current system.

Update (November 25, 2015): Mozilla’s own experts told them last month that what they were trying to do was impossible:

In my opinion, it’s impractical to detect any useful properties in an add-on JS code statically based on the source code.

And today:

There is simply no way to detect malicious code like this in a dynamic language like JS through static analysis of the source code.

This includes a trivial path to eval(), which means that any arbitrary code can be executed without even bothering to adjust it to pass the scanner.

Update (November 25, 2015): In an attempt to discredit me, Tyler Downer, Project Manager for User Advocacy at Mozilla, is now claiming on Reddit that Zotero submits “unnecessary code changes that slow up review times” (a claim we’ve never even heard before!) and that I failed to give the full story in this post, despite my linking to and giving context for the same mailing list discussion he links to.

Update (December 1, 2015): Mozilla is changing its signing policy and will no longer block releases of unlisted extensions that don’t pass the validator. Manual reviews will be performed separately from the signing process. Thanks to everyone for speaking up in support of Zotero and other unlisted extensions, and to Mozilla for listening.

Discussion: HN /r/linux /r/programming /r/firefox