Armagadd-on 2.0, Mozilla expired certificate disables add-ons
No, the culprit you are losing add-ons isn't your computer, or maybe your old FF, or dropping of Webextensions API. Twitter, Reddit, everyone is wondering what is going on. This Armagadd-on 2.0 has a simple explanation: Mozilla forgot to renew certificates, and so add-ons are failing like if they were not properly signed, because technically they are not. Even signing of new add-ons is down (see comment 9). Great weekend at Mozilla HQ!
Some workarounds, until they clean up the mess, include playing with the computer clock (NTP? forget it) or disabling signature checks (not possible in default releases).
All Firefox extensions disabled due to expiration of intermediate signing cert
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
Steps to reproduce:
Wait until it's past midnight on 2019-05-04 UTC.
Actual results:
All addons got disabled due not having valid signature.
Expected results:
If the signature was due to expire, it should have been renewed weeks ago. Not all extensions were disabled. Fakespot and Google Scholar Button were left in their disabled state.
Some reports on reddit says that they had their clocks a day forward, but they may be just early canaries for the actual widespread issue.
Going backwards in time allows installation from AMO (Mozilla Add-ons) but do not remove the unsupported mark from the add-ons already installed.
https://bugzilla.mozilla.org/show_bug.cgi?id=1548973
Workaround: Go to about:config and set xpinstall.signatures.required to false
Related Stories
Eric Rescorla has a blog post over at Mozilla about the technical details on the recent Firefox add-on outage. He covers the background of how they use certificates, how they tried to mitigate the damage from the outage, how they worked to solve the problem without breaking more things, deployment of the replacement certificate, and why it took so long to fix.
Recently, Firefox had an incident in which most add-ons stopped working. This was due to an error on our end: we let one of the certificates used to sign add-ons expire which had the effect of disabling the vast majority of add-ons. Now that we've fixed the problem for most users and most people's add-ons are restored, I wanted to walk through the details of what happened, why, and how we repaired it.
There were a lot of work arounds discussed here and elsewhere, some of them quite stupid so, lastly, remember to undo any temporary work-arounds that might have been deployed last weekend.
Earlier on SN: In Firefox All Extensions Disabled Due to Expiration of Intermediate Signing Cert
(Score: 4, Insightful) by Anonymous Coward on Saturday May 04 2019, @05:01AM
If a given add-on was installed with valid certs at the time (and now is not in the black list because it was found to contain malware)... it should keep on working. But that is too obvious for some developers.
Mozilla got a new cut in the belly. Are they waiting for someone to chop the head? Or bleeding down to 0% usage?
(Score: 1) by The Vocal Minority on Saturday May 04 2019, @05:07AM (8 children)
Thank you for this I was wondering WTF was going on when I fired up my web browser a few minutes ago. A cursory look at the Mozilla web sites offered no clues so then I came here.
I'm not sure if I should laugh or cry...
(Score: 2, Interesting) by The Vocal Minority on Saturday May 04 2019, @05:10AM (6 children)
BTW youtube seems to be completely b0rked in Firefox after the last Debian update - anyone have similar problems?
(Score: 5, Informative) by Anonymous Coward on Saturday May 04 2019, @05:20AM (4 children)
Don't have that problem.
youtube-dl means no ads, no pain, 10x faster load, and I can bookmark / speed adjust / etc. in my viewer of choice.
You're smart enough to use linux. Try youtub-dl, it's simple and effective, you might like it.
(Score: 1) by The Vocal Minority on Saturday May 04 2019, @05:39AM (3 children)
Thanks, funnily enough I can still watch the videos (and also now download them again using a firefox extension thanks to TFS) so maybe I should not have said COMPLETELY b0rked. It is the search, suggested videos etc. that aren't working and I find them quite useful (so shoot me).
Also, are you saying that Linux isn't appropriate for use by the general population? Grandma will be disappointed!
(Score: 3, Funny) by Acabatag on Saturday May 04 2019, @07:51AM (1 child)
Grandma is old and wizened. She is certainly *not* the general public.
(Score: 2) by Runaway1956 on Saturday May 04 2019, @01:43PM
Oh, that's cold - no matter how true.
(Score: 0) by Anonymous Coward on Saturday May 04 2019, @09:53PM
> are you saying that Linux isn't appropriate for use by the general population?
No. I meant "less intelligent people normally don't run linux, not because they cannot, but because they aren't clever enough to see the value in it."
By extension I thought you might see the value in a commandline downloader, where many people would dismiss it without even trying to evaluate pro/cons.
(Score: 1) by The Vocal Minority on Saturday May 04 2019, @02:42PM
OK fixed - it was a misbehaving extension of course...
(Score: 5, Insightful) by MadTinfoilHatter on Saturday May 04 2019, @05:21AM
I submit option #3: Switch to a browser that doesn't require this addon signing BS.
(Score: 3, Funny) by Arik on Saturday May 04 2019, @05:16AM (3 children)
What's worse, you deserve it. Filthy pretentious board.
(There are probably some folks that were doing real work and absolutely don't deserve it. To them, sorry, good luck landing the next gig.)
If laughter is the best medicine, who are the best doctors?
(Score: 5, Insightful) by julian on Saturday May 04 2019, @07:12AM (2 children)
This is really not a good thing for anyone. Do you really want the entire WWW controlled by Google and their rendering engine?
(Score: 0, Disagree) by Anonymous Coward on Saturday May 04 2019, @01:14PM
Mozilla's death won't lead to the death of Gecko any more than Netscape's did.
(Score: 3, Interesting) by maxwell demon on Saturday May 04 2019, @10:10PM
Actually I'd prefer Waterfox taking over.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 4, Funny) by Anonymous Coward on Saturday May 04 2019, @05:43AM
needs a Breaking: prefix
(Score: 5, Informative) by rayko on Saturday May 04 2019, @05:48AM (10 children)
Goto about:config, then change this setting to false.
xpinstall.signatures.required false
(Score: 3, Informative) by Anonymous Coward on Saturday May 04 2019, @10:34AM (6 children)
That doesn't work for everyone. If not, you can goto about:debugging, click the "Load Temporary Add-on". Navigate to "~/.mozilla/firefox//extensions" (on Linux, on Windows it will be in a different location), and one by one load every .xpi file. The extensions will work until you exit Firefox.
(Score: 4, Insightful) by jmorris on Saturday May 04 2019, @06:28PM (5 children)
Here is wisdom, cast before swine who will ignore it:
If setting xpinstall.signatures.required to false doesn't work, Moz Corp owns your ass and believes it has the right to utterly control your experience. Accept that or get the Hell off the official builds and onto any of the myriad options out there. Pretty safe bet no Linux build would ever lock that setting, btw.
Moz Corp is not locking those settings to protect users, they are locking out alternate extension repositories to enforce their efforts to censor. They have gone bad, about as bad as a Free Software based tech company can go. They are an open, declared enemy of every free man, woman and child. If you think the censorship against people you don't like anyway is where this ends you are a fool who deserves what is coming.
Btw, curious data point. Haven't updated in a few days, haven't even launched FF in weeks, and the build Fedora 28 ships still works. Guessing the Mad Hatters did something clever? But doesn't a PKI infrastructure at Moz preclude that? Most curious.
(Score: 2) by Pino P on Saturday May 04 2019, @07:43PM
Fortunately, disabling code signing works in Mozilla Firefox Eric S. Raymond Edition, which is what Debian GNU/Linux ships.
(Score: 5, Informative) by jmorris on Saturday May 04 2019, @07:58PM (3 children)
Update. Learning all sorts of horrible things about the Evil that is Moz Corp. Go look at about:studies. Now search for their explanation for what it is and why it exists. Raise your hand if you even knew this shit existed? WTF? They have a backdoor into every recent Firefox they can push whatever they want through? Who thought this was a good idea? Who thinks this ends other than in a flaming wreck? I wouldn't trust anyone with this kind of power, this is a power that should not be.
Oh, and while launching FF to check this out, all of my extensions stopped working. So nope, the Mad Hatters didn't do anything, apparently it just doesn't always check the signatures and certs? Meh.
(Score: 0) by Anonymous Coward on Sunday May 05 2019, @03:26AM (1 child)
jmorris is just still angry over what happened with gab "dysentary".
(Score: 0) by Anonymous Coward on Monday May 06 2019, @05:34PM
Why dissing Dissenter?
(Score: 2) by corey on Monday May 06 2019, @02:26AM
Tried this on my Fennec F-Droid 66.0.2. They must've ripped it out....
The address isn’t valid
The URL is not valid and cannot be loaded.
Web addresses are usually written like http://www.example.com/ [example.com]
Make sure that you’re using forward slashes (i.e. /).
(Score: 1, Informative) by Anonymous Coward on Saturday May 04 2019, @01:48PM
Also, disable auto update on each extension till the signing problem is sorted out.
(Score: 1, Insightful) by Anonymous Coward on Sunday May 05 2019, @12:38AM (1 child)
Goto Mozilla and set brendaneich.ceo = true
Everything went to shit since the SJW contingent set that preference to false.
(Score: 0) by Anonymous Coward on Sunday May 05 2019, @03:01PM
That may be so, but his new Brave browser is a money grabbing scam.
(Score: 5, Informative) by Bot on Saturday May 04 2019, @06:16AM (10 children)
an ordinary snafu? bigger fish than mozilla have fucked up with stupid things like renewals of certificates and even domains, but mozilla has already boycotted extensions in the past and:
- mozilla gets money from google, the competitor
- mozilla gets money from microsoft, the competitor
- advertisers do not like extensions
- deep state does not like extensions
never attribute to stupidity what is perfectly in line with past behavior and current objectives
Account abandoned.
(Score: 4, Insightful) by takyon on Saturday May 04 2019, @07:08AM
Mozilla is a founder of Let's Encrypt. How embarrassing.
[SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
(Score: 0) by Anonymous Coward on Saturday May 04 2019, @09:34AM (7 children)
"deep state"?
(Score: -1, Troll) by Anonymous Coward on Saturday May 04 2019, @01:20PM (6 children)
Ask Eisenhower or Kennedy. Or travel back in time 5 years ago before Democrats fell in love with the CIA and ask one of them.
(Score: 0) by Anonymous Coward on Saturday May 04 2019, @03:51PM (3 children)
Just give a straight answer? Or an alignment unknown or equal opportunity answer. Something.
(Score: 2, Informative) by Anonymous Coward on Saturday May 04 2019, @04:10PM (2 children)
The intricate web of corporate lobbyists, military interests, bankers, intelligence agencies and other non-elected (sometimes non-government) entities that control the country outside of the political process you learn in middle school civics class. Oil companies, tech giants, the CIA, Goldman Sachs, all are entities which could be labeled part of the "deep state".
It's inaccurate to describe it as a singular entity with a unified ideology and motive; it's more inaccurate to pretend that it doesn't exist.
Is that straight enough for you?
(Score: 1, Informative) by Anonymous Coward on Saturday May 04 2019, @04:16PM
https://en.m.wikipedia.org/wiki/Senior_Executive_Service_(United_States) [wikipedia.org]
(Score: 0, Flamebait) by Anonymous Coward on Saturday May 04 2019, @09:08PM
Of course the establishment ass-suckers here would mod this 'troll'. This might as well be the green site.
(Score: 2) by srobert on Saturday May 04 2019, @09:38PM (1 child)
LOL. The CIA lied us into war and got a few hundred thousand people killed.
But now that they parrot the Russiagate narrative, we trust them.
(Score: 0) by Anonymous Coward on Wednesday May 08 2019, @04:01PM
Because an intelligence agency needs to 1) always lie 2) always tell the truth. That will really keep the opponents guessing...
(Score: 0) by Anonymous Coward on Sunday May 05 2019, @12:47AM
There should many people shown the door and shamed by all for this stupid thing.
First should not have 1 cert but 2. That are 365 days out of sync (assuming 2 yr life).
If one does not validate - use the other. Then if the failure - idiot who cannot use a calendar, or file corruption, or ... to continue and allow for the processing to continue.
second use a FUCKING calendar!! It on your smart phone. The techs over this cert, probably treat their dentist better.
.
(Score: 0) by Anonymous Coward on Saturday May 04 2019, @09:00AM (2 children)
Laptop version of firefox extentions still there and working fine? Desktop machine? The Horror, the Horror. Going to Midori.
(Score: 0) by Anonymous Coward on Sunday May 05 2019, @03:20AM (1 child)
Update. Now laptop hosed, desktop box restored. And spent a lovely afternoon reading harsh words over at the Pale Moon forums, where they do not like NoScript, because it is too hard for users.
(Score: 2) by Reziac on Wednesday May 08 2019, @05:17AM
Try NewMoon instead... NoScript still plays nice with it (and with the old version of PrefBar). Compiled for XP, but apparently has more backward-compatibility in general.
http://rtfreesoft.blogspot.com/2019/05/weekly-browser-binaries-20190504.html [blogspot.com]
I use this one,
NM28XP build:
Win32 https://o.rths.cf/palemoon/palemoon-28.6.0a1.win32-git-20190504-d9d9d1ed8-xpmod.7z [o.rths.cf]
Win64 https://o.rths.cf/palemoon/palemoon-28.6.0a1.win64-git-20190504-d9d9d1ed8-xpmod.7z [o.rths.cf]
but this is also available:
New New Moon 27 Build!
32bit https://o.rths.cf/palemoon/palemoon-27.9.6.win32-git-20190504-249ad075c-xpmod.7z [o.rths.cf]
32bit SSE https://o.rths.cf/palemoon/palemoon-27.9.6.win32-git-20190504-249ad075c-xpmod-sse.7z [o.rths.cf]
32bit noSSE https://o.rths.cf/palemoon/palemoon-27.9.6.win32-git-20190504-249ad075c-xpmod-ia32.7z [o.rths.cf]
64bit https://o.rths.cf/palemoon/palemoon-27.9.6.win64-git-20190504-249ad075c-xpmod.7z [o.rths.cf]
Just unzip and run. NM picked up my existing Palemoon add-ons with no effort on my part, tho I don't know if that's typical.
And there is no Alkibiades to come back and save us from ourselves.
(Score: 2) by Runaway1956 on Saturday May 04 2019, @01:47PM (2 children)
My browsers have been up for weeks - I don't close them. Only time they go down, is when we lose power. I'll take this as a warning not to close any browsers until the certs are fixed.
(Score: 3, Informative) by Anonymous Coward on Saturday May 04 2019, @02:19PM
It will hit you when the browser decides to check, which is every 24 hours from whatever random start time you have.
(Score: 1, Informative) by Anonymous Coward on Sunday May 05 2019, @12:14AM
That won't do you any good. My Firefox has been running for a few weeks and just got hit by this drive-by extension-disabling. Bastards.
(Score: 2, Funny) by Anonymous Coward on Saturday May 04 2019, @02:06PM (2 children)
No LGBTQ2S were harmed by this incident.
(Score: 4, Insightful) by Anonymous Coward on Saturday May 04 2019, @06:14PM (1 child)
This is why they fucked up the certs. They were too busy trying to recruit underrepresented losers and arguing about who loved trannies more.
(Score: 1, Funny) by Anonymous Coward on Saturday May 04 2019, @10:27PM
"Hey, let's go buy some pink hair dye instead of working on this boring cert stuff"
(Score: 5, Informative) by Anonymous Coward on Saturday May 04 2019, @02:22PM (11 children)
Apparently as part of the bugfix, mozilla just "revealed" (it wasn't secret but not well publicized or known) they have a way to mess with your preferences without your knowledge. Basically a backdoor update mechanism.
https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout [mozilla.org]
(Score: 3, Insightful) by Azuma Hazuki on Saturday May 04 2019, @03:13PM (5 children)
O~kay, and that right there is getting me off of Firefox. Just need to find a way to bend Midori or Falkon to my will.
I am "that girl" your mother warned you about...
(Score: 4, Informative) by Anonymous Coward on Saturday May 04 2019, @03:54PM (1 child)
Go Palemoon?
Also, that bending to will stuff, can we watch?
(Score: 3, Interesting) by Azuma Hazuki on Sunday May 05 2019, @04:45AM
Falkon seems to be the best choice for me right now. Midori is...well, Midori, it's got Mad GNOME Disease or something. Pale Moon crashed a few times and the non-bootstrapping adblocker takes ages to load. Surf's adblocker isn't up to task, though I really like it otherwise. NetSurf is not really usable. It also has a surprisingly competent if small set of extensions available. Seems I'll be using more and more Qt apps as time goes on...
I am "that girl" your mother warned you about...
(Score: 5, Interesting) by jmorris on Saturday May 04 2019, @06:49PM (1 child)
My normie existence has been split between Seamonkey and Firefox for some time now. Thinking it is time to push the migration schedule up to "ASAP." Dissident life has been on and off with Brave for some time, it keeps getting better and more suitable as a default browser. Chrome is of course a non-starter.
Bottom line. We should have seen the writing on the wall with Moz Corp a few years ago, a lifetime of familiarity and goodwill with Netscape and Mozilla's children have lead us to greatly underestimate the danger they have become. The only reason Google is more dangerous is the nearly Trillion dollar market cap they can wield, which has allowed them to push Moz out of the way. But we need to wake up, lose the attachment to what they once were and let them die; for if they ever did write a new chapter in the Book of Mozilla, if they had another rebirth it would be in an awful and terrible form. The lizard has gone bad, we have to put it down, out of love for what it once stood for. Let not future generations remember Mozilla only for its modern sad and evil form in its twilight, let it be remembered as a powerful symbol of the first great age of the Internet. Let him be remembered for standing proudly and roaring his mighty challenge at Mammon.
(Score: 2) by urza9814 on Monday May 06 2019, @02:21PM
The problem with Brave is that it cements Google's control over the web -- it's just one more Chrome clone.
(Score: 2) by darkfeline on Sunday May 05 2019, @05:59AM
If the ability to change preferences is what stops you from trusting a piece of software, I hate to break this to you, but the devs can basically make their software do whatever they want, irrespective of whether they have a formal way of changing preferences. It's a pretty arbitrary and meaningless fact to base your trust or distrust on.
A feature like this has legitimate applications, for example if a serious bug is discovered in a setting, but a minority of users may still need/want that setting, then it makes sense to reset that setting and let users who want it set it back manually after reading the warning about the bug.
Join the SDF Public Access UNIX System today!
(Score: 0) by Anonymous Coward on Saturday May 04 2019, @09:32PM
I literally just removed Firefox and Thunderbird from my PC after reading the page you linked. Fuck you, Mozilla cunts.
(Score: 2) by darkfeline on Sunday May 05 2019, @06:04AM (3 children)
Apparently A/B testing is considered a backdoor now.
Join the SDF Public Access UNIX System today!
(Score: 0) by Anonymous Coward on Monday May 06 2019, @02:48AM
When I choose a setting, I expect it to remain set, even through an update. It definitely shouldn't be silently changing in the background.
(Score: 2) by urza9814 on Monday May 06 2019, @02:25PM (1 child)
The problem isn't that they're doing A/B testing, the problem is that they're remotely modifying software on other peoples' computers in order to do it.
(Score: 2) by Reziac on Wednesday May 08 2019, @05:22AM
And how is this not the biggest security hole in the history of browsers??
Hey Igor, let's change everyone's Mozilla to allow drive-by installs of our malware...
And there is no Alkibiades to come back and save us from ourselves.
(Score: 2) by corey on Saturday May 04 2019, @11:27PM (1 child)
Interestingly, my wife's Android phone Firefox had this issue automatically, she told me last night.
But my phone has Fennec F-Droid which is a modified Firefox for privacy and I haven't had this issue. I'd recommend it.
https://f-droid.org/en/packages/org.mozilla.fennec_fdroid/ [f-droid.org]
(Score: 2) by corey on Monday May 06 2019, @02:31AM
Update: it happened, just took longer.
Fixed with the xpinstall.signature toggle.
(Score: 2) by srobert on Sunday May 05 2019, @12:15AM (1 child)
Lot's of googling around today trying to get extensions back.
On my build of firefox the "studies" option can't be enabled. And I don't think I want to enable it anyway.
Installing the fix from this link seemed to restore everything. (without changing any about:config settings).
I hope that's safe.
https://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate%40mozilla.com-1.0.2-signed.xpi [googleapis.com]
I hope that's safe.
(Score: 2) by hendrikboom on Sunday May 05 2019, @12:18PM
Don't know whether it's safe, but firefox blocked it. Said it tried to install spftware on my system.
(Score: 2) by lentilla on Sunday May 05 2019, @02:07PM (4 children)
Here is the official fix. [mozilla.org] "The fix will be automatically applied in the background within the next few hours. No active steps need to be taken to make add-ons work again."
I am truly disappointed that Mozilla did not post that information on their front page. I went back to their front page (mozilla.org) and counted four clicks until the information for which I was searching was displayed. (It took me more than four clicks initially but I went back and counted them knowing where I wanted to end up. I am also aware that not everyone finds using computers particularly easy.) If we assume Firefox is used by 10% of web users, and we assume that maybe one billion people use the web on a daily basis... that means this issue has just affected one hundred million people. Do you not think that Mozilla might have woken their technical team, their legal team and their webmaster... and pasted something appropriate on their front page?
(Score: 0) by Anonymous Coward on Sunday May 05 2019, @03:04PM (1 child)
Perhabs this is just an april fools joke gone too far? (Devuan)
(Score: 0) by Anonymous Coward on Tuesday May 07 2019, @12:40AM
You certainly get that feeling because it's not on the front page of Mozilla. Talk about adding insult to injury...
(Score: 0) by Anonymous Coward on Sunday May 05 2019, @08:37PM
Running one of the esr (extended release) versions, I didn't want to wait around for Moz to get their act together.
This...
> Workaround: Go to about:config and set xpinstall.signatures.required to false
...came from The Fine Article and it worked for me.
All I wanted was to keep EFF's Privacy Badger working and that did it.
Q for anyone: is there any reason to set "xpinstall.signatures.required" back to true? I'm pretty happy to trust EFF on general principles.
(Score: 2) by corey on Monday May 06 2019, @02:49AM
The fix that worked well for me was as detailed here:
https://fossbytes.com/firefox-extensions-disabled-by-glitch-how-to-enable-firefox-extension-again/ [fossbytes.com]
Stuffed if I'm ever going to enable Studies. I'd actually found this fix on Reddit but since then the subreddit and my comment have disappeared...
I stumbled across this just before too:
https://www.reddit.com/r/firefox/comments/bkxa1h/on_the_privacy_implications_of_using_studies_to/ [reddit.com]
(Score: 2) by Luke on Monday May 06 2019, @04:37AM
The fix for me was to go looking at why the extensions weren't working and discover just how unsafe (for me) FF actually was.
I thought this was a browser that wasn't captured by outside interests, and which I had some control over, but i turns out I couldn't even say 'no' to updates any longer >:-(
On the positive side I discovered some derivatives of FF that I wasn't aware of, including Waterfox - which I'm using to write this short missive (I knew about Pale Moon but for various reasons hadn't gone down that path).
Although I've yet to fully grasp the differences it seems these alternatives may allow me to avoid being part of the chromium monoculture, yet bring back some of the things I used to like about FF - mostly around the ability to control what it does or doesn't do, including preventing it being profligate with my information, and not being beholden to 3rd parties in order for it to operate.
Thus this issue has educated me, let's hope it educates Mozilla too 'cos there's a lot to thank them for all the same.
I'm off now to look at how much in common there is with T-bird, something else I use and may need to be concerned about...
(Score: 0) by Anonymous Coward on Friday May 10 2019, @09:55PM
All the security engineers were away for two weeks of mandatory inclusiveness training, since one had violated the CoC by using the pronoun "he" instead of "ze" in a comment.