SoylentNews is people
SoylentNews
Hey Soylent community,
I’m thrilled to announce that we’ve successfully migrated our DNS to PDNS! This marks the final step in our infrastructure overhaul. Today, I pushed the big red button and turned off all the old systems. We are now operating 100% on the new infrastructure.
This transition brings us into uncharted territory, but so far, everything seems to be working perfectly. The new setup promises enhanced performance and reliability, and I’m optimistic about the improvements it will bring.
Thank you all for your patience and support throughout this process. Your feedback has been invaluable. If you have any questions or notice anything unusual, please let me know in the comments or on IRC.
Here’s to a smoother, faster, and more reliable SoylentNews!
(Score: 1, Informative) by Anonymous Coward on Saturday November 09, @09:54AM (14 children)
Your DNSSEC setup is giving me warnings. You definitely messed something up there. The RRSIG doesn't validate properly because it doesn't match your DNSKEY. The only reason I can connect to your site is because my resolver ignores that error because the org TLD has a valid NSEC3 record for soylentnews.org.
Additionally, you need to increase those SOA values, at least on the soylentnews.org 2LD zone. Your DNS server is going to be dealing with incessant traffic with it that short. That may also have something to do with my resolvers' problems to get the data over UDP.
I'd recommend using one of the various free DNS services too. They will cut down on your traffic and increase the reliability compared to rolling your own. Hurricane Electric's service might be worth looking into, especially since your BGP routing info seems to indicate you already use their infrastructure.
(Score: 3, Informative) by kolie on Saturday November 09, @10:10AM (9 children)
Thanks for pointing that out. DNS sec wasn't enabled prior, and it isn't now. There may be records historically that are interfering.
(Score: 0) by Anonymous Coward on Saturday November 09, @10:50AM (8 children)
That may have been your intent, but your DNS servers and the org TLD say otherwise. I really recommend you double check them. In fact, after my resolvers' expire the NSEC3, I'll lose access to this site completely since the .org TLD has now picked up an invalid DS signature to the wrong tag and dropped the NSEC3. Your keys on the two different NS*.soylentnews.org servers don't perfectly match and they don't match the TLD's records. Something between you, your registrar (Gandi it appears), and the .org infrastructure is not good. It appears that one of the name servers advertised an invalid CDS/CDNSKEY record somewhere in the changeover. Like I said, any software that hard fails on DNSSEC failures will lose access once they propagate the removal of the NSEC3. I'd suggest disabling DNSSEC at Gandi until you get your two name servers to agree on whether they should try to do DNSSEC or not.
(Score: 2) by janrinok on Saturday November 09, @10:57AM (7 children)
Are you saying that all 3 records are different from each other, our 2 servers and Gandi?
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 0) by Anonymous Coward on Saturday November 09, @12:09PM (6 children)
Looks like you got it sorted. There is a new NSEC3 now. I’ll just show you from a more reputable source than an AC: https://dnsviz.net/d/soylentnews.org/dnssec/ [dnsviz.net] You can page to the previous analysis at the top to see how it changed over the past few hours.
(Score: 2) by janrinok on Saturday November 09, @12:51PM (5 children)
We didn't doubt you, but thanks for the information. I think it was just taking a little time to propagate through the system. I don't think kolie changed anything and hopefully he is fast asleep by now. Thank you for your help.
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 2) by janrinok on Saturday November 09, @01:02PM (4 children)
I stand corrected - he has been working on it and has lost another 2 hours of his sleep :)
It seems we inherited the DNS misconfig from the original site...
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 1, Insightful) by Anonymous Coward on Saturday November 09, @11:48PM (3 children)
Yeah, your propagation time is 18 minutes worse case, so I figured it wasn't that. Glad you got it figured out. I tried to buy you guys a temporary solution by recommending disabling DNSSEC at your registrar. But it looks like you went above and beyond and got DNSSEC working properly instead. I'd be curious what exactly went wrong in the transition. I have my guesses considering that DNSSEC was turned on in one place but not another. The difficulty of modern DNS is often underestimated, but at least it automates well once you get it properly set up. But I would recommend looking into a more long-term solution to hosing DNS and fixing the various minor issues you have left (mostly the DANE rollover and the UDP listening).
Good job on the transfer. You all had very few problems from what I could see all things considered. Just a bit more cleanup of the historic baggage and you will be all set. A great example of what good planning can accomplish.
(Score: 3, Informative) by kolie on Sunday November 10, @12:45AM (2 children)
UDP listening works fine, the issue is dsnsviz is exclusively homed on HE v6, and our v6 is routed via cogent currently. Welcome to the world of peering disputes.
What happened was, And you can look back in the history of DNSviz going far back, DNSSEC was ina quasi weird states. We were operating with two NS records pointing to the same server helium, And then running linode as 5 secondaries. There seems to be records in the actual zone file, And while they exist they might have not been marked for publication or being sent out I'm not entirely certain at this point what the pre-position was. I just know when we turned it on, we had a bunch of records there was several DNS key values and the one when I started enabling DNSEC in our side officially was not the one being used to sign keys there was now a third signature other than the one that was previously unused in the old bind install. I enable DNSSEC at the registrar because it wasn't according to anything I had, And then disabled it and then removed all the keys. I manually wiped all DNSSEC anything in the zone completely and cycled PDNS, and just did a from scratch implementation. Total time once I identified the cluster was about 15 minutes, ten to clean up five to fix the domains and roll out new zones.
We have an actual secondary now turned on as well, And I think they got different information initially which amplified the issue. The current master was a secondary for helium. The second secondary referenced the first one I setup, because that would be its long-term configuration.
I come from AD and we have a mantra, but it actually applies everywhere and very APT here
"It is DNS. It is always DNS."
(Score: 0) by Anonymous Coward on Sunday November 10, @04:17AM
Are they still screwing with each other? I remember reading about it in 2011 and thinking it had gone on too long then. Jesus. The longest one we've ever had lasted less than a year.
(Score: 1) by pTamok on Monday November 11, @10:31AM
Except when it is the cabling.
Failover between diverse firewalls didn't. Turns out patch was bad *and nobody noticed*.
That little clip that always gets broken off on Ethernet cables without boots: turns out it's important. This is usually an end-user problem, because when you first put the cable into the socket, it works, but it works its way out of the socket relatively quickly and 'My Internet isn't working!!!".
Line errors that couldn't be tracked down: Bad patch cable (or maybe oxidised connection).
I've also had a bad *moulded* power cable, that, upon investigation, turned out to be incorrectly wired. That could have killed someone, but luckily didn't.
Intermittent connectivity errors are a pain to diagnose.
(Score: 2) by janrinok on Saturday November 09, @10:13AM (1 child)
Thanks for the email. Kolie has just gone to bed (around 02:00 his time) and he has a busy day ahead of him. I'm not sure how quick his response will be.
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 3, Funny) by janrinok on Saturday November 09, @10:16AM
He got my email as I was writing my response to you...
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 0) by Anonymous Coward on Saturday November 09, @10:49AM (1 child)
Haven't seen any warnings here - ISP is Spectrum cable (NE USA), browsing with recent Firefox on Win 7 Pro. If there were warnings, where would they appear?
What I'm seeing looks just like it looked last week (which I believe was the intent?)
Thanks to all who are keeping SN going...and now improving!
(Score: 1, Informative) by Anonymous Coward on Saturday November 09, @10:54AM
Your DNS resolver, probably at your ISP would be the one throwing errors. If they validate DNSSEC signatures, they would refuse to give you the address. Then you'd get an IP address not found error in FireFox. The nitty-gritty details would only show up if you manually checked the records, using something like dig or drill, or if you used one of the various places that check a site's DNSSEC status, such as zonemaster or DNSViz.
(Score: 1) by pTamok on Saturday November 09, @09:48PM (3 children)
Throws hat in the air*.
Well done (apart from the little DNS difficulty)!
*When you see old films of crowds of people throwing hats in the air, how did they get them back? Were people assiduous in putting contact details in their hats, and could trust that people would return them, or were hats disposable (unlikely, I would have thought), or any hat recovered would do (kind of common property), so you stood a chance of getting a better hat then you started with? Am I missing something?
(Score: 0) by Anonymous Coward on Saturday November 09, @10:10PM (2 children)
> *When you see old films ...
It may be safe to assume that the prop department collected the hats after the extras left the set?
Similar, iirc, we rented gowns for high school graduation and I don't think I bothered to try and find my "hat" (mortar board?) after we tossed them in the air for a picture. Either the rental company came and collected them or maybe they were trashed?
(Score: 1) by pTamok on Sunday November 10, @03:16PM
I was thinking of records of public celebrations, which don't look staged with a prop department. But, of course, Internet searches just show footage of graduation hat tosses, which are not the same thing.
(Score: 1, Informative) by Anonymous Coward on Sunday November 10, @11:26PM
I've gone to every commencement ceremony where I work. I asked the people collecting the regalia that very question. What they used to do is collect all the caps and gowns and verify their sizes when returned. Complete sets returned at each ceremony were immediately returned. They would then try to match loose, rented caps with rented gowns returned that night. If that didn't work, they set it aside until the end of the return period and just charged students the deposit. Eventually, the price between buying square hats and renting square hats was basically the same and an increasing number where nonreturnable due to their condition. Now, they don't even give you the option of renting square hats anymore. They still do the matching process for tams, since there is a lager difference in price, less people throw those, and good number just buy the whole regalia outright.