SoylentNews is people
SoylentNews
So, quick update here. The site was down for most of the night because the database cluster shot itself in the head. I had restarted a machine to install updates, and this caused the backend cluster to entire to entirely loose its mind. Unfortunately, I didn't have a manual dump of the database made, just a VM snapshot, since, well, I wasn't tinkering with it directly. I've mostly been trying to patch things to the point that I can sleep, and leaving things down like IRC and email which need to be seriously overhauled before they can go back up.
As far as damages go, it looks like we lost 10 or so days of messages, which uh, sucks for multiple reasons. We're currently on ##soylentnews on Libera.Chat while I pull bits of the site out of the flames, but I'm at the point that if I don't sleep, I will make things worse. Corruption in the production database is very much not what I wanted, and we're very much in limp mode for the moment. I'm going to let staff handle IRC and comments while I sleep, and then I'll post another update when I'm awake.
See you in a few hours
~ NCommander
(Score: 0) by Anonymous Coward on Saturday November 12 2022, @09:28AM (20 children)
Coincidentally I posted in my journal recently.
Similar thing happened some years(?) ago when I posted in my journal. Data loss included what I posted in my journal.
¯\_(ツ)_/¯
(Score: 1, Insightful) by Anonymous Coward on Saturday November 12 2022, @09:32AM (7 children)
I'm going with coincidence. I don't think there's any sort of conspiracy. Remember yesterday's meta? This site is held together with chewing gum and string.
(Score: 5, Insightful) by Anonymous Coward on Saturday November 12 2022, @10:07AM (5 children)
(Score: 2) by RS3 on Saturday November 12 2022, @05:59PM (4 children)
Hmmm. Blanket? As in, never ever anywhere?
I'm the only IT / admin for a smallish hosting / server operation. I took over for departed staff about 15 years ago. A bunch of the sites are WordPress, and I've never had any problems. I've updated, upgraded, migrated, no problems, ever, with mysql, or php or WordPress or plugins.
My only worry about mysql is Oracle, so I may switch to mariadb soon, but for now everything has been great.
What problems have you seen or had with mysql or derivatives?
(Score: 0) by Anonymous Coward on Sunday November 13 2022, @06:41AM (3 children)
Is your brain running on MySQL too? Has it lost info on the recent/regular noticeable data loss on SoylentNews?
If you store stuff in a real DB it won't get lost like that so easily.
There's no good reason to use MySQL other than "legacy" aka some fools made the wrong choice years ago and you're stuck with it.
(Score: 3, Funny) by janrinok on Sunday November 13 2022, @09:25AM (2 children)
What a load of rubbish. Have you bothered to find out why we lost data?
(Score: 0) by Anonymous Coward on Sunday November 13 2022, @10:44AM
(Score: 0) by Anonymous Coward on Sunday November 13 2022, @10:48AM
So what DB software is that DB cluster running?
(Score: 4, Funny) by janrinok on Saturday November 12 2022, @12:09PM
(Score: 5, Informative) by janrinok on Saturday November 12 2022, @10:17AM (9 children)
We have also lost 493 'user' accounts, over 50 current submissions, and all of the work that had been prepared for the front page to cover the weekend, etc
Yep, we might have also lost your journal.
Of the accounts that were lost, only a handful of genuine accounts were among them (no more than 5). There were also 5 sock puppet accounts but fortunately somebody's spam bot was filling up the vast majority of the remainder (475+). If you think that your account was among those lost please email your nickname to [my_user_name][at]protonmail.com and I'll see what I can do to recover it. Sock puppets need not bother applying.
(Score: 1) by Runaway1956 on Saturday November 12 2022, @10:59AM (7 children)
Journal entries are missing. McGrew posted a journal entry just a couple days ago. DannyB and I both commented in that journal, and I just looked to see what DannyB may have replied. It's gone.
I don't know how may others may have been affected.
(Score: 2) by turgid on Saturday November 12 2022, @11:23AM (6 children)
I've lost some journal entries too, but I can live without them. I am sorry to have lost the comment I posted on yesterday's Meta, though.
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 2) by janrinok on Saturday November 12 2022, @11:55AM (5 children)
https://web.archive.org/web/20221111011540/https://soylentnews.org/meta/article.pl?sid=22%2F11%2F10%2F1428220 [archive.org]
Enjoy! Thanks to the AC who provided the link.
(Score: 3, Insightful) by turgid on Saturday November 12 2022, @12:10PM
That is both impressive and very troubling.
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 2) by Unixnut on Saturday November 12 2022, @03:53PM (3 children)
oooh, I got responses to my post! Alas the site had the hiccup while I was asleep, so I didn't get to see any of that, let alone respond.
Thanks for the link! Shame can't respond on the thread though.
(Score: 2) by janrinok on Saturday November 12 2022, @03:59PM (2 children)
(Score: 3, Interesting) by maxwell demon on Sunday November 13 2022, @10:03AM (1 child)
But couldn't a new comment eventually get that same number? After all, the system doesn't know about that old comment any more.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by janrinok on Monday November 14 2022, @06:00AM
Yep - but the comment numbers printed on the Wayback machine copy are not going to change. The reference would be only be valid for the comment numbers shown on that unchanging page.
(Score: 1) by Runaway1956 on Saturday November 12 2022, @11:04AM
Also, unable to connect to IRC here. "Disconnected (Remote host closed socket)."
(Score: 3, Funny) by maxwell demon on Saturday November 12 2022, @10:49AM
So actually you are the one who caused the problem, by posting in your journal? :-)
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by mcgrew on Sunday November 13 2022, @08:45PM
Not the first time, my last journal I posted Friday was lost, too, but it's not really lost; it's at mcgrew.info. I had unread messages that were lost, too, but it's no big deal either, as they were all about a journal that wasn't lost.
My guess is only newer stuff was lost, and if they lost your stuff and it wasn't backed up, that's your own fault. I'm thankful that they posted this. Shit happens.
mcgrewbooks.com mcgrew.info nooze.org
(Score: 5, Insightful) by maxwell demon on Saturday November 12 2022, @10:22AM
Thank you for the work to get the site back up.
While it certainly sucks having lost those days of comments, it would have sucked much more to lose the entire site.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 1) by Runaway1956 on Saturday November 12 2022, @11:13AM
Closing Link: gateway/vpn/pia/account (SASL authentication to a NickServ account with a verified email address is required to connect from your current network. Please see https://libera.chat/guides/sasl [libera.chat] for configuration assistance.)>/quote>
(Score: 3, Informative) by Anonymous Coward on Saturday November 12 2022, @11:44AM (2 children)
FWIW it looks like archive.org's Wayback Machine took a snapshot of at least some of the site on the 11th - http://web.archive.org/web/20221111112621/https://soylentnews.org/ [archive.org]
(Score: 2) by janrinok on Saturday November 12 2022, @11:50AM (1 child)
(Score: 3, Funny) by kazzie on Sunday November 13 2022, @05:10AM
Cue comments about *real programmers* just uploading their code through FTP, and letting everyone else make backups. ;)
(Score: 4, Interesting) by RamiK on Saturday November 12 2022, @11:46AM (19 children)
Considering my non-existent level of commitment I should probably stay quiet, but have you considered running a Lemmy [join-lemmy.org] server like Beehaw [beehaw.org]?
I mean, originally I assumed you'll just update a few packages and tweak some function call and related logic where library APIs changed so I wasn't planning on saying anything... But if things are this fragile and it's really all you alone holding it together with duct tape and good intentions and no one else will be able and/or willing to keep things going in the future... Well...
compiling...
(Score: 4, Interesting) by janrinok on Saturday November 12 2022, @11:52AM (14 children)
The plan - if there is such a thing - is to rebuild everything with up-to-date software. The technical debt has been unmanageable for several years and we either update or close down.
(Score: 4, Interesting) by JoeMerchant on Saturday November 12 2022, @02:25PM (13 children)
So, disclaimer: this is armchair spit-balling, but....
Over my past 31 years of professional (well, paid at least) software development, the thing I have consistently hated throughout the decades has been "the treadmill" that constant need to update your own stuff to remain compatible with "their stuff". I do my best to choose "their stuff" with care to avoid short lifespan APIs and interfaces, but, being professionally paid for functionality today and tomorrow (but not caring about 5-10 years hence) I don't do too much gratuitous reinvention of the wheels.
Here, in a "mature" platform with well established bounds of functionality, I wonder how close to "bare metal" it might be practical to go? You need the multi user https interface, database, and code between to implement the page renders... Am I missing anything big?
Would the treadmill slow down if all you had to keep current with the world was the https interface?
Can you get bare metal hosting that avoids the constant updating of some underlying OS?
As for backups, if the database can mirror itself off-site while simultaneously serving the front end, and if the front end can hook up to any mirror site of choice, that would seem at least as robust as whatever is happening now...
I know: donations of time, money and code gratefully accepted. If I didn't need to work for food and shelter I seriously doubt I would have been doing it for the past 31+ years...
🌻🌻 [google.com]
(Score: 3, Interesting) by janrinok on Saturday November 12 2022, @02:43PM (11 children)
The idea is to containerise everything (docker/kubernetes) which makes it easier to move and separated from the hardware supporting it. One of the problems is that our software is so old it is not only unsupported and insecure, but it is also difficult to maintain. We have to update or, eventually, close down.
That doesn't mean that we will not have to maintain it, but it does mean that anyone can have the docker image to work with and change locally. Then they submit their proposed changes once they have tested their changes. As it stands it is very difficult to test changes unless it is on one of our servers which have been specially configured to work with out-of-date software. Quite a few people have tried to create their own local instance of the existing system and failed.
However, the technical support is not my forte so perhaps I may be misstating things.
(Score: 4, Informative) by JoeMerchant on Saturday November 12 2022, @05:18PM (7 children)
Containers are good, in theory, and in practice too if you have the support man hours to keep up with changes in the container ecosystem, which at least seems to usually move more slowly than the typical server OS distribution.
As for setting up bare metal OS to support "a thing" that's more or less been my primary paid responsibility since 2014. I build setup scripts that layer over Ubuntu, the basic schtick is: install the base OS, apt install git, clone the source repo and run the scripts which purge the unwanted / unneeded / unloved default packages, then install the custom requirements however that needs doing, preferably with apt install but also can get custom .deb packages and/or build from source, either cloned from others' repo or our own, copied or in house developed. For our systems the deployed image includes developer tools, so there's no headache maintaining required for maintaining a differently configured dev system, though about 70% of the developers on the project still choose to.
We're the SN container under my care, I would want a similar set of setup scripts that can setup the container from a well known base OS, rebuilding it from the ground up occasionally to ensure that the magic sauce mostly resides in the scripts in the source code repo and not some thing some sysadmin did to the master container image late one night and then forgot what they did.
Tangentially, I would look at this: https://www.google.com/amp/s/www.hackster.io/news/davide-eynard-s-picogopher-puts-a-90s-network-protocol-on-a-raspberry-pi-pico-w-and-in-a-backpack-dd3cc41995a6.amp [google.com]
Not so much as something to emulate too closely, but as inspiration of a direction to strive for...
🌻🌻 [google.com]
(Score: 2) by RS3 on Saturday November 12 2022, @06:24PM (6 children)
Wow, that's all very interesting, thanks for that. From your other postings, I had no idea this was your main gig.
It's a very interesting approach. Are you using any "automation" (puppet, chef, ansible, etc.)?
What hypervisor are you using?
(Score: 2) by JoeMerchant on Saturday November 12 2022, @06:57PM (2 children)
At one point we were using a bare metal hypervisor from some company in Switzerland, it preferred to work with CentOS so we were using Xfce over Cent over that hypervisor with a couple of cores devoted to running the GUI in Windows, trading control of the display back and forth between development and production modes...
Then we did a tech eval and decided that Virtual Box was good enough for our purposes, so the GUI moved in there and the host OS moved to Ubuntu.
The whole system communicates internally through a RabbitMQ server (Intel core based apps using AMQP, ST micro components using MQTT) and eventually that Rabbit MQ server found it's way into a docker container running on the generic dockerd community edition.
In other words it's a complicated mess, but I think a significant improvement over the mess it replaced.
🌻🌻 [google.com]
(Score: 2) by RS3 on Saturday November 12 2022, @07:54PM (1 child)
Ah, it's something special, not generic web / cloud hosting. Pretty cool! I do kind of remember you writing about it sometime long ago...
I'm deploying a new server, after keeping some old ones going perfectly well for years. Older CentOS (6), updates stopped, but I've had no problems. One has uptime of 390 days right now. Ain't broke, not fixing it... Well, new server is big CPU and RAM, so hypervisor is needed. I've run VMWare, Xen, messed with a few others, need to stay away from big $ software licensing / subscription costs. Most likely going with kvm. Trying to steer away from systemd- partly because I hear too much bad, and that dovetails with this IT gig not being full-time, nor any regular hours. The _last_ thing I need is emergencies 15+ miles from where I live or work. IE, I'm spoiled by systems that just run, and there have been time periods when I haven't visited the physical site in more than 1.5 years.
I've tried and like Devuan, and MX is pretty cool, but neither are really server-oriented. Love Alpine, wish for a better package manager. About to evaluate Void. Longtime Slackware user- I just worry that if I depart the situation, I don't want to leave something complicated for whoever takes over next...
(Score: 2) by JoeMerchant on Saturday November 12 2022, @08:52PM
Yeah, our system is single user but we have deployed thousands of systems around the world, different animal from a single server with thousands of users, but it's remarkable how many tools apply to both worlds.
We got a minor burn from our systemd service file behavior the other day, it had Rabbit / Docker as a dependancy for another service we run, never expected killing that service would also kill docker... Now that we know it's not a problem, but it was about 20 hours of developer investigation to deal with that bit of nonsense.
🌻🌻 [google.com]
(Score: 2) by JoeMerchant on Saturday November 12 2022, @07:33PM (2 children)
As for Ansible etc.... Nothing much along those lines yet, mostly just bash scripts for setup and Qt apps in the live system to take care of various system things as needed.
Dockerfiles are an interesting variation on the bash script approach, and we use them to (mildly) customize the Rabbit MQ server container.
The thing I like about bash scripts is that they easily encapsulate stuff you can try on the command line, and minor system image updates can be distributed to the team as a patch script that is usually identical to the modification of the system setup scripts that implements the same thing.
🌻🌻 [google.com]
(Score: 2) by RS3 on Saturday November 12 2022, @08:09PM (1 child)
Yeah, for what you're doing, esp. that you've been developing your scripts all along, it would probably be more effort to set up Ansible, et al. Those are more oriented to large stacks of duplicate servers.
That said, some of the "automation" packages make some things much easier, like samba, apache, mysql, and other configurations. I've occasionally used them on a test server just to get some information from them about their ideas for config files, in case I'm missing something, etc. But I already have a big head start on config files, so kind of like systemd, I'm not sure I want to trust someone else's idea of how my system's config files are written...
Since I don't have a stack of servers, just a few, I've done like you- written several simple scripts to automate simple things like updating WordPress plugins and themes. WordPress has built-in updating available, but it requires giving a site access password to the source for the update.
IIRC, Apache used to run per virtual hosts ownership / permissions. IE, each customer's /home directory is owned by that username, and each Apache process would only have access to those files. But unfortunately Apache runs as apache:apache, so as far as I can tell, giving that access password to one WordPress site would or could give access to all sites. Although it is difficult to get past the virtual root for Apache's files, still, it's a risk.
Some years ago there was a malicious WordPress plugin that used /tmp as a mechanism to do its dirtywork. I didn't study the thing- it was right around when I inherited the sites around 15 years ago. That was a pretty quick fix, including just some raw OS updates.
(Score: 2) by JoeMerchant on Saturday November 12 2022, @09:03PM
Just recently we updated Virtual Box to whatever installs in Ubuntu 22.04 with apt and they went and locked down (well, made less open) configuration of host only network addresses, now you have to create folder /etc/vbox and put a network.conf file in there to specify something other than their 192.168.56.1/20 default range...
It's always something.
For SN I would really try to avoid all the "cool" stuff that opens so many security holes like you describe. If you don't have fancy stuff, then you don't need to provide access for updating of the fancy stuff.
🌻🌻 [google.com]
(Score: 2) by RS3 on Saturday November 12 2022, @06:37PM (2 children)
Not knowing the details, I can't speak authoritatively, but moving an OS installation into a container requires some virtualization considerations, especially hardware, like Ethernet, USB (keyboard), graphics (for local terminals), etc. Theoretically a hypervisor or container should be able to emulate any hardware and run any OS, but of course, that never happens, so it's best to use ones that work well together.
It's possible that a docker image would work well on someone's local machine, but have problems when uploaded to the actual host.
But I doubt anything SN would be problematic, as it's pretty standard stuff (perl, mysql, nginx (or apache?)...
(Score: 2) by janrinok on Saturday November 12 2022, @06:46PM (1 child)
Understood
.
For me it is more that I can change the software locally, test it, and then put in a pull request to have my changes built into the next docker release. At present, having a full system configured with the same out-of-date software that our current servers use has proven impossible to achieve. And if I had have achieved it - it would be insecure and receiving no security updates.
We have been unable to do this for a long time now.
(Score: 2) by RS3 on Saturday November 12 2022, @08:14PM
Makes perfect sense.
At this moment I'm not sure how you could have a development server that's worked on by many developers / admins, without the file contention problem... I suppose you could run a git host on it, and then people could check out and own certain things. Just thinking out loud... setting up and adminning the git might be no fun... Your aforementioned more brute-force method is probably much better, just that someone has to play "traffic cop" and make sure to merge the updates, then when two or more people overlap efforts, communicate (irc?) and figure out how to merge. Thanks!! Major progress is being made!
(Score: 2, Insightful) by Anonymous Coward on Sunday November 13 2022, @07:51AM
Yeah, it seems as insane as building and maintaining factories on foundations that change every 18 months. But many even seem enthusiastic about it.
In many/most cases compatibility is broken often enough that it isn't a simple move.
I guess it's better job security but still crazy, disgusting and wasteful to me.
(Score: 2) by janrinok on Saturday November 12 2022, @12:03PM (3 children)
Thanks for the suggestion. There are lots of other similar systems. Many of our community do NOT want to install such programs on their phones or stand-along computers. NCommander already has his own Discord channel. What we finally end up with in the future is still not even at the discussion stage. We may end up going that way, but personally I for one hope not.
(Score: 4, Interesting) by RamiK on Saturday November 12 2022, @12:52PM (2 children)
Then I should have said something sooner.
Yes there are. Any specific feature missing in one of the other dozen systems that makes you want to write a new one?
It's just a website as far as the users are concerned. That is, you signup with an email ( https://beehaw.org/signup [beehaw.org] ), login with that email ( https://beehaw.org/login [beehaw.org] ) and then mod up/down, post comments and micro-blog. The federated and distributed self-hosting is equivalent to running a git server where it basically there to let you move your data and identity elsewhere as a design feature that's built into the protocols. e.g. if soylent goes belly-up / taken over by group you don't like, you'll be able to take your posts and comments with you when switching to a clone or even hosting one yourself similar to how projects switch from github to whatever.
compiling...
(Score: 3, Informative) by janrinok on Saturday November 12 2022, @01:08PM (1 child)
I wasn't suggesting that we rewrite that app from scratch, just that we will reconstruct what we have but with today's software so that we are not using repos, kernels and programs that are years out-of-date and no longer supported with security updates. Then we can look at how to move forward from that position.
We may well follow your suggestion - but we a not even close to that at the moment.
(Score: 3, Interesting) by RamiK on Saturday November 12 2022, @03:16PM
Yeah it sounds more like what I originally thought where you got stuck with old libraries and distro bases since there wasn't anyone comfortable enough to start rewriting logic to use the new APIs.
Got it. Just don't let Lemmy being a small project ( https://join-lemmy.org/instances [join-lemmy.org] ) fool you: It's low resources (written in Rust and uses PostgreSQL) and separates the front and back ends so there's nothing stopping you from sculpting a soylent-news like web facing frontend UI around it like how LemmyBB does for phpBB: https://join-lemmy.org/news/2022-11-02-_First_release_of_LemmyBB [join-lemmy.org]
But yeah, probably too soon to talk about this right now.
compiling...
(Score: 5, Insightful) by acid andy on Saturday November 12 2022, @12:06PM
NCommander, I'm not posting here so often at the moment, think it's (mild) mental health stuff, but it's wonderful to have you back here and I don't need to tell you that SoylentNews and this community really needs you right now. So I'll just welcome you back and echo the big thanks for your awesome work on the site and your support.
If a cat has kittens, does a rat have rittens, a bat bittens and a mat mittens?
(Score: -1, Troll) by HammeredGlass on Saturday November 12 2022, @01:46PM (4 children)
is that dumbass mastodon post made by some retarded tumblrina is gone, although it will probably be reposted as they seemed so proud of their virtue signaling against the bird app *massive applause*
(Score: 0) by Anonymous Coward on Saturday November 12 2022, @03:32PM (3 children)
https://www.theatlantic.com/technology/archive/2022/11/twitter-facebook-social-media-decline/672074/ [theatlantic.com]
have a nice day
(Score: -1, Flamebait) by HammeredGlass on Saturday November 12 2022, @06:39PM (2 children)
okay, uncle ted
i look forward to watching you try to put the genie back in the bottle
best we can do is use the genie and not allow it to interpret our wishes as it sees fit
(Score: 0) by Anonymous Coward on Saturday November 12 2022, @07:51PM (1 child)
"best we can do is use the genie and not allow it to interpret our wishes as it sees fit"
Just walk away not use the genie, and let Social (Broadcast) Media Die
(Score: 1) by HammeredGlass on Saturday November 12 2022, @10:38PM
you're a "the glass is always full" type of person, ay?
(Score: 3, Insightful) by Gaaark on Saturday November 12 2022, @02:14PM
Good to see someone with skills is back (wish i had some to help :{ ).
Good to see discussions happening again instead of just down-mods for idiots (there's a band name!)
Good to see all the people who give a sh*t about this site.
Welcome back, NCommander.
--- Please remind me if I haven't been civil to you: I'm channeling MDC. ---Gaaark 2.0 ---
(Score: 2, Insightful) by pTamok on Saturday November 12 2022, @04:03PM
Welcome back NCommander. I see you are proving that no good deed goes unpunished [wikipedia.org].
I hope this is but a minor speed bump in the rescue project.
(Score: 2) by PiMuNu on Saturday November 12 2022, @06:06PM
nomsg
(Score: 4, Interesting) by SomeGuy on Saturday November 12 2022, @07:25PM (1 child)
First order of business after returning: Break everything! :)
Data loss happens, but why weren't there more recent backups?
(Score: 3, Interesting) by coolgopher on Sunday November 13 2022, @12:26AM
Backups (correction - working backups) are gold. On my workstation at work, I'm running a zfs mirror with daily snapshots kept for a week. Snapshots are great, though can be dangerous with databases if they don't implement a write-ahead log. Has MySQL got that these days? I haven't looked in a very long time. Anyway, zfs. Snapshots. Good. And easy to set up.