SoylentNews is people
SoylentNews
Happy New Year!
As the final hours of 2021 here's wishing everyone a Happy New Year!
In light of the holiday, I am inviting the editorial staff to post stories on a weekend/holiday schedule. Thank you for all your hard work in 2021. Here's wishing for a better year to come! Enjoy!
We did it! [*]
([*] I think).
Current Status:
Thanks to a VERY generous subscription of nearly $1,000, we reached our fundraising goal for the second half of the year THANK YOU!: $4,132.81 on a goal of $3,500.00 (all amounts are estimates):
mysql> SELECT SUM(payment_net) AS Net, 100.0 * SUM(payment_net) / 3500.00 AS GoalPercent, MAX(ts), MAX(spid), NOW() FROM subscribe_payments WHERE ts > '2021-06-30' ;
+---------+-------------+---------------------+-----------+---------------------+
| Net | GoalPercent | MAX(ts) | MAX(spid) | NOW() |
+---------+-------------+---------------------+-----------+---------------------+
| 4132.81 | 118.0802857 | 2021-12-30 17:36:36 | 1744 | 2021-12-30 23:45:49 |
+---------+-------------+---------------------+-----------+---------------------+
1 row in set (0.00 sec)mysql>
And for those of you interested in the details:
mysql> SELECT spid, ts, payment_gross, payment_net, payment_type FROM subscribe_payments WHERE ts > '2021-12-29 22:06:03' AND payment_gross > 0 ORDER BY ts ;
+------+---------------------+---------------+-------------+--------------+
| spid | ts | payment_gross | payment_net | payment_type |
+------+---------------------+---------------+-------------+--------------+
| 1728 | 2021-12-29 23:16:21 | 20.00 | 18.81 | user |
| 1729 | 2021-12-30 00:15:05 | 100.00 | 96.80 | user |
| 1730 | 2021-12-30 01:08:02 | 20.00 | 19.12 | user |
| 1731 | 2021-12-30 01:13:58 | 30.00 | 28.01 | user |
| 1732 | 2021-12-30 01:45:50 | 50.00 | 48.25 | user |
| 1733 | 2021-12-30 02:35:54 | 40.00 | 38.54 | user |
| 1734 | 2021-12-30 03:12:48 | 20.00 | 18.81 | user |
| 1735 | 2021-12-30 04:24:07 | 924.43 | 897.32 | user |
| 1736 | 2021-12-30 07:05:37 | 20.00 | 18.51 | user |
| 1737 | 2021-12-30 07:50:05 | 20.00 | 18.51 | gift |
| 1738 | 2021-12-30 09:23:14 | 20.00 | 19.12 | gift |
| 1739 | 2021-12-30 12:22:42 | 20.00 | 18.51 | user |
| 1740 | 2021-12-30 12:24:24 | 20.00 | 18.81 | user |
| 1741 | 2021-12-30 13:59:52 | 40.00 | 38.11 | user |
| 1742 | 2021-12-30 17:33:36 | 20.00 | 19.12 | gift |
| 1743 | 2021-12-30 17:35:13 | 20.00 | 19.12 | gift |
| 1744 | 2021-12-30 17:36:36 | 20.00 | 19.12 | gift |
+------+---------------------+---------------+-------------+--------------+
17 rows in set (0.00 sec)mysql>
That's great news! So why the equivocation?
Looking Closer:
Actually, it's more of a stepping back to look at things over the course of the entire year:
mysql> SELECT SUM(payment_gross) AS Gross, SUM(payment_net) AS Net, ts, max(spid) AS SPID FROM subscribe_payments WHERE ts > '2020-12-31' ;
+---------+---------+---------------------+------+
| Gross | Net | ts | SPID |
+---------+---------+---------------------+------+
| 6916.61 | 6611.75 | 2020-12-31 21:47:25 | 1744 |
+---------+---------+---------------------+------+
1 row in set (0.00 sec)mysql>
The fundraising goal for the first half of the year was also $3,500.00. So... (2 x $3,500.00) is $7,000.00 but we have a total of... $6,916.61?
The Crash:
And then I remembered. Early this year we had a server (fluorine) crash. We had backups (yay!), but they were borken (Boo! Hiss!). We lost over a day's worth of activity, including a number of subscriptions. I *was* able to manually reconstruct people's subscriptions (time) based on information displayed on a window I just happened to have open at the time. But that was in a table separate from what is used to generate these numbers. After 3 days' effort, I'd patched things up as well as I could. Thankfully the official numbers (on which income and taxes are calculated) are kept on a completely separate server. Whew! One that I DO NOT have access. I'd concluded that we'll just have to sort things out at the end of the year. And that time has draw nigh.
tl;dr:
We're probably all set for the year, but there is also the matter that (unknown to me) we had previously been running at a deficit for a couple years. So anything additional you can contribute will go to replenish our funding base. (NCommander and Matt_ each put up $5,000.00 of their own money that to get us started.)
(Score: 0) by Anonymous Coward on Saturday January 01 2022, @03:17AM (24 children)
Which is why the rest of the stack was mentioned. Not that it really matters because if you read all their downtime notes, it was almost always the database going sideways.
(Score: 1) by khallow on Saturday January 01 2022, @03:20AM (16 children)
And which kills your argument.
(Score: 0) by Anonymous Coward on Saturday January 01 2022, @04:00AM (15 children)
Except it doesn't, especially when the source of the downtime is the one part you are completely unable to argue is not capable of five-nines. But no, keep on arguing that this spectacular devop of yours made the conscious decision to lower their downtime from the theoretical and practical maximum. To be fair, he made the decisions that caused the problems, but I would hope without the goal of causing problems.
(Score: 1) by khallow on Saturday January 01 2022, @04:22AM (14 children)
So you claim. But you already acknowledge that there's a lot glued to that one part.
(Score: 0) by Anonymous Coward on Saturday January 01 2022, @05:42AM (13 children)
https://soylentnews.org/meta/article.pl?sid=18/02/08/0115251 [soylentnews.org]
https://soylentnews.org/meta/article.pl?sid=19/08/15/1341210 [soylentnews.org]
https://soylentnews.org/meta/article.pl?sid=20/08/09/0550216 [soylentnews.org]
https://soylentnews.org/meta/article.pl?sid=21/05/21/0025201 [soylentnews.org]
All database. Those alone get you well below five nines. There are even more than that, but they are the biggies that remain in the database. And all thanks to the devop's inability to properly op, regardless of how good of a dev he is.
(Score: 2, Insightful) by khallow on Saturday January 01 2022, @12:27PM (11 children)
No. First, it was claimed that TMB didn't "actually understood how the site works". Now, the goalposts have moved to "inability to properly op, regardless of how good of a dev he is" which if you look at the description of the four outage writeups is due more to the absence of people than to any lack of understanding. For example, the second and third both were long as they were due to absence of admins ("several key admins being asleep", "failure to babysit the db node"). And the big outage didn't have TMB present at all (he had left SN before that point). And none of the outage descriptions describe the role of the rest of the SN system or their impact on that database. I'm not convinced these are strictly database issues, particularly the big one.
Without that big outage, BTW, SN would be pretty close and possibly better than 2 nines of reliability. The other three outages appear to be about 4-7 days in total outage duration (99% reliability over a seven year period would be about 25 days of outage or loss).
My point here is that this is a dishonest portrayal of the system, its staff, and our needs on that. There's no reason to expect five nines of anything because it's not staffed to support that level of reliability (and all the staff are volunteer!). And most of the outage duration is due to a single failure which happened after the loss of the very admin, TMB who is being criticized here. Finally, there's no need for that level of reliability for a discussion forum, even if it were a commercial product.
Blustering about how the admin who failed to deliver 5 nine SoylentNews would be blackballed in the industry is just over-the-top nonsense.
(Score: 2, Interesting) by Anonymous Coward on Saturday January 01 2022, @09:53PM (10 children)
Your counter was that he ran it successfully. Too bad he didn't. A proper setup doesn't require admins to be available to fix every problem or "babysit the nodes." That alone is a failure. And the design decisions on top of it exacerbated the problems thanks to not properly handling split brains. My point is that worshipping the ground he walked on because of his very visible efforts to keep the site afloat and his dev ability, doesn't change the fact that he was a shitty blowhard of an op that caused most of the large problems in the first place.
(Score: 1) by khallow on Sunday January 02 2022, @03:45PM (9 children)
Except, of course, when that "proper setup" does require them to be available. My point is that there would be a much higher availability for the system, if admins had higher availability - not a skill issue. A real world attempt at high availability would have the admins ready.
Then you are an idiot. Because nobody was doing that, including TMB. This is far from the first time I've seen a blowhard with some expertise tell some volunteer organization that they're doing it all wrong, without bothering to be helpful. They don't usually bother to spin some fanciful narrative in the process. I guess that means you're extra special.
(Score: 0) by Anonymous Coward on Sunday January 02 2022, @10:17PM (8 children)
I've tried to be helpful, now I don't care to help. If that makes me special because you don't understand server administration, then so be it.
(Score: 1) by khallow on Monday January 03 2022, @06:35PM (7 children)
Pull my other finger.
No, it makes you special because now you're harassing the admins rather than doing something useful for anyone, including yourself. Nobody here cares that you're allegedly another person on the internet with moderately useful technology skills.
(Score: 0) by Anonymous Coward on Tuesday January 04 2022, @12:13AM (6 children)
You seem to care an awful lot.
(Score: 1) by khallow on Tuesday January 04 2022, @01:44AM (5 children)
About something in this thread. It's not those mad tech sk1llz.
(Score: 0) by Anonymous Coward on Tuesday January 04 2022, @02:26AM (4 children)
There you go talking about them again. I'm blushing over here.
(Score: 1, Insightful) by Anonymous Coward on Tuesday January 04 2022, @03:20AM (3 children)
Perhaps it's because khallow cares about the success of this site and doesn't like trolls taking potshots at the admins.
I can't stand khallow's politics. Aside from his views on COVID, we agree on virtually nothing. That said, he seems to be a good guy who genuinely wants this site to succeed. I'm pretty sure he modded up three of my posts despite disagreeing with them, which earned a lot of respect from me. You, on the other hand, seem to only care about posting snarky comments about a service that's being provided to you for free.
Have you even subscribed? Although I'm posting this comment as an AC, I subscribed about ten days ago and gave $20 to the site again. I suspect you're just here to shit on work that people are doing for you for free.
If you think this site is so hideously awful, why are you so obsessed with shitting all over it? Nobody is forcing you to be here. Why do you care so much?
(Score: 0) by Anonymous Coward on Tuesday January 04 2022, @07:10AM
Definitely wasn't khallow. He don't roll that way. In fact, I have never come across a more vindictive down modder. Every time I point out how wrong he is, he throws several downmods, in bad faith. So your admiration is poorly placed, unless you are khallow? Could you be, AC?
(Score: 0) by Anonymous Coward on Wednesday January 05 2022, @02:23AM (1 child)
So you can only be a critic of the website if you have subscribed in the past. Good to know for future reference despite not being applicable now for multiple reasons. I'll add that to pile along with pointing out mistakes by the admins is the same as thinking their website is hideously awful and being obsessed with shitting all over it. Why do you care so much to make the world black and white or to feed what you think is a troll? No one is forcing you to do that either.
(Score: 1) by khallow on Wednesday January 19 2022, @04:19PM
Yet here you are whining and shitting all over that website. Looks like it was a good call to keep you out of SN admin.
(Score: 1) by khallow on Saturday January 01 2022, @12:30PM
(Score: 4, Informative) by janrinok on Saturday January 01 2022, @10:02AM (6 children)
Not entirely accurate. The database going sideways was often as a result of other software which was not playing nice with the database. On at least one occasion it was as a result of running out of disk space during a backup - which clearly should not have happened but is hardly the fault of the database per se. Earlier in the life of the site it was a result of various users trying to corrupt the database intentionally - as time went by we managed to find those gaps and plug them effectively. But this existed in the code that we inherited. There was also a huge amount of meta-moderation code which had to be ripped out because it was a bit of a rats nest and poorly documented - or at least nobody could figure out why it had been written the way that it had. NCommander and PJB took it out because it just didn't function the way it should and was preventing the site from working properly. This is also not the fault of the database but of slash/rehash code but the end result was that the database was being asked to do some pretty weird things and got itself tied in knots.
One of the major efforts over the last 12 months or so has been to simplify the entire structure and reduce the amount of resources that the site needed. The site is in a much better position now but there is still work to be done - and insufficient manpower to do it. The systems guys (both of them!) have worked hard behind the scenes just to keep it all working without interrupting the community's access to it. MySQL is sometimes but rarely a problem in itself - but more of a result of how our configuration was using it.
You will probably have noticed Martyb's 'Thanks to audioguy/mechanicjay' on here or on IRC, both of whom live in timezones only a few hours apart. That means that for a huge part of the day we have no support available unless they pause their jobs, private lives or sleep to help out. And the heartfelt and sincere 'Thank you' is often the only reward they get for their hard work.
Sure, you can maintain 5 nines availability at work - but can you do it for free in the middle of the night in addition to what you do during the day? You might say that you could - but words are cheap and unconvincing. Criticism is easy when you don't have to demonstrate that you have a better plan.
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 0) by Anonymous Coward on Saturday January 01 2022, @09:38PM (5 children)
That is the whole thing in a nutshell. I could maintain that for free in the middle of the night in addition to what I do during the day. The whole point of a high availability system is that once you set it up correctly it handles most of the problems for you automatically. I've offered multiple times to help out or how to fix the setup but when your devop already thinks they know it all about administration but causes most of the recent problems, all you can do is point out the idiocy and laugh.
(Score: 3, Touché) by janrinok on Sunday January 02 2022, @02:45AM (3 children)
We await your approach to the team to volunteer your services. Thank you.
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 0) by Anonymous Coward on Sunday January 02 2022, @04:02AM (2 children)
My very own ticket to the Titanic? How thoughtful of you!
(Score: 3, Insightful) by janrinok on Sunday January 02 2022, @04:14AM (1 child)
As I said in an earlier post - talk is cheap. If you wish to convince me that your actions would match your words then we offer a genuine invitation to join the team.
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 1, Touché) by Anonymous Coward on Sunday January 02 2022, @10:22PM
That offer is as genuine as my desire to disco.
(Score: 2, Insightful) by khallow on Sunday January 02 2022, @04:34PM
You "could", but you aren't.
I don't want to actually scare off someone who knows their shit, but the level of knowledge you've exhibited so far I could feign with some googling. We've had AC almost from day one harassing and making up shit about SN and its admins. Here's a great example [slashdot.org] from a bit more than a year in:
This incidentally just spontaneously appeared in a thread complaining about yet another potential sale of Slashdot. There was no mention of SN prior to it. Some of the replies further down the thread were interesting as well with accusations that a defending reply sounded "angry and extremist" even though it didn't. Or the heavy insinuation that SN would have a ban list because the SoylentNews code had some urls that had the word "ban" in them.
Well, it's been more than six years from that and things didn't turn out as foretold.
So maybe you're legit. But maybe you're just a continuation of a seven year old harassment campaign.