Slrpnk instance is down till mid July; they might relaunch their server on piefed.
Technology
89
Beiträge
55
Kommentatoren
1
Aufrufe
-
Slrpnk.net admin here.
The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.
There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time...
An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.
I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I'll know more when I have physical access again.
I've done a lot of SysAdmin and DCOps stuff in the past so, thought I'd give you some plausible suggestions (haven't dug deep into Lemmy DB stuff and DNS/Federation of the stack, so not sure all is practical).
Scenario 1 - Preserve and merge when access is restored
Setup
- Spin up two VMs/VPS (or one that has enough grunt for two Lemmy servers). Call them
robak.slrpnk.netandslrpnk.netand point DNS appropriately. - Pull federated content from other instances and place it on robak, set as read-only.
- Sync important comms to (new) slrpnk.net without content.
- Allow users to sign up, vetting as possible (all mods). Keep a list of those that are vetted (call it vetted.list). Inform all users that any non-vetted users will have their content dropped when access is restored.
Merge!
- Once access is restored, ensure that (old) slrpnk.net is set to read-only.
- Schedule a maintenance window (announce more time than you are likely to need).
- During the maintenance window, put (new) slrpnk.net into R/O, or just block external access.
- Query the db on (old) slrpnk.net for all users.
- Subtract the vetted users from vetted.list from the list.
- Drop all records from the resulting list of non-vetted users from (new) slrpnk.net.
- Insert the records from vetted and new users (those without conflicts) into the DB on (old) slrpnk.net.
- Validate that everything is working
- Cut over DNS and spin down the new VMs/VPS.
Scenario 2 - Server is in DC or Admin able to facilitate access
- Get a db dump/backup.
- Spin up temporary slrpnk.net on a VM/VPS.
- Use backup of temporary server to restore data to original, when possible.
- Spin up two VMs/VPS (or one that has enough grunt for two Lemmy servers). Call them
-
It's clearly "slurp nook," that nook where you can go to slurp soup as loudly as you want without being judged.
Makes most sense
-
Don't pull a feddit.de on people, alright?
What exactly happened there? It was the big thing, then I didn't use it for a month or so and then it was gone.
-
Nope they all use the public API. Even the default Lemmy web client.
well that's poor planning and why bots are such a problem.
I know CSRF tokens aren't a silver bullet, but doing nothing to stop them does nothing to stop them.
-
What is so special about piefed? I see a few communities moving there. The interface looks different from the original lemmy interface.
Well, you can add flairs
-
What exactly happened there? It was the big thing, then I didn't use it for a month or so and then it was gone.
The admin basically ran it as a one man show with only one other admin who had very limited privileges.
He then went on a "business trip" or workaction or longterm vacation - there were different stories.
Anyway, the database went belly up, the other admin couldn't do a thing and none could contact the admin.
There are some rumours that he wasn't who he claimed he was and actually was a Chinese national who simply returned home, but who knows that.
As a matter of fact none had any meaningful contact with him for months then and it appears he did not return. (But is alive)A Austrian NGO who amongst others does host some mastodon instances,etc. took over and now feddit.org is on a very productive, professional and transparent level.
-
Slrpnk.net admin here.
The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.
There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time...
An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.
I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I'll know more when I have physical access again.
Appreciate the answer and the detail. Good luck getting it all resolved.
-
The admin basically ran it as a one man show with only one other admin who had very limited privileges.
He then went on a "business trip" or workaction or longterm vacation - there were different stories.
Anyway, the database went belly up, the other admin couldn't do a thing and none could contact the admin.
There are some rumours that he wasn't who he claimed he was and actually was a Chinese national who simply returned home, but who knows that.
As a matter of fact none had any meaningful contact with him for months then and it appears he did not return. (But is alive)A Austrian NGO who amongst others does host some mastodon instances,etc. took over and now feddit.org is on a very productive, professional and transparent level.
Thanks for the summary! That sounds freaky!
Well, the trade-off between trusting a huge corporation or a single dude on the internet.
-
What is so special about piefed? I see a few communities moving there. The interface looks different from the original lemmy interface.
Our sysadmin explained some technical advantages here: https://feddit.org/post/13613230/7063696
-
You underestimate the userbase. I made a temp account in the mean time, but we are a hyper tight knit community. We will probably lose accounts - no question - but the core userbase will return
Count me in! Slrpnks all the way!
-
Thanks for the summary! That sounds freaky!
Well, the trade-off between trusting a huge corporation or a single dude on the internet.
XKCD #2347
-
Our sysadmin explained some technical advantages here: https://feddit.org/post/13613230/7063696
Postgres slowing factor
I'm pretty sure one of the best optimised free/libre DBMS's is faster than Python
EDIT: skimmed it wrongly, see corrections below
Also no one knows how Piefed scales, since it only has like 350 MAU
-
Postgres slowing factor
I'm pretty sure one of the best optimised free/libre DBMS's is faster than PythonEDIT: skimmed it wrongly, see corrections below
Also no one knows how Piefed scales, since it only has like 350 MAU
Piefed also uses PostgreSQL. He was mentioning that the limiting factor on either platform is the DB, meaning that the parts written in Python will likely not be a limiting factor.
Piefed also has quite impressive optimizations in other areas as well compared to Lemmy or even Mbin.
-
well that's poor planning and why bots are such a problem.
I know CSRF tokens aren't a silver bullet, but doing nothing to stop them does nothing to stop them.
CSRF protection is a security feature not bot prevention. A bot would just need to get a token first.
-
Postgres slowing factor
I'm pretty sure one of the best optimised free/libre DBMS's is faster than PythonEDIT: skimmed it wrongly, see corrections below
Also no one knows how Piefed scales, since it only has like 350 MAU
Reread it. They're saying "The slowest part is postgres, so even if python is slower than rust it won't make a difference."
-
XKCD #2347
Ya gotta give a line from it or something, I don't just see numbers and know what they are.
-
Did they run out of sun?
No, just punk.
-
After a month and a half downtime all the users will have moved on to other instances. This is essentially a death sentence for the instance and its communities.
I don't think so, I will definitely go back and do not wish to move permanently in another instance.
As others have said I think this instance has a strong base of dedicated users, it's a "specialized" instance that has no equivalent.
-
Ya gotta give a line from it or something, I don't just see numbers and know what they are.
from context, it's probably the "single dude in nebraska holding up the entire internet" one
e: holy shit I got the state right
-
Slrpnk.net admin here.
The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.
There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time...
An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.
I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I'll know more when I have physical access again.
Is it run out of a private residence? How could it happen if it’s in a real data center…?
