What exactly happened there? It was the big thing, then I didn't use it for a month or so and then it was gone.

Slrpnk.net admin here.

The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.

There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time...

An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.

I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I'll know more when I have physical access again.

The admin basically ran it as a one man show with only one other admin who had very limited privileges.
He then went on a "business trip" or workaction or longterm vacation - there were different stories.
Anyway, the database went belly up, the other admin couldn't do a thing and none could contact the admin.
There are some rumours that he wasn't who he claimed he was and actually was a Chinese national who simply returned home, but who knows that.
As a matter of fact none had any meaningful contact with him for months then and it appears he did not return. (But is alive)

A Austrian NGO who amongst others does host some mastodon instances,etc. took over and now feddit.org is on a very productive, professional and transparent level.

What is so special about piefed? I see a few communities moving there. The interface looks different from the original lemmy interface.

You underestimate the userbase. I made a temp account in the mean time, but we are a hyper tight knit community. We will probably lose accounts - no question - but the core userbase will return

Thanks for the summary! That sounds freaky!

Well, the trade-off between trusting a huge corporation or a single dude on the internet.

Our sysadmin explained some technical advantages here: https://feddit.org/post/13613230/7063696

Postgres slowing factor

I'm pretty sure one of the best optimised free/libre DBMS's is faster than Python

EDIT: skimmed it wrongly, see corrections below

Also no one knows how Piefed scales, since it only has like 350 MAU

well that's poor planning and why bots are such a problem.

I know CSRF tokens aren't a silver bullet, but doing nothing to stop them does nothing to stop them.

XKCD #2347

Did they run out of sun?

After a month and a half downtime all the users will have moved on to other instances. This is essentially a death sentence for the instance and its communities.

Ya gotta give a line from it or something, I don't just see numbers and know what they are.

Slrpnk.net admin here.

The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.

There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time...

An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.

I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I'll know more when I have physical access again.

Is it run out of a private residence? How could it happen if it’s in a real data center…?

It is run from a private residence in the DIY punk spririt (and this also allows us to run of a local solar PV system), but more or less the same would happen if you rent rack-space in a "real" data-center. Only if you rent a managed server or VPS someone else will be responsible to fix such issue and this comes at a significantly higher cost at the scale we operate at (slrpnk is part of a bigger project that also hosts other services).

Reread it. They're saying "The slowest part is postgres, so even if python is slower than rust it won't make a difference."

If you rent rack space, you can at least call the DC to send someone to reboot it or smt.

What is so special about piefed? I see a few communities moving there. The interface looks different from the original lemmy interface.