Hey all, most of today’s outage was caused by me. I implemented Anubis in attempt to differ some bot/scrapping traffic. We are presently serving over 350 million requests per month and its starting to show in the stability in the instance. I will be switching to the horizontal scaling approach in hopes to stabilize things more.

I’ll also be providing an long overdue update once I get everything back under control.

Sorry for the stability issues as of late, you all deserve better.

Hey, we did an upgrade earlier today. All tests were passed but its possible something was missed.

Could you tell us which app you are using?

Done

We have not upgraded yet but will be pushing the update this week.

Hey all,

I am very well aware of the slowness and timeouts that people are experiencing. I’m working on getting a few upgrade put into place to bring things back to where they need to be. Thanks for the patience!

This issue appears to have been fixed. Can anyone confirm whether they are still having any problems?

Our Terms of Service will be likely be reviewed in the future, but our core philosophy of “don’t get us in trouble” will stay the same. We’ve always believed that the actions of a few shouldn’t negatively impact the experience for everyone else—and that belief still stands.

When it comes to defederation, we handle it on a case-by-case basis. We only consider it under very specific circumstances, or if there’s a clear mandate from the community. For example, we may choose to defederate from an instance if it’s causing harm or violating our rules, especially if their admin team is unresponsive or unreachable.

Whenever we do take such action, we usually start a discussion with the community to keep everyone informed. In fact, this has happened recently—you can read more about it here.

sh.itjust.works

Give that link a try again otherwise as @InEnduringGrowStrong@sh.itjust.works mentioned send us a PM with your account and we’ll manually verify it.

Thanks

Hey there! I’m sorry that you didn’t receive the activation email. Can I ask how long ago this was? I recently changed our email provider and want to make sure it wasn’t after the fact. Thanks!

Hey!

Everything has been reviewed and we’re good to move over the 19.9. I’ll be posting an announcement in the coming days to give everyone a heads up. I’m aiming towards the end of this week.

This is odd. I did not upgrade pictrs (the service responsible for image hosting). There may have been an error for some specific tables rows during the database migrates. Another possibility could have been an error during the schema update that I believe the new version had to go through.

Would you be able to DM me links to other posts that you come across with broken images? I will look to see if I can identify any common trends that could tell us more.

Thank you

We sure are!

deleted by creator

Yes, I’ll get this done

UPDATE: here you go https://oldsh.itjust.works/

There was a little hiccup this morning unrelated to the migration tonight. The Lemmy services like to be restarted every once in a while. Once we move to the new hardware I’ll be able to look into implementing a better logging system and hopefully be more proactive when situations like these happen.

The biggest consumer of storage on this instance is related to the image hosting which we use an external object storage provider for. The second is the database which is no were near the 2TB capacity. 1TB SSDs are cheaper than 2TB SSDs and I also didn’t want to spend more than I needed. As other mentioned if we need more space or IOPs in the future, I could accomplish this by adding more drives as a quick fix. This server does not support NVME unless I leverage its PCIe ports but I don’t plan on doing that. By the time this instance gets to the point where 10 SSD drives just isn’t cutting it anymore I’ll probably have come across another opportunities on getting a new server with better NVME support.

I’ve started the preparation steps to minimize the maintenance window needed for this upgrade. These operations are running some background tasks and once they are done I’ll make an announcement to give everyone a few days heads up of the anticipated upgrade time and date. Should be soon.

Hey!

The best way is to @TheDude on our matrix channel. This action triggers notifications across all my devices, creating a barrage of alerts that almost always grab my attention, a.k.a a ddos to my sensory system.

@kersploosh@sh.itjust.works explained it well. It was related to a small bug on the pictrs service that I’ve already reported which they quickly fixed in a newer version. I will tackle this upgrade at the same time as I get this instance up to v19

Hi there

Sent you a message on the matrix chat. This should now be fixed.

Thank you