Unscheduled image outage report (9~10AM May 7)

Posted 1 year, 11 months ago by admin

Regarding 9:30~10:30AM image outage 

Hi all; this is a status report regarding the 1 hour image outage that happened this morning.

For about an hour (9:30AM~10:30AM site time), images on our second image host became inaccessible; the outage seems to have occurred on our image host's side, and was resolved by our host without any action from TH. We're still waiting on an update from our host regarding the root cause, but I wanted to post a status update to make sure people know the problem's being looked into and apologise for the service disruption and concern. 

Once we have more information regarding the cause, I’ll post a follow up as a reply to this thread, and we’ll also be doing our best from our end to prevent this from re-occurring in the future.

The symptoms look similar to an image outage we had last year in February 2021, when our image host went down for about an hour due to DNS resolution issues. In cases like these, please rest assured that your images are still safely hosted; DNS problems are temporary and your images will be immediately accessible again after they have been resolved.

Regarding TH shutdown rumours

The rest of this post is regarding rumours which we've received requests to clarify. 

  • Rumour 1: TH's domain has not been renewed, is expiring in July, and the site will be shut down in July

TH's domain registration is publicly accessible information and the domain does not expire until September 2023. To be clear, this does not mean the site will close in September 2023; our domain registration is on an automated one-year renewal period, and will renew an additional year every September. 

It's likely that confusion arose due to our SSL cert having an expiry date of July 12 2022, which is displayed in the browser when inspecting the certificate. I understand this date may look scary; however, this doesn't indicate when a service is closing down. SSL certs commonly have renewal periods of 90 days or less, so seeing an upcoming expiry isn't uncommon; for example, the cert for https://www.google.com/ expires July 11 2022, and the cert for https://www.facebook.com/ expires May 14 2022 at the time of posting.  

Neither domain registration or cert issuance actually indicate that a service is going to shut down or not, so to be explicitly clear; we don't have any plans to shut down and site operations are running as normal. 

  • Rumour 2: TH's servers are underprovisioned/underfunded/unpaid, and are failing due to over-crowding

This is an understandable concern to have due to the image outage from this morning, and I apologise sincerely for the confusion. Our images are hosted separately from our servers, and the outage was not caused due to a lack of server resources. Our servers are healthy and paid for in full. We have spare servers which are currently unused and can be spun up in case of emergency, and have plans to purchase additional servers later this year after I'm done with restructuring the codebase to better permit sharding. 

As an aside, I do live in a timezone that means I'm usually asleep during peak traffic (the time when server issues are most likely to occur), so I've been paranoid about the possibility that the site may be laggy or up-and-down when I'm not looking. As usual, I would encourage filing a ticket at the HelpDesk if you run into any problems or performance issues since this does help me identify problems!

I understand that a lot of the uncertainty here is being caused by a lack of movement in site development, and I'm aware there are many areas I need to do better in. I'll continue trying my best, am thankful for any and all trust the users place in TH, and fully encourage users to continue giving feedback via the HelpDesk 🙇 If you have any other concerns that are unaddressed, or if I've been unclear about anything mentioned, please do reach out and I'll do my best to clarify.

Thanks all as always; hope everyone's having a lovely weekend.

admin

A quick follow up! While our hosts weren't able to disclose the details of the root cause of the downtime, they've reassured us that this was a temporary and once-off internal issue that they don't anticipate re-occurring; the service is stable and in no danger of long-term or recurring problems. 

As always, I'll be keeping an eye out for any further disruptions, and please do reach out if you notice any unusual behaviour 🙇