Changelog

The upgrade has been rolled back due to unexpected behaviour; we'll be doing some further work on the code and re-releasing some time in the next week 🙇

We'll be proceeding with testing a major framework update today and will be monitoring over the next day or two. Thanks as always for your patience.

The updates continue; we're proceeding with a major platform upgrade today and will be monitoring. Thanks all, hope everyone's enjoying the weekend!

Updates are ongoing; we'll be doing another major platform upgrade today, and most likely another major framework upgrade later this week. 

As a heads up, after monitoring the current numbers, we've decided to add some database servers to the family and upgrade some of our existing ones. This will likely require a read-only maintenance some time within the next week or two; I'll update with an announcement once we have a timeframe for this.

Thanks all, hope everyone's weekend is going well.

We're currently doing some clean up operations which we see is causing some service degradation (lag and some timeouts). The operation should not take longer than a few more minutes to complete; I apologize in the meanwhile for the disruption. 

This has now been concluded, thanks for your patience!

We're re-distributing some load on our servers to see if we can improve site performance. Early numbers are showing a reduction in page load time, but I'll continue monitoring throughout the week to ensure it behaves as expected during peak hours. If you run into any issues particular to slow page load times, please reach out via the HelpDesk. 

Hi, happy weekend! 

We're releasing another update tonight; if this goes smoothly, this shouldn't cause any visible change, but this change touches a lot of pages (primarily related to form submission, such as character creation and upload), so if you notice anything strange please do reach out via the HelpDesk. 

Update - 

• Sorry for the disruption, the following related bugs have now been fixed
  • Posting new comments is fixed
  • Subscribing/unsubscribing to a user from a profile sub-page is now fixed
  • Bulletin deletions are now fixed
• If anyone's still having problems with submitting new comments or other content, please open a ticket via the HelpDesk.

We're rolling out our next upgrade in the series tonight; only a bit more until we're ready for our new servers. 

Again, this is a major upgrade that touches many pages; I'll be monitoring for any strange behaviours in case it needs to be reversed. Thanks all for your patience.

Updates on this upgrade - 

• We are currently aware of keyword search being broken; we'll need to run an operation on our search engine to get this working, but it's likely this will take some time as it will have to re-index the entire character database. Let me give an update on this once we have further details available - apologies for the inconvenience.

Our search engine has now been re-indexed and keyword search should be fully available again; please just let us know via the HelpDesk if you run into any unexpected behaviour. Thanks all for your patience.

Rolling out some code clean up currently; if done correctly, we shouldn't see any change in behaviour, but do reach out via the HelpDesk in case of any unexpected problems.

Hi, we've got two updates going out today - 

• Small restructuring of our image uploader 
  • This shouldn't have any visible changes, but I'll be monitoring in case of any errors. 
  • This is the first step we're taking to a larger recode, so there will be several more updates to the image uploader over the next few weeks. 
• Added upper limit of 120 images per page in character galleries 
  • Previously, it was possible to disable pagination to have thousands of images on a single page in a character's gallery, but this would cause the page to crash at large volumes of images. It was also possible for traffic to a very large gallery to eat up server resources.
  • This is being capped now to prevent page crashes. 
  • It's likely that we'll introduce an option for infinite loading in the future as a best attempt to support previous behaviour. Apologies for the inconvenience in the meanwhile. 

Some backend changes which were released earlier this morning - 

• There is now a 1 second cooldown on comments (you cannot post more than 1 comment per second)
• Added prevention for double-posting comments or characters (form will be disabled after submission)
• We've added more informative error for when PayPal rejects your payment method (previously it was just giving a generic error page)

Thanks all!

Some more changes going out tonight to see if we can ease lag spikes during peak traffic periods - 

• We are moving some heavy operations from the main servers to our secondary servers
  • Primary updates are in emails and notifications, so if you see any strange behaviour with mails, please do reach out
• We are also moving image processing off the main server, but this requires rewriting a big chunk of the codebase, so we'll instead be making small progressive updates towards this 
  • For today's update, the image thumbnailer has been restructured to prepare for re-writing

• Image thumbnailer has been re-implemented to prepare for moving it to our secondary servers.
• We've added automatic retries when an image fails to upload; this should reduce random errors but may have a negative impact on server performance. We'll be tuning these numbers over the next few days to find a good compromise. 

• Watermarker has been re-implemented to prepare for moving to our secondary servers
  • This should be the last update to our existing image uploader for awhile while we hammer out the new code to bring this over to the secondary servers
• We're continuing to tune the timeout and retry settings on image uploads, thanks for your patience
• Bug fixes
  • Fixed bug in thumbnailer causing blank thumbnails to be generated for large GIFs
  • Fixed bug on character credits page causing page to crash when providing an unexpected input