ChicagoServers - Database Servers outage – Incident details

Database Servers outage

Resolved
Operational
Started 4 months agoLasted about 17 hours

Affected

Website & Billing Panel

Operational from 1:12 PM to 12:23 AM, Major outage from 12:23 AM to 6:14 AM

Database Servers

Operational from 1:12 PM to 1:12 PM, Major outage from 1:12 PM to 12:23 AM, Degraded performance from 12:23 AM to 6:14 AM

Server Control Panel

Operational from 1:12 PM to 6:14 AM

Updates
  • Resolved
    Resolved

    Earlier today our entire database server experienced an issue caused on the Amazon AWS database hosting that we utilized for all of our various databases. This rendered all systems unusable for the remainder of the afternoon. Our team was working closely with Amazon AWS engineers to rectify the issues but after a while our team became skeptical of their troubleshooting techniques and they did not seem to understand the severity of the situation that affected all of our customers. Eventually, the decision was made to move our databases elsewhere from an earlier backup we had this morning. So thankfully no data loss occurred, but we understand that this still caused a severe inconvenience for our customers.

    We are pleased to announce that the systems are now fully restored, and the server control panel was restored several hours ago so customers could at least navigate through and configure/manage their server as usual.

    On behalf of our hardworking team that has been busy all day getting things moved over and stabilized on an entirely different platform, we appreciate your patience. Given the system-wide issue we could not send out email blasts containing updates.

    You may sign up for update emails should any future outages occur, and monitor the issues, here at our status page: https://chicagoservers.instatus.com/

    Please let us know if you have any questions or concerns and we'd be happy to address them :)

    Thank you for your continued business.

  • Monitoring
    Monitoring

    Our team is moving our system away from Amazon AWS due to their inability to fix this in a timely fashion.

    The Server Control Panel is back online but the main website and customer portal are down. Live chat is available through the control panel. We will post updates as soon as we can.

    Customers that use MySQL databases will be migrated within the next few hours.

  • Identified
    Identified

    We are investigating an Amazon Web Service (AWS) outage that's causing intermittent connectivity issues involving our website and server control panel. Game servers themselves are unaffected.

  • Investigating
    Investigating

    Database Servers cannot be accessed at the moment. This incident was created by an automated monitoring service.