Scheduled Site Maintenance (Resolved)
Affecting Other - Support and Billing Websites | Priority - Medium

In an effort to improve our customer support services we will be scheduling a maintenance event for Monday, June 26th 2017 from 5:00 a.m. to 1:00 p.m. Mountain Standard Time (MST).

Through an extensive review of our company earlier this year and to align with our commitment to provide the best support services we can deliver to our customers, we will be taking efforts to upgrade our customer billing panel, support systems and ticketing systems with the latest offering provided by our software vendor.  This upgrade will require the migration of our billing site to our latest software build and new IP where we can deliver our service in the full confidence of our latest in a high-performance and secure software platform.

This maintenance window is currently an extended schedule with an estimated timeframe of 8 hours, but maintenance may extend further due to geographical DNS replications.  During this 8 hour maintenance window, our ticketing, invoicing and billing systems will currently be unavailable.  For immediate support assistance during this event, customers are recommended to email support services directly at support at thirdhosting.com.  This maintenance action will not affect any active services on customer nodes and no customer outage is expected.  We appreciate your patience and understanding as we continue to work to improve our support systems and welcome any feedback you may have.  If you have any questions or concerns regarding this maintenance schedule, please contact our Support Department at any time.

Sincerely,
Third Generation Internet Services

06-26-2017 11:45 MST - Migration of our billing software and support site has been completed.  This maintenance window is being extended however while we review new configuration options supplied by our vendor as well reviewing possible upgrades to theme support for the client area.  No major interruptions to support services are expected.  At this time we're also reviewing an issue with the order form not functioning due to out of date theme controls. Date - 2017-06-26 05:00 - 2017-10-16 14:25
Last Updated - 2017-06-26 11:47

Scheduled Network Maintenance 6/22/2017 (Resolved)
Affecting System - Network - OVH BHS Datacenter | Priority - Medium

Datacenter engineers will be performing network maintenance on Thursday, June 22nd from 5:00 to 6:00 a.m. Eastern Daylight Time (EDT) which may affect public connectivity for a brief duration.  During this one-hour maintenance window, engineers will be performing scheduled maintenance to upgrade site routers.  Due to the software issues of the June 6th network outage, to improve services network engineers will be replacing routers at BHS-3.  This maintenance may result in a brief network disruption or periods of increased latency as traffic is rerouted.  Our provider has made all necessary adjustments to minimize any impact to service.  This maintenance window only affects services located at the OVH data center and our East Coast based services.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services Date - 2017-06-22 05:00 - 2017-06-23 11:27
Last Updated - 2017-06-21 07:30

Goldmoon Server - Database Maintenance (Resolved)
Affecting Server - goldmoon | Priority - Medium

In preparation of an upcoming account migration, technicians will be performing database maintenance on the Goldmoon cPanel web server Monday, June 19th 2017 from 8:00 to 10:00 a.m. Mountain Standard Time (MST).  During this maintenance window, system administrators will be restarting the MySQL service on the Goldmoon server and launching the service in a maintenance mode to review the integrity and consistency of customer databases that are currently using InnoDB tables.  Throughout this maintenance window services dependent upon the databases may be temporarily unavailable while system administrators rebuild the InnoDB tables and log cache to improve the reliability and consistency of the service.  Although this maintenance window has been scheduled for 2 hours, the databases located on the Goldmoon server are minimal and the expected interruption is anticipated at approximately 15 minutes while database caches are dumped and re-imported.

We appreciate your patience and understanding as we work to improve our service reliability.  If you have any questions or concerns regarding our maintenance schedule, please feel free to contact Customer Support at any time.

Sincerely,
Third Generation Internet Services

6-19-2017 8:00: This maintenance is now in progress.  Technicians are now reviewing databases and checking the integrity of tables.
6-19-2017 9:40: This maintenance window is now closed.  All maintenance has been successfully completed.  All database issues have been identified and repaired.
Date - 2017-06-19 08:00 - 2017-06-19 13:09
Last Updated - 2017-06-19 09:41

Scheduled Server Maintenance (Resolved)
Affecting Server - goldmoon | Priority - High

Our next scheduled maintenance will take place on Monday, May 22nd, 2017 from 1:00 a.m. to 2:00 a.m. (EDT) Eastern Daylight Time.

This maintenance will affect VPS customers located at the BHS datacenter and customers of the Goldmoon hosting server.  During this scheduled maintenance event we will be checking the performance, reliability and current state of the system hardware due to a suspected raid array degradation.  To accomplish this we will be migrating customer servers to a new upgraded hardware node.  A service interruption of up to 20 minutes is anticipated throughout this 1 hour maintenance window as we live migrate customer systems and hosting servers to the new hardware.  A brief interruption of up to twenty minutes is anticipated as we relocate existing IPs and re-establish connectivity.

There is no further action is necessary for customers for this event.  All customer data has already been synced in preparation of the relocation and customer services will be migrated in a live state.

We appreciate your patience and understanding as we continue to work to improve our services and server reliability.  If you have any questions or concerns regarding this maintenance action, please contact our Support Department.

Best Regards,
Third Generation Internet Services
support@thirdhosting.com

2
017-05-22 01:05 EDT - This scheduled maintenance event is now in progress.  Further updates will be noted here.
2017-05-22 01:55 EDT - This maintenance window is being extended to 3:00 am EDT.
2017-05-22 02:35 EDT - Scheduled server moves were successful with the exception of a live migration for customers of server Goldmoon.  We're currently in the process of restarting all background game processes manually.
2017-05-22 03:00 EDT - This maintenance has been completed.  All game processes on goldmoon have been restarted.

Date - 2017-05-22 01:00 - 2017-06-16 21:05
Last Updated - 2017-05-22 00:03

srv04 node down (Resolved)
Affecting System - srv04ssphxaz | Priority - Critical

Services Affected:  Riverwind, Nightshade, VPS Cluster Control Panel and VPS Services at Phoenix.

We are currently investigating issues across multiple virtual servers with service failure and database crash.

These services affect the srv04ssphzaz hardware node as well as Riverwind and Nightshade hosting servers.

We apologize for the inconvience and appreciate everyone's patience as we work to restore services.
Email for support has been routed off to a backup MX host in the mean time as we work to reply to support requests.
We will restore any background processes that were running prior to issue.
Service credits will be issues for the outage, a follow-up email will be released once outage event has been resolved.

2017-05-15 20:50 MST - An emergency reboot of the hardware node is necessary.  All services on across the node are experiencing issues and the database problems are belived to be due to disk cache issues.
2017-05-15 21:19 MST - The server has failed to restart from reboot and we're currently investigating the cause.  At this time no ETR.
2017-05-16 01:30 MST - We're currently reviewing options of migration of customer data to new hardware.  The server is experiencing hardware issues at this time.
2017-05-16 04:00 MST - New hardware is being procured and setup.  Data backup off of the previous server has begun to be used for restoration.
2017-06-16 06:50 MST - Data restoration is at approximiately 25%.  Recovering the system's data from the raid disks has yielded in large amount of 'recovery' files that has the potential for issues.  As we're restoring servers, we are manually comparing customer data for missing or corrupted files.  Due to the data corruption, the last snapshot backup from earlier in the day was corrupted/missing data and we'll compare/restore against the previous daily.  This may yield is previously deleted files being restored.  These files can be safely deleted again.  This may be necessary to verify that customers have all necessary files for operation.
2017-06-16 11:45 MST - Data restoration still in progress. 
2017-06-16 12:30 MST - Unfortunately, after making a detailed review of the file system we're finding widespead file corruption randomly throughout files in 100,000s of inodes.  Most of the corruption does fortunately reside in operating system files and there is only sparse corruption throughout customer profiles.  After reviewing the data and options, we're unfortunately making the decision to restore the last known good backup from May 13.  We'll then work through each individual customer profile to try and determine if we can restore the live data to their profile.  This process will take significantly longer due to the manual checking process, but we believe it will yield a more consistent result.  
2017-06-16 14:20 MST - We have begun restoring services to customer VPS services.  At this time, the VPS Control Panel, HN cluster and customer VMs are back online.  Data was restored from the last backup from yesterday and all servers are publically responding to pings. A copy of your data prior to the raid crash is available upon request.  Due to the block corruption that we're seeing and the extensity of the corruption, it is recommended to check the file prior to use.
2017-06-16 14:30 MST - We are currently in the process of restoring services to the Nightshade and Riverwind web and game servers.  Container services have been brought online while we work to restore data. Soft ETA of about 6 to 8 hours as we work to relcoate recovery data from our other data center.
2017-06-16 18:30 MST - All DNS servers part of our DNS clusters for VPS and Hosting services that resided on the node have been restored at this time.  All records have been sync successfully.
2017-06-16 20:00 MST - To clarify, services hosted on the Riverwind and Nightshade servers are still down at this time.  All other services have been recovered, but data restoration is still in progress for these two servers.  Due to having to make a separate pull from the backup servers the restoration time of these servers has been extended.  ETA for restoration for all services on these server is 8 hours.  All game processes, websites and databases will be started and verified once these servers come online.
2017-06-16 21:30 MST - Services for our web hosting servers have been offline for 24 hours.  We are hard at work still restoring services.
2017-06-17 05:45 MST - Data has been restored to the Riverwind server.  In the process of bringing the server online we're unfortunately encountering an error with MySQL services and InnoDB database corruption still.  The database issue appears to unrelated and will need further restoration.  At this time, we are not unlocking customer services until fully restored.
2017-06-17 12:55 MST - Services for the Nightshade server have been fully restored.  All recent game databases that could be located have been started.  Database services for Nightshade have been recovered.
2017-06-17 13:00 MST - We're still working to restore the MySQL databases on the Riverwind server and currently at about 80% restoration.  Meanwhile all game databases that could be located have been started in preparation of returning the server to production ready status.  The server is not available for access at this time.
2017-06-17 15:30 MST - The MySQL database server has been restored on Riverwind.  All web services and service processes have been restored.  Server access has been re-granted.  If you have a game process and we were able to determine the correct folder, it has been started.  Any customer needing assistance is encouraged to submit a support request at this time.  

STATUS: All servers and services are back online.

Thank you everyone for your patience.  We will be following up with a detailed email in regards to the issues, how we plan to mitigate and improve in the future as well as making service credits as per our SLA/TOS in the following day.  We still have a bit of cleanup work to do and tuning for the new hardware.  These changes should have no/minimal impact during the process.  Any customer needing assistance at this time is encouraged to submit a support request.  A current copy of customer data off previous hardware is available upon request.  Given that the data was throughly corrupted, we have opted to not grant this data to the customer unless upon request.  The silent file corruption issue requires manual review of each file and difficult to detect issues with encoded or binary files.  For this reason we do not recommend attempting to restore this data in mass.  This data can be used however to recover file changes, profiles, or work that may have occured between May 14th and 15th. Date - 2017-05-15 21:30 - 2017-05-19 23:12
Last Updated - 2017-05-17 16:16

Scheduled Network Maintenance 2/16/2017 (Resolved)
Affecting System - Network - Phoenix NAP Datacenter | Priority - Medium

Datacenter engineers will be performing network maintenance on Thursday, February 16th from 12:00 to 8:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this eight-hour maintenance window, engineers will be performing scheduled maintenance on the backened network in our Phoenix location.  The maintenance may result in up to three brief network disruptions for up to 5 minutes each.  This outage will only affect services located at the Phoenix NAP data center.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services Date - 2017-02-16 00:00 - 2017-03-04 18:08
Last Updated - 2017-02-12 12:08

Emergency Server Maintenance (Resolved)
Affecting System - Node srv03 - BHS Datacenter | Priority - Critical

System administrators will be performing emergency scheduled maintenance on Friday, August 5th from 2:00 a.m. to 6:00 a.m. (EDT).

Our monitoring systems have reported a hardware failure on node srv03 and the system is currently operating on redundant hardware in a degraded status.  Due to the service life of the hardware, our system administrators will be migrating customer services to a new srv05 hardware node.  This 4 hour maintenance event will result in a service interruption while IP fail-over schemes are migrated.  A brief service interruption of up to 1 hour for network connectivity is anticipated throughout this 4 hour maintenance window.  Throughout this event every effort will be made to migrate operational servers and services in an online status.  All system services will be suspended briefly and resumed from snapshot in their operational state.  Updates for this maintenance action will be posted to this notice at http://maintenance.thirdhosting.com.

We appreciate your patience and understanding as we continue to work to provide a reliable and fault tolerant service environment.  If you have any questions or concerns regarding this maintenance action, please feel free to contact our Support Department.

Best Regards,
Third Generation Internet Services

Update: 8/4/2016 @ 7:30 p.m. :: Premigration of data has been started.  No impact to customer performance or service is expected at this time.

Update: 8/5/2016 @ 3:15 a.m. :: System migration has begun.  Customer services will loose network connectivity briefly while the IP fail-over scheme is migrated.

Update: 8/5/2016 @ 3:50 a.m. :: All systems have been successfully migrated at this time. Date - 2016-08-05 02:00 - 2017-02-12 12:03
Last Updated - 2016-08-05 00:54

Scheduled Site Maintenance (Resolved)
Affecting Other - Billing Control Panel and Support Site | Priority - Medium

As we continue to update our systems to provide our customers with the latest software and support, we will be scheduling an extended maintenance window for our support site on Thursday, July 7th 2016 from 17:00 to Friday, July 8th 2016 till 02:00 MST.

Systems administrators will be upgrading our primary web site, customer billing panel, support system and ticketing system with the latest offering provided by our vendor.  This latest change will require migration of our primary website to a new server and IP.  This maintenance window is currently scheduled with an estimated timeframe of 6 hours but may be extended as well as affected by geographical DNS replication.  For immediate support assistance during this window customers are recommended to email the support email address directly at support at thirdhosting.com.

This maintenance action will not affect any active services or processing on customer nodes and no customer outage is expected.  We appreciate your patience and understanding as we work to improve our customer support systems and welcome any feedback our customers may have.  If you have any questions or concerns regarding this maintenance schedule, please contact our Support Department at any time.

Sincerely,
Third Generation Internet Services Date - 2016-07-07 19:00 - 2016-07-09 20:29
Last Updated - 2016-07-05 12:28

Routing Issues to Phoenix NAP for Level3 ISP (Resolved)
Affecting Other - Phoenix NAP | Priority - Critical

Customers originating on the East Coast who utilize ISPs with Level3-based backend routing (Comcast and Century Link) are currently experiencing up to a 50% packet loss reaching services at the Phoenix Datacenter. 

This outage affects the eastern United States regions for Level3 based customers.  Customers performing trace rouets will notice a significant packet loss at bar2.phoenix1.level.net.

Customers currently affected by their ISP can use a Proxy, VPN or other service to route around their ISP's routing issues.  US Based Open Proxy List: http://www.xroxy.com/proxy---US-ssl.htm

Datacenter engineers have been notified and are currently working with the ISP to mitigate the routing issue.

Sincerely,
Third Generation Date - 2016-06-16 18:30 - 2016-07-05 12:27
Last Updated - 2016-06-17 01:01

Routing Issues to Phoenix NAP for COX ISP (Resolved)
Affecting System - Phoenix NAP | Priority - Critical

Customers using Cox Communications ISP are reporting connectivity issues to services located at the Phoenix Datacenter.  Datacenter engineers have been notified but the issue originates with the customer's ISP routing.  

The outage affects the southwestern United States regions.  Customers performing trace routes will notice their last hop within Cox Communications as 68.105.31.41 before being routed to geographical routes outside of the United States.  The last hops of the trace route will show a flapping typically between the 202.97.50.74 and 202.97.50.73 addresses resulting in a failure to route service.

These addresses are external to our services, https://whois.arin.net/rest/net/NET-68-105-30-0-1/pft?s=68.105.31.41 and https://whois.arin.net/rest/net/NET-202-0-0-0-1/pft?s=202.97.50.77

Customers currently affected by their ISP can use a Proxy, VPN or other service to route around their ISP's routing issues.  US Based Open Proxy List: http://www.xroxy.com/proxy---US-ssl.htm

Regards,
Third Generation Internet Services

Update: Routing issues are reported to be between Cox and China Telecom.

Update: 3/23/2016 @ 16:00 MST - Emerngecy maintenance wil be conducted to re-route BGP traffic from China Telecom to Level3 @ 1 AM MST 3/24/2016.  This maintenance will result in a brief disconnect for customers of Cox Cable Communications ISP.

Date - 2016-03-22 06:02 - 2016-04-07 16:21
Last Updated - 2016-03-23 17:07

« Prev Page   Next Page »