Network Status

Mail Server Certificate Issues

Jul 16, 2020 12:00 EDT

Investigating – We are currently experiencing issues with 2 of our legacy mail servers, MAIL9 and MAIL10.

For iPhone and MacOS users please follow the instructions here in order to resolve certificate errors: https://dsonedesign.freshdesk.com/support/solutions/articles/9000190583
For users who cannot login to POP e-mail (MAIL9) we are investigating further and will post an update here once resolved.

 

Jul 17, 2020 12:30 EDT

Resolved – We have pushed new SSL certificates to the server and following the above steps has resolved the issues for our clients,




ca-cpanel-web1 - Maintenance

Sep 4, 23:30 EDT
Completed - The scheduled maintenance has been completed.

Sep 4, 23:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Sep 2, 16:25 EDT
Scheduled - We will be performing maintenance on ca-web1.cpanel server in our Toronto, ON facility on Sept 4th 2020 at 11PM Eastern. The maintenance period is between 15-30 minutes, you may experience small service interruption. If you have any issues or questions, please contact us at [email protected]

Thank you,



Scheduled Windows Update Maintenance

Jul 31, 00:00 EDT
Completed - The scheduled maintenance has been completed.

Jul 30, 19:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 30, 16:55 EDT
Scheduled - Our network operations team will be performing updates to all of our managed Windows servers tonight to patch for security updates.

We apologize for any inconvenience this may cause. DS Digital Media takes Security including Zero Day exploits extremely seriously & this maintenance will be proceeding with this as quickly as possible.



Hsphere Control Panel - Connectivity Issues

Jul 30, 18:56 EDT
Resolved - This incident has been resolved.

Jul 30, 17:41 EDT
Investigating - Our team is currently investigating an issue with client access to our Hsphere control panel.

We apologize for the inconvenience and will share an update once we have more information.



Hsphere Control Panel - Maintenance

Jul 25, 02:00 EDT
Completed - The scheduled maintenance has been completed.

Jul 24, 18:01 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 24, 11:36 EDT
Scheduled - During this period the DS Digital Media Hsphere Control Panel will be migrated to new hardware. Access to the control panel may be restricted during the maintenance period.

No web services aside from the control panel itself will be impacted

Updates will be posed to this status page as the team progresses with the migration.



US/Canada - ActiveDirectory Maintenance

Jul 15, 21:00 EDT
Completed - The scheduled maintenance has been completed.

Jul 15, 17:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 15, 14:43 EDT
Scheduled - During this maintenance period, our network operations team will be applying updates to our Active Directory clusters in both Toronto and Dallas.

Expected Impact: As the servers are being upgraded, clients using Active Directory to connect to servers, or utilize our Microsoft Exchange platform may see connections log out as loads balance through the cluster. . Please sign-out, then sign-in to refresh your connection.

If you have any questions or concerns, please contact us via email, phone, or chat.



Cartika Accounts Portal - Unavailable

Jul 10, 16:00 EDT
Resolved - Our Engineering team has resolved the issue impacting the control panel and services should now be functioning normally. Thank you for your patience and understanding throughout this process.

Jul 10, 15:49 EDT
Investigating - Our Engineering team has identified an issue with loading our Control Panel located at https://accounts.dsdigitalmedia.ca. We apologize for the inconvenience and will share an update once we have more information.



Dallas - Microsoft Exchange Connectivity Issues

May 22, 11:46 EDT
Resolved - This incident has been resolved.

May 22, 09:10 EDT
Investigating - Our exchange team is currently investigating an issue with client access for Cloud Email clusters. At this time, users may experience issues using OWA or Exchange mail clients to connect to Cloud Email clusters. We apologize for the inconvenience and will share an update once this issue is resolved.

Thank you,



Scheduled Backup System Maintenance

May 17, 10:00 EDT
Completed - The scheduled maintenance has been completed.

May 16, 10:01 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

May 12, 14:13 EDT
Scheduled - Our developers will be performing maintenance on our backup system In Toronto,On Facility. During this period, you will not have the ability to perform backups or restores through our control panel.

This maintenance will take place between May 16th 10AM ET to May 17th 10AM ET

Updates will be posted here on our Status Page as the maintenance is completed.


Thank you,
DS Digital Media



WEB21 - Physical Maintenance

Apr 13, 20:00 EDT
Completed - The scheduled maintenance has been completed.

Apr 13, 19:01 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Apr 12, 16:32 EDT
Scheduled - We will be performing physical maintenance on the WEB21 server in our Dallas, TX facility on April 13th at 7PM Eastern.

The server will be powered off for about 30 minutes during this time.



Network Connectivity Impacted

Apr 13, 18:21 EDT
Resolved - We are declaring this incident as resolved.

The outage stemmed from one of the Transit carriers (Telia) having a significant routing issue impacting connections to/from our Toronto network. Our team is engaging with Telia for a full RCA on the outage.

We apologize for the inconvenience!

Apr 13, 18:12 EDT
Monitoring - At this point in time we are seeing all network routing returning to normal. We are engaged with network vendors at this time to fully declare this incident as resolved.

Apr 13, 17:47 EDT
Update - We are continuing to investigate this issue.

Apr 13, 17:45 EDT
Investigating - We are currently investigating reports of degraded network connectivity in our Toronto, Canada facility. We will update this incident page as investigation occurs



Toronto Cloud Maintenance

Apr 12, 23:00 EDT
Completed - The scheduled maintenance has been completed.

Apr 12, 22:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Apr 12, 12:34 EDT
Scheduled - Please be advised that we will be performing hardware maintenance on one of our Cloud Nodes in the Toronto facility this evening at 10PM (Eastern).

VMs will be powered off for up to 45 minutes during this time while we swap a critical component that we are proactively replacing.

Individual VM owners have been notified

Additionally the below Services will be impacted:

dsdigitalmedia.ca site
CA-MYSQL (Shared)



Web29 - Server Down

Mar 23, 16:36 EDT
Resolved - IP filtering has been removed and service has been restored. Please contact our support team if you continue to have any issues

Mar 23, 14:31 EDT
Monitoring - Network isolation and filtering have been put in place and the majority of impacted sites are back online. We are continuing to work with network vendors and are further monitoring the situation.

Mar 23, 13:19 EDT
Identified - Our team is working with network vendors to mitigate a large DDOS attack causing congestion in our Dallas, TX facility. Updates will follow.

Mar 23, 12:43 EDT
Update - We are continuing to investigate this issue.

Mar 23, 12:38 EDT
Investigating - We are currently experiencing network issues with 'Web29, Win18, Win19' server. Please be patient while we investigate this issue.



Dallas - Microsoft Exchange Connectivity Issues

Mar 7, 22:04 EST
Resolved - This incident has been resolved

Mar 7, 19:50 EST
Monitoring - Our Engineering team has implemented an in-place fix, and all users should now be able to connect to Microsoft Exchange services at this time.

We apologize for any inconvenience and thank you for your patience throughout this process.

Mar 7, 15:59 EST
Investigating - Our exchange team is currently investigating an issue with client access for Cloud Email clusters. At this time, users may experience issues using OWA or Exchange mail clients to connect to Cloud Email clusters. We apologize for the inconvenience and will share an update once we have more information.



Toronto & Dallas - Public Cloud Maintenance

Mar 7, 00:21 EST
Completed - The scheduled maintenance has been completed. Should you have any residual issues please contact our support team via email ([email protected])

Mar 6, 22:23 EST
Update - We have extended this maintenance period for an additional 90 minutes at this time. Maintenance is progressing successfully, however taking a bit longer than anticipated.

Mar 6, 21:00 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Mar 2, 14:37 EST
Scheduled - On Friday March 6th 2020, we will be applying updates to all Hypervisors running our Public Cloud in both Toronto and Dallas.

During this time all Virtual Machines will be rebooted by our team. We expect on average about 15 minutes of total impact per VM.



Dallas - ch-web - Scheduled Maintenance

Mar 3, 19:30 EST
Completed - The scheduled maintenance has been completed.

Mar 3, 19:00 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Mar 2, 17:26 EST
Scheduled - The Web Server in our Dallas, Tx Facility will be taken offline for physical hardware maintenance at 7PM ET for 5-30 mins

We anticipate total downtime to be less than 30 minutes

Updates will be posted here on our Status Page as the maintenance is completed.

Impacted Servers
ch-web



Power Maintenance - Dallas, TX

Feb 29, 02:00 EST
Completed - The scheduled maintenance has been completed.

Feb 28, 23:00 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Feb 26, 22:41 EST
Scheduled - Site: DAL01 (DS Digital Media Dallas)
Services Impacted: Various Shared, VM and Dedicated Servers
Window: February 28th - 11PM - 2am EST/Eastern

Description: During the aforementioned window, DS Digital Media Engineers will be replacing a Power Distribution Unit located in our Dallas, TX facility.

During this process, we may be required to re-cable and re-distribute power used for specific devices.

We anticipate the impact to be as simple as a Reboot for the impacted clients and should last no more than 30 minutes to one hour.

Impacted Shared Servers:
-win19
-master7
-web22



Network Outage

Feb 21, 12:42 EST
Resolved - This incident has been resolved.

Feb 21, 11:27 EST
Investigating - We are currently experiencing a network related outage in Dallas. 1 cabinet seems to be impacted. We are investigating the switching in the cabinet and will provide updates as they become available”



Toronto - Web1 - Scheduled Maintenance

Jan 24, 19:30 EST
Completed - The scheduled maintenance has been completed.

Jan 24, 19:00 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jan 20, 21:37 EST
Scheduled - The Web1 Server in our Toronto, ON Facility will be taken offline for physical hardware maintenance between 7PM - 9PM Eastern.

We anticipate total downtime to be less than 30 minutes

Updates will be posted here on our Status Page as the maintenance is completed.



Win19 - Server Down

Jan 7, 21:58 EST
Resolved - This incident has been resolved.

Jan 7, 15:35 EST
Monitoring - A fix has been implemented and we are monitoring the results.

Jan 7, 12:30 EST
Identified - Windows update has caused our server to become unresponsive, we are currently restoring our services. Please check again for another update. We apologize for the inconvenience and delay.

Jan 7, 09:43 EST
Investigating - We are currently investigating this issue.



Dallas - SQL4 - Scheduled Maintenance

Dec 24, 17:15 EST
Completed - The scheduled maintenance has been completed.

Dec 24, 17:00 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Dec 24, 07:16 EST
Scheduled - The SQL4 Server in our Toronto, ON Facility will be taken offline for physical hardware maintenance between 4PM - 5PM Central Time (CT).

We anticipate total downtime to be less than 15 minutes

Updates will be posted here on our Status Page as the maintenance is completed.



Toronto - Web1 - Scheduled Maintenance

Dec 11, 07:30 EST
Completed - The scheduled maintenance has been completed.

Dec 11, 07:00 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Dec 9, 09:58 EST
Update - We will be undergoing scheduled maintenance during this time.

Dec 9, 09:58 EST
Scheduled - The Web1 Server in our Toronto, ON Facility will be taken offline for physical hardware maintenance between 7PM - 9PM Eastern.

We anticipate total downtime to be less than 30 minutes

Updates will be posted here on our Status Page as the maintenance is completed.



Web 23 - Down

Nov 23, 11:05 EST
Resolved - This incident has been resolved. Services on web23 are back online.

Nov 23, 09:28 EST
Investigating - Web 23 is currently down. We are working to restore services



MSSQL5 Data Integrity

Nov 1, 23:57 EDT
Resolved - This incident has been resolved.

Nov 1, 23:26 EDT
Identified - The issue with MSSQL5 data appearing to be out of date has been identified and a fix is being implemented. During this time, databases on MSSQL5 may be unavailable. We apologize for the inconvenience and will share an update once we have more information.



US Shared WIndows Servers

Nov 1, 19:29 EDT
Resolved - This incident has been resolved.

Nov 1, 19:28 EDT
Update - We are continuing to investigate this issue.

Oct 31, 19:04 EDT
Update - we have downgraded this incident to minor

WE WILL leave this incident open until tomorrow when we can post a formal and detailed reason for outage and analysis

as of right now, all services have been restored. we have a final shared windows web server that will be BMR’ed here shortly (win16)

at this time, all US shared/reseller services are back online. If you have any specific issues with a website (outside of win16),please contact our support team for assistance

Oct 31, 15:40 EDT
Update - Restores are progressing. At this point, some services are starting to come back online. because of the nature of this incident, we needed to do a combination of fresh Operating System Restores and manual data backup restoration along with BMR’s in some scenarios.

A full Reason For Outage (RFO) will be provided once we have restored all services

Oct 31, 11:04 EDT
Update - we have restored some of the services here

mssql5 and some web servers are still impacted in the Dallas Shared/Reseller services

Oct 31, 04:49 EDT
Update - we are working to recover the impacted dallas shared/reseller services. because of the nature of this incident, we will need to bare metal restore some servers here. once all services are recovered, we will restore the latest backup data at that time

Oct 30, 23:12 EDT
Investigating - We currently are experiencing a partial outage of our dallas shared/reseller hosting infrastructure

we will continue to post updates here as they become available



[Canada] RDS Issues

Oct 30, 17:46 EDT
Resolved - Hello

this issue has been resolved

we will engage the RDS cluster again overnight and customers can once again log in via RDS APPs in the morning.

Please continue to utilize the RDS users via RDP until then as the backup method to access your remote desktop and apps

Oct 30, 09:38 EDT
Identified - The issue has been identified and a fix is being implemented.

Oct 29, 18:01 EDT
Update - We have engaged Microsoft and are working to identify and resolve the problem.

Oct 29, 14:42 EDT
Investigating - We are currently investigating reports of RDS applications errors for some servers hosted in our Toronto, Canada facility.