Yesterday was a frustrating and disappointing day for us and especially for our customers. The scheduled maintenance of our technical partner, Cygate, completely failed causing our payment service to be down most of the day.
Summary of situation
- Cygate scheduled a service maintenance 12.01.2014 from 00:00-07:00. Based on this information we expected only a few minutes of downtime during their maintenance.
- At 03:08, the first payments through our service started to fail and from then on our payment service fluctuated.
- At 11:37, our payment service went completely down.
- At 21:26, our payment service and other services were up and running again.
- It is estimated that throughout the day, only about one third of payments were successful.
Their service maintenance was supposed to correct the problems associated with the downtimes experienced 9.11.2013 and 9.12.2013. After several attempts to fix these problems, Cygate was unable to correct the situation and had to restore the system to its previous configuration. At this point the payments and other services functioned normally.
Based on the information we received from Cygate, we expected the situation to be rectified quickly but that was not the case. We were unable to provide additional information because we had not received any. We expected a quick resolution and waited to pass this information on. Moving forward we will communicate more frequently and transparently so that our customers receive accurate and real-time information. Information will be also be available on our website.
Future plans
Currently our payment service is duplicated and located in different addresses in the data centers managed by Cygate. Now corrective actions were taken to the guiding devices controlling the data centers. We thought we had purchased the best, most professional services on the market but unfortunately this didn’t prevent this incident.
We have begun researching our options and looking for other technical providers. Our plan is to share our services with two different providers so that if one provider fails our payment service will remain unaffected.
Again our sincerest apologies to all effected parties and measures to prevent any additional situations have already been taken.
Markus Laurio, CEO
Written by Markus Laurio - Jan 13, 2014
