Update - We are continuing to work on a fix for this issue.
Jan 21, 04:13 CET

Update - All Best Western Interfaces are up and running again according to the standard protocol.
Jan 16, 18:39 CET

Update - Update 4.30 pm

protel Air PMS/Front Office continues to work normally (Note that Front Office was not impacted by this issue). The only impediment at this time is that new objects (users, hotels) can not be created right now.

*protel Cloud Center (with the exception of messenger), IDS, EFT, WBE, Voyager, 3rd party interfaces (including Kiosk, Ariane etc) are now returned to service. *

For protel messenger, event-based emails are being sent by the system, but the system currently does not show the confirmation. Some customer-created messages are currently not fully working yet. We continue to work to bring these back to service.

The Best Western IFC is being brought back online in batches. When the BW Interfaces are working for all hotels, we will amend the status here to operational later this evening.

We continue to work on this issue to fix the remaining items and update this status page as these are completed.

As previously mentioned, we will automatically provide a full and detailed root cause analysis (RCA) via this channel once the incident has been resolved and we have had the chance to perform a complete analysis of the issue and determine the steps to be taken to ensure that such a case can not reoccur in future.
Jan 16, 16:46 CET

Update - 2 Databases of 3 are now fully up and running. The third is in the process of starting. With the 2 DBs that are now online, we have initiated the process of restarting selected services and extensively testing the quality of the data and responses. As the QA process finishes on each individual item, we will inform you via this channel as the various components are returned to service.
Jan 16, 15:21 CET

Update - IDS Services are now partially restored for PULL instances.
As further services come back online during the course of the afternoon, we will provide further information here.
Jan 16, 13:22 CET

Update - EFT services are now restored and fully functional. PCI is back online for IDS and protel i/o connections.
As further services come back online we will provide further Information here.
Jan 16, 11:54 CET

Update - Update 10am CET
During the course of the night and morning, the main system database has now been fully repaired and restored to service by our systems engineering team. This will enable us to start bringing the first services back online during the course of this morning. In parallel, we are already in the process of cloning the restored database to create a cluster in order to bring full stability back to the system and to enable us to return all remaining components back to service. Information will follow in this channel as services start to come back online during the course of the morning and afternoon. The time that this will take depends on the speed of the cloning exercise and the numbers of suspended messages from all systems that need to be processed. All available system (processing) resources within the data centre have been deployed to ensure that any running processes complete as quickly as possible.
Jan 16, 10:12 CET

Update - Our development and engineering teams are continuing to work around the clock on returning the impacted components to service. The database rebuild continues to run and once completed, our operations teams will be in a position to restart the impacted services. We do expect that it will still take a number of hours until all work has completed and all services are fully operational. We will continue to keep you informed about the process and the overall status via this channel. Thank you.
Jan 16, 03:02 CET

Update - Update 23.00 CET
The rebuild of the database is continuing and our engineering teams are doing all that they can and using all resources at their disposal to ensure that this process competes as soon as possible. We will continue to work on this throughout the night and until the issue is brought to resolution. Any changes to the status, further information or updates regarding timelines will be posted here as soon as they become available. Thank you for your continued patience.
Jan 15, 23:07 CET

Update - The teams are continuing to work on the issue. Progress is being made with the database rebuild and resync.
Jan 15, 21:10 CET

Update - Update 17.30 CET.

At this point, we would like to provide you with more detailed information concerning this ongoing incident, based on the information currently available to us.

Firstly, we would again like to reassure you that we understand the urgency and importance of the issue and that all available resources have been working on this issue since the outset. We will continue to work on this around the clock until full service is restored.

The problem that we are facing is database related. The complexity of this incident is being compounded by the fact that several interrelated components are impacted. We have had to perform a time-consuming repair of the so-called Mongo databases and reinstall these within the AWS environment. A very substantial amount of data has had to be restored into the recreated databases. Each individual component or service must be reconnected to the database layer. Because the Mongo databases are mirrored services, these need to be brought back into sync with one another, which is also very time-consuming. Once they are up and running, individual components need to be restarted, and we need to be mindful that they can not be "switched on" simultaneously. The switch must be completed in a controlled manner - one by one - so that further issues or exacerbation of the existing problem are not caused by the amount of data traffic being processed.

Because of this ongoing task's complexity, we can not yet provide you with guidance as to when the system will return to normal, in part or as a whole. We remain optimistic that the work should complete over the next few hours and are doing our very best to meet this target.

We will continue to keep you updated via this channel whenever further information comes to light.

We will automatically provide a full and detailed root cause analysis (RCA) via this channel once the incident has been resolved and we have had the chance to perform a complete analysis of the issue and determine the steps to be taken to ensure that such a case can not reoccur in future.

Thank you.
Jan 15, 17:30 CET

Update - Update 3.45 pm CET. Work to restore services is still ongoing.

Any changes to the status, further information or updates regarding timelines will be posted here as soon as they become available - next update at 5.30 pm today. We thank you for your continued patience while we work to resolve this problem.
Jan 15, 15:53 CET

Update - Update 1 pm CET. The Database repair continues to progress but unfortunately, we can still not provide guidance as to how long this will take to complete. EFT and CD-Proxy have been separated from the main fault and we hope to be able to provide guidance about how long it will take to bring these services back online latest during the next update. Any changes to the status, further information or updates regarding timelines will be posted here as soon as they become available - next update at 3 pm today. We thank you for your continued patience while we work to resolve this problem.
Jan 15, 13:13 CET

Update - Update 11.00 CET. The Database repair is progressing but we can not, as yet, say how long it will take to complete. A separate team continues to focus on EFT and Booking related services and are testing a newly staged system. Any changes to the status, further information or updates regarding timelines will be posted here as soon as they become available - next update at 1 pm today. We thank you for your continued patience while we work to resolve this problem.
Jan 15, 11:05 CET

Update - We are continuing to work on a fix for this issue.
Jan 15, 10:01 CET

Update - Update 09.30 CET. While the cause of the problem IS known, resolving the issue is proving to be time-consuming. Our engineering teams and database specialists are working to resolve the problem as soon as possible, with priority being placed on the EFT and booking related services. Any changes to the status, further information or updates regarding timelines will be posted here as soon as they become available, latest by 11 am CET this morning. We thank you for your continued patience while we work to resolve this problem.
Jan 15, 09:38 CET

Update - As previously advised, our development teams have identified the cause of the problem and continue to take all appropriate steps to resolve the issue. The team have been working around the clock in order to bring impacted components back to service. We will provide guidance around the expected time still required to return the system as soon as it is available. This page will be updated with further information at 09.30 CET latest again. View the current status and impacted services via https://cloudstatus.protel.net.
Jan 15, 08:01 CET

Update - Access to all systems with the exception of protel Messenger has been restored. We expect Messenger to be back online shortly - a further message will follow as soon as this is the case.

We will continue to closely monitor all systems and would like to apologise for any inconvenience caused and thank you for your understanding.
Jan 14, 11:27 CET