On Wednesday 3rd of August 2022, protel AIR customers were not able to access the NG environment, as well as connected applications, starting approx. 11:55 AM UTC+2:00.
The applications were reachable again, approx. 12:20 PM UTC+2:00.
In order to prevent such an incident from reoccurring, we have performed a thorough analysis of what happened and the corrective measures were taken. Our assessment of the fault, including future preventative actions, can be found below.
Affected systems
IAM and connected applications of the NG environment (for example pAir, dSignature, SMP)
Impact
Login to the applications was intermittently not possible.
Root Cause
One instance of the Identity Server, WSO2, ran out of disk storage due to continued Log file writing.
Ultimately that server didn’t accept any new login attempts.
The effect of this was that logging into IAM was not possible anymore for roughly half the user base, as one of the Identity Servers couldn’t be reached, affecting all connected applications.
In order to fix the problem the following steps had to be taken:
At approx. 12:20 PM UTC+2:00, the failed Identity Server had been restarted and as a result all systems were restored and applications were once again accessible.
Upon completion, the functionality of the Identity Server was closely monitored.
The following actions will be taken in order to reduce the chance of recurrence:
Disclaimer: This document has been compiled for information purposes only by protel Hotelsoftware GmbH (protel) to the best of its knowledge and belief based on information currently available and at hand. However, protel does not guarantee that the information is correct, complete, up-to-date and/or in the correct order. Protel reserves the right to make changes and/or additions without prior notice. Protel makes no express or implied warranty (including but not limited to any warranty or merchantability or fitness for a particular purpose or use, etc.) with respect to this information. Information from Protel is provided to users "as is". Protel shall not be liable to users or any other person for any interruptions, inaccuracies, errors or omissions etc. in protel's information, regardless of the cause, for any resulting damages (including, without limitation, direct or indirect, consequential damages, etc.). In all other respects, protel's General Terms and Conditions, which can be downloaded from the protel website at http://www.protel.net/de/agb/ , shall apply.