[RESOLVED] - SaaS-EU (Paris) GUI Access degradation on 2019/07/01 for TWA applications - #20190701A

 

Incident Description  

On 2019-07-01 - 04:35 AM (GMT), the SaaS-EU had an issue when accessing the GUI, using OSS API and using DX API. 

Because it was a random issue without malfunction or failed stop of a server, there was no switch-over between sites.  

Incident information and Service impact

• Incident Start Time: 2019-07-01 - 04:35 AM (GMT).
  
• Restoration Time: 2019-07-01 - 05:06 AM (GMT). 
  
• Service(s) Impact Duration: 31 minutes.
  
• Severe service impact: OSS (GUI and API), DX. 
  
• Non impacted services : Network Server (LRC).
  

Timeline (GMT) 

• [ 2019-07-01 - 04:35 AM ] : Monitoring probes detected connection issue on GUI 
  
• [ 2019-07-01 - 04:36 AM ] : On-call engineer acknowledged the alarm.
  
• [ 2019-07-01 - 04:36 AM ] : New alarms on DX API server (B-DXAPI-E2E Roundtrip).
  
• [ 2019-07-01 - 04:40 AM ] : New alarms on DX API server (A-DXAPI-E2E Roundtrip). 
  
• [ 2019-07-01 - 04:51 AM ]  : New alarm on SMP server (SMP FD COUNT), Investigations point to SMP.   
  
• [ 2019-07-01 - 04:56 AM ] : DX servers were re-started. The issue remained. 
  
• [ 2019-07-01 - 05:01 AM ] : Restart of SMP (A then B). 
  
• [ 2019-07-01 - 05:06 AM ] : All alarms cleared.  
  

We apologizes for not having post an article on status portal this time.

As part of our quality improvement we will update our on-call procedure to update status.thingpark.com prior our investigations.

Root Cause Analysis

The main root cause is a load issue on A-SMP server.

We are currently investigating to find the cause of this overload by:

1. analyzing logs we gather from both DX servers,
   
2. analyzing traffic caught by our traffic monitoring probes,
   
3. we suspect extra load from DX-API to SMP to be the root cause of this overload. 
   

Actility Support

• Related Articles

• [RESOLVED] - SaaS-EU (PAris) GUI Access degradation on 2019/05/30 for TWA applications - #20190530A

        Incident Description   On  2019-30-05 - 03:29 AM (CEST), the SaaS-EU had a random issue when accessing the GUI, using OSS and using DX API.  Because it was a random issue without malfunction or fail stop of a server, there was no switchover ...

• [RESOLVED] - SaaS-EU (Paris) GUI Access degradation on 2019/06/20 for TWA applications - #20190620A

        Incident Description   On  2019-20-06   - 11:50   GMT,  SaaS-EU had an issue when accessing GUI, using OSS API, DX API  Incident Information and Service impact     Incident Start Time:  2019-06-20  - 11:50 GMT Restoration Time:  2019-06-20 ...

• [RESOLVED] - AU SaaS GUI Access degradation on 2019/04/16 for TWA applications

  AU SaaS GUI Access degradation on 2019/04/16 for TWA applications  On the 2019/04/16 our AU datacenter is experiencing GUI access service degradation since 05:30 AM Singapore Time (9:30  PM UTC) the GUI access is unreachable through regular Tab ...

• [RESOLVED] SaaS - EU (Paris) outage on 2019/03/05 - #20190305A

        Incident Description On 3/5/2019 11:30:00 GMT an incident alarm was raised on Actility EU SaaS plaform. The following component was impacted: TWA, SMP and DX-API Incident information and Service impact Incident Start Time : 3/5/2019 11:30:00 PM ...

• [RESOLVED] SaaS - EU outage on 2019-05-22 - #20190522[A]

        Incident Description  On  2019-05-22 - 03:08 AM, our SaaS-EU datacenter became unreachable.  From the outside world, it looks like a complete outage. Incident information and Service impact Incident Start Time: 2019-05-22 - 03:08 AM. ...