[on-going][2021-03-10] ECO platform is down: ThingPark Community Portal, POC, Dev1 total outage

[on-going][2021-03-10] ECO platform is down: ThingPark Community Portal, POC, Dev1 total outage


  • ๐Ÿ›  community.thingpark.org:
    • โœ” subscriber creation is working again, but still not available for customers as Interop Engine is not working yet,
    • โœ” Single Sign-On (SSO) to interop engine is also working
    • ๐Ÿ›  Interop engine validation needed.
  • โœ” TPXLE + ADA working for dev-ope instance
    • ๐Ÿ›  TPXLE not working for community instance without portal
  • โœ” All needed operator instances are now created.
  • ๐Ÿšฅ Next in line:
    • Subscribers, basestations and devices migration.
    • TPX decoders and drivers
    • Restore Partners sites
Please refer to the status page: https://status.thingpark.com/
Incident history:

Thingpark Community, PoCs and Dev1
โŒNetwork Server for PoCs Major incident
โŒNetwork Server for Dev1 Major incident
โŒOSS API and GUI Major incident
โŒThingpark X Location Engine Major incident
โŒThingPark X IoT-Flow Major incident
โŒDX API Major incident
โŒdocumentation platform Major incident

  1. 2021-03-10T09:35:00Z: the hosting company (OVH) just communicated they won't be able to restore their datacenter today.

2021-03-10T06:51:00Z (morning, Paris time) : ๐Ÿ”ฅ A datacenter hosting some of our services has burnt down ๐Ÿ”ฅ

        after the fire ๐Ÿ”ฅ that damaged multiple datacentres in Strasbourg, France, yesterday, the entire ECO environment is still down.
        OVH is in disaster-recovery mode. In the next few days, they will let us know what services / data are gone and what we can restore. We will then assess the situation.
        Our teams are working on restoring our own manual backups to a different location.

2021-03-18, morning, Paris time. Disaster recovery plan is ongoing.
- โœ” The new community platform is up, including DX-API and IPSec.
- ๐Ÿ›  IoTFlow, ThingPark X Location Engine, Interoperability Engine are not back online, yet.
- ๐Ÿšฅ next in line for re-establishment:
  - TPX IoTFlow
  - ThingPark X Location Engine (TPXLE) + Abeeway Device Analyzer (ADA)
  - migration of operator instances, subscribers, basestations (gateways), devices
  - provisioning of Community offers (including Interop)

2021-03-19, morning, Paris time. Disaster recovery plan is ongoing.
- โœ” IoTFlow is installed
- โœ” DX-API was updated to facilitate reprovisioning of subscribers and end-user accounts
- ๐Ÿ›  ThingPark X Location Engine, Interoperability Engine are not back online, yet.
-๐Ÿšฅnext in line:
  - validate TPXLE and ADA
  - achieve creation of all operator instances
  - migrate subscribers, basestations (gateways), devices
  - provision Community and Partner portals offers (including Interop)
    • Related Articles

    • [RESOLVED] - OVH (our datacenter) outage on 2017/11/09 - #201701109A

      Since the 2017/11/09 07:43 AM (CET) our datacenter at OVH is unreachable and this impact big part of Actility services. Actility services are accessible from certain locations/networks. Issues are located on some global routers by OVH (out of our ...
    • [RESOLVED] SaaS - EU outage on 2019-05-22 - #20190522[A]

            Incident Description  On  2019-05-22 - 03:08 AM, our SaaS-EU datacenter became unreachable.  From the outside world, it looks like a complete outage. Incident information and Service impact Incident Start Time: 2019-05-22 - 03:08 AM. ...
    • [RESOLVED] - LRC outage for SaaS-EU on 2020-04-25

      On 2020-04-25, our LRC end-to-end probe detects a traffic issue. The alarm is automatically cleared within five minutes: the LRC process was restarted after catastrophic memory failure. Incident information and Timeline (UTC) [summary] The problem ...
    • EU Platform 2020 11 30 Outage Post Mortem

      Please find below attachment for EU SaaS Platform Post-Mortem report of 2020/11/30 Incident v03 from December 15th 2020
    • ThingPark Wireless- 5.2.2 Maintenance Release Notes

      SCOPE This document describes the content of the maintenance release for the 5.2.2 version, cumulating with previous versions. Versions Version Date Author Details 01 2019-05-20 Nicolas Jombart First maintenance release 02 2019-05-27 Nicolas Jombart ...