Sub-optimal routing - 07/12 - Status Page

Resolved Dec 8, 2023 10:34 AM NZDT

Network has continued to remain stable since yesterday afternoon, during the night and this morning.
While the network will continue to remain under close observation, this incident is now resolved and our new core which was deployed has been rolled back to our previous network core.
We apologise for the inconvenience caused to you and your business(es) yesterday.
Our network core replacement & upgrade works have now been paused while our team, consultants and vendors work to fully understand all aspects of this incident to review our path forward to our new network core.
Once this process and external review has been completed, an incident report will become available for our impacted customers.

Monitoring Dec 7, 2023 5:21 PM NZDT

Network is presently stable and connectivity should have been restored.
If you are without connectivity, please turn off your router, turn back on, and wait 5 minutes.
We’ll be continuing to closely monitor all aspects of the network overnight and will continue to keep this incident open and in a monitoring state.
Changes to our network will only be made if absolutely necessary to restore service as a remedial activity, else we are in change freeze until such time as a full review and debrief has been completed internally.
We apologise for the inconvenience caused to you and your business(es) during the day today. Rest assured our team will be working to fully understand all aspects of this incident to review our path forward to our new network core.

Monitoring Dec 7, 2023 3:39 PM NZDT

Network is now reaching stability and connectivity should be restoring for any customers still impacted.
If you are without connectivity, please turn off your router, turn back on, and wait 5 minutes.
We’re continuing to closely monitor all aspects of the network and will continue to keep this incident updated.

Problem Identified Dec 7, 2023 2:16 PM NZDT

Rollback is continuing and we are working through full restoration. Apologies for the inconvenience caused by this incident.

Problem Identified Dec 7, 2023 12:41 PM NZDT

Our network team has made the decision to rollback our network migration completed overnight.
This will cause an interruption to services while network elements are rolled back to their previous states.
Again, we apologise for the inconvenience caused. A full debrief of todays events will be conducted to determine the root cause of these issues before completing this migration again.
Updates to follow as this rollback is completed.

Investigating Dec 7, 2023 11:12 AM NZDT

Our team are continuing to work on this connectivity issue at the highest priority.
We apologise for the inconvenience caused while this is ongoing.

Investigating Dec 7, 2023 10:19 AM NZDT

Following our network migration overnight, some customers may be experiencing degraded routing or connection performance. Our team are investigating to resolve as soon as possible.
We apologise for the inconvenience caused.