Incident - Line Card Reset - 23/11 - Status Page

Resolved Nov 23, 2023 8:18 AM NZDT

Services have remained stable since last update and we are comfortable this issue is now resolved.
Root cause investigations will now begin to determine the initial cause of this incident and subsequent issues that followed, which will then be implemented in our ongoing network upgrades.
If you are still without service, please power down your router, turn back on and wait 5 minutes.
We again apologise for the inconvenience caused and appreciate your patience while our teams worked through this complicated incident.

Monitoring Nov 23, 2023 7:22 AM NZDT

Database connectivity has been restored and remaining impacted connections are observed to be restoring to service.
If you are without service, please attempt to turn off and on your router and wait 5 minutes.
Our team are continuing to monitor to confirm services are restored.

Investigating Nov 23, 2023 7:17 AM NZDT

Core network has obtained stability. Core database nodes servicing authentication for circuits are encountering communication issues which is preventing new sessions from connecting.
We are continuing to work through to resolution as a top priority, and apologise for the inconvenience caused.

Updated Nov 23, 2023 6:58 AM NZDT

We are observing increased numbers of customers impacted due to a supplemental card reset.
We are continuing to work through this incident as our top priority.

Problem Identified Nov 23, 2023 6:42 AM NZDT

Continued impacts have been observed for some customers, our team are continuing to fully resolve this incident and apologise for any inconvenience.

Problem Identified Nov 23, 2023 5:55 AM NZDT

Root cause has been identified and appears to be a switch port malfunction.
A patch has been implemented and services are showing restoration, with the majority of customers now back in service.
We are aware of some isolated cases of customers without service and are continuing to work through these instances.

Problem Identified Nov 23, 2023 5:10 AM NZDT

Our team are continuing to monitor restoration post-reset.

Investigating Nov 23, 2023 4:35 AM NZDT

A core network element has performed a line card reset which has resulted in some instability for customers traversing this network segment.
Our team are investigating and will perform remedial activities as required to restore full service. We apologise for any inconvenience.