Incident - Connectivity Disruption - 23/04

Postmortem

Starting approximately 2:58pm 23rd April, we began receiving reports of abnormal activity within a segment of our network.
The decision was made to assemble our action team at 3:00pm, which was completed at 3:06pm, and status page incident created at 3:11pm.
Investigations concluded at 3:17pm that a single network element was creating issues, which had full connectivity within the network, however lost connectivity to both of our metro route reflectors via IPv4, resulting in links appearing up and active, but return path traffic being unable to locate the destination path.
This impacted some select downstream peering sessions, and internal tools using this element as a primary path, with the largest impact being felt by internet subscribers and attempts to renew their DHCP leases being unsuccessful.
At 3:19pm the decision was made to reboot this network element, and also simultaneously complete a minor firmware upgrade.
This was completed at 3:21pm and our action team began confirming service restoration, which was completed by 3:31pm. Monitoring continued showing no further outlying impacts or issues.
Investigations will continue into why connections to these metro route reflectors will continue and any appropriate adjustments completed to ensure this doesn’t reoccur.
We apologise for any inconvenience experienced to our customers as a result of this incident.

Resolved

Monitoring has shown no abnormalities following restoration and this incident is now resolved.
Apologies for any interruptions to connectivity or destinations for the select customers who were impacted by this incident.
If you are experiencing any remaining issues, please restart your router before reaching out to our team.

Monitoring

Degraded connectivity caused by a degraded network element has been restored.
Our team are monitoring for any outliers to ensure full restoration.

Investigating

Some customers may be experiencing issues accessing specific destinations or websites.
Our team are investigating and updates to follow.

10 Affected Services: