Incident - bng3 connectivity issue - 13/03 - Status Page

Postmortem Mar 13, 2024 4:30 PM NZDT

Due to a firmware bug, unfortunately a single routing filter line which distributes routes between bng3-quic and our core was not being respected by our core network elements.
This line resulted in a subset of customers on bng3-quic, within a single /24 subnet, to have reduced, or no connectivity while this issue was located and routing filter line removed and re-added.
We have been in touch with our vendor and advised this bug has been fixed in a beta firmware release. Once released into a stable version, we will complete the necessary upgrades to this version to ensure this doesn’t reoccur.
We apologise to impacted customers for the inconvenience caused by this incident.

Resolved Mar 13, 2024 1:19 PM NZDT

Issue confirmed resolved as at 10:00am today. No further occurrences since monitoring.

Monitoring Mar 13, 2024 9:56 AM NZDT

We are observing impacted circuits recovering and are monitoring for any further occurrences.

Problem Identified Mar 13, 2024 9:44 AM NZDT

We have detected an issue impacting some customers connected to bng3-quic are experiencing connectivity issues.
We are working through draining these customers onto an alternate BNG to provide resolution.