Lumen codfw-ulsfo down (Nov 2024)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ayounsi
	Thu, Nov 21, 10:18 AM

Event Timeline

ayounsi triaged this task as High priority.Thu, Nov 21, 10:18 AM

ayounsi created this task.

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptThu, Nov 21, 10:18 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Automated diagnostic for Netbox circuit ID 102

Interface cr4-ulsfo:xe-0/1/1

admin-status: up
⚠️ oper-status: down
interface-flapped: 2024-11-20 17:04:03 UTC (17:15:19 ago)
⚠️ errors: {'input-errors': 1077, 'framing-errors': 1077, 'carrier-transitions': 136, 'output-errors': 24}
laser-output-power: 0.7030
laser-output-power-dbm: -1.53
rx-signal-avg-optical-power: 0.0004
⚠️ rx-signal-avg-optical-power-dbm: -33.98

Ticket ID 30654682 has been successfully created.

Seems to be back up:

Nov 21 13:51:02  cr4-ulsfo rpd[29932]: RPD_OSPF_NBRUP: OSPF neighbor 198.35.26.203 (realm ospf-v2 xe-0/1/1.0 area 0.0.0.0) state changed from Exchange to Full due to ExchangeDone (event reason: DBD exchange of slave completed)

I guess we can wait for confirmation on the ticket before closing.

From Lumen:

2024-11-21 12:38:45 GMT - Splicing has commenced at the south end. Service affecting alarms are expected to begin clearing.

So sounds like it was a cut on a multi-core fiber, clearly ours has been fixed as it's been stable for over an hour now:

cmooney@cr4-ulsfo> show ospf neighbor interface xe-0/1/1.0 detail    
Address          Interface              State           ID               Pri  Dead
198.35.26.203    xe-0/1/1.0             Full            208.80.153.192   128    37
  Area 0.0.0.0, opt 0x52, DR 0.0.0.0, BDR 0.0.0.0
  Up 01:21:04, adjacent 01:21:03

Lumen codfw-ulsfo down (Nov 2024)Closed, ResolvedPublicActions

Event Timeline

Automated diagnostic for Netbox circuit ID 102

Lumen codfw-ulsfo down (Nov 2024)
Closed, ResolvedPublic
Actions