OCF 2025 Performance Incidents

July ?

Spanning Tree Incident - Looped non-IT switch

July ?

Possible L3 routing problems with the removal of the traffic shaping on WBR and/or ROUTE

July 11th ~6PM

Impact:

• SIP phones
• WB office
• Main Stage (video, etc.)

Timeline

1. Clif saw it go down. He did a soft reload on WBR
2. Hard reload on WBR
3. Then he cold rebooted on both switches, and things came back up.

Clif spoke with Darren about the SIP phones and was informed that they are currently used for production purposes. He was under the impression that this was a “trial”.

TODO

July 11th - 11:30-ish

Impact:

• SIP phones

Timeline:

1. Shell TXT Shellbell and saw it post midnight after getting out of the Ritz and saw the TXT. He called Fair Central.
2. Clif came by and power cycled the ATAs and the switch. Noticed the ‘lights” slowly came back up. Clif tested the SIP phone at the office, and it worked.
3. There is a DID with an urgent inbound call to OCF. (Eg. The Sheriff uses this.) This was tested for inbound calls.

Todo

• Understand how the SIP is being deployed.
• We need a failure analysis on the SIP rollout to have higher reliability.
• There was a “process” that was missing about conveying how this was deployed, as IT didn’t understand that the SIP deployment was mission-critical.