Cisco Catalyst 9000 Series switches have become the switch of choice for many enterprises, including the environment that I work in, where Cisco Catalyst 9300 24- and 48-port switches running Gibraltar-16.12.3 code had become the standard for the access layer when more than 12 ports were needed.
That was about two years ago, and a year or so after that we began receiving notifications from an onsite location that there were intermittent network outages and performance degradation at the site. This is an account of how we found workarounds to the problem until Cisco provided a permanent fix.
We started troubleshooting the issue and found the following syslog messages that we had never seen before: