Spike: Tiered status of alerts

[matt4su]

Issue Description

The purpose of this ticket is to track multiple categories of alerts to provide better insight into which are actionable and which are non actionable. Currently responding to alerts from production monitors is time consuming, and the majority of alerts are non-actionable.

Guidance on the approach from Steve Albers here

Note:
Steve and Sanja would like us to tackle this late in December

Tasks

• Review guidance from rom Steve Albers here
• Update the system to include a separate Logger statement that tracks three categories of errors for applications
• Update existing Monitors to more clearly delineate between these three types of issues
• Plan a leadership review of SLOs with Product Owners on a scheduled basis to see if the Product team wants to help address systemic problems with API partners.

Acceptance Criteria

• The systems and monitors are updated per documented guidance.
• Leadership review is planned

---

[mjknight50]

This is being addressed in various ways but should remain open.
One way it is being addressed is
department-of-veterans-affairs/vets-website#35027