Script author here :-) This was due to two different GitHub Action workflows:
(Workflow 1): Remove the need-triage label under certain conditions.
(Workflow 2): If anyone outside a project maintainer removes a label, re-add it with a friendly message explaining why.
Submitted those at like 10 or 11 pm and went to sleep. Woke up to all issues that got changed overnight with dozens, hundreds, or thousands of these messages.
Cause: Workflow 2 should have checked for project maintainers but also other bots and automation that might also be clearing labels. It got fixed immediately once we realized the issue.
I learned the hard way to always implement a circuit breaker for event-driven triggers like this. We use a simple Redis counter with a short TTL to rate limit execution and fail fast if it detects a loop. It is standard practice in backend queues like Celery but easy to overlook in CI configurations.
(Workflow 1): Remove the need-triage label under certain conditions.
(Workflow 2): If anyone outside a project maintainer removes a label, re-add it with a friendly message explaining why.
Submitted those at like 10 or 11 pm and went to sleep. Woke up to all issues that got changed overnight with dozens, hundreds, or thousands of these messages.
Cause: Workflow 2 should have checked for project maintainers but also other bots and automation that might also be clearing labels. It got fixed immediately once we realized the issue.