ERI-SUPPORT-1001: Webshop orders fail with HTTP 503 and are very slow
Details
Status
IN PROGRESS
Severity
medium
Affected system
order-api
Created
Last updated
Description
Since about 09:10 customers get HTTP 503 errors when they place an order in the webshop, and the checkout takes over 30 seconds. It started shortly after this morning's release of the Order API.
Updates
Ticketing Tool
Ticket logged
Orchestrator
Resolution plan created: Customers get HTTP 503 errors and a slow checkout on order-api (severity medium) since a release this morning. A configuration or capacity change in that release is the likely cause. Establish the cause from logs and metrics, find the approved fix, and propose three reversible changes, one per Action Agent: roll back, scale up, restart.
Orchestrator
Perception: anomaly started 09:11, after release 2.14.3 reduced db.pool.maxSize from 50 to 10. The DB connection pool is exhausted (10 of 10 in use), causing 30s timeouts and HTTP 503. CPU and memory are normal. Confidence high.
Orchestrator
Knowledge search: runbook RB-114 and similar incident INC-20418 (similarity 0.91) recommend rolling back the release that changed the pool size. A restart is only needed if connections stay blocked after the rollback. Next: three reversible change proposals (rollback, scale up, restart), none executed.
Orchestrator
Action Agent proposed rolling back order-api to 2.14.2 with rollback-service (risk medium), not executed. I asked for the scale-up and the rolling restart twice each, and both times it returned the same rollback proposal. No separate scale-up or restart pipeline was proposed, so those two steps need a human decision. RB-114 says to restart only if connections stay blocked after the rollback.
