ERI-SUPPORT-1006: Webshop orders fail with HTTP 503 and are very slow
Details
Status
IN PROGRESS
Severity
medium
Affected system
order-api
Created
Last updated
Description
Since about 09:10 customers get HTTP 503 errors when they place an order in the webshop, and the checkout takes over 30 seconds. It started shortly after this morning's release of the Order API.
Updates
Ticketing Tool
Ticket logged
Orchestrator
Resolution plan created: Customers get HTTP 503 errors and a slow checkout on order-api (severity medium) since a release this morning. A configuration or capacity change in that release is the likely cause. Establish the cause from logs and metrics, find the approved fix, and propose three reversible changes, one per Action Agent: roll back, scale up, restart.
Orchestrator
Perception: release 2.14.3 (09:05) cut db.pool.maxSize from 50 to 10; pool exhausted (10/10), 503s and 30s timeouts since 09:11, error rate 18.2%. CPU and memory normal.
Orchestrator
Knowledge: RB-114 and similar incident INC-20418 recommend rolling back the release that changed the pool size. A restart is advised only if connections stay blocked after the rollback. Now requesting the three proposals (rollback, scale up, restart).
Orchestrator
Action Agent proposed rollback-service to 2.14.2 in production (medium risk). The scale-up and restart steps, retried once each, returned the same rollback proposal, so no separate scale or restart pipeline was proposed. Nothing has been executed. All proposals need critique, human approval and Policy Engine.
