How we diagnose and fix a "too many connections" incident for Cloud Run + Postgres

“Too many connections” names where the system finally refused more work. It doesn’t identify what created the pressure.

Cloud Run may have multiplied an oversized pool. A rollout may have overlapped revisions. Connections may be leaking. Long transactions or slow queries may be keeping ordinary sessions alive too long. Retries may be widening the fleet while Postgres is already saturated.

The incident response has three jobs: contain growth, preserve evidence, and classify the mechanism before making the temporary fix permanent.

Contain new pressure

Stop the system from increasing its claim on Postgres.

Reduce the affected service’s maximum instances when the fleet is still widening. Pause backfills, workers, admin jobs, or retry loops that aren’t required to restore the primary path. If one revision introduced the change, route traffic back or stop its rollout.

Shorten connection-acquisition waits so requests fail before they accumulate as a hidden queue inside the application. Shed non-essential traffic if the database can’t serve everything safely.

Killing sessions or restarting instances may be necessary, but first capture enough state to understand what will disappear with them.

Preserve the shape of the incident

Record a timestamped snapshot of:

• Cloud Run instance count by revision
• deployment and traffic-split changes
• request concurrency, latency, and retry rate
• pool size, checked-out sessions, waiters, and acquisition latency
• total Postgres sessions by application name and state
• oldest transactions, blocked sessions, and long-running queries
• database CPU, memory, I/O, and connection ceiling

Use a distinct PostgreSQL application_name per service and revision where practical. Otherwise several elastic workloads arrive as one anonymous client and the first useful question becomes needlessly expensive.

A basic classification query is enough to start:

select
  application_name,
  state,
  wait_event_type,
  wait_event,
  count(*) as sessions,
  max(now() - xact_start) as oldest_transaction
from pg_stat_activity
where datname = current_database()
group by application_name, state, wait_event_type, wait_event
order by sessions desc;

Then inspect the oldest transactions and queries for the groups that explain the pressure.

Classify the mechanism

Fleet multiplication

Instance count and session count rise together after traffic, retries, or a rollout. Each instance may be behaving exactly as configured. The fleet-wide connection claim was never safe.

Contain with a lower scale cap. Repair the connection budget, pool size, revision-overlap allowance, and retry behavior.

Oversized per-instance pools

The fleet is modest but every instance opens or reserves too many sessions. Framework defaults and multiple worker processes are common sources. A configured pool of five multiplied across four processes is twenty, not five.

Shrink the pool and expose pool metrics. Database-heavy requests don’t need a session per concurrent HTTP request.

Connection leak

Session count trends upward without matching useful traffic and falls after restarts. Inspect error paths, cancelled futures, transaction wrappers, and code that checks out a connection before work that can return early.

Fix ownership in the application. An idle timeout may reduce damage, but it doesn’t repair leaked lifecycle.

Long or idle transactions

A smaller number of sessions hold transactions, locks, or snapshots for too long. Waiting work then consumes the remaining pool and makes the incident look like pure connection volume.

Remove external calls and non-database processing from transaction scope. Set transaction and statement timeouts appropriate to the workload. Investigate lock chains before terminating sessions indiscriminately.

Slow queries or database saturation

Queries take longer, so each connection remains occupied longer. Pool waiters grow, requests slow down, and Cloud Run may add instances, creating a feedback loop.

Fix the query, index, lock, I/O, or capacity problem first. Increasing pool size gives a saturated database more concurrent work and often makes completion slower.

Separate containment from repair

Temporary actions may include reducing max instances, pausing secondary traffic, terminating clearly stuck sessions, restarting a leaking revision, or raising the database ceiling to restore access.

A durable repair changes the contract:

• explicit fleet-wide connection budgets
• smaller pools and short acquisition deadlines
• bounded Cloud Run scaling
• separate budgets for API, workers, jobs, migrations, and admin access
• tighter transaction scope
• corrected queries and indexes
• idempotent retries with backoff
• asynchronous execution for work that shouldn’t hold a request and session
• a pooler only when measured session churn or multiplexing justifies it

Raising max_connections can be part of capacity planning. It isn’t a substitute for knowing the maximum claim each workload can create.

Prove the fix under the failure shape

Reproduce the relevant burst, rollout overlap, slow query, or retry pattern in a controlled environment. Watch instance count, session count, pool acquisition latency, request failures, transaction age, and database saturation.

The service should degrade at an intentional boundary. It may queue briefly, reject work, or hand it to an asynchronous path. Postgres should retain headroom for admin access and recovery.

Close the incident only when the team can explain why connection growth is bounded and what will happen at the boundary. A restart that made the graph prettier restored service. It didn’t explain the failure.

The error came from Postgres. The system allowed it. The repair belongs wherever that uncontrolled pressure began.