Cloud Run Google Cloud Serverless Background Jobs DevOps

Leveraging Google Cloud Run for Auto-Scaling Background Workers

Core Concept LearningAugust 3, 20265 min read

A background worker that resizes uploaded images looks simple until traffic is spiky: quiet all morning, then two hundred uploads in the same minute after a marketing email goes out. A fixed pool of worker VMs is either oversized (idle all morning, wasted cost) or undersized (a queue that backs up for twenty minutes after the email send). Cloud Run's scale-to-zero and scale-out-on-demand model exists precisely for this shape of workload — instances appear when there's work and disappear when there isn't, without capacity planning for a load pattern nobody can predict a week in advance.

This guide builds one concrete pipeline: an image-resize worker triggered by a Pub/Sub message, running as a Cloud Run service that scales from zero to N instances under load. Every section adds a production concern the demo path skips — concurrency limits, the idempotency requirement that Pub/Sub's at-least-once delivery makes mandatory, and the difference between a Cloud Run service and a Cloud Run job that decides which one you actually want. For the event-driven system this worker plugs into, see Event-driven architecture patterns.

Cloud Run service vs. job for background work

Service, job, or queue-driven worker

Cloud Run offers two distinct primitives that get conflated in casual conversation. A Cloud Run service is request-driven: it accepts HTTP requests and scales based on concurrent requests in flight, including scaling to zero when idle. A Cloud Run job is task-driven: it runs a fixed number of tasks to completion and exits — no inbound HTTP, no scale-to-zero-and-wait, just "run this batch and stop."

For an image-resize worker triggered by new uploads arriving continuously, a service behind a Pub/Sub push subscription is the right fit — HTTP requests arrive as messages are published, and Cloud Run scales instances to match. For a nightly batch reprocessing job with a known, finite input set, a Cloud Run job is the better fit — there's no ongoing request stream to scale against, just a task to run and confirm finished.

Quick reference

Cloud Run service: request-driven, scales on concurrent requests, scale-to-zero when idle — fits a continuous stream of triggering events.
Cloud Run job: task-driven, runs to a fixed completion, no scaling logic to reason about — fits a known, bounded batch.
A push subscription from Pub/Sub to a Cloud Run service turns "new message published" into "new HTTP request received."
Choosing the wrong primitive shows up as either a service that never scales to zero (because you modeled a batch as continuous) or a job that can't react to new work arriving mid-run.

Remember this

A continuous stream of triggering events fits a Cloud Run service behind a push subscription; a fixed, known batch fits a Cloud Run job — modeling one as the other produces either wasted idle capacity or a worker that can't react to new work.

One message from publish to completion

Trace one upload through the pipeline: a client uploads an image, your API publishes a message to a Pub/Sub topic with the object's storage path, a push subscription delivers that message as a signed HTTP POST to the Cloud Run service, and the service resizes the image and writes the result back to storage. Cloud Run scales instances up as concurrent pushes increase and back down to zero once uploads stop.

The part most first attempts get wrong is the acknowledgment contract: Pub/Sub expects an HTTP 200 to acknowledge the message within the subscription's ack deadline. Return anything else — a 500, a timeout, a crash — and Pub/Sub will redeliver the same message, which is exactly why the handler must be idempotent (next section) rather than assuming each message arrives exactly once.

One background job from publish to completion

Quick reference

Configure the push subscription with a dedicated service account and verify its OIDC token — an unauthenticated public endpoint will process forged messages.
Return 200 only after work genuinely succeeds; returning 200 early to "stop retries" silently drops failed work.
Set min-instances above zero only if cold-start latency for the first message after idle is unacceptable for your SLA — it costs continuous baseline spend.
Configure --concurrency based on whether the resize operation is CPU-bound (low concurrency) or I/O-bound waiting on storage (higher concurrency is safe).

Push handler — no idempotency, no explicit ack semantics

1import express from "express";2 3const app = express();4app.use(express.json());5 6app.post("/pubsub-push", async (req, res) => {7  const message = JSON.parse(8    Buffer.from(req.body.message.data, "base64").toString()9  );10  await resizeImage(message.objectPath); // if this throws, message redelivers11  res.status(200).send();12});

Verify the push subscription's OIDC token first

1import { OAuth2Client } from "google-auth-library";2 3const authClient = new OAuth2Client();4const EXPECTED_AUDIENCE = process.env.PUSH_ENDPOINT_URL!;5 6async function verifyPushRequest(authHeader?: string) {7  if (!authHeader?.startsWith("Bearer ")) throw new Error("missing_token");8  const token = authHeader.slice("Bearer ".length);9  const ticket = await authClient.verifyIdToken({10    idToken: token,11    audience: EXPECTED_AUDIENCE,12  });13  return ticket.getPayload();14}15 16app.post("/pubsub-push", async (req, res) => {17  try {18    await verifyPushRequest(req.header("authorization"));19  } catch {20    return res.status(401).send(); // Expected: unsigned requests are rejected, not processed.21  }22  // ...proceed to handler23});24// Break it: send a POST without the Authorization header —25// expected: 401, not a resize attempt on unverified input.

Remember this

Only return HTTP 200 once the work has actually completed — returning early to silence retries doesn't fix the failure, it just makes it invisible until someone notices the output never arrived.

Pub/Sub's at-least-once delivery makes idempotency mandatory

Pub/Sub guarantees at-least-once delivery, not exactly-once — a message can be redelivered after a crash, a timeout, or a network blip between Cloud Run acknowledging and Pub/Sub recording that acknowledgment. If the resize handler is not idempotent, a redelivered message means the same image gets resized and stored twice, or worse, a payment or notification handler double-charges or double-sends.

The fix is a deterministic idempotency key derived from the message content (the object path, or Pub/Sub's own message ID) checked against a small store before doing the real work. If the key was already processed, acknowledge immediately without repeating the side effect.

Zoom: one message redelivery after a crash

Quick reference

Use Pub/Sub's messageId as the idempotency key when the message itself is unique per event; derive your own key when messages can legitimately repeat with the same ID for different logical events.
Expire idempotency keys after a bounded window (hours to a day) — don't grow the dedup store unbounded.
A dedup store outage should fail toward re-processing (safe if idempotent) rather than silently dropping messages.
Idempotency is required for any handler with a side effect (writes, charges, sends) — a pure read-only handler doesn't need it.
This is the same guarantee gap covered generally in Exactly-once delivery: myth vs reality — Pub/Sub is not an exception to it.

No idempotency — a crash mid-resize causes a duplicate

1app.post("/pubsub-push", async (req, res) => {2  const { objectPath } = decodeMessage(req.body);3  await resizeImage(objectPath);           // crashes here...4  await storage.write(`resized/${objectPath}`, /* ... */);5  res.status(200).send();6  // ...Pub/Sub redelivers, resizeImage runs again on the same object.7});

Idempotency key backed by a small key-value store

1async function alreadyProcessed(messageId: string): Promise<boolean> {2  const existing = await redis.get(`processed:${messageId}`);3  return existing !== null;4}5 6async function markProcessed(messageId: string) {7  await redis.set(`processed:${messageId}`, "1", { EX: 60 * 60 * 24 });8}9 10app.post("/pubsub-push", async (req, res) => {11  const { message } = req.body;12  const messageId = message.messageId as string;13 14  if (await alreadyProcessed(messageId)) {15    return res.status(200).send(); // Expected: duplicate acknowledged, no re-work.16  }17 18  const { objectPath } = decodeMessage(req.body);19  await resizeImage(objectPath);20  await storage.write(`resized/${objectPath}`, /* ... */);21  await markProcessed(messageId);22  res.status(200).send();23});24// Break it: manually re-POST the same message body twice —25// expected: the second call skips resizeImage and still returns 200.

Remember this

At-least-once delivery means a redelivered message is a normal, expected event, not an edge case — every handler with a side effect needs a deterministic idempotency key checked before the side effect runs, not after.

Key takeaway

Build the pipeline against a stub image-resize handler: publish a Pub/Sub message with an object path, confirm the push subscription delivers it to your Cloud Run service, and confirm the resized output appears in storage. Verify the OIDC token check rejects an unsigned request with a 401.

Then force the redelivery case directly: re-publish the identical message (or replay the same push payload) and confirm your idempotency key check skips the resize the second time, still returning 200. Pass criterion: the first delivery produces exactly one resized output, the duplicate delivery produces zero additional side effects, and an unauthenticated push request is rejected before it reaches the handler at all.

Enterprise Claude Code

Enterprise Claude Apps Gateway: Deploying Agents on AWS & GCP

Jul 31, 2026 · 4 min read

As AI coding tools transition from individual developer utility to organization-wide engineering infrastructure, enterpr…

Read

Gemini Firebase

Integrating Gemini AI Logic into Firebase Functions – Best Practices

Aug 3, 2026 · 9 min read

Calling the Gemini API from a Firebase Cloud Function looks like three lines of code — grab the API key, send a prompt, …

Read

Security Microservices

Security & Zero Trust Microservices: mTLS Guide