A print queue that can’t print the same job twice

The problem

Campus printing works like this: you send a document to a shared queue, walk to the printer, and find it jammed, out of paper, or already printing someone else’s job. You send it again to a different printer. Pages pile up uncollected, and occasionally somebody walks off with yours.

The fix is to separate submitting a job from printing it. You upload from any device, and the job sits in a HELD state for as long as it needs to. Nothing is queued at any printer yet. When you’re standing at one, you release the job to that machine, and only then does it print.

That moves the interesting failure from the printer to the release. A student taps release twice. A phone on campus Wi‑Fi retries a request that actually succeeded. A queue hands the same message to a worker twice. Each of those, handled naively, is a second copy of somebody’s 40‑page thesis.

Constraints

• A printer has a fixed speed, so adding workers can’t drain a long queue. Each printer gets exactly one worker, and jobs wait their turn.
• Standard SQS can hand a worker the same message twice, so every step has to be safe to repeat.
• Two people can release the same job at the same moment. Exactly one may win, and we didn’t want a lock service that could become its own outage.
• The stack runs a few hours at a time on a course budget, so fixed monthly costs dominate. A NAT gateway alone would have been the biggest line item.
• AWS Academy allows one pre-made IAM role, so the Terraform had to fit inside it instead of creating its own roles.

Design

A Go (Gin) API runs as two tasks on ECS Fargate behind an Application Load Balancer. Uploads go to S3 and a job record goes into DynamoDB as HELD. Each printer has its own SQS queue, its own dead-letter queue, and one Python worker that long-polls it.

Every state change is a DynamoDB conditional write. Releasing a job is an update to RELEASED that only succeeds if the item is still HELD. If two releases race, DynamoDB serializes them; one succeeds and the other gets a ConditionalCheckFailedException, which the API returns as 409 Conflict. There is no lock to acquire, expire, or lose.

The worker uses the same idea. It claims a job by moving it from RELEASED to PROCESSING, and finishes by moving it to DONE. If a message arrives for a job that is already DONE, it deletes the message and does nothing. If it arrives for a job stuck in PROCESSING on a second delivery, the previous worker crashed mid-job and this one picks it up. That gives exactly-once printing on top of at-least-once delivery.

My part was the platform underneath. Nine Terraform modules (networking, ECR, IAM, DynamoDB, S3, SQS, ALB, ECS and CloudWatch) bring the whole stack up from nothing in about five minutes and tear it down with one command. Adding a printer is one entry in a list: a for_each creates its queue, dead-letter queue and worker service. I also moved the API onto a resilience layer: every AWS call sits behind a circuit breaker that opens after five consecutive failures and fails fast with 503 for 15 seconds, a token-bucket rate limiter caps traffic at 100 requests a second, and a bulkhead bounds concurrent work. DynamoDB’s conflict responses are excluded from the breaker’s failure count, so a busy release race never trips it.

Trade-offs

• Per-printer queues over one global queue. A jammed printer only blocks its own queue, and queue depth per printer is a free CloudWatch metric. The cost is more resources: six queues instead of two. SQS queues cost nothing when idle.
• Standard SQS over FIFO. Print order between different jobs doesn’t matter, and the worker has to be idempotent anyway to survive crashes. FIFO’s deduplication would have been redundant, with a lower throughput ceiling.
• No auto-scaling for workers. A printer is a physical device; another worker can’t make it faster. Under load, jobs wait and latency grows with queue depth. For virtual printers (print to PDF), scaling on queue depth would be the right call.
• Public subnets, no NAT gateway. That saved about $33 a month on a course budget. Workers have no listening ports and the API only accepts traffic from the load balancer, but in production I’d use private subnets with VPC endpoints for defence in depth.

How we tested it

Thirty-five unit tests run in GitHub Actions on every push, including the worker’s state machine (idempotency, redelivery, failure handling) and the Go handlers with the race detector on. On top of that we ran four experiments against the deployed stack: a Locust load test, a release race, a saturation test that sends 50 jobs to one printer, and fault injection that kills a printer task in the middle of a job.

Results

• Contention: 50 concurrent releases of the same job produced exactly one success, 49 409 conflicts and no server errors.
• Load: 50 users for 60 seconds sent 1,484 requests at about 25 a second with no failures. p50 settled near 110 ms and p99 stayed under 470 ms.
• Fault injection: after a printer task was killed mid-processing, ECS replaced it, the message reappeared when its visibility timeout expired, and all 10 jobs reached DONE with no duplicate prints. Recovery took about 60 to 95 seconds, most of it the visibility timeout.
• Cost: about $0.08 an hour while running, and nothing when torn down.

What I’d change

I’d shorten the base visibility timeout. The worker already extends it with a heartbeat while it prints, so a short base value would only cut the wait after a crash, from about a minute to seconds. I’d move to private subnets with VPC endpoints. And I’d write the latency budget and rollback criteria down before the experiments, so the load test had a pass or fail line instead of a result we interpreted afterwards.