Queue tuning notes: from 40 seconds to 400ms

A client's nightly import was drifting later and later into the morning — 40 seconds per job, thousands of jobs. The reflex answer is horizontal: more workers. Here's why we didn't, and the three changes that actually fixed it.

1. Stop hydrating the world

The job's constructor was accepting a full Eloquent model, which meant every payload serialised the model and its loaded relations — some of them huge. Passing the ID and re-fetching inside handle() cut payload size by 98%.

// Before: serialises the model + 4 eager-loaded relations
public function __construct(public Shipment $shipment) {}

// After: 8 bytes of payload
public function __construct(public int $shipmentId) {}

2. Chunk the writes, not just the reads

Everyone chunks reads. The import was still doing one INSERT per row inside a loop — 3,000 round trips. A single upsert() with 1,000-row chunks brought DB time from 31s to 1.2s.

3. Let the database do the diffing

The job compared incoming rows against existing ones in PHP. Replacing that with a WHERE ... NOT IN anti-join let MySQL skip 90% of rows before PHP ever saw them.

The scoreboard

The lesson that keeps re-teaching itself: before scaling out, look at what each unit of work is actually doing. It's usually doing too much.