Batch AVIF Conversion with Sharp in CI

A single AVIF encode is trivial. Five thousand of them inside a CI job is a resource-scheduling problem: libaom peaks around 310 MB of resident memory per concurrent encode, a ubuntu-latest runner gives you 4 vCPU and 16 GB, and the naive await Promise.all(files.map(toAvif)) will either thrash the machine or get the Node process SIGKILLed with no error text. This page is the batch-execution layer that sits on top of the pipeline described in Sharp and FFmpeg media build pipelines: how to pick a pool width from measured memory rather than from os.cpus().length, how to split the batch across matrix jobs without destroying cache hit rates, and how to make one corrupt master fail one file instead of the whole run.

The target is a batch that is bounded (peak RSS stays under the runner’s ceiling), observable (you can see progress and an ETA while it runs), resumable (a cancelled job does not start from zero), and fault-isolated (a Git LFS pointer file that never got pulled does not abort 4,999 good encodes).

Prerequisite checklist

Step 1 — Derive the pool width from measured RSS, not core count

The instinct is to set the pool to os.availableParallelism(). That is the CPU bound. AVIF batches are usually bound by memory first, and the two ceilings are far apart. Measure both, then take the minimum.

On a 4 vCPU / 16 GB runner encoding 4032×3024 JPEG masters down to 1600px AVIF at quality: 50, effort: 4, throughput plateaus at a pool width of four while peak RSS keeps climbing linearly:

Throughput plateaus at four workers while memory keeps climbing Horizontal bars show encodes per minute for pool widths of 1, 2, 3, 4, 6, 8 and 12 on a four vCPU runner. Throughput rises from 77 to 265 encodes per minute between widths one and four, then flattens at 271 for width six and falls back to 241 at width twelve. A right-hand column lists peak resident memory rising steadily from 0.4 GB to 3.7 GB across the same range. AVIF effort 4, 1600px derivatives, 4 vCPU / 16 GB runner pool encodes / minute peak RSS 1 77 0.4 GB 2 150 0.7 GB 3 218 1.0 GB 4 265 1.3 GB 6 271 1.9 GB 8 262 2.5 GB 12 241 3.7 GB Width 4 buys 3.4x the throughput of width 1. Width 12 buys nothing and costs 2.8x the memory. Same numbers on a 2 GB container: width 6 is already past the cgroup limit and the run is killed.

Two things follow. First, past the core count you are only paying for context switches — the encoder is CPU-saturated, not latency-bound, so there is no idle time for extra workers to fill. Second, the memory ceiling is what actually kills builds, and it is the one people never measure because os.totalmem() inside a container reports the host’s memory, not the cgroup limit. The pool width should be:

pool = min( availableParallelism(),
            floor((cgroupLimit - 512 MB headroom) / measuredPeakPerWorker) )

For a 2 GB container that is floor((2048 - 512) / 310) = 4 by memory but capped to 2 by CPU — so 2. For the 16 GB / 4 vCPU runner it is 4 by CPU. The memory term only binds on small containers, but that is exactly where builds mysteriously die.

Step 2 — The batch script

One file, no dependencies beyond sharp. It reads a shard assignment from the environment, keeps a fixed-width pool, isolates every failure, appends a ledger line per completed unit so a cancelled run resumes, and prints a heartbeat so a 40-minute step does not look hung.

// scripts/avif-batch.mjs — run: node scripts/avif-batch.mjs media/**/*.{jpg,png}
import { createHash } from 'node:crypto';
import { readFile, writeFile, appendFile, mkdir, rename, stat, open } from 'node:fs/promises';
import { existsSync, readFileSync } from 'node:fs';
import { dirname, relative } from 'node:path';
import os from 'node:os';
import sharp from 'sharp';

// --- global libvips configuration -------------------------------------------
// One libvips thread per pipeline. We parallelise at the task level below, so
// per-operation threading would only multiply RSS and add sync overhead.
sharp.concurrency(1);
// The libvips operation cache memoises intermediate results across calls. Every
// input in a batch is distinct, so its hit rate is 0% and it is pure resident
// memory. Turn it off before the first sharp() call, not after.
sharp.cache(false);
// Decompression-bomb guard: 16384^2 pixels. A 60000x60000 PNG in a user-supplied
// catalogue would otherwise try to allocate ~14 GB and take the runner with it.
const MAX_PIXELS = 268402689;

// --- pool sizing -------------------------------------------------------------
// cgroup v2 exposes the *real* limit inside a container; os.totalmem() does not.
function memoryLimitBytes() {
  for (const p of ['/sys/fs/cgroup/memory.max', '/sys/fs/cgroup/memory/memory.limit_in_bytes']) {
    if (existsSync(p)) {
      const v = readFileSync(p, 'utf8').trim();
      if (v !== 'max' && Number(v) > 0 && Number(v) < 2 ** 53) return Number(v);
    }
  }
  return os.totalmem();
}
const PEAK_PER_WORKER = 320 * 1024 ** 2;   // measured, not guessed — see Step 1
const HEADROOM        = 512 * 1024 ** 2;   // Node heap + the rest of the build
const POOL = Number(process.env.AVIF_POOL) ||
  Math.max(1, Math.min(
    os.availableParallelism(),
    Math.floor((memoryLimitBytes() - HEADROOM) / PEAK_PER_WORKER)
  ));

// --- deterministic sharding --------------------------------------------------
// Hash the PATH, not the array index. Index modulo reshuffles every assignment
// the moment a file is added or removed, which empties every shard's cache.
const SHARD_INDEX = Number(process.env.SHARD_INDEX ?? 0);
const SHARD_TOTAL = Number(process.env.SHARD_TOTAL ?? 1);
const shardOf = (p) =>
  parseInt(createHash('sha1').update(relative(process.cwd(), p)).digest('hex').slice(0, 8), 16)
    % SHARD_TOTAL;

// --- ledger ------------------------------------------------------------------
// Append-only JSONL. Each line is one finished unit of work. On resume we skip
// anything already listed, so a cancelled 40-minute job restarts near where it
// stopped instead of at zero.
const LEDGER = `.cache/avif-ledger-${SHARD_INDEX}.jsonl`;
const done = new Set(
  existsSync(LEDGER)
    ? readFileSync(LEDGER, 'utf8').split('\n').filter(Boolean).map((l) => JSON.parse(l).out)
    : []
);

const WIDTHS = [400, 800, 1200, 1600, 2400];
// 4:2:0 halves chroma resolution — invisible on photographs, destructive on
// screenshots and coloured text. `classify()` below picks per file.
const PHOTO = { quality: 50, effort: 4, chromaSubsampling: '4:2:0' };
const GRAPHIC = { quality: 62, effort: 4, chromaSubsampling: '4:4:4' };

async function encodeOne(src, width) {
  const out = `dist/media/${relative('media', src).replace(/\.\w+$/, '')}-${width}.avif`;
  if (done.has(out)) return { out, status: 'resumed' };

  const img = sharp(src, {
    sequentialRead: true,   // stream scanlines instead of random-access; ~half the RSS
    limitInputPixels: MAX_PIXELS,
    failOn: 'error'         // reject truncated files instead of emitting a grey frame
  });
  const meta = await img.metadata();
  if (width > meta.width) return { out, status: 'skipped-upscale' };

  const opts = meta.channels >= 3 && meta.density !== 72 ? PHOTO : classify(meta);
  await mkdir(dirname(out), { recursive: true });
  const tmp = `${out}.${process.pid}.tmp`;

  const { size } = await img
    .rotate()                                        // bake in EXIF orientation first
    .resize({ width, withoutEnlargement: true })
    .withIccProfile('srgb')                          // sharp strips ICC by default;
                                                     // a P3 master without it renders flat
    .avif({ ...opts, bitdepth: 8 })                  // 10-bit doubles encode time here
    .toFile(tmp);

  // AVIF is not universally smaller. Flat UI graphics and small PNGs often lose
  // to WebP or to the original. Reject rather than ship a regression.
  const srcSize = (await stat(src)).size;
  if (width === meta.width && size >= srcSize) {
    return { out, status: 'rejected-larger', size, srcSize };
  }

  await rename(tmp, out);   // atomic within one filesystem: a killed job never
                            // leaves a truncated file that looks like a cache hit
  await appendFile(LEDGER, JSON.stringify({ out, size, at: Date.now() }) + '\n');
  return { out, status: 'encoded', size };
}

function classify(meta) {
  // Palette PNGs and images with an alpha channel are almost always UI graphics.
  return meta.palette || meta.hasAlpha ? GRAPHIC : PHOTO;
}

// --- driver ------------------------------------------------------------------
const sources = process.argv.slice(2).filter((p) => shardOf(p) === SHARD_INDEX);
const jobs = sources.flatMap((src) => WIDTHS.map((w) => ({ src, w })));
const failures = [];
const counts = { encoded: 0, resumed: 0, 'skipped-upscale': 0, 'rejected-larger': 0 };
let cursor = 0;
const started = Date.now();

// Heartbeat: a CI step that prints nothing for ten minutes reads as hung to
// every human looking at it, and some runners reap silent steps outright.
const beat = setInterval(() => {
  const pct = ((cursor / jobs.length) * 100).toFixed(1);
  const rate = cursor / ((Date.now() - started) / 1000);
  const eta = rate > 0 ? Math.round((jobs.length - cursor) / rate) : 0;
  console.log(`[avif] shard ${SHARD_INDEX}/${SHARD_TOTAL} ${cursor}/${jobs.length} (${pct}%) eta ${eta}s rss ${(process.memoryUsage.rss() / 1e6) | 0}MB`);
}, 15_000).unref();

await Promise.all(Array.from({ length: POOL }, async () => {
  while (cursor < jobs.length) {
    const job = jobs[cursor++];              // single-threaded JS: no lock needed
    try {
      const r = await encodeOne(job.src, job.w);
      counts[r.status] = (counts[r.status] ?? 0) + 1;
    } catch (err) {
      // Per-file isolation. One LFS pointer, one CMYK TIFF libvips cannot open,
      // one zero-byte file must not abort the other 4,999 encodes.
      failures.push({ src: job.src, width: job.w, message: err.message });
    }
  }
}));
clearInterval(beat);

console.log(`[avif] pool=${POOL} ${JSON.stringify(counts)} failures=${failures.length}`);
await writeFile(`.cache/avif-failures-${SHARD_INDEX}.json`, JSON.stringify(failures, null, 2));
// Fail the step once, at the end, with the full list — not on the first bad file.
if (failures.length) {
  for (const f of failures) console.error(`  ✗ ${f.src} @${f.width}: ${f.message}`);
  process.exitCode = 1;
}

Wire it into a matrix so wall clock divides by the shard count. The shards share one cache path and never collide, because every output name is derived from the source path:

      - name: Convert to AVIF
        strategy:
          matrix:
            shard: [0, 1, 2, 3]
        run: node scripts/avif-batch.mjs media/**/*.{jpg,png}
        env:
          SHARD_INDEX: ${{ matrix.shard }}
          SHARD_TOTAL: '4'
          UV_THREADPOOL_SIZE: '4'   # must be >= pool width or sharp() calls queue
Path-hash sharding across four matrix jobs A list of master files enters a hash function that takes the first eight hex digits of the SHA-1 of the file path modulo four. Each resulting shard number routes the file to one of four parallel matrix jobs, each running its own bounded worker pool and appending to its own ledger file. All four upload derivative artifacts that merge into a single dist directory. A note explains that adding a new file changes only that file's shard. 5,214 masters media/a/hero.jpg media/a/tile.png media/b/spec.jpg … media/z/logo.png sha1(path)[0:8] mod 4 shard 0 — pool 4 1,309 masters, own ledger shard 1 — pool 4 1,288 masters, own ledger shard 2 — pool 4 1,331 masters, own ledger shard 3 — pool 4 1,286 masters, own ledger dist/ media Hashing the path: adding one master moves exactly one file. Cache hit rate on the other three shards stays 100%. Hashing the array index: adding one master reshuffles ~75% of assignments and every shard starts cold.

Tradeoff: sharding divides wall clock but multiplies cache-restore cost — each of the four jobs restores and saves its own copy of the derivative cache. Below roughly 300 derivatives the restore overhead exceeds the encode time you saved, and a single job is faster. Shard when the batch is large enough that encoding, not I/O, dominates.

Step 3 — Understand what each job can do

Every unit of work has more than two outcomes, and conflating them is what turns a green build into shipped regressions. rejected-larger is a success for the pipeline and a failure for the file; skipped-upscale is neither. Model them explicitly:

Lifecycle of one AVIF encode job A state machine. A planned job either resolves from the ledger as resumed, is skipped when the requested width exceeds the master width, or enters the encoding state. Encoding leads to a size check on success, or to failure recorded in the failures file on a decode error. The size check commits the file with an atomic rename when the AVIF is smaller than the source, and rejects it otherwise so the WebP fallback ships instead. planned src × width in ledger width > master else resumed skipped encoding → .tmp file throws — LFS pointer, CMYK, truncated ok size check avif < source ? yes no committed rename + ledger rejected ship WebP instead failed collected, step exits 1 at the end

The rejected branch is the one most batches omit, and it matters more than it sounds. AVIF’s entropy coder is tuned for photographic content; a 6 KB flat-colour PNG logo frequently comes out at 9 KB as AVIF because the container overhead alone (ftyp + meta + iprp boxes) is around 300 bytes before a single pixel is coded. Encode it anyway, compare, and drop the loser — the AVIF vs WebP compression benchmarks page has the crossover points by image class, and the AVIF vs JPEG XL decision guide covers the cases where neither is the right answer.

Verification

1. Prove every emitted file actually decodes

An encoder that exits zero has not proven it wrote a valid file — a full disk or a cancelled job produces a truncated AVIF that opens in some decoders and fails in others.

# Round-trip every output through libvips. A truncated or malformed file fails
# here with a non-zero exit; -P4 keeps the check itself from dominating the job.
find dist/media -name '*.avif' -print0 \
  | xargs -0 -n1 -P4 sh -c 'vipsheader -a "$0" >/dev/null || { echo "BAD: $0"; exit 1; }' \
  && echo "all AVIFs decode"

# Spot-check the container fields that browsers care about: real dimensions,
# the loader that produced it, and the interpretation (should be srgb).
vipsheader -a dist/media/hero-1600.avif \
  | grep -E '^(width|height|bands|interpretation|vips-loader|icc-profile-data)'

2. Confirm the pool never approached the memory ceiling

# Peak RSS of the whole batch, in kilobytes (GNU time, not the shell builtin).
/usr/bin/time -v node scripts/avif-batch.mjs media/**/*.jpg 2>&1 \
  | grep -E 'Maximum resident set size|Elapsed \(wall'

# Inside a container, compare against the cgroup ceiling rather than free -m.
cat /sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory.max

# If a run died with no output, confirm whether the kernel killed it.
dmesg -T | grep -iE 'out of memory|killed process' | tail -5

Peak RSS should land near POOL × 320 MB + 200 MB. If it is far higher, sharp.cache(false) is being called after the first sharp() invocation, or something is reading masters into buffers before dispatch.

3. Check the batch is actually shrinking bytes

# Total AVIF weight versus total source weight, per shard.
find dist/media -name '*.avif' -printf '%s\n' | awk '{s+=$1} END {printf "avif %.1f MB\n", s/1048576}'
find media       -type f       -printf '%s\n' | awk '{s+=$1} END {printf "src  %.1f MB\n", s/1048576}'

# List every file where AVIF lost to the source — these should all be in the
# rejected bucket; anything here that shipped is a byte regression.
jq -r 'group_by(.status)[] | "\(.[0].status): \(length)"' .cache/avif-ledger-0.jsonl 2>/dev/null \
  || wc -l .cache/avif-ledger-*.jsonl

Pair the totals with a page-level budget so a bad quality edit fails a check instead of quietly landing; Lighthouse CI budget enforcement for image weight is the natural place to assert it.

Common mistakes

1. await Promise.all(files.map(toAvif))

This looks bounded because Node’s libuv thread pool is four wide by default, so “only four encodes run at once”. Two things break that reasoning. Every promise in the array is created immediately, so any per-job setup — most commonly await readFile(src) to get a buffer — allocates for all 5,000 files before the first encode finishes. And the moment someone raises UV_THREADPOOL_SIZE to speed things up, the encode fan-out really is unbounded.

// WRONG — 5,000 input buffers resident before the first encode completes.
await Promise.all(files.map(async (f) => sharp(await readFile(f)).avif().toFile(out(f))));

// CORRECT — fixed-width pool, sharp reads from the path so libvips streams it.
let i = 0;
await Promise.all(Array.from({ length: POOL }, async () => {
  while (i < files.length) { const f = files[i++]; await sharp(f).avif().toFile(out(f)); }
}));

2. Task-level pool on top of default libvips concurrency

sharp.concurrency() defaults to the detected core count. Leaving it there while running a four-wide task pool on a 4 vCPU runner gives you 16 libvips threads for 4 cores — measurably slower than either dial alone, with roughly double the peak RSS.

// WRONG — 4 tasks × 4 libvips threads = 16 threads contending for 4 cores.
const POOL = os.availableParallelism();

// CORRECT — one thread per pipeline, parallelism comes from the pool.
sharp.concurrency(1);
const POOL = os.availableParallelism();

3. Shipping Sharp’s default AVIF chroma for photographs

Sharp’s .avif() defaults to chromaSubsampling: '4:4:4' — full chroma resolution. That is the correct default for a library that does not know what you are encoding, and the wrong setting for a photo catalogue: 4:2:0 is typically 25–30% smaller on photographic content with no perceptible difference. The inverse error is just as common — forcing 4:2:0 globally smears coloured text and thin UI strokes in screenshots.

// WRONG — one setting for a mixed library.
.avif({ quality: 50, effort: 4 })                              // silently 4:4:4

// CORRECT — branch on content class (palette/alpha ⇒ graphic).
.avif({ quality: 50, effort: 4, chromaSubsampling: '4:2:0' })   // photographs
.avif({ quality: 62, effort: 4, chromaSubsampling: '4:4:4' })   // screenshots, logos

4. Letting one bad master abort the batch

Promise.all rejects on the first failure, and every other in-flight encode is abandoned mid-write. In a 5,000-file catalogue there is always at least one CMYK TIFF, one zero-byte placeholder, or one Git LFS pointer that was never pulled.

// WRONG — the first throw discards 40 minutes of successful work.
await Promise.all(jobs.map(run));

// CORRECT — isolate per job, collect, and fail once with the full list.
const failures = [];
for (const job of jobs) {
  try { await run(job); } catch (e) { failures.push({ job, message: e.message }); }
}
if (failures.length) { console.error(failures); process.exitCode = 1; }

Add a pre-flight guard for the LFS case specifically, since its error message (Input buffer contains unsupported image format) points nowhere useful: a 130-byte file starting with version https://git-lfs is a pointer, not an image.

5. Sharding by array index

files[i] → i % SHARD_TOTAL looks equivalent to hashing the path and is not. Insert one image near the front of a sorted list and roughly (N-1)/N of all assignments shift by one shard, so every shard’s derivative cache misses on almost everything it is asked for. The symptom is a batch that is instant for weeks and then takes 40 minutes after a one-image commit.

// WRONG — assignment depends on every other file in the list.
const mine = files.filter((_, i) => i % SHARD_TOTAL === SHARD_INDEX);

// CORRECT — assignment depends only on this file's own path.
const mine = files.filter((f) =>
  parseInt(createHash('sha1').update(f).digest('hex').slice(0, 8), 16) % SHARD_TOTAL === SHARD_INDEX);

Warning: the same stability argument applies to the shard count. Changing SHARD_TOTAL from 4 to 6 remaps every file exactly once. That is a one-off cost, but schedule it on a default-branch push so the warm cache is rebuilt where every PR can inherit it, rather than discovering it on a release branch.