Responsive Video Delivery in Next.js and React

Video delivery in component-driven applications demands more discipline than static <video> embeds. As part of the broader Responsive Image & Video Delivery architecture, this guide focuses on the layer between your transcode pipeline and the React component tree: source negotiation across codecs, deferred hydration to protect Largest Contentful Paint, poster pre-loading, and Cache-Control header configuration for edge caching. The output is a <ResponsiveVideo> component that ships zero bytes of player JS until the element enters the viewport, supports the full AV1 → VP9 → H.264 fallback chain, and passes WCAG 2.2 AA audit.

Concept & Architecture

The browser’s built-in <video> codec negotiation works identically to <picture> format negotiation: it reads <source> elements in document order and plays the first src whose type codec string passes HTMLMediaElement.canPlayType(). This means the media pipeline has three distinct responsibilities that must be designed separately.

Stage 1 — Transcode. FFmpeg converts source media into codec/container pairs during your build or CI job. Each output targets a different browser generation.

Stage 2 — Serve. Next.js route handlers and CDN edge rules apply Cache-Control: immutable, enforce Accept-Ranges: bytes (mandatory for mobile seek), and set correct MIME types for WebM and MP4 containers.

Stage 3 — Negotiate. The browser evaluates <source> elements in order. Explicit codecs= strings in the type attribute let the browser decide without a network round-trip.

Hydration control lives across stages 2 and 3: React defers mounting the <video> element until IntersectionObserver signals viewport entry, eliminating eager preload requests for below-the-fold video.

The resource selection algorithm, precisely

The HTML specification defines media loading as a two-phase state machine, and the difference between the phases explains most “the wrong file downloaded” bugs. In the resource selection phase the element walks its child <source> nodes in tree order. For each candidate it evaluates the type attribute through the same logic that backs canPlayType(). Three outcomes are possible: "probably" (container and every listed codec are recognised), "maybe" (container is recognised but the codec list is absent or unverifiable), and "" (definitely unsupported). Only the empty string causes the browser to move to the next <source> — "maybe" is treated as a match and commits the element to that URL.

That single rule has two practical consequences. First, omitting codecs= downgrades every candidate to "maybe", so the browser commits to the first <source> in the list and never evaluates the rest; an AV1-first list without codec strings will hand an AV1 file to a browser that cannot decode it, and the element will fire error on the <source> rather than falling through cleanly. Second, a malformed codec string is not an error — it simply returns "", and the source is skipped silently. A typo in av01.0.05M.08 costs you AV1 delivery with no console warning anywhere.

Once a candidate is committed, the resource fetch phase begins. preload governs how far it runs: none stops after the URL is resolved, metadata fetches enough of the container to populate duration, videoWidth, and videoHeight and then halts, and auto lets the browser buffer as aggressively as it likes. metadata is not free — it issues a ranged GET for the head of the file, and if the moov atom is at the tail (no +faststart), a second range request for the last few hundred kilobytes follows. On a page with eight below-the-fold clips that is sixteen requests before the user has scrolled.

Codec string grammar

The codecs= parameter is an RFC 6381 list, and each family has its own grammar. Getting the fields right is what lets negotiation resolve locally instead of round-tripping to the network.

Warning: the audio codec belongs in the same codecs list as the video codec, comma-separated. A type='video/mp4; codecs="avc1.42E01E"' string on a file that also carries AAC still returns "probably", but you have told the browser less than you know — and on Safari, an mp4 declared with a video-only codec list while carrying an unsupported audio track can select the source and then stall at readyState 1.

Why range requests decide mobile behaviour

Progressive <video> playback is built entirely on HTTP range requests. The element issues Range: bytes=0- first, expects 206 Partial Content with a Content-Range header, and thereafter seeks by issuing fresh ranges against the byte offsets it computed from the index. If the origin answers 200 OK with the whole body — which is what most naive Node handlers and misconfigured object-storage proxies do — the scrub bar becomes non-interactive on iOS Safari, and Chrome buffers the entire asset before it will let the user jump forward. This is why Accept-Ranges: bytes is not an optimisation but a correctness requirement, and why it belongs in the same header block as your Cache-Control policy for media assets. Container choice interacts here too: an mp4 without +faststart cannot be seeked until the tail arrives, while WebM stores its cues in a SeekHead element near the front by default.

Benchmark Reference

Codec selection has measurable file-size and decode-latency consequences. Use these baselines to justify the transcode overhead in your pipeline.

Tradeoff: AV1 encode time is 5–20x slower than H.264 at equivalent quality. Run AV1 encodes on fast presets (-preset 6 for libsvtav1) in CI and reserve slower presets for offline archival masters. If your build runs on ephemeral compute, the AV1 vs VP9 encode-time comparison on AWS Lambda quantifies where the per-invocation ceiling bites.

Read the table as a floor, not a promise. The percentages describe bitrate at matched objective quality — typically VMAF 93 or SSIM 0.93 — on mid-motion 1080p content. Three things move the number materially. Grain and fine texture favour AV1’s film-grain synthesis and push its advantage toward the top of the range; flat synthetic content such as screen recordings compresses so well in every codec that the absolute byte saving becomes irrelevant. Short clips under about ten seconds pay a fixed keyframe and header cost that dilutes the saving. And any codec’s advantage collapses if you compare at matched bitrate rather than matched quality — a mistake that makes VP9 look better than it is because the reference H.264 encode was quality-starved.

Hardware decode availability matters more than compression ratio on low-end mobile. A device that decodes AV1 in software will burn 15–30 % of a battery-limited CPU budget to save 200 KB of transfer, and on sustained playback it will thermally throttle before the saving pays for itself; the hardware AV1 decode support matrix by device class is the right input to that decision, not the compression table above.

Step-by-Step Implementation

Step 1 — Transcode Multi-Codec Variants

Run FFmpeg once per source file in your build pipeline. Output three variants: AV1 in mp4 (highest compression, modern browsers), VP9 in webm (broad coverage), and H.264 in mp4 (universal fallback).

# AV1 in MP4 container via libsvtav1
# -crf 30     — constant-rate factor: lower = higher quality (0–63 scale, NOT inverted like avifenc)
# -preset 6   — speed/quality tradeoff; 0=slowest, 12=fastest; 6 is a CI-safe default
# -c:a libopus — Opus audio; required for AV1 containers in Chrome
ffmpeg -i input.mp4 \
  -c:v libsvtav1 -crf 30 -preset 6 \
  -c:a libopus -b:a 96k \
  -movflags +faststart \
  output_av1.mp4

# VP9 in WebM — broader Safari 14+ support than AV1
# -b:v 0      — enable constant-quality mode (required when -crf is set for VP9)
# -row-mt 1   — row-based multi-threading; reduces encode time ~40% on multi-core CI
ffmpeg -i input.mp4 \
  -c:v libvpx-vp9 -crf 30 -b:v 0 -row-mt 1 \
  -c:a libopus -b:a 96k \
  output_vp9.webm

# H.264 in MP4 — IE11+ universal fallback
# -movflags +faststart moves the moov atom to the front for progressive streaming
ffmpeg -i input.mp4 \
  -c:v libx264 -crf 23 -preset fast -profile:v high \
  -c:a aac -b:a 128k \
  -movflags +faststart \
  output_h264.mp4

Step 2 — Configure Next.js Caching Headers

Set Cache-Control: immutable on all video routes. Use content-hashed filenames (e.g. hero-a1b2c3.mp4) so the immutable directive is safe to deploy.

// next.config.js
// All /videos/* paths receive 1-year immutable caching.
// Warning: only use immutable on content-hashed filenames — a filename collision
// will serve stale video from CDN edge until the hash changes.
module.exports = {
  async headers() {
    return [
      {
        source: '/videos/:path*',
        headers: [
          {
            key: 'Cache-Control',
            value: 'public, max-age=31536000, immutable'
            // 31536000 seconds = 1 year; immutable tells CDNs never to revalidate
          },
          {
            key: 'Accept-Ranges',
            value: 'bytes'
            // Required for mobile seek — without this iOS Safari cannot scrub
          }
        ]
      }
    ];
  }
};

Step 3 — Build the <ResponsiveVideo> Component

The component uses IntersectionObserver to defer mounting the <video> element until it is within 200px of the viewport. Before intersection, only the poster <img> is rendered — no video bytes are fetched.

// components/ResponsiveVideo.tsx
'use client'; // Next.js App Router: this component requires browser APIs

import { useEffect, useRef, useState } from 'react';

interface VideoSources {
  av1: string;   // AV1 in mp4 container — primary, highest compression
  vp9?: string;  // VP9 in webm — broad coverage; optional but strongly recommended
  h264: string;  // H.264 in mp4 — universal fallback; always required
}

interface VideoProps extends React.VideoHTMLAttributes<HTMLVideoElement> {
  src: VideoSources;
  poster: string;
  aspectRatio?: string; // e.g. '16/9', '4/3'; defaults to '16/9'
}

export function ResponsiveVideo({
  src,
  poster,
  aspectRatio = '16/9',
  ...props
}: VideoProps) {
  const containerRef = useRef<HTMLDivElement>(null);
  const [active, setActive] = useState(false);

  useEffect(() => {
    if (!containerRef.current) return;
    const io = new IntersectionObserver(
      ([entry]) => {
        if (entry.isIntersecting) {
          setActive(true);
          io.disconnect(); // one-shot — stop observing after first intersection
        }
      },
      { rootMargin: '200px' } // pre-load 200px before entering viewport to avoid flash
    );
    io.observe(containerRef.current);
    return () => io.disconnect();
  }, []);

  return (
    <div
      ref={containerRef}
      style={{
        position: 'relative',
        width: '100%',
        aspectRatio, // reserve layout space before video loads → CLS = 0
        background: '#000',
        overflow: 'hidden'
      }}
    >
      {active ? (
        <video
          {...props}
          preload="metadata"  // fetch only duration/dimensions, not payload
          playsInline          // mandatory for iOS Safari inline playback (no fullscreen forced)
          controls
          poster={poster}
          aria-label={props['aria-label'] ?? 'Video player'}
          style={{ width: '100%', height: '100%', display: 'block' }}
        >
          {/* Explicit codecs= string lets canPlayType() decide without a network probe */}
          <source src={src.av1} type='video/mp4; codecs="av01.0.05M.08"' />
          {src.vp9 && (
            <source src={src.vp9} type='video/webm; codecs="vp9"' />
          )}
          <source src={src.h264} type='video/mp4; codecs="avc1.42E01E"' />
          {/* Static fallback when JS/video is unavailable */}
          <img src={poster} alt={props['aria-label'] ?? 'Video placeholder'} loading="lazy" />
        </video>
      ) : (
        // Poster-only state: renders until IO triggers, zero video bytes fetched
        <img
          src={poster}
          alt=""                // decorative — the video element will replace this
          aria-hidden="true"
          fetchPriority={props.autoPlay ? 'high' : 'auto'}
          style={{ width: '100%', height: '100%', objectFit: 'cover' }}
        />
      )}
    </div>
  );
}

The component is best understood as a five-state machine whose transitions are driven by two independent event sources: the observer callback (which flips active) and the media element’s own readiness events (loadedmetadata, canplay). Keeping those sources separate is what prevents the classic bug where a component re-renders on every timeupdate and tears down its own player. Note that the first transition is not server-side: on the server active is always false, so the SSR payload contains only the poster <img>, and the observer is created in the effect after hydration.

Two transitions deserve extra care. The armed → mounting hop is one-way in this implementation because io.disconnect() runs inside the callback; if you instead keep observing, a user who scrolls a long list past the element repeatedly will re-trigger setActive(true) on every pass and thrash React’s reconciler for no benefit. The mounting → error branch is the one nobody instruments: the error event fires on the last <source>, not on the <video>, so a listener attached to the video element alone will never see it. Attach the handler to the final <source> node, or poll video.networkState === 3 (NETWORK_NO_SOURCE) after the first animation frame. Where multiple players share a page, the observer bookkeeping is worth extracting into the reusable IntersectionObserver patterns for media rather than duplicating it per component.

Step 4 — Add Captions and Reduced-Motion Support

Captions are a WCAG 2.2 Level AA requirement (Success Criterion 1.2.2). The prefers-reduced-motion media query disables autoplay for users who have requested reduced animation.

// Usage: wrap with reduced-motion guard at the call site
import { useMediaQuery } from '@/hooks/useMediaQuery'; // project-specific hook

export function HeroVideo() {
  // prefers-reduced-motion: disable autoplay for accessibility
  const reduced = useMediaQuery('(prefers-reduced-motion: reduce)');

  return (
    <ResponsiveVideo
      src={{
        av1: '/videos/hero-a1b2c3_av1.mp4',
        vp9: '/videos/hero-a1b2c3_vp9.webm',
        h264: '/videos/hero-a1b2c3_h264.mp4'
      }}
      poster="/videos/hero-poster.jpg"
      aria-label="Product overview video"
      autoPlay={!reduced}   // honour OS-level animation preference
      muted                 // required for autoPlay in any browser
      loop
    >
      {/* VTT captions — WCAG 2.2 AA SC 1.2.2 */}
      <track
        kind="captions"
        src="/videos/hero-captions-en.vtt"
        srclang="en"
        label="English"
        default
      />
    </ResponsiveVideo>
  );
}

Step 5 — Adapt the Source Set to Connection and Container Width

<video> has no srcset. There is no declarative way to say “serve the 720p rendition below 800 px of container width” — the element negotiates codec, never resolution. Resolution selection therefore has to be a render-time decision in the component, taken before the <source> elements are emitted. Two signals are worth reading: the container’s own width (measured once, at mount, from the same ref the observer uses) and navigator.connection.

// components/useVideoRendition.ts
// Chooses a rendition tier once, at mount. Deliberately NOT reactive:
// swapping <source> after the element has committed a resource forces a
// full load() + buffer flush and restarts playback from zero.
import { useRef, useState, useEffect, type RefObject } from 'react';

type Tier = '480p' | '720p' | '1080p';

export function useVideoRendition(ref: RefObject<HTMLElement>): Tier {
  const [tier, setTier] = useState<Tier>('720p'); // SSR-safe default; matches most cards
  const decided = useRef(false);

  useEffect(() => {
    if (decided.current || !ref.current) return;
    decided.current = true;

    // devicePixelRatio matters: a 400 CSS-px card on a 3x phone still wants 1080p-class
    // pixels, but capping at 2 avoids paying for a rendition the panel cannot resolve.
    const dpr = Math.min(window.devicePixelRatio || 1, 2);
    const cssWidth = ref.current.getBoundingClientRect().width;
    const targetPx = cssWidth * dpr;

    // NetworkInformation is Chromium-only; every branch must survive it being undefined.
    const conn = (navigator as any).connection;
    const saveData: boolean = conn?.saveData ?? false;
    const effective: string = conn?.effectiveType ?? '4g';

    if (saveData || effective === '2g' || effective === 'slow-2g') {
      setTier('480p');           // hard cap — respect an explicit user preference
    } else if (effective === '3g' || targetPx < 720) {
      setTier('720p');
    } else if (targetPx >= 1280) {
      setTier('1080p');
    }
  }, [ref]);

  return tier;
}

Feed the tier into the filename convention your transcode step already produces (hero-a1b2c3_720p_av1.mp4), so the source list stays three entries long regardless of tier. The rule that makes this safe is the one encoded in the decided ref: decide once. Changing a <source> src after the element has committed does nothing at all until you call load(), and load() resets currentTime, discards the buffer, and re-runs resource selection from the top.

Tradeoff: reading getBoundingClientRect() in an effect forces a synchronous layout on mount. With a handful of videos per page the cost is negligible; in a virtualised feed of dozens, hoist the measurement to the list container and pass the width down as a prop so layout is forced once rather than once per row. Where the surrounding layout is itself fluid, container queries for dynamic media sizing can carry the CSS half of the same decision without any JavaScript measurement at all.

Warning: effectiveType is a rolling estimate derived from recent transfers, not a live measurement. Immediately after a cold navigation it frequently reports 4g on a genuinely slow link because the only samples so far were small, cached responses. Treat saveData as authoritative and effectiveType as a hint.

Parameter Reference

Tradeoffs & Edge Cases

Tradeoff: AV1 encode time. libsvtav1 at -preset 6 is 3–5x slower than libx264 at -preset fast. For CI pipelines with many video assets, parallelise FFmpeg jobs and cache outputs by content hash. Reserve AV1 generation for assets over 10 seconds; shorter clips may not recoup the transcode cost.

Warning: Safari 16.4 AV1 software fallback. Safari 16.4 supports AV1 decode but falls back to software decoding on Macs without Apple silicon or Intel Ice Lake+. Software AV1 decode can spike CPU to 100% on a 1080p clip, causing frame drops. Test on real hardware; for Safari-heavy audiences keep VP9 in the source order.

Warning: fetchpriority="high" starvation. Applying fetchpriority="high" to more than one element on a page causes the browser’s preloader to deprioritise CSS and fonts. Limit high-priority poster fetches to the single above-the-fold video element.

Tradeoff: rootMargin size. A 200px root margin pre-fetches video on fast connections before the user sees the element, but on a slow 3G connection it can initiate a large download the user may never watch. Consider reading navigator.connection.saveData and reducing the root margin to 0px when saveData === true.

Warning: SSR hydration mismatch. The active state in <ResponsiveVideo> starts as false on the server and true only after IO triggers on the client. React 18 Suspense boundaries and startTransition do not help here — use a suppressHydrationWarning prop on the container div if the poster/video difference triggers a hydration error in strict mode.

Tradeoff: HEVC/H.265 as a second fallback. H.265 achieves better compression than VP9 on Apple hardware but is absent from Chrome and Firefox. Adding it between VP9 and H.264 in the source list has no effect on Chrome/Firefox (they skip to H.264), while Safari selects it over H.264. Only add H.265 if your analytics show >20% Safari share; the extra transcode/storage cost is rarely justified otherwise.

Warning: autoPlay returns a promise that can reject. React’s autoPlay prop maps to the attribute, and when an autoplay policy blocks the attempt the element quietly stays paused — no error surfaces in React. If the video is decorative (a muted loop behind a headline) that is acceptable; if it is content, call videoEl.play() explicitly in an effect and catch the rejection, then reveal the poster and controls so the user has a way forward. The rejection reason is NotAllowedError for policy blocks and AbortError when a pending load() superseded the request — only the first should surface UI.

Warning: multiple players competing for decoder slots. Mobile Safari historically limited the page to a small number of simultaneously loaded media elements, and Android WebView imposes a hardware-decoder-instance cap that varies by SoC. Beyond the cap, additional elements either fail to render a first frame or silently drop to software decode. In a feed layout, unmount players that have scrolled well out of view rather than merely pausing them — the observer that mounted them can run a second, larger-margin rule that tears them down again.

Tradeoff: poster format versus poster weight. The poster is almost always the LCP element in this design, so its encode matters more than the video’s. An AVIF poster at quality 50 typically lands 40–60 % smaller than the equivalent WebP, but Safari 14 cannot decode it. Because poster accepts a single URL and no <source> negotiation, either accept WebP as the universal choice or render your own <img> with a <picture> wrapper as the idle state and drop the poster attribute entirely — the component above already renders exactly such an <img>, which makes the second option nearly free.

Warning: preload="metadata" still costs a request per element. Eight below-the-fold clips with preload="metadata" will issue eight ranged requests during page load if they are all mounted, competing with fonts and CSS for connections. The deferred mount in Step 3 is what prevents that — but only if the poster-only branch really renders no <video> element. A common regression is refactoring the ternary into a hidden prop on an always-mounted <video>, which restores every one of those requests while looking identical on screen.

Tradeoff: caching immutable video behind a signature. Cache-Control: public, max-age=31536000, immutable and time-limited signed URLs pull against each other — the signature changes the cache key on every rotation, so the edge stores a new copy of a multi-megabyte object each time. If the assets are not genuinely private, prefer content-hashed public paths; if they must be signed, extend the signature lifetime past the CDN’s object TTL and layer stale-while-revalidate for media assets so a rotation never produces a synchronous origin fetch on the critical path.

Debugging & Validation

Confirm codec negotiation via canPlayType() in the browser console before filing a “video won’t play” bug:

// Run in the browser console to check what the current UA actually supports
const v = document.createElement('video');
console.table({
  'AV1 mp4':  v.canPlayType('video/mp4; codecs="av01.0.05M.08"'),
  'VP9 webm': v.canPlayType('video/webm; codecs="vp9"'),
  'H264 mp4': v.canPlayType('video/mp4; codecs="avc1.42E01E"')
  // Returns: "probably", "maybe", or "" (empty = not supported)
});

Inspect response headers to confirm Accept-Ranges and Cache-Control are set correctly:

# -sI: silent mode, headers only; replace with your actual video URL
curl -sI https://example.com/videos/hero_av1.mp4 | grep -E 'Content-Type|Cache-Control|Accept-Ranges'
# Expected:
# Content-Type: video/mp4
# Cache-Control: public, max-age=31536000, immutable
# Accept-Ranges: bytes

Check for eager preload in the Network panel. In Chrome DevTools → Network → filter by Media: if video requests fire on page load for below-the-fold elements, IntersectionObserver is not wiring up correctly — the most common cause is the containerRef being null on mount due to a conditional render.

Lighthouse audit. Run Lighthouse in mobile simulation. Key signals: LCP should be driven by the poster <img> (not the video element itself); aspect-ratio on the container ensures CLS = 0 across all viewports. Any CLS > 0 on a video element usually means the container is missing its aspect-ratio rule.

Validate VTT captions with the W3C validator at https://quuz.org/webvtt/ or via the vtt npm package. Missing or malformed cues cause the captions track to silently fail in Safari — it will show no error in the console.

Prove that range requests actually work. A Accept-Ranges: bytes header is a claim; the status code is the evidence. Ask for a byte range explicitly and check that the origin honours it:

# Request the first 1024 bytes only.
# -r 0-1023 sets the Range header; -o /dev/null discards the body; -D - dumps headers.
curl -s -r 0-1023 -o /dev/null -D - https://example.com/videos/hero_av1.mp4
# Required:
#   HTTP/2 206                       <- 200 here means seeking is broken
#   content-range: bytes 0-1023/4823914
#   content-length: 1024

# Confirm the moov atom is at the head (i.e. +faststart was applied).
# In a faststart mp4, "moov" appears within the first few hundred bytes,
# right after the ftyp box; otherwise it sits at the end of the file.
curl -s -r 0-512 https://example.com/videos/hero_av1.mp4 | xxd | grep -m1 moov

Measure what the poster actually cost you. The LCP entry names the element it measured, which is the fastest way to confirm the poster — not the video — is the candidate:

// Paste into the console before a reload, or ship inside a web-vitals wrapper.
new PerformanceObserver((list) => {
  const e = list.getEntries().at(-1);           // LCP reports repeatedly; the last wins
  console.log('LCP', Math.round(e.startTime), 'ms', e.element?.tagName, e.url ?? '');
  // Expected: "LCP 1180 ms IMG https://.../hero-poster.webp"
  // A VIDEO tagName here means the poster never painted and the first decoded
  // frame became the candidate — check that the poster URL returns 200.
}).observe({ type: 'largest-contentful-paint', buffered: true });

// Byte accounting: how much media did the page pull before any user interaction?
const media = performance.getEntriesByType('resource')
  .filter(r => r.initiatorType === 'video' || /\.(mp4|webm)$/.test(r.name));
console.table(media.map(r => ({ url: r.name.split('/').pop(), kb: Math.round(r.transferSize / 1024) })));
// Below-the-fold clips should be absent entirely. A ~30–90 KB entry is a
// metadata fetch; anything megabyte-scale means preload="auto" leaked in.

Check the negotiated codec at runtime, not the one you intended. video.currentSrc reports the URL the element actually committed to after resource selection, which is the only trustworthy answer to “did this browser take the AV1?” Compare it against getVideoPlaybackQuality().droppedVideoFrames after ten seconds of playback: a rising dropped-frame count alongside an AV1 currentSrc is the signature of software decode, and the fix is to reorder VP9 ahead of AV1 for that user agent rather than to lower the bitrate.

Frequently Asked Questions

Related

• Using Next/Image with Custom Loader Configurations — apply the same loader pattern to still images in the same Next.js pipeline
• Implementing Responsive Video with Video.js — add a full-featured player UI while keeping the same codec negotiation strategy
• Art Direction with the HTML Picture Element — translate art-direction breakpoint logic from <picture> to <video> source elements
• Understanding Video Codecs: VP9 vs H.265 vs AV1 — codec fundamentals and encode-time benchmarks behind the fallback chain used here
• Cache-Control Headers for Image and Video Assets — configure edge caching for the immutable video assets served by this pipeline
• Responsive Image & Video Delivery — parent section covering srcset, art direction, and container queries alongside this video guide