VideoSnap Vision: Real-Time Object Recognition PWA Architecture

Posted on by — No Comments ↓

Real-Time Object Recognition in a React PWA with Hugging Face Transformers

Hey folks! I recently built a super fun Progressive Web App (PWA) that does real-time object recognition using a small multimodal LLM from Hugging Face. Picture this: point your webcam at something, and the app instantly tells you what it sees—a dog, a cup, or even your favorite sneaker! It works right in your browser, even offline, and feels just like a native app. Pretty cool, right? Here’s how I pulled it off with React, TensorFlow.js, and a dash of PWA magic. Let’s dive in!

Why Real-Time Video in a PWA?

I’m a big fan of apps that are always ready to go, even if my internet connection isn’t. Plus, who wants to rely on a beefy server for live video processing if you can do it on the device? PWAs are fantastic for this: they’re installable, cache what they need for offline use, and work across all sorts of devices. For the brains of the operation—the Machine Learning part—I picked a small multimodal LLM from Hugging Face (think a lightweight version of CLIP). These models are champs at recognizing objects in images or video frames and are nimble enough to run smoothly in the browser.

Setting Up the React PWA

First things first, I got my React PWA project started using Create React App’s PWA template: 

npx create-react-app video-object-pwa --template cra-template-pwa
cd video-object-pwa
npm start

This command set me up with a service-worker.js for handling offline caching and a manifest.json to give it that authentic app-like feel (like being installable on your home screen!). I popped into the manifest.json to name my app “VideoSnap” and gave it a snazzy icon.

Our App’s Blueprint: The Architecture

Before we get into the nitty-gritty of code, let’s take a bird’s-eye view of how VideoSnap is put together. A picture is worth a thousand words, so here’s a diagram (imagine this rendered beautifully with Eraser.io!):

Image description

Let’s break down what’s happening:

  1. User’s Device: Everything happens right here! No servers involved for the core functionality.
  2. Web Browser: This is our app’s home.
    • VideoSnap PWA (React App): This is our actual application code.
      • App Shell & UI: The main interface you see and interact with.
      • VideoRecognizer Component: The star of the show, handling webcam input and displaying predictions.
      • PWA Features:
        • Manifest.json: Tells the browser how to treat our app (icon, name, installability).
        • Service Worker: The background hero that caches assets and the ML model, enabling offline use and speeding up subsequent loads. It intercepts network requests and can serve files directly from the…
    • Browser Caches:
      • PWAAssetsCache: Stores our app’s code (JS, CSS, images).
      • ModelCache: Crucially, this holds the downloaded ML model files. Once downloaded, they’re available offline!
    • In-Browser ML Stack:
      • 🤗 Transformers.js: Makes it easy to use Hugging Face models in JavaScript. It handles loading the model and processor, and helps with preprocessing data.
      • TensorFlow.js: The underlying library that runs the ML model computations efficiently in the browser.
      • WebGL Backend: TensorFlow.js uses WebGL to tap into your device’s GPU for much faster calculations.
    • Browser APIs:
      • Webcam (getUserMedia): Lets our app access the camera.
      • Cache/Storage API: Used by the Service Worker to store and retrieve files.

Key Interactions:

  • Opening the App (1): You open VideoSnap. The Manifest.json helps it look and feel like an app.
  • Accessing Webcam (2): The VideoRecognizer component asks for permission to use your webcam via getUserMedia.
  • Service Worker Magic (3): The Service Worker gets registered. On first load, it fetches all app assets and the ML model, then tucks them away in the Browser Caches. On later visits (or when offline), it serves these directly from the cache – super fast!
  • Loading the Model (4): Our component uses Transformers.js to load the object recognition model. The Service Worker might intercept this request and serve the model from its cache.
  • Real-Time Loop (5-7):
    1. A video frame is captured.
    2. Transformers.js preprocesses this frame.
    3. TensorFlow.js (using the WebGL backend for speed) runs the model to get a prediction.
    4. Transformers.js translates this into a human-readable label.
    5. The UI updates to show you what it “sees”! This loop repeats, giving you real-time object recognition.

This architecture allows VideoSnap to be fast, offline-capable, and process video directly on your device, which is pretty powerful stuff for a web app!

Grabbing the Hugging Face Model

I chose a compact multimodal LLM from Hugging Face, specifically openai/clip-vit-base-patch32 (or you could go for an even smaller, distilled variant if speed is paramount). These CLIP-style models are great because they can compare an image (or a video frame) to a list of text descriptions and tell you which description fits best.

To use it in the browser with TensorFlow.js, we need to convert it.

First, install the necessary Python libraries: 

pip install transformers tensorflow

Next, we’ll write a small Python script to download the model and processor from Hugging Face and save them in a format we can then convert. 

# export_model.py
from transformers import TFCLIPModel, CLIPProcessor # For this example, using the TF variant for direct Keras save

MODEL_NAME = "openai/clip-vit-base-patch32"
EXPORT_DIR_BASE = "./clip_export_temp" # Temporary base directory for raw exports
MODEL_SAVE_DIR = f"{EXPORT_DIR_BASE}/model_files" # For Keras model (tf_model.h5) and its config.json
PROCESSOR_SAVE_DIR = f"{EXPORT_DIR_BASE}/processor_files" # For processor configuration files

# Create directories if they don't exist
import os
os.makedirs(MODEL_SAVE_DIR, exist_ok=True)
os.makedirs(PROCESSOR_SAVE_DIR, exist_ok=True)

# Load pre-trained model and processor
print(f"Loading model: {MODEL_NAME}")
model = TFCLIPModel.from_pretrained(MODEL_NAME)
print(f"Loading processor: {MODEL_NAME}")
processor = CLIPProcessor.from_pretrained(MODEL_NAME)

# Save Keras model (tf_model.h5) and its config.json
# This is what Transformers.js (AutoModel) will need for the model architecture.
model.save_pretrained(MODEL_SAVE_DIR)
print(f"Model files (incl. config.json and tf_model.h5) saved to {MODEL_SAVE_DIR}")

# Save processor configuration files (preprocessor_config.json, tokenizer files, etc.)
# These are what Transformers.js (AutoProcessor) will need.
processor.save_pretrained(PROCESSOR_SAVE_DIR)
print(f"Processor files saved to {PROCESSOR_SAVE_DIR}")

Run this Python script (python export_model.py). It will download the model files (including tf_model.h5 and config.json) into ./clip_export_temp/model_files/ and the processor files (like preprocessor_config.json, tokenizer.json, etc.) into ./clip_export_temp/processor_files/.

Now, convert the Keras model (tf_model.h5) to the TensorFlow.js web-friendly format: 

# Make sure you have tensorflowjs_converter installed:
# pip install tensorflowjs

tensorflowjs_converter --input_format=keras 
                       ./clip_export_temp/model_files/tf_model.h5 
                       ./public/model

This command takes the tf_model.h5 file and spits out a model.json file (the model architecture) and one or more binary weight files (.bin) into your PWA’s public/model directory.

Crucial Step for AutoModel and AutoProcessor:
You need to manually copy some files into that same public/model directory so Transformers.js can find them:

  1. Copy config.json from ./clip_export_temp/model_files/ into ./public/model/.
  2. Copy all the processor configuration files (e.g., preprocessor_config.json, tokenizer.json, vocab.json, merges.txt) from ./clip_export_temp/processor_files/ into ./public/model/.

After this, your public/model directory should contain model.json, the *.bin weight files, config.json, and all the processor files. This is what our React app will load.

Building the Real-Time Video Component

This is where the real magic happens! I created a React component (VideoRecognizer.js) that:

  1. Accesses the user’s webcam.
  2. Loads our Hugging Face model and processor using Transformers.js.
  3. Continuously grabs frames from the video.
  4. Preprocesses these frames.
  5. Runs them through the model for object recognition.
  6. Displays the prediction.

I’m using @tensorflow/tfjs for the core ML operations and @huggingface/transformers to easily work with the model. The browser’s built-in navigator.mediaDevices.getUserMedia API handles webcam access. 

// src/components/VideoRecognizer.js
import React, { useEffect, useRef, useState } from 'react';
import *alsot * as tf from '@tensorflow/tfjs';
// Using AutoModel and AutoProcessor for flexibility with Hugging Face models
import { AutoProcessor, AutoModel } from '@huggingface/transformers';

// Define the labels our CLIP model will try to match against.
// For CLIP, descriptive prompts usually work best!
const CANDIDATE_LABELS = [
  'a photo of a cat',
  'a photo of a dog',
  'a photo of a car',
  'a photo of a tree',
  'a photo of a coffee cup',
  'a photo of a sneaker',
  'a photo of a human face',
  'a photo of a laptop',
  'a photo of a keyboard',
  'a photo of a bottle of water'
];
// Helper to get a cleaner display name from the label
const getDisplayName = (labelText) => labelText.replace("a photo of a ", "");

const VideoRecognizer = () => {
  const videoRef = useRef(null);
  const [model, setModel] = useState(null);
  const [processor, setProcessor] = useState(null);
  const [prediction, setPrediction] = useState('');
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState(null); // For displaying errors

  // Effect to load the model and processor
  useEffect(() => {
    const loadModelAndProcessor = async () => {
      try {
        console.log("Setting TF.js backend to WebGL...");
        await tf.setBackend('webgl'); // Use WebGL for GPU acceleration

        console.log("Loading model and processor from /model...");
        // '/model' points to the public/model directory where we placed all our files
        const loadedModel = await AutoModel.from_pretrained('/model');
        const loadedProcessor = await AutoProcessor.from_pretrained('/model');

        setModel(loadedModel);
        setProcessor(loadedProcessor);
        setLoading(false);
        console.log('Model and processor loaded successfully!');
      } catch (err) {
        console.error('Failed to load model or processor:', err);
        setError(`Oops! Model load failed: ${err.message}. Ensure all model files are in public/model/ and reachable.`);
        setLoading(false);
      }
    };
    loadModelAndProcessor();
  }, []);

  // Effect to start the webcam video stream
  useEffect(() => {
    const startVideo = async () => {
      // Don't try to start video if the model is still loading or if there was an error
      if (loading || error) return; 
      try {
        const stream = await navigator.mediaDevices.getUserMedia({ video: true });
        if (videoRef.current) {
          videoRef.current.srcObject = stream;
        }
      } catch (err) {
        console.error('Error accessing webcam:', err);
        setError(`Webcam error: ${err.message}. Please grant permission and ensure no other app is using it.`);
      }
    };
    startVideo();

    // Cleanup: stop video tracks when component unmounts
    return () => {
      if (videoRef.current && videoRef.current.srcObject) {
        videoRef.current.srcObject.getTracks().forEach(track => track.stop());
      }
    };
  }, [loading, error]); // Re-run if loading state changes or an error occurs

  // Effect for real-time frame processing and inference
  useEffect(() => {
    if (!model || !processor || !videoRef.current || !videoRef.current.srcObject || videoRef.current.paused) {
      return; // Exit if model/processor not ready, or video not playing
    }

    const processFrame = async () => {
      const video = videoRef.current;
      // Ensure video is ready and has dimensions before processing
      if (!video || video.readyState < video.HAVE_ENOUGH_DATA || video.videoWidth === 0) {
        requestAnimationFrame(processFrame); // Wait for next frame
        return;
      }

      try {
        // Create a temporary canvas to draw the current video frame
        const canvas = document.createElement('canvas');
        canvas.width = video.videoWidth;
        canvas.height = video.videoHeight;
        const ctx = canvas.getContext('2d');
        ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

        // Process the image (from canvas) and text labels with the CLIP processor
        const inputs = await processor(
          /*text=*/ CANDIDATE_LABELS,
          /*images=*/ canvas, // Pass the canvas directly
          { return_tensors: 'tf', padding: true, truncation: true }
        );

        let topLabel = '';
        // tf.tidy helps manage memory by auto-disposing intermediate tensors
        tf.tidy(() => {
          // Run inference
          const outputs = model(inputs); // For some models, you might need model(**inputs)
          // CLIP outputs logits_per_image which indicate similarity between image and each text label
          const logitsPerImage = outputs.logitsPerImage; // Shape: [batch_size, num_labels]
          const probabilities = tf.softmax(logitsPerImage.squeeze()); // Squeeze to [num_labels] then apply softmax

          const topProbIndex = probabilities.argMax().dataSync()[0]; // Get index of highest probability
          topLabel = getDisplayName(CANDIDATE_LABELS[topProbIndex]);
        });

        setPrediction(`I see... a ${topLabel}!`);

      } catch (err) {
        console.error('Inference error:', err);
        // You could set an error state here for the user too
      }
      // Request the next frame for continuous processing
      requestAnimationFrame(processFrame);
    };

    // Start the processing loop
    const animationFrameId = requestAnimationFrame(processFrame);

    // Cleanup: cancel animation frame when component unmounts or dependencies change
    return () => cancelAnimationFrame(animationFrameId);

  }, [model, processor]); // Re-run this effect if the model or processor changes

  // Render the UI
  if (error) {
    return (
      <div style={{ textAlign: 'center', padding: '20px', color: 'red', border: '1px solid red', margin: '10px' }}>
        <h1>VideoSnap</h1>
        <p><strong>Error:</strong> {error}</p>
        <p>Please check the console for more details. Try refreshing or ensuring model files are correctly placed.</p>
      </div>
    );
  }

  if (loading && !navigator.onLine && !model) {
     return (
      <div style={{ textAlign: 'center', padding: '20px' }}>
        <h1>VideoSnap</h1>
        <p>You seem to be offline. Please connect to the internet to download the AI model for the first time.</p>
        <p>Once downloaded, it will be available offline thanks to PWA magic!</p>
      </div>
    );
  }

  return (
    <div style={{ textAlign: 'center', padding: '20px' }}>
      <h1>VideoSnap - What Do I See?</h1>
      {loading ? (
        <p>🧠 Loading the AI model, please wait... (This might take a moment on your first visit, especially for the model download!)</p>
      ) : (
        <>
          <video 
            ref={videoRef} 
            autoPlay 
            playsInline 
            muted /* Muting is often required for autoplay in browsers */
            style={{ width: '100%', maxWidth: '640px', border: '2px solid #007bff', borderRadius: '8px', display: error ? 'none' : 'block' }} 
          />
          {prediction && <p style={{ fontSize: '1.8em', fontWeight: 'bold', marginTop: '15px', color: '#28a745' }}>{prediction}</p>}
          {!prediction && !error && <p>Point your camera at an object!</p>}
        </>
      )}
    </div>
  );
};

export default VideoRecognizer;

Note that I’ve switched setInterval to requestAnimationFrame for smoother video processing. This ties the processing to the browser’s display refresh rate, which is generally better for animations and video.

I then plugged this VideoRecognizer component into my main App.js: 

// src/App.js
import React from 'react';
import VideoRecognizer from './components/VideoRecognizer';
import './App.css'; // For any global styling

function App() {
  return (
    <div className="App">
      <header className="App-header">
        {/* You could add a title or nav bar here */}
      </header>
      <main>
        <VideoRecognizer />
      </main>
      <footer style={{ textAlign: 'center', padding: '10px', fontSize: '0.8em', color: '#777' }}>
        Built with React, Hugging Face Transformers.js, and TensorFlow.js
      </footer>
    </div>
  );
}

export default App;

Making It Offline-Ready

To make sure VideoSnap truly shines as a PWA (and keeps rocking even when the internet flakes out), I updated the service-worker.js file. The goal is to cache all our app’s assets and those crucial model files.

The Create React App PWA template gives you a service-worker.js (often src/service-worker.js or public/service-worker.js depending on your setup which gets built into build/service-worker.js). We need to ensure it precaches our model and processor files from the /model/ directory.

Here’s how you can ensure your model files are part of the precache manifest, or add a custom caching strategy. If you’re using CRA’s Workbox setup, it typically uses self.__WB_MANIFEST which includes files from the public folder. You might need to explicitly list them or use a runtime caching strategy if they are numerous or large.

A good approach for model files within a Workbox-powered service worker: 

// src/service-worker.js (Modify the one generated by Create React App)
import { clientsClaim } from 'workbox-core';
import { ExpirationPlugin } from 'workbox-expiration';
import { precacheAndRoute, createHandlerBoundToURL } from 'workbox-precaching';
import { registerRoute } from 'workbox-routing';
import { CacheFirst, StaleWhileRevalidate } from 'workbox-strategies';

clientsClaim();

// Precache all of the assets generated by your build process.
// Their URLs are injected into the manifest variable below.
// This variable must be present somewhere in your service worker file,
// even if you decide not to use precaching. See https://cra.link/PWA
const manifestEntries = self.__WB_MANIFEST || [];

// Files from the public folder are typically included in __WB_MANIFEST by CRA's build process.
// Double-check if your model files in `public/model` are automatically added.
// If not, or for more control, you can add them manually or use runtime caching.

// Example: Add model files to precache if not automatically included.
// Best practice is to let the build process hash these files for revision control.
// If CRA includes `public` folder contents, this might be redundant.
const modelFilesToPrecache = [
  // Ensure these paths match exactly how they are in your public/model folder
  { url: '/model/model.json', revision: null }, // 'null' means don't version based on content hash here, if already versioned by filename or build process.
  { url: '/model/config.json', revision: null },
  { url: '/model/preprocessor_config.json', revision: null },
  { url: '/model/tokenizer.json', revision: null },
  { url: '/model/vocab.json', revision: null },
  { url: '/model/merges.txt', revision: null },
  // IMPORTANT: List ALL your .bin files (TensorFlow.js weight shards)
  // For example, if you have one shard:
  { url: '/model/group1-shard1of1.bin', revision: null },
  // If you have multiple, list them all:
  // { url: '/model/group1-shard1ofN.bin', revision: null },
  // { url: '/model/group1-shard2ofN.bin', revision: null },
  // ... etc.
];

// Combine CRA's manifest with our custom model files
// Ensure no duplicates if CRA already includes them.
const allFilesToPrecache = [...manifestEntries, ...modelFilesToPrecache.filter(
  modelFile => !manifestEntries.find(entry => typeof entry === 'string' ? entry === modelFile.url : entry.url === modelFile.url)
)];

precacheAndRoute(allFilesToPrecache);

// You can also use a runtime caching strategy for model files, especially if they are very large
// or you want to fetch them on demand and cache them with specific rules.
// Example: Cache model files with a CacheFirst strategy if not precached.
registerRoute(
  ({url}) => url.pathname.startsWith('/model/'),
  new CacheFirst({
    cacheName: 'ml-model-cache',
    plugins: [
      new ExpirationPlugin({
        maxEntries: 20, // Cache up to 20 model-related files
        maxAgeSeconds: 30 * 24 * 60 * 60, // Cache for 30 Days
      }),
    ],
  })
);


// The rest of CRA's default service worker (routing for index.html, etc.) usually follows...
// This allows the PWA to function as a single-page application.
const fileExtensionRegexp = new RegExp('/[^/?]+\.[^/]+$');
registerRoute(
  ({ request, url }) => {
    if (request.mode !== 'navigate') {
      return false;
    }
    if (url.pathname.startsWith('/_')) {
      return false;
    }
    if (url.pathname.match(fileExtensionRegexp)) {
      return false;
    }
    return true;
  },
  createHandlerBoundToURL(process.env.PUBLIC_URL + '/index.html')
);

With this service worker in place, after the first visit, the app loads blazing fast, and the model is ready to go even if you’re on a desert island (as long as your device has power!). My component already shows a gentle message if you’re offline before the model’s first download.

Performance Hacks & Tips

Running ML in the browser needs a bit of care to keep things snappy:

  • WebGL is Your Friend: tf.setBackend('webgl') is key. It lets TensorFlow.js use the GPU, which is way faster for these kinds_of tasks than the CPU.
  • Smooth Processing with requestAnimationFrame: Instead of a fixed setInterval, using requestAnimationFrame(processFrame) syncs processing with the browser’s refresh rate. This generally leads to smoother visuals and better resource management, as the browser can optimize when to run the frame processing.
  • Model Size: I aimed for a model around ~100MB. Smaller models load faster and run quicker. If your chosen model is hefty, look into quantization techniques (like tf.quantization.quantize_weights) which can shrink model size, often with minimal impact on accuracy.
  • Memory Management with tf.tidy(): In the processFrame function, wrapping the TensorFlow.js operations within tf.tidy(() => { ... }) is a lifesaver. It automatically cleans up (disposes) any intermediate tensors created during the model inference, preventing memory leaks that could crash your app over time, especially with continuous video processing.
  • Webcam & Component Cleanup: Always clean up! In useEffect hooks, the return function is perfect for stopping the webcam stream (track.stop()) and canceling animation frames (cancelAnimationFrame) when the component unmounts. This prevents resource leaks and weird background activity.

Testing and Deploying

Testing is super important! I spent a good amount of time in Chrome DevTools:

  • Lighthouse Audits: To check PWA compliance (installability, offline support, performance).
  • Network Tab: Throttling to simulate slower connections and using the “Offline” checkbox to rigorously test the service worker and caching.
  • Performance Tab: To profile JavaScript execution and identify any bottlenecks in the frame processing loop.
  • Console: Watching for any errors from TensorFlow.js or Transformers.js.

On my laptop, inference was taking a fraction of a second per frame with requestAnimationFrame, making it feel very responsive. The initial app load (including model download) was a bit longer on mobile (5-10 seconds depending on network), but subsequent loads were near-instant thanks to the service worker.

For deployment, I’m a fan of Vercel for its simplicity with frontend projects: 

npm run build
vercel --prod

And just like that, VideoSnap was live! I could install it on my phone from the browser and show off its real-time, offline object recognition skills. It really feels like magic having a mini AI sidekick in your pocket.

What’s Next?

This project was a ton of fun, and it’s amazing what you can do in the browser these days. But my mind is already buzzing with ideas:

  • More Labels: Expand the CANDIDATE_LABELS list to recognize an even wider array of objects.
  • Fine-tuning Performance: Experiment more with model quantization or different small model architectures for even faster inference or lower resource usage.
  • User-Provided Labels: Allow users to type in what they’re looking for, turning it into a “visual search” tool.
  • Accessibility: Ensuring the app is usable for everyone, including providing feedback via ARIA attributes.

If you’re inspired to build something similar, I highly recommend diving into the documentation for Hugging Face Transformers.js and TensorFlow.js. The possibilities are incredible.

Got questions, cool ideas, or hit a snag trying this out? Drop a comment below – I’d love to hear from you!

Happy coding, and let’s keep building PWAs that can see and understand the world in real time!

Crafted with curiosity and a love for tech that runs in the browser

Leave a Reply