Pagination — Architecture Series: Part 1

🚀 Pagination — The Complete MERN Stack Guide

In large-scale applications, managing massive datasets efficiently is critical. Whether it’s displaying hundreds of blog posts, thousands of users, or millions of transactions, fetching everything at once is both impractical and wasteful.
Pagination is the architectural pattern that solves this—by dividing data into discrete, manageable pages for optimal performance, scalability, and user experience.

In this article, we’ll break down the WHAT, WHY, and HOW of pagination — covering both backend and frontend implementations, exploring offset-based, cursor-based, and keyset (seek) strategies. You’ll also learn about edge cases, performance tuning, database indexing, and best practices used in production systems.

🔹 1. What is Pagination?

Pagination means dividing large datasets into smaller, digestible pieces (pages).
Instead of sending 10,000 records in one response, we send, for example, 10 or 20 per request.

Real-world analogy:

Google doesn’t show all results at once — it shows 10 per page with “Next” & “Prev”.

🔹 2. Why Pagination Matters

Reason	Description
⚡ Performance	Limits DB load — query small slices instead of all records
🧠 Memory Efficiency	Prevents browser & server from crashing on large responses
🧍‍♂️ Better UX	Users digest info easier, faster initial loads
📡 Bandwidth	Reduces unnecessary data transfer
📈 Scalability	Apps handle millions of rows smoothly
🔐 Security & Control	Prevents abuse (e.g., scraping entire datasets)

🔹 3. Pagination Types (and When to Use Them)

Type	Description	Best For
Offset / Page-based	`page` + `limit`, uses `.skip()` & `.limit()`	Dashboards, Admin Panels
Cursor-based	Uses `_id` or timestamp to fetch next batch	Infinite Scroll, Real-time Feeds
Keyset-based	Combines sort + cursor for precise ordering	Large ordered datasets

🔹 4. Offset-Based Pagination (Classic)

🧠 How it works:

You send:
GET /api/users?page=2&limit=10

The server calculates:

skip = (page - 1) * limit
limit = 10

🧩 Backend (Node.js + Express + MongoDB)

import express from "express";
import mongoose from "mongoose";
import User from "./models/User.js"; // assume name, email, createdAt
const app = express();

app.get("/api/users", async (req, res) => {
  try {
    let page = parseInt(req.query.page) || 1;
    let limit = parseInt(req.query.limit) || 10;

    // Validation
    if (page < 1 || limit < 1 || limit > 100) {
      return res.status(400).json({ error: "Invalid pagination params" });
    }

    const skip = (page - 1) * limit;
    const total = await User.countDocuments();

    const users = await User.find()
      .sort({ createdAt: -1 }) // always sort for consistent results
      .skip(skip)
      .limit(limit);

    const totalPages = Math.ceil(total / limit);

    res.json({
      data: users,
      pagination: {
        currentPage: page,
        totalPages,
        totalItems: total,
        itemsPerPage: limit,
        hasNextPage: page < totalPages,
        hasPrevPage: page > 1,
      },
    });
  } catch (err) {
    res.status(500).json({ error: "Server Error" });
  }
});

⚛️ Frontend (React Example)

import { useState, useEffect } from "react";
import axios from "axios";

export default function PaginatedUsers() {
  const [users, setUsers] = useState([]);
  const [pagination, setPagination] = useState({});
  const [loading, setLoading] = useState(false);

  const fetchUsers = async (page = 1) => {
    setLoading(true);
    const res = await axios.get(`/api/users?page=${page}&limit=10`);
    setUsers(res.data.data);
    setPagination(res.data.pagination);
    setLoading(false);
  };

  useEffect(() => {
    fetchUsers(1);
  }, []);

  const goToPage = (p) => {
    if (p >= 1 && p <= pagination.totalPages) fetchUsers(p);
  };

  return (
    <div>
      <h2>Users (Page {pagination.currentPage}/{pagination.totalPages})</h2>
      {loading && <p>Loading...</p>}
      <ul>
        {users.map(u => <li key={u._id}>{u.name} — {u.email}</li>)}
      </ul>

      <div className="flex gap-2 mt-3">
        <button disabled={!pagination.hasPrevPage} onClick={() => goToPage(pagination.currentPage - 1)}>Prev</button>
        <button disabled={!pagination.hasNextPage} onClick={() => goToPage(pagination.currentPage + 1)}>Next</button>
      </div>
    </div>
  );
}

✅ Pros:

Easy to implement
Supports jumping to any page

❌ Cons:

skip() becomes slow for large collections (e.g., skip(100000) scans 100k docs)
Inconsistent if data changes while paging

🔹 5. Cursor-Based Pagination (Efficient for Feeds)

Instead of page, send a cursor (usually last item’s _id or timestamp).

Example:

GET /api/users?cursor=652aab234b8f6&limit=10

🧩 Backend (MongoDB + Express)

app.get("/api/users", async (req, res) => {
  try {
    const limit = parseInt(req.query.limit) || 10;
    const cursor = req.query.cursor;

    let query = {};
    if (cursor) query = { _id: { $gt: cursor } };

    const users = await User.find(query)
      .sort({ _id: 1 })
      .limit(limit + 1); // one extra to detect next page

    const hasMore = users.length > limit;
    const sliced = hasMore ? users.slice(0, -1) : users;
    const nextCursor = hasMore ? sliced[sliced.length - 1]._id : null;

    res.json({ 
      data: sliced, 
      pagination: { nextCursor, hasMore } 
    });
  } catch (err) {
    res.status(500).json({ error: "Server Error" });
  }
});

⚛️ Frontend (Infinite Scroll Example)

import { useState, useEffect } from "react";
import axios from "axios";

export default function InfiniteScrollUsers() {
  const [users, setUsers] = useState([]);
  const [nextCursor, setNextCursor] = useState(null);
  const [hasMore, setHasMore] = useState(true);

  const fetchUsers = async () => {
    if (!hasMore) return;
    const url = nextCursor
      ? `/api/users?cursor=${nextCursor}&limit=10`
      : `/api/users?limit=10`;
    const res = await axios.get(url);
    setUsers(prev => [...prev, ...res.data.data]);
    setNextCursor(res.data.pagination.nextCursor);
    setHasMore(res.data.pagination.hasMore);
  };

  useEffect(() => { fetchUsers(); }, []);

  useEffect(() => {
    const observer = new IntersectionObserver(entries => {
      if (entries[0].isIntersecting && hasMore) fetchUsers();
    });
    const sentinel = document.getElementById("sentinel");
    if (sentinel) observer.observe(sentinel);
    return () => observer.disconnect();
  }, [hasMore]);

  return (
    <div>
      {users.map(u => <div key={u._id}>{u.name}</div>)}
      <div id="sentinel">{hasMore ? "Loading more..." : "No more users"}</div>
    </div>
  );
}

✅ Pros:

Scales to millions of records
Consistent even when data updates

❌ Cons:

Can’t jump to arbitrary pages (like page 8)
Requires ordering by unique key

🔹 6. Keyset Pagination (For Sorted Data)

Used when you sort by a column (like createdAt) and want consistent ordering.

Example Query (MongoDB)

const posts = await Post.find({
  $or: [
    { createdAt: { $lt: lastCreatedAt } },
    { createdAt: lastCreatedAt, _id: { $lt: lastId } }
  ]
})
.sort({ createdAt: -1, _id: -1 })
.limit(limit + 1);

🔹 7. Edge Cases & Best Practices

✅ Handle Empty Dataset

if (total === 0) return res.json({ data: [], message: "No items found" });

✅ Out-of-Range Page
If page > totalPages, return empty list or redirect to last page.

✅ Invalid Inputs

if (isNaN(page) || page < 1) page = 1;
if (limit > 100) limit = 100;

✅ Deleted or Added Items
Prefer cursor-based pagination if frequent changes happen.

✅ Indexing

await db.collection("users").createIndex({ createdAt: -1 });

✅ Limit Deep Pagination

if (page > 100) 
  return res.status(400).json({ error: "Page limit exceeded" });

✅ Frontend Race Condition

const controller = new AbortController();
fetch(url, { signal: controller.signal });
controller.abort(); // cancel old requests

✅ URL Syncing
Keep page in URL query to enable reload and shareable links.

✅ Prefetch Next Page
While user views current page, silently fetch next one in background.

✅ Accessibility
Add aria-label="Next page" etc., for buttons.

🔹 8. Pagination + Search/Filter

Combine safely:

const { page = 1, limit = 10, search = "" } = req.query;
const regex = new RegExp(search, "i");
const total = await User.countDocuments({ name: regex });
const users = await User.find({ name: regex }).skip(skip).limit(limit);

Always recalculate pagination when filters change.

🔹 9. Performance Tips

⚡ Use .lean() in Mongoose to skip hydration (faster):

const users = await User.find().skip(skip).limit(limit).lean();

⚡ Cache first page using Redis or in-memory:

if (page === 1) cache.set("users_page1", users);

⚡ Paginate at DB level, not in code (avoid slicing arrays in JS).

🔹 10. Choosing the Right Type

Use Case	Recommended Type
Admin table	Offset
Social feed	Cursor
Chat messages	Keyset / Cursor
Infinite scroll	Cursor
Analytics data	Keyset
Static lists (few pages)	Offset

⚡ Final Summary

Category	Concept	Backend	Frontend	Edge Cases
Pagination Type	Offset / Cursor / Keyset	`.skip().limit()` / `_id` / timestamps	Paginated or infinite scroll	Out-of-range, Empty, Deep pagination
Why	Performance, UX, scalability	Reduced DB load	Faster rendering	–
When to Use	Always on large datasets	Limit to 10–50 per page	Provide navigation	Reset on filters
Best Practices	Validate params, use indexes, cache, sort consistently	Use `.lean()`	AbortController, Prefetch	Handle concurrent updates

Summary

Pagination may seem simple, but under the hood, it’s a foundational performance pattern every scalable system relies on. From admin dashboards to social media feeds, the way you design your pagination determines how efficiently your application handles growth.

Use offset pagination for classic dashboards and tables.
Use cursor or keyset pagination for real-time feeds or large datasets.
Always validate, cache, and index your queries.
Handle empty, deleted, or concurrent updates gracefully.
On the frontend, sync pagination state with the URL and ensure responsive, accessible navigation.

This marks the first chapter in the Architecture Series — exploring real-world, production-grade MERN stack scalability patterns.
Next up in Part 2, we’ll go deeper into Caching and Data Layer Optimization — how to reduce redundant queries and speed up response times across the stack.