How to Build a Competitor Monitoring Pipeline in 30 Minutes

Posted on by — No Comments ↓

How to Build a Competitor Monitoring Pipeline in 30 Minutes

Stop checking competitor prices manually. Here’s the exact stack I use to monitor 50+ competitor URLs in real-time, without getting blocked.

What We’re Building

A system that:

  • Checks competitor URLs every 6 hours
  • Extracts prices, titles, and availability
  • Alerts you only when something significant changes
  • Doesn’t get blocked by anti-bot protection

The Stack

  • XCrawl — residential proxies + scraping API
  • Airtable — free, simple data storage with nice UI
  • GitHub Actions — free cron scheduling
  • Telegram bot — instant alerts on your phone

Total cost: under $20/month for 50 URLs.

Step 1: Define What You’re Monitoring

Create a simple Airtable table with columns:

  • url — competitor product page
  • name — competitor or product name
  • last_price — price from last check
  • last_checked — timestamp
  • change_alert — threshold % for alerts

Start with 10-20 URLs. You can always add more later.

Step 2: The Scraping Function

const { XCrawlScraper } = require('xcrawl-scraper');

const xcrawl = new XCrawlScraper({
  apiKey: process.env.XCRAWL_API_KEY,
});

async function checkCompetitor(url) {
  try {
    const result = await xcrawl.scrapeMarkdown(url, {
      render: true,
    });

    const markdown = result.data.markdown;

    const priceMatch = markdown.match(/$[d,]+.?d*/);
    const price = priceMatch
      ? parseFloat(priceMatch[0].replace(/[$,]/g, ''))
      : null;

    const titleMatch = markdown.match(/#s+(.+)/);
    const title = titleMatch ? titleMatch[1].trim() : 'Unknown';

    return {
      url,
      price,
      title,
      checkedAt: new Date().toISOString(),
      success: true,
    };
  } catch (err) {
    return {
      url,
      price: null,
      title: null,
      checkedAt: new Date().toISOString(),
      success: false,
      error: err.message,
    };
  }
}

Step 3: Alert Logic

You don’t want to be alerted every time a price changes by $0.01. You want to know when something significant happens: 

function shouldAlert(lastPrice, currentPrice, threshold = 5) {
  if (!lastPrice || !currentPrice) return false;

  const change = Math.abs(((currentPrice - lastPrice) / lastPrice) * 100);
  return change >= threshold;
}

function formatAlert(competitor, lastPrice, currentPrice) {
  const direction = currentPrice < lastPrice ? 'DROPPED' : 'INCREASED';
  const change = Math.abs(((currentPrice - lastPrice) / lastPrice) * 100);

  return direction + ' ' + competitor.name + '
' +
    'Price: $' + lastPrice + ' -> ' + currentPrice + ' (' + change.toFixed(1) + '% change)
' +
    'URL: ' + competitor.url;
}

Step 4: GitHub Actions Scheduling

Create .github/workflows/competitor-check.yml: 

name: Competitor Price Monitor
on:
  schedule:
    - cron: '0 */6 * * *'  # Every 6 hours
  workflow_dispatch:        # Manual trigger

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install xcrawl-scraper airtable

      - name: Run competitor checks
        env:
          XCRAWL_API_KEY: ${{ secrets.XCRAWL_API_KEY }}
          AIRTABLE_API_KEY: ${{ secrets.AIRTABLE_API_KEY }}
        run: node competitor-check.js

      - name: Send alert to Telegram
        if: matrix.changed == true
        env:
          TG_BOT_TOKEN: ${{ secrets.TG_BOT_TOKEN }}
          TG_CHAT_ID: ${{ secrets.TG_CHAT_ID }}
        run: |
          curl -s -X POST "https://api.telegram.org/bot$TG_BOT_TOKEN/sendMessage" -d "chat_id=$TG_CHAT_ID" -d "text=${{ matrix.alertMessage }}"

Step 5: The Full Check Loop

async function runChecks() {
  const competitors = await airtable.fetchCompetitors();
  const alerts = [];

  for (const competitor of competitors) {
    const result = await checkCompetitor(competitor.url);

    if (result.success) {
      await airtable.updateRecord(competitor.id, {
        last_price: result.price,
        last_checked: result.checkedAt,
        last_title: result.title,
      });

      if (shouldAlert(competitor.last_price, result.price)) {
        alerts.push(formatAlert(competitor, competitor.last_price, result.price));
      }
    } else {
      console.log('Failed: ' + competitor.url + ' - ' + result.error);
    }

    // Be respectful - don't hammer the servers
    await sleep(2000 + Math.random() * 3000);
  }

  if (alerts.length > 0) {
    await sendTelegramAlerts(alerts);
  }

  console.log('Checked ' + competitors.length + ' competitors. ' + alerts.length + ' alerts.');
}

What This Actually Looks Like in Practice

Here’s the alert you’d get on your phone: 

DROPPED Amazon Basics Mouse
Price: $29.99 -> $24.99 (16.7% change)
URL: amazon.com/dp/B07ZVPQR5M

That’s it. No dashboards to check. No manual research. Just the signal you need to act.

Key Decisions That Made This Work

  1. Residential proxies — datacenter IPs get blocked immediately on Amazon, Best Buy, etc.
  2. 6-hour check interval — fast enough to catch changes, slow enough to not look like a bot
  3. Store everything in Airtable — the UI makes it easy to spot patterns over time
  4. Telegram over email — faster, and your phone will buzz to wake you up if needed

The ROI

My client monitors 47 competitor URLs across 8 product categories. In the first month:

  • Caught 3 competitor price drops within hours
  • Adjusted their own pricing proactively
  • Estimated incremental revenue: $2,400 that month

System cost: $49/month (XCrawl) + free (GitHub Actions, Airtable, Telegram).

Building something similar? I’ve open-sourced the monitoring template. Link in bio. Questions below.

Leave a Reply