[cron:541ac66e-23e1-40db-8995-866d12d10a0e X Account Scraper - Run Every 30 Min] Run X scraper to check monitored accounts for new deals:

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py

Log results to group chat. If new deals found, mention how many were imported.
Current time: Sunday, February 15th, 2026 — 6:11 AM (UTC)

Return your summary as plain text; it will be delivered automatically. If the task explicitly calls for messaging a specific external recipient, note who/where it should go instead of sending it yourself.


The user is asking me to run an X account scraper to check for new deals. Let me execute this command and see what results we get.

exec

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py


Command still running (session crisp-orbit-2, pid 3080553). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


The command is still running. Let me check the log to see what's happening.


(no output yet)


Let me wait a bit and poll again to see if there's output.

sleep 2 && cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py


Command still running (session warm-ember, pid 3082977). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


Let me poll the second session to see if there's output.


(no new output)

Process still running.


Let me wait a bit more and check again. The script might be taking time to run. Let me check if the file exists first.


#!/usr/bin/env python3
"""
X (Twitter) Deal Scraper for GottaHaveCoups
============================================

Scrapes specified X accounts for deal posts and imports them into the GHC system.

Usage:
    python3 x_deal_scraper.py                    # Run scrape on all accounts
    python3 x_deal_scraper.py --add @account      # Add account to scrape list
    python3 x_deal_scraper.py --remove @account   # Remove account from list
    python3 x_deal_scraper.py --list              # List all monitored accounts
    python3 x_deal_scraper.py --dashboard         # Launch web dashboard
"""

import json
import os
import sys
import re
import argparse
import subprocess
from datetime import datetime, timezone
from urllib.request import Request, urlopen
from typing import List, Dict, Optional

# Config paths
CONFIG_DIR = "/home/crogers2287/openclaw-workspaces/ghc-marketing/config"
CONFIG_FILE = os.path.join(CONFIG_DIR, "x_scraper_accounts.json")
SCRAPED_DEALS_FILE = os.path.join(CONFIG_DIR, "x_scraped_deals.json")

# GHC API endpoints
GHC_API_BASE = "https://www.gottahavecoups.com/api/v1"
GHC_DEALS_API = f"{GHC_API_BASE}/deals"
GHC_IMPORT_API = f"{GHC_API_BASE}/admin/import-deal"  # Adjust based on actual API

# Maya VM connection for twikit
MAYA_HOST = "maya@maya"

def load_config() -> Dict:
    """Load scraper configuration."""
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, 'r') as f:
            return json.load(f)
    return {
        "version": 1,
        "accounts": [],
        "last_scrape": None,
        "scrape_interval_minutes": 30,
        "auto_import": True,
        "priority_boost": True,
        "min_deal_score": 30
    }

def save_config(config: Dict):
    """Save scraper configuration."""
    os.makedirs(CONFIG_DIR, exist_ok=True)
    with open(CONFIG_FILE, 'w') as f:
        json.dump(config, f, indent=2)

def load_scraped_deals() -> List[Dict]:
    """Load previously scraped deals to avoid duplicates."""
    if os.path.exists(SCRAPED_DEALS_FILE):
        with open(SCRAPED_DEALS_FILE, 'r') as f:
            return json.load(f)
    return []

def save_scraped_deals(deals: List[Dict]):
    """Save scraped deals history."""
    with open(SCRAPED_DEALS_FILE, 'w') as f:
        json.dump(deals, f, indent=2)

def add_account(username: str, category: str = "general", notes: str = ""):
    """Add an X account to the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    # Check if already exists
    for acc in config["accounts"]:
        if acc["username"].lower() == username:
            print(f"⚠  @{username} is already in the scrape list")
            return False

    config["accounts"].append({
        "username": username,
        "category": category,
        "notes": notes,
        "added_at": datetime.now(timezone.utc).isoformat(),
        "last_scraped": None,
        "active": True,
        "deals_found": 0
    })

    save_config(config)
    print(f"✅ Added @{username} to scrape list")
    return True

def remove_account(username: str):
    """Remove an X account from the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    original_count = len(config["accounts"])
    config["accounts"] = [a for a in config["accounts"] if a["username"].lower() != username]

    if len(config["accounts"]) < original_count:
        save_config(config)
        print(f"✅ Removed @{username} from scrape list")
        return True
    else:
        print(f"⚠  @{username} not found in scrape list")
        return False

def list_accounts():
    """List all accounts being scraped."""
    config = load_config()

    if not config["accounts"]:
        print("\n📭 No X accounts configured for scraping")
        print("\nAdd accounts with: python3 x_deal_scraper.py --add @username")
        return

    print(f"\n📱 X Accounts to Scrape ({len(config['accounts'])} total)\n")
    print(f"{'Username':<20} {'Category':<12} {'Status':<8} {'Deals':<6} {'Last Scraped'}")
    print("-" * 70)

    for acc in config["accounts"]:
        status = "🟢" if acc.get("active", True) else "🔴"
        last_scraped = acc.get("last_scraped", "Never")[:16] if acc.get("last_scraped") else "Never"
        deals = acc.get("deals_found", 0)
        print(f"@{acc['username']:<19} {acc.get('category', 'general'):<12} {status:<8} {deals:<6} {last_scraped}")

    print("\n" + "-" * 70)
    print(f"Auto-import: {'✅ ON' if config.get('auto_import') else '❌ OFF'}")
    print(f"Priority boost: {'✅ ON' if config.get('priority_boost') else '❌ OFF'}")
    print(f"Min deal score: {config.get('min_deal_score', 30)}")

def extract_deal_info(tweet_text: str, tweet_media: List[str] = None) -> Optional[Dict]:
    """
    Extract deal information from a tweet text.
    Returns deal dict or None if no deal detected.
    """
    deal = {
        "title": None,
        "price": None,
        "original_price": None,
        "discount_percentage": None,
        "store": None,
        "product_url": None,
        "image_url": None,
        "category": None,
        "source_tweet": None
    }

    # Price patterns
    price_patterns = [
        r'\$([\d,]+\.?\d*)',  # $XX.XX
        r'([\d,]+)\s*dollars',  # XX dollars
    ]

    # Look for prices
    prices = []
    for pattern in price_patterns:
        matches = re.findall(pattern, tweet_text, re.IGNORECASE)
        prices.extend([float(p.replace(',', '')) for p in matches])

    if len(prices) >= 1:
        deal["price"] = prices[0]
    if len(prices) >= 2:
        # Assume higher price is original
        deal["original_price"] = max(prices)
        deal["price"] = min(prices)
        if deal["original_price"] > 0:
            deal["discount_percentage"] = round(
                (deal["original_price"] - deal["price"]) / deal["original_price"] * 100
            )

    # Store detection
    store_patterns = {
        "Amazon": [r'amazon\.com', r'amzn\.to', r'@amazon'],
        "Walmart": [r'walmart\.com', r'w-mt\.co', r'@walmart'],
        "Target": [r'target\.com', r'@target'],
        "Best Buy": [r'bestbuy\.com', r'@bestbuy'],
        "Costco": [r'costco\.com', r'@costco'],
        "Home Depot": [r'homedepot\.com', r'@homedepot'],
    }

    for store, patterns in store_patterns.items():
        for pattern in patterns:
            if re.search(pattern, tweet_text, re.IGNORECASE):
                deal["store"] = store
                break
        if deal["store"]:
            break

    # URL extraction
    url_pattern = r'https?://[^\s<>"{}|\\^`\[\]]+'
    urls = re.findall(url_pattern, tweet_text)
    if urls:
        deal["product_url"] = urls[0]

    # Title extraction - look for product mentions
    title_patterns = [
        r'(?:Get|Grab|Score| snag )\s*([^\n.]+?)(?:\s+for\s+|\s+at\s+|\s+-\s+|\s+–\s+)',
        r'([^\n.]{20,80}?)(?:\s+is\s+\$|\s+only\s+\$|\s+just\s+\$)',
    ]

    for pattern in title_patterns:
        match = re.search(pattern, tweet_text, re.IGNORECASE)
        if match:
            deal["title"] = match.group(1).strip()[:150]
            break

    # If no title found, use first line
    if not deal["title"]:
        first_line = tweet_text.split('\n')[0][:150]
        deal["title"] = first_line

    # Category detection
    category_keywords = {
        1: ["tech", "electronics", "laptop", "phone", "headphones", "gaming", "tv", "computer"],
        2: ["home", "kitchen", "furniture", "decor", "appliance", "tool"],
        3: ["fashion", "clothing", "shoes", "apparel", "jewelry", "watch"],
        4: ["toy", "game", "lego", "nintendo", "playstation", "xbox"],
        5: ["sports", "outdoor", "fitness", "gym", "camping", "exercise"],
        8: ["book", "kindle", "novel", "audiobook"],
        9: ["food", "grocery", "snack", "coffee", "tea"],
        10: ["pet", "dog", "cat", "animal"],
    }

    text_lower = tweet_text.lower()
    for cat_id, keywords in category_keywords.items():
        for kw in keywords:
            if kw in text_lower:
                deal["category"] = cat_id
                break
        if deal["category"]:
            break

    # Check if this looks like a deal
    if deal["price"] and (deal["store"] or deal["discount_percentage"]):
        return deal

    return None

def scrape_account_tweets(username: str, limit: int = 20) -> List[Dict]:
    """
    Scrape recent tweets from an X account using Maya VM Chrome/Playwright.
    Returns list of tweet dicts.
    """
    print(f"🔍 Scraping @{username}...")

    try:
        # Create a scraping script to run on Maya VM
        scrape_script = f'''
import asyncio
from playwright.async_api import async_playwright
import json

async def scrape():
    tweets = []
    async with async_playwright() as p:
        try:
            browser = await p.chromium.connect_over_cdp("http://localhost:9222")
            context = browser.contexts[0]
            page = await context.new_page()

            # Go to user's profile
            await page.goto(f"https://x.com/{username}", wait_until="networkidle", timeout=30000)
            await asyncio.sleep(3)

            # Extract tweets
            tweet_elements = await page.locator('article[data-testid="tweet"]').all()

            for tweet in tweet_elements[:{limit}]:
                try:
                    # Get tweet text
                    text_elem = tweet.locator('[data-testid="tweetText"]')
                    text = await text_elem.inner_text() if await text_elem.count() > 0 else ""

                    # Get timestamp
                    time_elem = tweet.locator('time')
                    timestamp = await time_elem.get_attribute('datetime') if await time_elem.count() > 0 else ""

                    if text:
                        tweets.append({{
                            "text": text,
                            "created_at": timestamp,
                            "media": []
                        }})
                except Exception as e:
                    pass

            await page.close()
        except Exception as e:
            print(f"Browser error: {{e}}")

    print(json.dumps(tweets, default=str))

asyncio.run(scrape())
'''

        # Save script to Maya VM
        script_path = f"/tmp/scrape_{username}.py"
        with open(script_path, 'w') as f:
            f.write(scrape_script)

        # Copy to Maya
        subprocess.run(['scp', '-q', script_path, f"{MAYA_HOST}:/tmp/scrape_{username}.py"], check=True)

        # Run the scraper on Maya
        result = subprocess.run(
            ['ssh', MAYA_HOST, f'python3 /tmp/scrape_{username}.py'],
            capture_output=True,
            text=True,
            timeout=60
        )

        if result.returncode != 0:
            print(f"⚠  Error scraping @{username}: {result.stderr}")
            return []

        # Parse output
        output = result.stdout.strip()
        if output:
            try:
                tweets = json.loads(output)
                print(f"✅ Found {len(tweets)} tweets from @{username}")
                return tweets
            except json.JSONDecodeError:
                print(f"⚠  Invalid JSON from @{username}")
                return []

        return []

    except subprocess.TimeoutExpired:
        print(f"⏱  Timeout scraping @{username}")
        return []
    except Exception as e:
        print(f"⚠  Error scraping @{username}: {e}")
        return []

def import_deal_to_ghc(deal: Dict, source_account: str) -> bool:
    """
    Import a scraped deal into the GottaHaveCoups system.
    Returns True if successful.
    """
    # For now, we'll create a draft deal that can be reviewed
    # In production, this would POST to the GHC API

    draft_deal = {
        "title": deal["title"],
        "description": f"Scraped from X @{source_account}",
        "price": deal["price"],
        "original_price": deal.get("original_price"),
        "discount_percentage": deal.get("discount_percentage"),
        "store_name": deal.get("store", "Unknown"),
        "category_id": deal.get("category", 1),
        "source": f"X:@{source_account}",
        "deal_url": deal.get("product_url", ""),
        "image_url": deal.get("image_url", ""),
        "quality_score": 50,  # Boost priority
        "is_hot": True,
        "is_featured": True,
        "scraped_at": datetime.now(timezone.utc).isoformat(),
        "needs_review": True  # Flag for manual review
    }

    # Save to scraped deals file
    scraped = load_scraped_deals()

    # Check for duplicates (by title + price)
    for existing in scraped:
        if (existing.get("title") == draft_deal["title"] and
            existing.get("price") == draft_deal["price"]):
            print(f"   ⚠  Duplicate deal skipped: {draft_deal['title'][:50]}...")
            return False

    scraped.append(draft_deal)
    save_scraped_deals(scraped)

    print(f"   ✅ Imported: {draft_deal['title'][:60]}...")
    return True

def run_scraper():
    """Main scraper routine."""
    config = load_config()

    if not config["accounts"]:
        print("📭 No accounts configured. Add accounts with --add @username")
        return

    print(f"\n🚀 Starting X Deal Scraper")
    print(f"   Accounts: {len(config['accounts'])}")
    print(f"   Auto-import: {'ON' if config.get('auto_import') else 'OFF'}")
    print()

    total_deals = 0
    new_deals = 0

    for account in config["accounts"]:
        if not account.get("active", True):
            continue

        username = account["username"]
        tweets = scrape_account_tweets(username, limit=20)

        account_deals = 0
        for tweet in tweets:
            deal_info = extract_deal_info(tweet.get("text", ""), tweet.get("media", []))
            if deal_info:
                total_deals += 1
                if import_deal_to_ghc(deal_info, username):
                    new_deals += 1
                    account_deals += 1

        # Update account stats
        account["last_scraped"] = datetime.now(timezone.utc).isoformat()
        account["deals_found"] = account.get("deals_found", 0) + account_deals

        print()

    config["last_scrape"] = datetime.now(timezone.utc).isoformat()
    save_config(config)

    print(f"📊 Scrape Complete")
    print(f"   Total deals found: {total_deals}")
    print(f"   New deals imported: {new_deals}")
    print(f"   Next scrape: ~{config['scrape_interval_minutes']} minutes")

def launch_dashboard():
    """Launch the web dashboard for managing accounts."""
    dashboard_path = "/home/crogers2287/openclaw-workspaces/ghc-marketing/x-scraper-dashboard.html"
    print(f"🌐 Launching dashboard...")
    print(f"   File: {dashboard_path}")

    # Create dashboard HTML if it doesn't exist
    if not os.path.exists(dashboard_path):
        create_dashboard_html(dashboard_path)

    # Open in browser
    os.system(f"xdg-open {dashboard_path} 2>/dev/null || open {dashboard_path} 2>/dev/null || echo 'Open {dashboard_path} in your browser'")

def create_dashboard_html(path: str):
    """Create the HTML dashboard file."""
    html = '''<!DOCTYPE html>

    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>X Deal Scraper Dashboard</title>

🐦 X Deal Scraper

Monitor X accounts for deals and import to GottaHaveCoups

📊 Stats

0

Accounts

0

Deals Found

--

Last Scrape

➕ Add Account

                <input type="text" id="new-username" placeholder="@username (without @)">
                <input type="text" id="new-category" placeholder="Category (optional)" style="flex: 0.5;">
                <button onclick="addAccount()">Add Account</button>

📱 Monitored Accounts

No accounts configured

⚙ Settings

                Auto-import deals

                    <input type="checkbox" id="auto-import" checked>

                Priority boost (post first)

                    <input type="checkbox" id="priority-boost" checked>

                Scrape interval: 30 minutes

        <button class="run-btn" onclick="runScraper()">🚀 Run Scraper Now</button>

'''

    with open(path, 'w') as f:
        f.write(html)

def main():
    parser = argparse.ArgumentParser(
        description="X (Twitter) Deal Scraper for GottaHaveCoups",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
    python3 x_deal_scraper.py --add @deals_account
    python3 x_deal_scraper.py --add @techdeals --category tech
    python3 x_deal_scraper.py --list
    python3 x_deal_scraper.py --remove @deals_account
    python3 x_deal_scraper.py                    # Run scraper
    python3 x_deal_scraper.py --dashboard       # Open web dashboard
        """
    )

    parser.add_argument('--add', metavar='@username', help='Add an X account to scrape')
    parser.add_argument('--remove', metavar='@username', help='Remove an X account from scraping')
    parser.add_argument('--list', action='store_true', help='List all monitored accounts')
    parser.add_argument('--category', default='general', help='Category for the account (with --add)')
    parser.add_argument('--notes', default='', help='Notes about the account (with --add)')
    parser.add_argument('--dashboard', action='store_true', help='Launch web dashboard')
    parser.add_argument('--run', action='store_true', help='Run scraper (default behavior)')

    args = parser.parse_args()

    if args.add:
        add_account(args.add, args.category, args.notes)
    elif args.remove:
        remove_account(args.remove)
    elif args.list:
        list_accounts()
    elif args.dashboard:
        launch_dashboard()
    else:
        # Default: run scraper
        run_scraper()

if __name__ == "__main__":
    main()


This is a Python script that scrapes X (Twitter) for deals. It uses a Maya VM to do the actual scraping via Playwright/Chrome. Let me check if the config directory exists and see the current state of accounts.

Let me check the current state and wait for the scraper to complete:


X (Twitter) Accounts to Scrape for Deals


Let me wait a bit more for the process to complete and check its output.


No active session found for crisp-orbit-2


Killed session warm-ember.


It looks like the process was taking too long. Let me run it fresh with a longer timeout.

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && timeout 90 python3 scripts/x_deal_scraper.py 2>&1


Command still running (session clear-lobster, pid 3086905). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


Let me wait and then check the output again.


Command still running (session tender-shoal, pid 3089228). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


Let me poll the process to see if it finished.


(no output yet)


The script seems to be stuck waiting for something. Let me kill it and run it differently, maybe checking if there are any dependency issues. Let me try running the script with unbuffered output to see what's happening.


No active session found for clear-lobster


Killed session tender-shoal.


The script seems to be hanging. Based on my reading of the code, it's trying to SSH to a Maya VM that may not be accessible. Let me try running a simpler version or check if Maya is reachable.

ssh -o ConnectTimeout=5 maya@maya "echo 'Maya VM is reachable'" 2>&1


Warning: Permanently added 'maya' (ED25519) to the list of known hosts.
Maya VM is reachable


Maya is reachable. Let me try running the script with a timeout to capture output, or run it differently to see what's happening.

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 -u scripts/x_deal_scraper.py


Command still running (session tide-comet, pid 3092781). Use process (list/poll/log/write/kill/clear/remove) for follow-up.

🚀 Starting X Deal Scraper
   Accounts: 3
   Auto-import: ON

🔍 Scraping @dealsfinderio...


Let me wait a moment and then check the log.


Command still running (session salty-summit, pid 3094934). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


🚀 Starting X Deal Scraper
   Accounts: 3
   Auto-import: ON

🔍 Scraping @dealsfinderio...
✅ Found 0 tweets from @dealsfinderio

🔍 Scraping @glitcheddeals...
✅ Found 0 tweets from @glitcheddeals

🔍 Scraping @fatkiddeals...


Ah, the script is running! It's just slow. Let me wait a bit more for it to finish the last account.


Command still running (session clear-reef, pid 3097586). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


🚀 Starting X Deal Scraper
   Accounts: 3
   Auto-import: ON

🔍 Scraping @dealsfinderio...
✅ Found 0 tweets from @dealsfinderio

🔍 Scraping @glitcheddeals...
✅ Found 0 tweets from @glitcheddeals

🔍 Scraping @fatkiddeals...
✅ Found 0 tweets from @fatkiddeals

📊 Scrape Complete
   Total deals found: 0
   New deals imported: 0
   Next scrape: ~30 minutes


The scraper completed successfully. It found 0 tweets from all 3 accounts. This is likely because X/Twitter requires authentication to view tweets now, and the scraper isn't authenticated. Let me update the config to show when the last scrape was done.

Now let me summarize the results for the user.

X Deal Scraper completed successfully.

**Results for Sunday, February 15th, 2026 — 6:11 AM (UTC):**

📊 **Scrape Summary**
- Accounts monitored: 3
  - @dealsfinderio
  - @glitcheddeals
  - @fatkiddeals
- Total deals found: **0**
- New deals imported: **0**

All accounts were successfully scraped, but no tweets were retrieved. X/Twitter now requires authentication to view tweet content, which may be limiting the scraper's ability to fetch posts. The scraper connects to the Maya VM's Chrome/Playwright instance but returned empty results for all accounts.

**Next scrape:** ~30 minutes (scheduled)