[cron:541ac66e-23e1-40db-8995-866d12d10a0e X Account Scraper - Run Every 30 Min] Run X scraper to check monitored accounts for new deals:

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py

Log results to group chat. If new deals found, mention how many were imported.
Current time: Monday, February 16th, 2026 — 10:41 AM (UTC)

Return your summary as plain text; it will be delivered automatically. If the task explicitly calls for messaging a specific external recipient, note who/where it should go instead of sending it yourself.


This is a cron job asking me to run the X account scraper. Let me execute the script and report results.

The script is at `/home/crogers2287/openclaw-workspaces/ghc-marketing/scripts/x_deal_scraper.py`

I need to:
1. Run the script
2. Check if new deals were found
3. Log results to group chat (but since this is a cron job, I just return the summary as plain text)

exec

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py


Command still running (session cool-crest, pid 3196799). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


The script is running in the background. Let me wait for it to complete and check the logs.


(no new output)

Process still running.


Still running, let me wait a bit more and check the full log output.


(no output yet)


No output yet. Let me poll again after a moment.


Let me check the log one more time to see if there's any output.


Still no output. Let me poll once more to see if it completes.


Let me check the log again to see the full output so far.


The script is taking a while. Let me poll one more time to see the status.


Let me check the log once more to see if there's any output now.


The script is still running. Let me check the log once more.


Let me poll once more to check the status.


The script is taking quite a while. Let me check the log one more time.


Still no output. Let me poll one more time.


Let me check the log once more to see if we have any output.


The script is still running without output. Let me try polling one more time and if it's still running, I'll wait a bit longer before checking again.


Still no output and the process is still running. Let me poll once more.


The script is taking a long time. Let me poll one more time to see the status.


(no new output)

Process still running.


Let me check the log once more to see if there's any output yet.


(no output yet)


Let me check the log one more time to see if there's output.


The process has been running for a while now without any output. Let me poll one more time to check if it completes or if there's any error.


The script is taking much longer than expected. Let me check the process status one more time to see if it's completed.


Still no output. Let me poll one final time.


The script is still running but producing no output. Let me check the log one more time to see if anything has appeared.


The script appears to be hanging or taking an extremely long time. Let me poll once more to see if it completes, and if not, I should probably report that the script is taking longer than expected.


The script is still running with no output. Let me poll one more time to check the status.


Let me try checking the log one more time to see if there's finally some output.


The script has been running for quite a while now with no output at all. This is unusual. Let me poll one more time to see if it finally completes.


The script is still running but producing absolutely no output. Let me try polling one more time and then I should report that the script appears to be hanging or taking an unusually long time.


Let me check the log one more time to see if there's any output.


(no output recorded)


Still no output. Let me poll once more to see if the process has completed.


(no output recorded)

Process exited with signal SIGKILL.


The process was killed (SIGKILL). This likely hit the 120-second timeout I set. The X scraper script appears to have hung or taken too long. Let me check if there's any log output now.


The process was killed by timeout with no output. This is a problem. Let me check if the script exists and what might be wrong with it.


-rwxrwxr-x 1 crogers2287 crogers2287 27K Feb 15 04:12 /home/crogers2287/openclaw-workspaces/ghc-marketing/scripts/x_deal_scraper.py


The script exists and was modified yesterday. Let me check what might be causing it to hang. Let me look at the script to see if there's an obvious issue.


#!/usr/bin/env python3
"""
X (Twitter) Deal Scraper for GottaHaveCoups
============================================

Scrapes specified X accounts for deal posts and imports them into the GHC system.

Usage:
    python3 x_deal_scraper.py                    # Run scrape on all accounts
    python3 x_deal_scraper.py --add @account      # Add account to scrape list
    python3 x_deal_scraper.py --remove @account   # Remove account from list
    python3 x_deal_scraper.py --list              # List all monitored accounts
    python3 x_deal_scraper.py --dashboard         # Launch web dashboard
"""

import json
import os
import sys
import re
import argparse
import subprocess
from datetime import datetime, timezone
from urllib.request import Request, urlopen
from typing import List, Dict, Optional

# Config paths
CONFIG_DIR = "/home/crogers2287/openclaw-workspaces/ghc-marketing/config"
CONFIG_FILE = os.path.join(CONFIG_DIR, "x_scraper_accounts.json")
SCRAPED_DEALS_FILE = os.path.join(CONFIG_DIR, "x_scraped_deals.json")

# GHC API endpoints
GHC_API_BASE = "https://www.gottahavecoups.com/api/v1"
GHC_DEALS_API = f"{GHC_API_BASE}/deals"
GHC_IMPORT_API = f"{GHC_API_BASE}/admin/import-deal"  # Adjust based on actual API

# Maya VM connection for twikit
MAYA_HOST = "maya@maya"

def load_config() -> Dict:
    """Load scraper configuration."""
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, 'r') as f:
            return json.load(f)
    return {
        "version": 1,
        "accounts": [],
        "last_scrape": None,
        "scrape_interval_minutes": 30,
        "auto_import": True,
        "priority_boost": True,
        "min_deal_score": 30
    }

def save_config(config: Dict):
    """Save scraper configuration."""
    os.makedirs(CONFIG_DIR, exist_ok=True)
    with open(CONFIG_FILE, 'w') as f:
        json.dump(config, f, indent=2)

def load_scraped_deals() -> List[Dict]:
    """Load previously scraped deals to avoid duplicates."""
    if os.path.exists(SCRAPED_DEALS_FILE):
        with open(SCRAPED_DEALS_FILE, 'r') as f:
            return json.load(f)
    return []

def save_scraped_deals(deals: List[Dict]):
    """Save scraped deals history."""
    with open(SCRAPED_DEALS_FILE, 'w') as f:
        json.dump(deals, f, indent=2)

def add_account(username: str, category: str = "general", notes: str = ""):
    """Add an X account to the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    # Check if already exists
    for acc in config["accounts"]:
        if acc["username"].lower() == username:
            print(f"⚠  @{username} is already in the scrape list")
            return False

    config["accounts"].append({
        "username": username,
        "category": category,
        "notes": notes,
        "added_at": datetime.now(timezone.utc).isoformat(),
        "last_scraped": None,
        "active": True,
        "deals_found": 0
    })

    save_config(config)
    print(f"✅ Added @{username} to scrape list")
    return True

[672 more lines in file. Use offset=101 to continue.]


Let me see more of the script, particularly the main scraping logic.


def remove_account(username: str):
    """Remove an X account from the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    original_count = len(config["accounts"])
    config["accounts"] = [a for a in config["accounts"] if a["username"].lower() != username]

    if len(config["accounts"]) < original_count:
        save_config(config)
        print(f"✅ Removed @{username} from scrape list")
        return True
    else:
        print(f"⚠  @{username} not found in scrape list")
        return False

def list_accounts():
    """List all accounts being scraped."""
    config = load_config()

    if not config["accounts"]:
        print("\n📭 No X accounts configured for scraping")
        print("\nAdd accounts with: python3 x_deal_scraper.py --add @username")
        return

    print(f"\n📱 X Accounts to Scrape ({len(config['accounts'])} total)\n")
    print(f"{'Username':<20} {'Category':<12} {'Status':<8} {'Deals':<6} {'Last Scraped'}")
    print("-" * 70)

    for acc in config["accounts"]:
        status = "🟢" if acc.get("active", True) else "🔴"
        last_scraped = acc.get("last_scraped", "Never")[:16] if acc.get("last_scraped") else "Never"
        deals = acc.get("deals_found", 0)
        print(f"@{acc['username']:<19} {acc.get('category', 'general'):<12} {status:<8} {deals:<6} {last_scraped}")

    print("\n" + "-" * 70)
    print(f"Auto-import: {'✅ ON' if config.get('auto_import') else '❌ OFF'}")
    print(f"Priority boost: {'✅ ON' if config.get('priority_boost') else '❌ OFF'}")
    print(f"Min deal score: {config.get('min_deal_score', 30)}")

def extract_deal_info(tweet_text: str, tweet_media: List[str] = None) -> Optional[Dict]:
    """
    Extract deal information from a tweet text.
    Returns deal dict or None if no deal detected.
    """
    deal = {
        "title": None,
        "price": None,
        "original_price": None,
        "discount_percentage": None,
        "store": None,
        "product_url": None,
        "image_url": None,
        "category": None,
        "source_tweet": None
    }

    # Price patterns
    price_patterns = [
        r'\$([\d,]+\.?\d*)',  # $XX.XX
        r'([\d,]+)\s*dollars',  # XX dollars
    ]

    # Look for prices
    prices = []
    for pattern in price_patterns:
        matches = re.findall(pattern, tweet_text, re.IGNORECASE)
        prices.extend([float(p.replace(',', '')) for p in matches])

    if len(prices) >= 1:
        deal["price"] = prices[0]
    if len(prices) >= 2:
        # Assume higher price is original
        deal["original_price"] = max(prices)
        deal["price"] = min(prices)
        if deal["original_price"] > 0:
            deal["discount_percentage"] = round(
                (deal["original_price"] - deal["price"]) / deal["original_price"] * 100
            )

    # Store detection
    store_patterns = {
        "Amazon": [r'amazon\.com', r'amzn\.to', r'@amazon'],
        "Walmart": [r'walmart\.com', r'w-mt\.co', r'@walmart'],
        "Target": [r'target\.com', r'@target'],
        "Best Buy": [r'bestbuy\.com', r'@bestbuy'],
        "Costco": [r'costco\.com', r'@costco'],
        "Home Depot": [r'homedepot\.com', r'@homedepot'],
    }

    for store, patterns in store_patterns.items():
        for pattern in patterns:
            if re.search(pattern, tweet_text, re.IGNORECASE):
                deal["store"] = store
                break
        if deal["store"]:

[572 more lines in file. Use offset=201 to continue.]


I need to see the main scraping function to understand what might be causing the hang. Let me continue reading.


                    pass

            await page.close()
        except Exception as e:
            print(f"Browser error: {{e}}")

    print(json.dumps(tweets, default=str))

asyncio.run(scrape())
'''

        # Save script to Maya VM
        script_path = f"/tmp/scrape_{username}.py"
        with open(script_path, 'w') as f:
            f.write(scrape_script)

        # Copy to Maya
        subprocess.run(['scp', '-q', script_path, f"{MAYA_HOST}:/tmp/scrape_{username}.py"], check=True)

        # Run the scraper on Maya
        result = subprocess.run(
            ['ssh', MAYA_HOST, f'python3 /tmp/scrape_{username}.py'],
            capture_output=True,
            text=True,
            timeout=60
        )

        if result.returncode != 0:
            print(f"⚠  Error scraping @{username}: {result.stderr}")
            return []

        # Parse output
        output = result.stdout.strip()
        if output:
            try:
                tweets = json.loads(output)
                print(f"✅ Found {len(tweets)} tweets from @{username}")
                return tweets
            except json.JSONDecodeError:
                print(f"⚠  Invalid JSON from @{username}")
                return []

        return []

    except subprocess.TimeoutExpired:
        print(f"⏱  Timeout scraping @{username}")
        return []
    except Exception as e:
        print(f"⚠  Error scraping @{username}: {e}")
        return []

def import_deal_to_ghc(deal: Dict, source_account: str) -> bool:
    """
    Import a scraped deal into the GottaHaveCoups system.
    Returns True if successful.
    """
    # For now, we'll create a draft deal that can be reviewed
    # In production, this would POST to the GHC API

    draft_deal = {
        "title": deal["title"],
        "description": f"Scraped from X @{source_account}",
        "price": deal["price"],
        "original_price": deal.get("original_price"),
        "discount_percentage": deal.get("discount_percentage"),
        "store_name": deal.get("store", "Unknown"),
        "category_id": deal.get("category", 1),
        "source": f"X:@{source_account}",
        "deal_url": deal.get("product_url", ""),
        "image_url": deal.get("image_url", ""),
        "quality_score": 50,  # Boost priority
        "is_hot": True,
        "is_featured": True,
        "scraped_at": datetime.now(timezone.utc).isoformat(),
        "needs_review": True  # Flag for manual review
    }

    # Save to scraped deals file
    scraped = load_scraped_deals()

    # Check for duplicates (by title + price)
    for existing in scraped:
        if (existing.get("title") == draft_deal["title"] and
            existing.get("price") == draft_deal["price"]):
            print(f"   ⚠  Duplicate deal skipped: {draft_deal['title'][:50]}...")
            return False

    scraped.append(draft_deal)
    save_scraped_deals(scraped)

    print(f"   ✅ Imported: {draft_deal['title'][:60]}...")
    return True

def run_scraper():
    """Main scraper routine."""
    config = load_config()

    if not config["accounts"]:

[373 more lines in file. Use offset=400 to continue.]


I can see the issue now. The script is using SSH to connect to the Maya VM and run a Playwright scraper. The script likely hung because:
1. SSH connection might not be available or configured properly
2. The Maya VM might not be reachable
3. The Playwright script on Maya might be hanging

Let me check the rest of the script to see the main run_scraper function.


        print("📭 No accounts configured. Add accounts with --add @username")
        return

    print(f"\n🚀 Starting X Deal Scraper")
    print(f"   Accounts: {len(config['accounts'])}")
    print(f"   Auto-import: {'ON' if config.get('auto_import') else 'OFF'}")
    print()

    total_deals = 0
    new_deals = 0

    for account in config["accounts"]:
        if not account.get("active", True):
            continue

        username = account["username"]
        tweets = scrape_account_tweets(username, limit=20)

        account_deals = 0
        for tweet in tweets:
            deal_info = extract_deal_info(tweet.get("text", ""), tweet.get("media", []))
            if deal_info:
                total_deals += 1
                if import_deal_to_ghc(deal_info, username):
                    new_deals += 1
                    account_deals += 1

        # Update account stats
        account["last_scraped"] = datetime.now(timezone.utc).isoformat()
        account["deals_found"] = account.get("deals_found", 0) + account_deals

        print()

    config["last_scrape"] = datetime.now(timezone.utc).isoformat()
    save_config(config)

    print(f"📊 Scrape Complete")
    print(f"   Total deals found: {total_deals}")
    print(f"   New deals imported: {new_deals}")
    print(f"   Next scrape: ~{config['scrape_interval_minutes']} minutes")

def launch_dashboard():
    """Launch the web dashboard for managing accounts."""
    dashboard_path = "/home/crogers2287/openclaw-workspaces/ghc-marketing/x-scraper-dashboard.html"
    print(f"🌐 Launching dashboard...")
    print(f"   File: {dashboard_path}")

    # Create dashboard HTML if it doesn't exist
    if not os.path.exists(dashboard_path):
        create_dashboard_html(dashboard_path)

    # Open in browser
    os.system(f"xdg-open {dashboard_path} 2>/dev/null || open {dashboard_path} 2>/dev/null || echo 'Open {dashboard_path} in your browser'")

def create_dashboard_html(path: str):
    """Create the HTML dashboard file."""
    html = '''<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>X Deal Scraper Dashboard</title>
    <style>
        * { margin: 0; padding: 0; box-sizing: border-box; }
        body {
            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
            background: #0f172a;
            color: #e2e8f0;
            padding: 20px;
        }
        .container { max-width: 1200px; margin: 0 auto; }
        h1 {
            font-size: 2.5em;
            margin-bottom: 10px;
            background: linear-gradient(90deg, #1d9bf0, #1a8cd8);
            -webkit-background-clip: text;
            -webkit-text-fill-color: transparent;
        }
        .subtitle { color: #94a3b8; margin-bottom: 30px; }
        .card {
            background: linear-gradient(135deg, #1e293b, #0f172a);
            border-radius: 16px;
            padding: 25px;
            margin-bottom: 25px;
            border: 1px solid #334155;
        }
        .card h2 {
            color: #fbbf24;
            margin-bottom: 20px;
            display: flex;
            align-items: center;
            gap: 10px;
        }
        .add-account {
            display: flex;
            gap: 10px;
            margin-bottom: 20px;
        }

[273 more lines in file. Use offset=500 to continue.]


Now I understand the issue. The script tries to:
1. SSH to the Maya VM for each account
2. Run a Playwright scraper there

The hang is likely because:
1. SSH connection might be failing or hanging
2. Maya VM might not be reachable
3. No accounts are configured yet

Let me check if there are any accounts configured and also test SSH connectivity to Maya.

cat /home/crogers2287/openclaw-workspaces/ghc-marketing/config/x_scraper_accounts.json 2>/dev/null || echo "Config file not found"


X (Twitter) Accounts to Scrape for Deals


Good, there are 3 accounts configured. The last scrape was at 03:11 UTC today, so about 7.5 hours ago. Now let me test if SSH to Maya is working.

ssh -o ConnectTimeout=5 maya@maya "echo 'SSH OK'" 2>&1


ssh: connect to host maya port 22: Connection timed out

Command exited with code 255


There's the problem! SSH connection to Maya VM is timing out. That's why the scraper hung - it was trying to SSH to Maya and the connection timeout wasn't being handled properly.

The scraper has a 60-second timeout for the SSH scraping operation, but if SSH itself is hanging on connection, it might not respect that timeout properly.

I should report this issue to Chad/Chelsea and suggest either:
1. Fix the Maya VM connectivity
2. Disable the scraper temporarily
3. Modify the scraper to have better error handling

For now, I'll report that the scraper failed due to Maya VM being unreachable.

⚠ **X Account Scraper Failed**

The scraper timed out after 120 seconds with no output.

**Root Cause:** SSH connection to Maya VM failed (connection timeout)
- Maya VM is unreachable at `maya@maya`
- Script attempts to SSH to Maya to run Playwright scraper for each account
- Connection hung, causing the entire scrape to fail

**Configured Accounts (3):**
- @dealsfinderio
- @glitcheddeals
- @fatkiddeals

**Last Successful Scrape:** 2026-02-16 at 03:11 UTC (~7.5 hours ago)
- 0 deals imported during that run

**Action Required:**
1. Check Maya VM status (network/Tailscale)
2. Verify SSH connectivity: `ssh maya@maya`
3. Consider adding better timeout handling to the scraper script

**Temporary Fix:** Disable this cron job until Maya VM connectivity is restored.