[cron:541ac66e-23e1-40db-8995-866d12d10a0e X Account Scraper - Run Every 30 Min] Run X scraper to check monitored accounts for new deals:

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py

Log results to group chat. If new deals found, mention how many were imported.
Current time: Monday, February 16th, 2026 — 4:41 PM (UTC)

Return your summary as plain text; it will be delivered automatically. If the task explicitly calls for messaging a specific external recipient, note who/where it should go instead of sending it yourself.


This is a cron job that wants me to run the X account scraper script and report results. I need to:
1. Run the Python script from the correct directory
2. Check the output for new deals found
3. Report the results to the group chat

Let me execute the script first.

I'll run the X account scraper to check for new deals.

exec

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py


Command still running (session vivid-tidepool, pid 2015196). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


The script is running in the background. Let me poll it to see the results.


(no new output)

Process still running.


Still running, let me wait a bit more and check the full log.


(no output yet)


No output yet. Let me wait for it to finish.


Let me wait a bit longer and then check the log again.


The script is still running. Let me check the log to see if there's any output now.


The script is still running with no output. This might take a while since it's scraping multiple accounts. Let me poll once more and then wait for completion.


Still running. Let me wait a bit more. I'll poll again.


Let me check the log one more time to see if there's any output at all.


The script is still running with no output. Let me poll again to see if it completes.


The script is taking a while. Let me continue polling.


Still running. Let me check the log again to see if there's any output now.


I'll continue polling. The script might be doing network requests which can take time.


Let me check the log to see if there's output now.


This is taking a while. Let me continue polling.


Still running. Let me check log one more time.


Still no output. Let me poll one more time.


(no new output)

Process still running.


The script is still running. Let me continue polling to see when it completes.


(no output yet)


Still running. Let me check the log one more time.


Let me poll again to see if the process has finished.


The script is still running. Let me poll one more time.


This is taking quite a while. Let me poll one more time.


I'll continue polling. The script is definitely still running.


(no output recorded)

Process exited with signal SIGKILL.


The process was killed, probably because it hit the timeout (120 seconds). The script seems to be taking longer than expected to run. This could be because:
1. Network requests to X are slow
2. There are many tweets to process
3. The script is hanging somewhere

Let me check if the script file exists and see what it does, then try running it with a longer timeout or check if there are any issues.

The X scraper script timed out after 2 minutes and was killed. This suggests the script is taking longer than expected to run, possibly due to:
- Slow network requests to X/Twitter
- Rate limiting
- Large number of tweets to process

Let me check the script to see what might be causing the delay:


#!/usr/bin/env python3
"""
X (Twitter) Deal Scraper for GottaHaveCoups
============================================

Scrapes specified X accounts for deal posts and imports them into the GHC system.

Usage:
    python3 x_deal_scraper.py                    # Run scrape on all accounts
    python3 x_deal_scraper.py --add @account      # Add account to scrape list
    python3 x_deal_scraper.py --remove @account   # Remove account from list
    python3 x_deal_scraper.py --list              # List all monitored accounts
    python3 x_deal_scraper.py --dashboard         # Launch web dashboard
"""

import json
import os
import sys
import re
import argparse
import subprocess
from datetime import datetime, timezone
from urllib.request import Request, urlopen
from typing import List, Dict, Optional

# Config paths
CONFIG_DIR = "/home/crogers2287/openclaw-workspaces/ghc-marketing/config"
CONFIG_FILE = os.path.join(CONFIG_DIR, "x_scraper_accounts.json")
SCRAPED_DEALS_FILE = os.path.join(CONFIG_DIR, "x_scraped_deals.json")

# GHC API endpoints
GHC_API_BASE = "https://www.gottahavecoups.com/api/v1"
GHC_DEALS_API = f"{GHC_API_BASE}/deals"
GHC_IMPORT_API = f"{GHC_API_BASE}/admin/import-deal"  # Adjust based on actual API

# Maya VM connection for twikit
MAYA_HOST = "maya@maya"

def load_config() -> Dict:
    """Load scraper configuration."""
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, 'r') as f:
            return json.load(f)
    return {
        "version": 1,
        "accounts": [],
        "last_scrape": None,
        "scrape_interval_minutes": 30,
        "auto_import": True,
        "priority_boost": True,
        "min_deal_score": 30
    }

def save_config(config: Dict):
    """Save scraper configuration."""
    os.makedirs(CONFIG_DIR, exist_ok=True)
    with open(CONFIG_FILE, 'w') as f:
        json.dump(config, f, indent=2)

def load_scraped_deals() -> List[Dict]:
    """Load previously scraped deals to avoid duplicates."""
    if os.path.exists(SCRAPED_DEALS_FILE):
        with open(SCRAPED_DEALS_FILE, 'r') as f:
            return json.load(f)
    return []

def save_scraped_deals(deals: List[Dict]):
    """Save scraped deals history."""
    with open(SCRAPED_DEALS_FILE, 'w') as f:
        json.dump(deals, f, indent=2)

def add_account(username: str, category: str = "general", notes: str = ""):
    """Add an X account to the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    # Check if already exists
    for acc in config["accounts"]:
        if acc["username"].lower() == username:
            print(f"⚠  @{username} is already in the scrape list")
            return False

    config["accounts"].append({
        "username": username,
        "category": category,
        "notes": notes,
        "added_at": datetime.now(timezone.utc).isoformat(),
        "last_scraped": None,
        "active": True,
        "deals_found": 0
    })

    save_config(config)
    print(f"✅ Added @{username} to scrape list")
    return True

def remove_account(username: str):
    """Remove an X account from the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    original_count = len(config["accounts"])
    config["accounts"] = [a for a in config["accounts"] if a["username"].lower() != username]

    if len(config["accounts"]) < original_count:
        save_config(config)
        print(f"✅ Removed @{username} from scrape list")
        return True
    else:
        print(f"⚠  @{username} not found in scrape list")
        return False

def list_accounts():
    """List all accounts being scraped."""
    config = load_config()

    if not config["accounts"]:
        print("\n📭 No X accounts configured for scraping")
        print("\nAdd accounts with: python3 x_deal_scraper.py --add @username")
        return

    print(f"\n📱 X Accounts to Scrape ({len(config['accounts'])} total)\n")
    print(f"{'Username':<20} {'Category':<12} {'Status':<8} {'Deals':<6} {'Last Scraped'}")
    print("-" * 70)

    for acc in config["accounts"]:
        status = "🟢" if acc.get("active", True) else "🔴"
        last_scraped = acc.get("last_scraped", "Never")[:16] if acc.get("last_scraped") else "Never"
        deals = acc.get("deals_found", 0)
        print(f"@{acc['username']:<19} {acc.get('category', 'general'):<12} {status:<8} {deals:<6} {last_scraped}")

    print("\n" + "-" * 70)
    print(f"Auto-import: {'✅ ON' if config.get('auto_import') else '❌ OFF'}")
    print(f"Priority boost: {'✅ ON' if config.get('priority_boost') else '❌ OFF'}")
    print(f"Min deal score: {config.get('min_deal_score', 30)}")

def extract_deal_info(tweet_text: str, tweet_media: List[str] = None) -> Optional[Dict]:
    """
    Extract deal information from a tweet text.
    Returns deal dict or None if no deal detected.
    """
    deal = {
        "title": None,
        "price": None,
        "original_price": None,
        "discount_percentage": None,
        "store": None,
        "product_url": None,
        "image_url": None,
        "category": None,
        "source_tweet": None
    }

    # Price patterns
    price_patterns = [
        r'\$([\d,]+\.?\d*)',  # $XX.XX
        r'([\d,]+)\s*dollars',  # XX dollars
    ]

    # Look for prices
    prices = []
    for pattern in price_patterns:
        matches = re.findall(pattern, tweet_text, re.IGNORECASE)
        prices.extend([float(p.replace(',', '')) for p in matches])

    if len(prices) >= 1:
        deal["price"] = prices[0]
    if len(prices) >= 2:
        # Assume higher price is original
        deal["original_price"] = max(prices)
        deal["price"] = min(prices)
        if deal["original_price"] > 0:
            deal["discount_percentage"] = round(
                (deal["original_price"] - deal["price"]) / deal["original_price"] * 100
            )

    # Store detection
    store_patterns = {
        "Amazon": [r'amazon\.com', r'amzn\.to', r'@amazon'],
        "Walmart": [r'walmart\.com', r'w-mt\.co', r'@walmart'],
        "Target": [r'target\.com', r'@target'],
        "Best Buy": [r'bestbuy\.com', r'@bestbuy'],
        "Costco": [r'costco\.com', r'@costco'],
        "Home Depot": [r'homedepot\.com', r'@homedepot'],
    }

    for store, patterns in store_patterns.items():
        for pattern in patterns:
            if re.search(pattern, tweet_text, re.IGNORECASE):
                deal["store"] = store
                break
        if deal["store"]:
            break

    # URL extraction
    url_pattern = r'https?://[^\s<>"{}|\\^`\[\]]+'
    urls = re.findall(url_pattern, tweet_text)
    if urls:
        deal["product_url"] = urls[0]

    # Title extraction - look for product mentions
    title_patterns = [
        r'(?:Get|Grab|Score| snag )\s*([^\n.]+?)(?:\s+for\s+|\s+at\s+|\s+-\s+|\s+–\s+)',
        r'([^\n.]{20,80}?)(?:\s+is\s+\$|\s+only\s+\$|\s+just\s+\$)',
    ]

    for pattern in title_patterns:
        match = re.search(pattern, tweet_text, re.IGNORECASE)
        if match:
            deal["title"] = match.group(1).strip()[:150]
            break

    # If no title found, use first line
    if not deal["title"]:
        first_line = tweet_text.split('\n')[0][:150]
        deal["title"] = first_line

    # Category detection
    category_keywords = {
        1: ["tech", "electronics", "laptop", "phone", "headphones", "gaming", "tv", "computer"],
        2: ["home", "kitchen", "furniture", "decor", "appliance", "tool"],
        3: ["fashion", "clothing", "shoes", "apparel", "jewelry", "watch"],
        4: ["toy", "game", "lego", "nintendo", "playstation", "xbox"],
        5: ["sports", "outdoor", "fitness", "gym", "camping", "exercise"],
        8: ["book", "kindle", "novel", "audiobook"],
        9: ["food", "grocery", "snack", "coffee", "tea"],
        10: ["pet", "dog", "cat", "animal"],
    }

    text_lower = tweet_text.lower()
    for cat_id, keywords in category_keywords.items():
        for kw in keywords:
            if kw in text_lower:
                deal["category"] = cat_id
                break
        if deal["category"]:
            break

    # Check if this looks like a deal
    if deal["price"] and (deal["store"] or deal["discount_percentage"]):
        return deal

    return None

def scrape_account_tweets(username: str, limit: int = 20) -> List[Dict]:
    """
    Scrape recent tweets from an X account using Maya VM Chrome/Playwright.
    Returns list of tweet dicts.
    """
    print(f"🔍 Scraping @{username}...")

    try:
        # Create a scraping script to run on Maya VM
        scrape_script = f'''
import asyncio
from playwright.async_api import async_playwright
import json

async def scrape():
    tweets = []
    async with async_playwright() as p:
        try:
            browser = await p.chromium.connect_over_cdp("http://localhost:9222")
            context = browser.contexts[0]
            page = await context.new_page()

            # Go to user's profile
            await page.goto(f"https://x.com/{username}", wait_until="networkidle", timeout=30000)
            await asyncio.sleep(3)

            # Extract tweets
            tweet_elements = await page.locator('article[data-testid="tweet"]').all()

            for tweet in tweet_elements[:{limit}]:
                try:
                    # Get tweet text
                    text_elem = tweet.locator('[data-testid="tweetText"]')
                    text = await text_elem.inner_text() if await text_elem.count() > 0 else ""

                    # Get timestamp
                    time_elem = tweet.locator('time')
                    timestamp = await time_elem.get_attribute('datetime') if await time_elem.count() > 0 else ""

                    if text:
                        tweets.append({{
                            "text": text,
                            "created_at": timestamp,
                            "media": []
                        }})
                except Exception as e:
                    pass

            await page.close()
        except Exception as e:
            print(f"Browser error: {{e}}")

    print(json.dumps(tweets, default=str))

asyncio.run(scrape())
'''

        # Save script to Maya VM
        script_path = f"/tmp/scrape_{username}.py"
        with open(script_path, 'w') as f:
            f.write(scrape_script)

        # Copy to Maya
        subprocess.run(['scp', '-q', script_path, f"{MAYA_HOST}:/tmp/scrape_{username}.py"], check=True)

        # Run the scraper on Maya
        result = subprocess.run(
            ['ssh', MAYA_HOST, f'python3 /tmp/scrape_{username}.py'],
            capture_output=True,
            text=True,
            timeout=60
        )

        if result.returncode != 0:
            print(f"⚠  Error scraping @{username}: {result.stderr}")
            return []

        # Parse output
        output = result.stdout.strip()
        if output:
            try:
                tweets = json.loads(output)
                print(f"✅ Found {len(tweets)} tweets from @{username}")
                return tweets
            except json.JSONDecodeError:
                print(f"⚠  Invalid JSON from @{username}")
                return []

        return []

    except subprocess.TimeoutExpired:
        print(f"⏱  Timeout scraping @{username}")
        return []
    except Exception as e:
        print(f"⚠  Error scraping @{username}: {e}")
        return []

def import_deal_to_ghc(deal: Dict, source_account: str) -> bool:
    """
    Import a scraped deal into the GottaHaveCoups system.
    Returns True if successful.
    """
    # For now, we'll create a draft deal that can be reviewed
    # In production, this would POST to the GHC API

    draft_deal = {
        "title": deal["title"],
        "description": f"Scraped from X @{source_account}",
        "price": deal["price"],
        "original_price": deal.get("original_price"),
        "discount_percentage": deal.get("discount_percentage"),
        "store_name": deal.get("store", "Unknown"),
        "category_id": deal.get("category", 1),
        "source": f"X:@{source_account}",
        "deal_url": deal.get("product_url", ""),
        "image_url": deal.get("image_url", ""),
        "quality_score": 50,  # Boost priority
        "is_hot": True,
        "is_featured": True,
        "scraped_at": datetime.now(timezone.utc).isoformat(),
        "needs_review": True  # Flag for manual review
    }

    # Save to scraped deals file
    scraped = load_scraped_deals()

    # Check for duplicates (by title + price)
    for existing in scraped:
        if (existing.get("title") == draft_deal["title"] and
            existing.get("price") == draft_deal["price"]):
            print(f"   ⚠  Duplicate deal skipped: {draft_deal['title'][:50]}...")
            return False

    scraped.append(draft_deal)
    save_scraped_deals(scraped)

    print(f"   ✅ Imported: {draft_deal['title'][:60]}...")
    return True

def run_scraper():
    """Main scraper routine."""
    config = load_config()

    if not config["accounts"]:
        print("📭 No accounts configured. Add accounts with --add @username")
        return

    print(f"\n🚀 Starting X Deal Scraper")
    print(f"   Accounts: {len(config['accounts'])}")
    print(f"   Auto-import: {'ON' if config.get('auto_import') else 'OFF'}")
    print()

    total_deals = 0
    new_deals = 0

    for account in config["accounts"]:
        if not account.get("active", True):
            continue

        username = account["username"]
        tweets = scrape_account_tweets(username, limit=20)

        account_deals = 0
        for tweet in tweets:
            deal_info = extract_deal_info(tweet.get("text", ""), tweet.get("media", []))
            if deal_info:
                total_deals += 1
                if import_deal_to_ghc(deal_info, username):
                    new_deals += 1
                    account_deals += 1

        # Update account stats
        account["last_scraped"] = datetime.now(timezone.utc).isoformat()
        account["deals_found"] = account.get("deals_found", 0) + account_deals

        print()

    config["last_scrape"] = datetime.now(timezone.utc).isoformat()
    save_config(config)

    print(f"📊 Scrape Complete")
    print(f"   Total deals found: {total_deals}")
    print(f"   New deals imported: {new_deals}")
    print(f"   Next scrape: ~{config['scrape_interval_minutes']} minutes")

def launch_dashboard():
    """Launch the web dashboard for managing accounts."""
    dashboard_path = "/home/crogers2287/openclaw-workspaces/ghc-marketing/x-scraper-dashboard.html"
    print(f"🌐 Launching dashboard...")
    print(f"   File: {dashboard_path}")

    # Create dashboard HTML if it doesn't exist
    if not os.path.exists(dashboard_path):
        create_dashboard_html(dashboard_path)

    # Open in browser
    os.system(f"xdg-open {dashboard_path} 2>/dev/null || open {dashboard_path} 2>/dev/null || echo 'Open {dashboard_path} in your browser'")

def create_dashboard_html(path: str):
    """Create the HTML dashboard file."""
    html = '''<!DOCTYPE html>

    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>X Deal Scraper Dashboard</title>

🐦 X Deal Scraper

Monitor X accounts for deals and import to GottaHaveCoups

📊 Stats

0

Accounts

0

Deals Found

--

Last Scrape

➕ Add Account

                <input type="text" id="new-username" placeholder="@username (without @)">
                <input type="text" id="new-category" placeholder="Category (optional)" style="flex: 0.5;">
                <button onclick="addAccount()">Add Account</button>

📱 Monitored Accounts

No accounts configured

⚙ Settings

                Auto-import deals

                    <input type="checkbox" id="auto-import" checked>

                Priority boost (post first)

                    <input type="checkbox" id="priority-boost" checked>

                Scrape interval: 30 minutes

        <button class="run-btn" onclick="runScraper()">🚀 Run Scraper Now</button>

'''

    with open(path, 'w') as f:
        f.write(html)

def main():
    parser = argparse.ArgumentParser(
        description="X (Twitter) Deal Scraper for GottaHaveCoups",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
    python3 x_deal_scraper.py --add @deals_account
    python3 x_deal_scraper.py --add @techdeals --category tech
    python3 x_deal_scraper.py --list
    python3 x_deal_scraper.py --remove @deals_account
    python3 x_deal_scraper.py                    # Run scraper
    python3 x_deal_scraper.py --dashboard       # Open web dashboard
        """
    )

    parser.add_argument('--add', metavar='@username', help='Add an X account to scrape')
    parser.add_argument('--remove', metavar='@username', help='Remove an X account from scraping')
    parser.add_argument('--list', action='store_true', help='List all monitored accounts')
    parser.add_argument('--category', default='general', help='Category for the account (with --add)')
    parser.add_argument('--notes', default='', help='Notes about the account (with --add)')
    parser.add_argument('--dashboard', action='store_true', help='Launch web dashboard')
    parser.add_argument('--run', action='store_true', help='Run scraper (default behavior)')

    args = parser.parse_args()

    if args.add:
        add_account(args.add, args.category, args.notes)
    elif args.remove:
        remove_account(args.remove)
    elif args.list:
        list_accounts()
    elif args.dashboard:
        launch_dashboard()
    else:
        # Default: run scraper
        run_scraper()

if __name__ == "__main__":
    main()


I see the issue. The scraper is trying to SSH to the Maya VM and run Playwright scripts, which:
1. Requires SSH access to Maya
2. Requires Chrome remote debugging on port 9222
3. Requires Playwright to be installed on Maya
4. Is slow due to network requests and browser automation

Looking at the script, it needs accounts to be configured first. Let me check if there's a config file with accounts to scrape.

Let me check if accounts are configured:


X (Twitter) Accounts to Scrape for Deals


OK, so there are 3 accounts configured and the last scrape was around 3:11 AM today. The script times out because it's trying to SSH to Maya and run Playwright scripts, which takes time. The issue is:

1. The script requires SSH access to Maya VM
2. It tries to use Playwright with Chrome remote debugging
3. Network requests to X can be slow
4. The 120-second timeout I set was too short

The script timed out because of the network operations. Let me check what the actual error might have been by looking at what happened in the most recent successful run. Let me check if there's a log or scraped deals file.

Let me check if any deals were scraped recently:


ENOENT: no such file or directory, access '/home/crogers2287/openclaw-workspaces/ghc-marketing/config/x_scraped_deals.json'


The scraped deals file doesn't exist yet, which means no deals have been successfully imported. The script timed out while trying to scrape. The issue is that the script:1. SSHs to Maya VM (network delay)2. Runs Playwright browser automation (slow)
3. Scrapes multiple accounts (3 accounts × ~60 seconds each)That's why it hit the 120-second timeout. The script is designed to work but needs a longer timeout. Let me try running it again with a much longer timeout (5 minutes) to let it complete.

I see the problem. The X scraper script uses Playwright browser automation via SSH to the Maya VM, which is slow. It timed out after 2 minutes while trying to scrape 3 accounts (@dealsfinderio, @glitcheddeals, @fatkiddeals).

Let me try running it again with a longer timeout:

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && timeout 300 python3 scripts/x_deal_scraper.py 2>&1


Command still running (session briny-crest, pid 2060310). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


It's running again. Let me wait and check for output.


(no new output)

Process still running.


Still running. Let me wait a bit and check the log.


(no output yet)


No output yet. The script might still be setting up or making initial SSH connections. Let me wait a bit more.


Let me check the log again to see if there's any output now.


Still no output. This is taking a long time. Let me poll again.


Still running with no output. This suggests the script might be stuck or having issues connecting to Maya. Let me wait a bit more and then check again.


The script is still running with no output. This could mean:
1. SSH connection to Maya is slow/hanging
2. The script is working but not producing output yet
3. There's an error but it's not being printed

Let me check the log one more time, and if still nothing, I might need to report that the script is having issues.


Still no output after quite a while. Let me poll once more.


The process is still running but with no output. Let me try a different approach - let me test if I can actually SSH to Maya first to see if that's the issue.

Let me check if SSH to Maya is working:

ssh -o ConnectTimeout=10 maya@maya 'echo "SSH connection successful"' 2>&1


Command still running (session quiet-crest, pid 2071053). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


The SSH test is also hanging. This suggests Maya VM is either:1. Not running2. Not accessible via SSH3. Network issues with TailscaleLet me check if the SSH command times out or succeeds.


ssh: connect to host maya port 22: Connection timed out

Process exited with code 255.


Aha! SSH connection to Maya is timing out. That's the root cause. The X scraper depends on SSHing to Maya VM to run Playwright scripts, but Maya is unreachable. This explains why the scraper hangs with no output.I should:
1. Kill the hanging scraper process
2. Report that the scraper can't run because Maya VM is unreachable
3. Suggest checking if Maya is running or if there's a network/Tailscale issue

Found the issue! SSH connection to Maya VM is timing out. Let me kill the hanging scraper:


Killed session briny-crest.


Good, killed the hanging process. Now I need to report the issue clearly to Chad/the group chat.

**🚨 X Scraper Failed - Maya VM Unreachable**

The X account scraper couldn't run because the Maya VM is offline or unreachable via SSH.

**Issue:**
- Script requires SSH connection to `maya@maya` to run Playwright browser automation
- SSH connection timeout: `Connection timed out` on port 22

**Configured Accounts (not scraped):**
- @dealsfinderio
- @glitcheddeals
- @fatkiddeals

**Last successful scrape:** Today at 3:11 AM UTC (5+ hours ago)

**Next Steps:**
1. Check if Maya VM is running
2. Verify Tailscale connectivity to Maya
3. Or disable this cron job until Maya is accessible

The scraper will continue to fail until Maya VM is back online.