[cron:541ac66e-23e1-40db-8995-866d12d10a0e X Account Scraper - Run Every 30 Min] Run X scraper to check monitored accounts for new deals:

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py

Log results to group chat. If new deals found, mention how many were imported.
Current time: Monday, February 16th, 2026 — 5:11 AM (UTC)

Return your summary as plain text; it will be delivered automatically. If the task explicitly calls for messaging a specific external recipient, note who/where it should go instead of sending it yourself.


This is a cron job that wants me to run the X/Twitter account scraper to check for new deals from monitored accounts. I need to:

1. Run the script at `/home/crogers2287/openclaw-workspaces/ghc-marketing/scripts/x_deal_scraper.py`
2. Log the results
3. Report how many new deals were found (if any)

Let me execute this script and see what happens.

exec

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py


Command still running (session fresh-seaslug, pid 632432). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


The script is running in the background. Let me poll for results to see what the scraper found.


(no new output)

Process still running.


Still running. Let me wait a bit more and check the log to see if there's any output.


(no output yet)


No output yet. Let me wait a bit longer since scraping can take time. I'll poll again.


Still running. Let me check the log again to see if there's any output now.


Hmm, no output yet. Let me wait a bit longer and poll again. Scraping Twitter can take time especially if checking multiple accounts.


Let me check the log one more time to see if there's any output.


Still no output. Let me poll once more to see if the process has completed.


The process is still running. Let me check the log again - sometimes output can be buffered.


Let me try polling one more time to see if it's completed.


Let me check the log to see if there's any output now.


The process is still running and there's no output. Let me check the log one more time to see if anything has appeared.


Still no output. Let me try polling again.


Let me check the log again. The script seems to be taking a while. It might be doing a lot of scraping.


(no output recorded)

Process exited with signal SIGKILL.


The process was killed, probably by the timeout (120 seconds). Let me check if there's any log output that was captured.


(no output recorded)


The script timed out after 120 seconds with no output. This suggests either:
1. The script is hanging or stuck
2. There's an issue with the Twitter authentication
3. The script needs more time to complete

Let me check if the script exists and what it does.


#!/usr/bin/env python3
"""
X (Twitter) Deal Scraper for GottaHaveCoups
============================================

Scrapes specified X accounts for deal posts and imports them into the GHC system.

Usage:
    python3 x_deal_scraper.py                    # Run scrape on all accounts
    python3 x_deal_scraper.py --add @account      # Add account to scrape list
    python3 x_deal_scraper.py --remove @account   # Remove account from list
    python3 x_deal_scraper.py --list              # List all monitored accounts
    python3 x_deal_scraper.py --dashboard         # Launch web dashboard
"""

import json
import os
import sys
import re
import argparse
import subprocess
from datetime import datetime, timezone
from urllib.request import Request, urlopen
from typing import List, Dict, Optional

# Config paths
CONFIG_DIR = "/home/crogers2287/openclaw-workspaces/ghc-marketing/config"
CONFIG_FILE = os.path.join(CONFIG_DIR, "x_scraper_accounts.json")
SCRAPED_DEALS_FILE = os.path.join(CONFIG_DIR, "x_scraped_deals.json")

# GHC API endpoints
GHC_API_BASE = "https://www.gottahavecoups.com/api/v1"
GHC_DEALS_API = f"{GHC_API_BASE}/deals"
GHC_IMPORT_API = f"{GHC_API_BASE}/admin/import-deal"  # Adjust based on actual API

# Maya VM connection for twikit
MAYA_HOST = "maya@maya"

def load_config() -> Dict:
    """Load scraper configuration."""
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, 'r') as f:
            return json.load(f)
    return {
        "version": 1,
        "accounts": [],
        "last_scrape": None,
        "scrape_interval_minutes": 30,
        "auto_import": True,

[722 more lines in file. Use offset=51 to continue.]


The script exists. Let me see more of it to understand what might be going wrong. Let me check the main execution logic.


                category: category || 'general',
                added_at: new Date().toISOString(),
                deals_found: 0,
                active: true
            });

            localStorage.setItem('x_scraper_config', JSON.stringify(config));
            document.getElementById('new-username').value = '';
            document.getElementById('new-category').value = '';
            renderAccounts();
        }

        function removeAccount(username) {
            config.accounts = config.accounts.filter(a => a.username !== username);
            localStorage.setItem('x_scraper_config', JSON.stringify(config));
            renderAccounts();
        }

        function runScraper() {
            alert('In production, this would trigger the scraper. Run: python3 x_deal_scraper.py');
        }

        // Initial render
        renderAccounts();
    </script>
</body>
</html>'''

    with open(path, 'w') as f:
        f.write(html)

def main():
    parser = argparse.ArgumentParser(
        description="X (Twitter) Deal Scraper for GottaHaveCoups",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
    python3 x_deal_scraper.py --add @deals_account
    python3 x_deal_scraper.py --add @techdeals --category tech
    python3 x_deal_scraper.py --list
    python3 x_deal_scraper.py --remove @deals_account
    python3 x_deal_scraper.py                    # Run scraper
    python3 x_deal_scraper.py --dashboard       # Open web dashboard
        """
    )

    parser.add_argument('--add', metavar='@username', help='Add an X account to scrape')
    parser.add_argument('--remove', metavar='@username', help='Remove an X account from scraping')
    parser.add_argument('--list', action='store_true', help='List all monitored accounts')

[23 more lines in file. Use offset=750 to continue.]


Let me check the rest of the main function to see what happens when we run the scraper without arguments.


    parser.add_argument('--category', default='general', help='Category for the account (with --add)')
    parser.add_argument('--notes', default='', help='Notes about the account (with --add)')
    parser.add_argument('--dashboard', action='store_true', help='Launch web dashboard')
    parser.add_argument('--run', action='store_true', help='Run scraper (default behavior)')

    args = parser.parse_args()

    if args.add:
        add_account(args.add, args.category, args.notes)
    elif args.remove:
        remove_account(args.remove)
    elif args.list:
        list_accounts()
    elif args.dashboard:
        launch_dashboard()
    else:
        # Default: run scraper
        run_scraper()

if __name__ == "__main__":
    main()


So when we run the script without arguments, it calls `run_scraper()`. Let me find that function to see what it does.

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && grep -n "^def run_scraper" scripts/x_deal_scraper.py


395:def run_scraper():


def run_scraper():
    """Main scraper routine."""
    config = load_config()

    if not config["accounts"]:
        print("📭 No accounts configured. Add accounts with --add @username")
        return

    print(f"\n🚀 Starting X Deal Scraper")
    print(f"   Accounts: {len(config['accounts'])}")
    print(f"   Auto-import: {'ON' if config.get('auto_import') else 'OFF'}")
    print()

    total_deals = 0
    new_deals = 0

    for account in config["accounts"]:
        if not account.get("active", True):
            continue

        username = account["username"]
        tweets = scrape_account_tweets(username, limit=20)

        account_deals = 0
        for tweet in tweets:
            deal_info = extract_deal_info(tweet.get("text", ""), tweet.get("media", []))
            if deal_info:
                total_deals += 1
                if import_deal_to_ghc(deal_info, username):
                    new_deals += 1
                    account_deals += 1

        # Update account stats
        account["last_scraped"] = datetime.now(timezone.utc).isoformat()
        account["deals_found"] = account.get("deals_found", 0) + account_deals

        print()

    config["last_scrape"] = datetime.now(timezone.utc).isoformat()
    save_config(config)

    print(f"📊 Scrape Complete")
    print(f"   Total deals found: {total_deals}")
    print(f"   New deals imported: {new_deals}")
    print(f"   Next scrape: ~{config['scrape_interval_minutes']} minutes")

def launch_dashboard():
    """Launch the web dashboard for managing accounts."""
    dashboard_path = "/home/crogers2287/openclaw-workspaces/ghc-marketing/x-scraper-dashboard.html"
    print(f"🌐 Launching dashboard...")
    print(f"   File: {dashboard_path}")

    # Create dashboard HTML if it doesn't exist
    if not os.path.exists(dashboard_path):
        create_dashboard_html(dashboard_path)

    # Open in browser
    os.system(f"xdg-open {dashboard_path} 2>/dev/null || open {dashboard_path} 2>/dev/null || echo 'Open {dashboard_path} in your browser'")

def create_dashboard_html(path: str):
    """Create the HTML dashboard file."""
    html = '''<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>X Deal Scraper Dashboard</title>
    <style>
        * { margin: 0; padding: 0; box-sizing: border-box; }
        body {
            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
            background: #0f172a;
            color: #e2e8f0;
            padding: 20px;
        }
        .container { max-width: 1200px; margin: 0 auto; }
        h1 {
            font-size: 2.5em;
            margin-bottom: 10px;
            background: linear-gradient(90deg, #1d9bf0, #1a8cd8);
            -webkit-background-clip: text;
            -webkit-text-fill-color: transparent;
        }
        .subtitle { color: #94a3b8; margin-bottom: 30px; }
        .card {
            background: linear-gradient(135deg, #1e293b, #0f172a);
            border-radius: 16px;
            padding: 25px;
            margin-bottom: 25px;
            border: 1px solid #334155;
        }
        .card h2 {
            color: #fbbf24;
            margin-bottom: 20px;
            display: flex;
            align-items: center;
            gap: 10px;
        }

[278 more lines in file. Use offset=495 to continue.]


I can see the run_scraper function. It's calling `scrape_account_tweets()` which is likely where it's hanging. Let me find that function.

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && grep -n "^def scrape_account_tweets" scripts/x_deal_scraper.py


254:def scrape_account_tweets(username: str, limit: int = 20) -> List[Dict]:


def scrape_account_tweets(username: str, limit: int = 20) -> List[Dict]:
    """
    Scrape recent tweets from an X account using Maya VM Chrome/Playwright.
    Returns list of tweet dicts.
    """
    print(f"🔍 Scraping @{username}...")

    try:
        # Create a scraping script to run on Maya VM
        scrape_script = f'''
import asyncio
from playwright.async_api import async_playwright
import json

async def scrape():
    tweets = []
    async with async_playwright() as p:
        try:
            browser = await p.chromium.connect_over_cdp("http://localhost:9222")
            context = browser.contexts[0]
            page = await context.new_page()

            # Go to user's profile
            await page.goto(f"https://x.com/{username}", wait_until="networkidle", timeout=30000)
            await asyncio.sleep(3)

            # Extract tweets
            tweet_elements = await page.locator('article[data-testid="tweet"]').all()

            for tweet in tweet_elements[:{limit}]:
                try:
                    # Get tweet text
                    text_elem = tweet.locator('[data-testid="tweetText"]')
                    text = await text_elem.inner_text() if await text_elem.count() > 0 else ""

                    # Get timestamp
                    time_elem = tweet.locator('time')
                    timestamp = await time_elem.get_attribute('datetime') if await time_elem.count() > 0 else ""

                    if text:
                        tweets.append({{
                            "text": text,
                            "created_at": timestamp,
                            "media": []
                        }})
                except Exception as e:
                    pass

            await page.close()
        except Exception as e:
            print(f"Browser error: {{e}}")

    print(json.dumps(tweets, default=str))

asyncio.run(scrape())
'''

        # Save script to Maya VM
        script_path = f"/tmp/scrape_{username}.py"
        with open(script_path, 'w') as f:
            f.write(scrape_script)

        # Copy to Maya
        subprocess.run(['scp', '-q', script_path, f"{MAYA_HOST}:/tmp/scrape_{username}.py"], check=True)

        # Run the scraper on Maya
        result = subprocess.run(
            ['ssh', MAYA_HOST, f'python3 /tmp/scrape_{username}.py'],
            capture_output=True,
            text=True,
            timeout=60
        )

        if result.returncode != 0:
            print(f"⚠  Error scraping @{username}: {result.stderr}")
            return []

        # Parse output
        output = result.stdout.strip()
        if output:

[439 more lines in file. Use offset=334 to continue.]


I see the issue now. The scraper is trying to SSH into the Maya VM and run Playwright to scrape Twitter/X. This is likely timing out. Let me check if there are any accounts configured to scrape first.


X (Twitter) Accounts to Scrape for Deals


OK, so there are 3 accounts configured and the last scrape was at 03:11 UTC. The current time is 05:11 UTC so it's been 2 hours. The scraper is probably hanging on the SSH/Playwright connection to Maya VM. Let me try running it with a longer timeout or check if Maya is accessible. First, let me try just listing the accounts to confirm the script works at a basic level.

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py --list


📱 X Accounts to Scrape (3 total)

Username             Category     Status   Deals  Last Scraped
----------------------------------------------------------------------
@dealsfinderio       deals        🟢        0      2026-02-16T03:11
@glitcheddeals       deals        🟢        0      2026-02-16T03:11
@fatkiddeals         deals        🟢        0      2026-02-16T03:11

----------------------------------------------------------------------
Auto-import: ✅ ON
Priority boost: ✅ ON
Min deal score: 30


Good, the script itself works. The issue is with the scraping process - it's trying to SSH to Maya and run Playwright. Let me check if Maya is accessible via SSH and if Chrome/Playwright is set up properly. First, let me ping Maya.

ssh maya@maya "echo 'Maya VM is accessible' && which python3 && python3 -c 'import playwright; print(\"Playwright installed\")' 2>&1"


Command still running (session tender-crest, pid 683922). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


The SSH connection is hanging too. Let me poll to see if it times out.


The SSH connection to Maya is timing out. This is why the scraper is failing - it can't reach the Maya VM. The issue is infrastructure, not the script itself. Let me report this properly to Chad via the cron job delivery.

⚠ **X Account Scraper Failed**

The scraper timed out trying to connect to Maya VM for browser automation.

**Issue:** SSH connection to `maya@maya` is hanging/unreachable
**Impact:** Cannot scrape X accounts (3 accounts monitored: @dealsfinderio, @glitcheddeals, @fatkiddeals)
**Last successful scrape:** 2026-02-16 03:11 UTC (2 hours ago)

**Next steps:**
1. Verify Maya VM is running and accessible on Tailscale
2. Check if Chrome/Playwright browser automation is set up on Maya
3. May need to restart Maya VM or fix SSH config

The scraper itself is functional - account management works, just the browser scraping step fails due to Maya being unreachable.