[cron:541ac66e-23e1-40db-8995-866d12d10a0e X Account Scraper - Run Every 30 Min] Run X scraper to check monitored accounts for new deals:

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py

Log results to group chat. If new deals found, mention how many were imported.
Current time: Monday, February 16th, 2026 — 11:11 AM (UTC)

Return your summary as plain text; it will be delivered automatically. If the task explicitly calls for messaging a specific external recipient, note who/where it should go instead of sending it yourself.


This is a cron job asking me to run the X (Twitter) deal scraper. Let me execute the script and report back with results.

I need to:
1. Run the Python script at the specified path
2. Check the output for any new deals
3. Report results to the group chat (this response will be delivered automatically)

Let me run the script.

I'll run the X scraper to check for new deals from monitored accounts.

exec

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py


Command still running (session cool-shore, pid 3803128). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


The script is running in the background. Let me poll it to see the output.


(no new output)

Process still running.


Still running. Let me wait a bit and check again.


Let me check the log to see if there's any output accumulated.


(no output yet)


No output yet. Let me wait a bit more and poll again.


Let me check the log again to see if there's any output now.


Still no output. Let me poll once more and then check the log.


Let me check the log one more time to see if there's output.


The process is running but not producing output yet. Let me wait a bit longer and poll again.


The script is taking a while. Let me poll again.


Let me check the log again for any accumulated output.


The script is still running but hasn't produced any output yet. Let me check the log once more.


The script is still running but producing no output. Let me poll one more time.


This is taking a while. Let me poll again.


The script is taking longer than expected. Let me poll one more time and see if it completes.


The script is taking quite a while. Let me check the log one more time.


(no output recorded)

Process exited with signal SIGKILL.


The process was killed, likely by the timeout. The 120-second timeout I set must have been exceeded. Let me check the log to see if there's any output before it was killed.


(no output recorded)


The script timed out with no output. This suggests the script might be hanging or taking too long to connect to X/Twitter. Let me try running it without a timeout to see what happens, or check if there's an issue with the script.

Actually, let me first check if the script exists and what it does.


#!/usr/bin/env python3
"""
X (Twitter) Deal Scraper for GottaHaveCoups
============================================

Scrapes specified X accounts for deal posts and imports them into the GHC system.

Usage:
    python3 x_deal_scraper.py                    # Run scrape on all accounts
    python3 x_deal_scraper.py --add @account      # Add account to scrape list
    python3 x_deal_scraper.py --remove @account   # Remove account from list
    python3 x_deal_scraper.py --list              # List all monitored accounts
    python3 x_deal_scraper.py --dashboard         # Launch web dashboard
"""

import json
import os
import sys
import re
import argparse
import subprocess
from datetime import datetime, timezone
from urllib.request import Request, urlopen
from typing import List, Dict, Optional

# Config paths
CONFIG_DIR = "/home/crogers2287/openclaw-workspaces/ghc-marketing/config"
CONFIG_FILE = os.path.join(CONFIG_DIR, "x_scraper_accounts.json")
SCRAPED_DEALS_FILE = os.path.join(CONFIG_DIR, "x_scraped_deals.json")

# GHC API endpoints
GHC_API_BASE = "https://www.gottahavecoups.com/api/v1"
GHC_DEALS_API = f"{GHC_API_BASE}/deals"
GHC_IMPORT_API = f"{GHC_API_BASE}/admin/import-deal"  # Adjust based on actual API

# Maya VM connection for twikit
MAYA_HOST = "maya@maya"

def load_config() -> Dict:
    """Load scraper configuration."""
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, 'r') as f:
            return json.load(f)
    return {
        "version": 1,
        "accounts": [],
        "last_scrape": None,
        "scrape_interval_minutes": 30,
        "auto_import": True,
        "priority_boost": True,
        "min_deal_score": 30
    }

def save_config(config: Dict):
    """Save scraper configuration."""
    os.makedirs(CONFIG_DIR, exist_ok=True)
    with open(CONFIG_FILE, 'w') as f:
        json.dump(config, f, indent=2)

def load_scraped_deals() -> List[Dict]:
    """Load previously scraped deals to avoid duplicates."""
    if os.path.exists(SCRAPED_DEALS_FILE):
        with open(SCRAPED_DEALS_FILE, 'r') as f:
            return json.load(f)
    return []

def save_scraped_deals(deals: List[Dict]):
    """Save scraped deals history."""
    with open(SCRAPED_DEALS_FILE, 'w') as f:
        json.dump(deals, f, indent=2)

def add_account(username: str, category: str = "general", notes: str = ""):
    """Add an X account to the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    # Check if already exists
    for acc in config["accounts"]:
        if acc["username"].lower() == username:
            print(f"⚠  @{username} is already in the scrape list")
            return False

    config["accounts"].append({
        "username": username,
        "category": category,
        "notes": notes,
        "added_at": datetime.now(timezone.utc).isoformat(),
        "last_scraped": None,
        "active": True,
        "deals_found": 0
    })

    save_config(config)
    print(f"✅ Added @{username} to scrape list")
    return True

def remove_account(username: str):
    """Remove an X account from the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    original_count = len(config["accounts"])
    config["accounts"] = [a for a in config["accounts"] if a["username"].lower() != username]

    if len(config["accounts"]) < original_count:
        save_config(config)
        print(f"✅ Removed @{username} from scrape list")
        return True
    else:
        print(f"⚠  @{username} not found in scrape list")
        return False

def list_accounts():
    """List all accounts being scraped."""
    config = load_config()

    if not config["accounts"]:
        print("\n📭 No X accounts configured for scraping")
        print("\nAdd accounts with: python3 x_deal_scraper.py --add @username")
        return

    print(f"\n📱 X Accounts to Scrape ({len(config['accounts'])} total)\n")
    print(f"{'Username':<20} {'Category':<12} {'Status':<8} {'Deals':<6} {'Last Scraped'}")
    print("-" * 70)

    for acc in config["accounts"]:
        status = "🟢" if acc.get("active", True) else "🔴"
        last_scraped = acc.get("last_scraped", "Never")[:16] if acc.get("last_scraped") else "Never"
        deals = acc.get("deals_found", 0)
        print(f"@{acc['username']:<19} {acc.get('category', 'general'):<12} {status:<8} {deals:<6} {last_scraped}")

    print("\n" + "-" * 70)
    print(f"Auto-import: {'✅ ON' if config.get('auto_import') else '❌ OFF'}")
    print(f"Priority boost: {'✅ ON' if config.get('priority_boost') else '❌ OFF'}")
    print(f"Min deal score: {config.get('min_deal_score', 30)}")

def extract_deal_info(tweet_text: str, tweet_media: List[str] = None) -> Optional[Dict]:
    """
    Extract deal information from a tweet text.
    Returns deal dict or None if no deal detected.
    """
    deal = {
        "title": None,
        "price": None,
        "original_price": None,
        "discount_percentage": None,
        "store": None,
        "product_url": None,
        "image_url": None,
        "category": None,
        "source_tweet": None
    }

    # Price patterns
    price_patterns = [
        r'\$([\d,]+\.?\d*)',  # $XX.XX
        r'([\d,]+)\s*dollars',  # XX dollars
    ]

    # Look for prices
    prices = []
    for pattern in price_patterns:
        matches = re.findall(pattern, tweet_text, re.IGNORECASE)
        prices.extend([float(p.replace(',', '')) for p in matches])

    if len(prices) >= 1:
        deal["price"] = prices[0]
    if len(prices) >= 2:
        # Assume higher price is original
        deal["original_price"] = max(prices)
        deal["price"] = min(prices)
        if deal["original_price"] > 0:
            deal["discount_percentage"] = round(
                (deal["original_price"] - deal["price"]) / deal["original_price"] * 100
            )

    # Store detection
    store_patterns = {
        "Amazon": [r'amazon\.com', r'amzn\.to', r'@amazon'],
        "Walmart": [r'walmart\.com', r'w-mt\.co', r'@walmart'],
        "Target": [r'target\.com', r'@target'],
        "Best Buy": [r'bestbuy\.com', r'@bestbuy'],
        "Costco": [r'costco\.com', r'@costco'],
        "Home Depot": [r'homedepot\.com', r'@homedepot'],
    }

    for store, patterns in store_patterns.items():
        for pattern in patterns:
            if re.search(pattern, tweet_text, re.IGNORECASE):
                deal["store"] = store
                break
        if deal["store"]:
            break

    # URL extraction
    url_pattern = r'https?://[^\s<>"{}|\\^`\[\]]+'
    urls = re.findall(url_pattern, tweet_text)
    if urls:
        deal["product_url"] = urls[0]

    # Title extraction - look for product mentions
    title_patterns = [
        r'(?:Get|Grab|Score| snag )\s*([^\n.]+?)(?:\s+for\s+|\s+at\s+|\s+-\s+|\s+–\s+)',
        r'([^\n.]{20,80}?)(?:\s+is\s+\$|\s+only\s+\$|\s+just\s+\$)',
    ]

    for pattern in title_patterns:
        match = re.search(pattern, tweet_text, re.IGNORECASE)
        if match:
            deal["title"] = match.group(1).strip()[:150]
            break

    # If no title found, use first line
    if not deal["title"]:
        first_line = tweet_text.split('\n')[0][:150]
        deal["title"] = first_line

    # Category detection
    category_keywords = {
        1: ["tech", "electronics", "laptop", "phone", "headphones", "gaming", "tv", "computer"],
        2: ["home", "kitchen", "furniture", "decor", "appliance", "tool"],
        3: ["fashion", "clothing", "shoes", "apparel", "jewelry", "watch"],
        4: ["toy", "game", "lego", "nintendo", "playstation", "xbox"],
        5: ["sports", "outdoor", "fitness", "gym", "camping", "exercise"],
        8: ["book", "kindle", "novel", "audiobook"],
        9: ["food", "grocery", "snack", "coffee", "tea"],
        10: ["pet", "dog", "cat", "animal"],
    }

    text_lower = tweet_text.lower()
    for cat_id, keywords in category_keywords.items():
        for kw in keywords:
            if kw in text_lower:
                deal["category"] = cat_id
                break
        if deal["category"]:
            break

    # Check if this looks like a deal
    if deal["price"] and (deal["store"] or deal["discount_percentage"]):
        return deal

    return None

def scrape_account_tweets(username: str, limit: int = 20) -> List[Dict]:
    """
    Scrape recent tweets from an X account using Maya VM Chrome/Playwright.
    Returns list of tweet dicts.
    """
    print(f"🔍 Scraping @{username}...")

    try:
        # Create a scraping script to run on Maya VM
        scrape_script = f'''
import asyncio
from playwright.async_api import async_playwright
import json

async def scrape():
    tweets = []
    async with async_playwright() as p:
        try:
            browser = await p.chromium.connect_over_cdp("http://localhost:9222")
            context = browser.contexts[0]
            page = await context.new_page()

            # Go to user's profile
            await page.goto(f"https://x.com/{username}", wait_until="networkidle", timeout=30000)
            await asyncio.sleep(3)

            # Extract tweets
            tweet_elements = await page.locator('article[data-testid="tweet"]').all()

            for tweet in tweet_elements[:{limit}]:
                try:
                    # Get tweet text
                    text_elem = tweet.locator('[data-testid="tweetText"]')
                    text = await text_elem.inner_text() if await text_elem.count() > 0 else ""

                    # Get timestamp
                    time_elem = tweet.locator('time')
                    timestamp = await time_elem.get_attribute('datetime') if await time_elem.count() > 0 else ""

                    if text:
                        tweets.append({{
                            "text": text,
                            "created_at": timestamp,
                            "media": []
                        }})
                except Exception as e:
                    pass

            await page.close()
        except Exception as e:
            print(f"Browser error: {{e}}")

    print(json.dumps(tweets, default=str))

asyncio.run(scrape())
'''

        # Save script to Maya VM
        script_path = f"/tmp/scrape_{username}.py"
        with open(script_path, 'w') as f:
            f.write(scrape_script)

        # Copy to Maya
        subprocess.run(['scp', '-q', script_path, f"{MAYA_HOST}:/tmp/scrape_{username}.py"], check=True)

        # Run the scraper on Maya
        result = subprocess.run(
            ['ssh', MAYA_HOST, f'python3 /tmp/scrape_{username}.py'],
            capture_output=True,
            text=True,
            timeout=60
        )

        if result.returncode != 0:
            print(f"⚠  Error scraping @{username}: {result.stderr}")
            return []

        # Parse output
        output = result.stdout.strip()
        if output:
            try:
                tweets = json.loads(output)
                print(f"✅ Found {len(tweets)} tweets from @{username}")
                return tweets
            except json.JSONDecodeError:
                print(f"⚠  Invalid JSON from @{username}")
                return []

        return []

    except subprocess.TimeoutExpired:
        print(f"⏱  Timeout scraping @{username}")
        return []
    except Exception as e:
        print(f"⚠  Error scraping @{username}: {e}")
        return []

def import_deal_to_ghc(deal: Dict, source_account: str) -> bool:
    """
    Import a scraped deal into the GottaHaveCoups system.
    Returns True if successful.
    """
    # For now, we'll create a draft deal that can be reviewed
    # In production, this would POST to the GHC API

    draft_deal = {
        "title": deal["title"],
        "description": f"Scraped from X @{source_account}",
        "price": deal["price"],
        "original_price": deal.get("original_price"),
        "discount_percentage": deal.get("discount_percentage"),
        "store_name": deal.get("store", "Unknown"),
        "category_id": deal.get("category", 1),
        "source": f"X:@{source_account}",
        "deal_url": deal.get("product_url", ""),
        "image_url": deal.get("image_url", ""),
        "quality_score": 50,  # Boost priority
        "is_hot": True,
        "is_featured": True,
        "scraped_at": datetime.now(timezone.utc).isoformat(),
        "needs_review": True  # Flag for manual review
    }

    # Save to scraped deals file
    scraped = load_scraped_deals()

    # Check for duplicates (by title + price)
    for existing in scraped:
        if (existing.get("title") == draft_deal["title"] and
            existing.get("price") == draft_deal["price"]):
            print(f"   ⚠  Duplicate deal skipped: {draft_deal['title'][:50]}...")
            return False

    scraped.append(draft_deal)
    save_scraped_deals(scraped)

    print(f"   ✅ Imported: {draft_deal['title'][:60]}...")
    return True

def run_scraper():
    """Main scraper routine."""
    config = load_config()

    if not config["accounts"]:
        print("📭 No accounts configured. Add accounts with --add @username")
        return

    print(f"\n🚀 Starting X Deal Scraper")
    print(f"   Accounts: {len(config['accounts'])}")
    print(f"   Auto-import: {'ON' if config.get('auto_import') else 'OFF'}")
    print()

    total_deals = 0
    new_deals = 0

    for account in config["accounts"]:
        if not account.get("active", True):
            continue

        username = account["username"]
        tweets = scrape_account_tweets(username, limit=20)

        account_deals = 0
        for tweet in tweets:
            deal_info = extract_deal_info(tweet.get("text", ""), tweet.get("media", []))
            if deal_info:
                total_deals += 1
                if import_deal_to_ghc(deal_info, username):
                    new_deals += 1
                    account_deals += 1

        # Update account stats
        account["last_scraped"] = datetime.now(timezone.utc).isoformat()
        account["deals_found"] = account.get("deals_found", 0) + account_deals

        print()

    config["last_scrape"] = datetime.now(timezone.utc).isoformat()
    save_config(config)

    print(f"📊 Scrape Complete")
    print(f"   Total deals found: {total_deals}")
    print(f"   New deals imported: {new_deals}")
    print(f"   Next scrape: ~{config['scrape_interval_minutes']} minutes")

def launch_dashboard():
    """Launch the web dashboard for managing accounts."""
    dashboard_path = "/home/crogers2287/openclaw-workspaces/ghc-marketing/x-scraper-dashboard.html"
    print(f"🌐 Launching dashboard...")
    print(f"   File: {dashboard_path}")

    # Create dashboard HTML if it doesn't exist
    if not os.path.exists(dashboard_path):
        create_dashboard_html(dashboard_path)

    # Open in browser
    os.system(f"xdg-open {dashboard_path} 2>/dev/null || open {dashboard_path} 2>/dev/null || echo 'Open {dashboard_path} in your browser'")

def create_dashboard_html(path: str):
    """Create the HTML dashboard file."""
    html = '''<!DOCTYPE html>

    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>X Deal Scraper Dashboard</title>

🐦 X Deal Scraper

Monitor X accounts for deals and import to GottaHaveCoups

📊 Stats

0

Accounts

0

Deals Found

--

Last Scrape

➕ Add Account

                <input type="text" id="new-username" placeholder="@username (without @)">
                <input type="text" id="new-category" placeholder="Category (optional)" style="flex: 0.5;">
                <button onclick="addAccount()">Add Account</button>

📱 Monitored Accounts

No accounts configured

⚙ Settings

                Auto-import deals

                    <input type="checkbox" id="auto-import" checked>

                Priority boost (post first)

                    <input type="checkbox" id="priority-boost" checked>

                Scrape interval: 30 minutes

        <button class="run-btn" onclick="runScraper()">🚀 Run Scraper Now</button>

'''

    with open(path, 'w') as f:
        f.write(html)

def main():
    parser = argparse.ArgumentParser(
        description="X (Twitter) Deal Scraper for GottaHaveCoups",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
    python3 x_deal_scraper.py --add @deals_account
    python3 x_deal_scraper.py --add @techdeals --category tech
    python3 x_deal_scraper.py --list
    python3 x_deal_scraper.py --remove @deals_account
    python3 x_deal_scraper.py                    # Run scraper
    python3 x_deal_scraper.py --dashboard       # Open web dashboard
        """
    )

    parser.add_argument('--add', metavar='@username', help='Add an X account to scrape')
    parser.add_argument('--remove', metavar='@username', help='Remove an X account from scraping')
    parser.add_argument('--list', action='store_true', help='List all monitored accounts')
    parser.add_argument('--category', default='general', help='Category for the account (with --add)')
    parser.add_argument('--notes', default='', help='Notes about the account (with --add)')
    parser.add_argument('--dashboard', action='store_true', help='Launch web dashboard')
    parser.add_argument('--run', action='store_true', help='Run scraper (default behavior)')

    args = parser.parse_args()

    if args.add:
        add_account(args.add, args.category, args.notes)
    elif args.remove:
        remove_account(args.remove)
    elif args.list:
        list_accounts()
    elif args.dashboard:
        launch_dashboard()
    else:
        # Default: run scraper
        run_scraper()

if __name__ == "__main__":
    main()


I see the issue. The script is trying to SSH to Maya VM and run Playwright/Chrome scraping, which is likely hanging because it's trying to connect to Chrome via CDP on port 9222. The timeout killed the process.

Let me check if there are any accounts configured first.


X (Twitter) Accounts to Scrape for Deals


Good, there are 3 accounts configured and the last scrape was about 8 hours ago (3:11 AM UTC, now it's 11:11 AM UTC). The issue is that the scraper is trying to use Playwright on Maya VM but the connection is timing out (likely Chrome isn't running on port 9222 or SSH isn't working properly).

Let me report the issue and suggest a fix. The scraper timed out after 120 seconds without producing any output, which means it's stuck trying to connect to Chrome.

**X Deal Scraper - Run Failed** ⚠

The scraper timed out after 2 minutes without producing output.

**Issue:** Script is trying to connect to Chrome via Playwright on Maya VM (port 9222), but the connection is hanging.

**Monitored accounts:** 3 accounts (@dealsfinderio, @glitcheddeals, @fatkiddeals)
**Last successful scrape:** 8 hours ago (2026-02-16 03:11 UTC)

**Likely causes:**
- Chrome isn't running with remote debugging on Maya VM
- SSH connection to Maya is slow/blocked
- Playwright CDP connection timing out

**Next steps:**
1. Check if Chrome is running on Maya with `--remote-debugging-port=9222`
2. Test SSH connection: `ssh maya@maya 'echo OK'`
3. Consider switching to twikit API instead of browser scraping

Should I investigate the connection to Maya VM or disable this cron job until the issue is fixed?