[cron:541ac66e-23e1-40db-8995-866d12d10a0e X Account Scraper - Run Every 30 Min] Run X scraper to check monitored accounts for new deals:

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py

Log results to group chat. If new deals found, mention how many were imported.
Current time: Monday, February 16th, 2026 — 10:11 AM (UTC)

Return your summary as plain text; it will be delivered automatically. If the task explicitly calls for messaging a specific external recipient, note who/where it should go instead of sending it yourself.


This is a cron job that wants me to run the X scraper script and report results. Let me execute the command and see what happens.

The script is located at `/home/crogers2287/openclaw-workspaces/ghc-marketing/scripts/x_deal_scraper.py`.

I should run it and capture the output, then summarize the results.

exec

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py


Command still running (session briny-fjord, pid 2553088). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


The script is running in the background. Let me wait a moment and then check the logs to see the output.


(no new output)

Process still running.


Still running. Let me wait a bit more and check the logs.


(no output yet)


No output yet. Let me wait a bit longer and poll again.


The script is still running. Let me wait a bit more and check logs again to see if there's any output.


Still no output. Let me give it more time and poll once more.


The process is still running. Let me check the log with a longer offset or wait a bit more.


Still no output. Let me poll one more time to see if it completes.


The process is taking a while. Let me check the log again to see if there's any output now.


Still no output. Let me poll one more time.


The process is still running. Let me give it a bit more time and check the log again.


Let me poll one more time to see if it has finished.


The script is taking longer than expected. Let me check the log one more time.


Let me try checking the log one more time.


The script has been running for a while without output. Let me poll one final time to see if it completes soon.


The process is still running. Let me check the log again to see if there's output now.


Still no output. Let me give it one more poll and then check the status again.


(no new output)

Process still running.


The script is still running. Let me check logs one more time.


(no output yet)


Let me poll once more to see if it's done.


The script is taking longer than expected. Let me check the logs one more time to see if there's any output.


The process is still running with no output. Let me try polling one more time.


The script is running longer than expected. Let me check the log one more time and see if there's output now.


Still no output. Let me poll one final time to see the status.


The script has been running for a while now. Let me check the log once more.


The script has been running for quite a while. Let me check the log one more time to see if there's any output now.


Let me poll one more time to check status.


The script is taking a very long time. Let me check the log once more.


Let me try polling one final time to see if the script has finished.


The script has been running for quite some time without producing any output. This is unusual. Let me check the log one more time to see if anything has changed.


Let me try polling again to see if the process has completed.


The script is still running. Let me check the log once more.


The script has been running for a while without output. Let me check the log one more time.


(no output recorded)


Let me poll once more to see if the process has finished.


(no output recorded)

Process exited with signal SIGKILL.


The process was killed (SIGKILL). This likely means it exceeded the 120-second timeout I set. The X scraper script is taking longer than expected to run. Let me check the log to see if there was any output before it was killed.


The process was killed by timeout (120 seconds) and produced no output. This suggests the script either:
1. Got stuck somewhere
2. Takes longer than 2 minutes to run
3. Has an issue that prevents it from producing output

Let me try running it again with a longer timeout and see what happens. Or maybe I should check if the script file exists and what it does first.


#!/usr/bin/env python3
"""
X (Twitter) Deal Scraper for GottaHaveCoups
============================================

Scrapes specified X accounts for deal posts and imports them into the GHC system.

Usage:
    python3 x_deal_scraper.py                    # Run scrape on all accounts
    python3 x_deal_scraper.py --add @account      # Add account to scrape list
    python3 x_deal_scraper.py --remove @account   # Remove account from list
    python3 x_deal_scraper.py --list              # List all monitored accounts
    python3 x_deal_scraper.py --dashboard         # Launch web dashboard
"""

import json
import os
import sys
import re
import argparse
import subprocess
from datetime import datetime, timezone
from urllib.request import Request, urlopen
from typing import List, Dict, Optional

# Config paths
CONFIG_DIR = "/home/crogers2287/openclaw-workspaces/ghc-marketing/config"
CONFIG_FILE = os.path.join(CONFIG_DIR, "x_scraper_accounts.json")
SCRAPED_DEALS_FILE = os.path.join(CONFIG_DIR, "x_scraped_deals.json")

# GHC API endpoints
GHC_API_BASE = "https://www.gottahavecoups.com/api/v1"
GHC_DEALS_API = f"{GHC_API_BASE}/deals"
GHC_IMPORT_API = f"{GHC_API_BASE}/admin/import-deal"  # Adjust based on actual API

# Maya VM connection for twikit
MAYA_HOST = "maya@maya"

def load_config() -> Dict:
    """Load scraper configuration."""
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, 'r') as f:
            return json.load(f)
    return {
        "version": 1,
        "accounts": [],
        "last_scrape": None,
        "scrape_interval_minutes": 30,
        "auto_import": True,

[722 more lines in file. Use offset=51 to continue.]


I can see the script exists and it's designed to scrape X accounts for deals. It mentions using "Maya VM connection for twikit" and runs remote commands on maya@maya. This is probably what's taking time - it's connecting to the Maya VM via SSH to scrape Twitter.

Let me run it again with a much longer timeout (5 minutes should be enough) to see if it completes successfully.


Command still running (session dawn-lobster, pid 2600384). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


Now it's running with a 5-minute timeout. Let me wait a bit and check for output.


(no new output)

Process still running.


Let me check the log to see if there's any output yet.


(no output yet)


Still no output. Let me wait a bit more and poll again.


The script has been running for a while still with no output. Let me try to see if maybe it's just not producing stdout output. Let me check if there's a log file it might be writing to instead, or try running it with verbose output.

Actually, let me look at the script more carefully to see what it does and if there's a flag for verbose output.


        "priority_boost": True,
        "min_deal_score": 30
    }

def save_config(config: Dict):
    """Save scraper configuration."""
    os.makedirs(CONFIG_DIR, exist_ok=True)
    with open(CONFIG_FILE, 'w') as f:
        json.dump(config, f, indent=2)

def load_scraped_deals() -> List[Dict]:
    """Load previously scraped deals to avoid duplicates."""
    if os.path.exists(SCRAPED_DEALS_FILE):
        with open(SCRAPED_DEALS_FILE, 'r') as f:
            return json.load(f)
    return []

def save_scraped_deals(deals: List[Dict]):
    """Save scraped deals history."""
    with open(SCRAPED_DEALS_FILE, 'w') as f:
        json.dump(deals, f, indent=2)

def add_account(username: str, category: str = "general", notes: str = ""):
    """Add an X account to the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    # Check if already exists
    for acc in config["accounts"]:
        if acc["username"].lower() == username:
            print(f"⚠  @{username} is already in the scrape list")
            return False

    config["accounts"].append({
        "username": username,
        "category": category,
        "notes": notes,
        "added_at": datetime.now(timezone.utc).isoformat(),
        "last_scraped": None,
        "active": True,
        "deals_found": 0
    })

    save_config(config)
    print(f"✅ Added @{username} to scrape list")
    return True

def remove_account(username: str):
    """Remove an X account from the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    original_count = len(config["accounts"])
    config["accounts"] = [a for a in config["accounts"] if a["username"].lower() != username]

    if len(config["accounts"]) < original_count:
        save_config(config)
        print(f"✅ Removed @{username} from scrape list")
        return True
    else:
        print(f"⚠  @{username} not found in scrape list")
        return False

def list_accounts():
    """List all accounts being scraped."""
    config = load_config()

    if not config["accounts"]:
        print("\n📭 No X accounts configured for scraping")
        print("\nAdd accounts with: python3 x_deal_scraper.py --add @username")
        return

    print(f"\n📱 X Accounts to Scrape ({len(config['accounts'])} total)\n")
    print(f"{'Username':<20} {'Category':<12} {'Status':<8} {'Deals':<6} {'Last Scraped'}")
    print("-" * 70)

    for acc in config["accounts"]:
        status = "🟢" if acc.get("active", True) else "🔴"
        last_scraped = acc.get("last_scraped", "Never")[:16] if acc.get("last_scraped") else "Never"
        deals = acc.get("deals_found", 0)
        print(f"@{acc['username']:<19} {acc.get('category', 'general'):<12} {status:<8} {deals:<6} {last_scraped}")

    print("\n" + "-" * 70)
    print(f"Auto-import: {'✅ ON' if config.get('auto_import') else '❌ OFF'}")
    print(f"Priority boost: {'✅ ON' if config.get('priority_boost') else '❌ OFF'}")
    print(f"Min deal score: {config.get('min_deal_score', 30)}")

def extract_deal_info(tweet_text: str, tweet_media: List[str] = None) -> Optional[Dict]:
    """
    Extract deal information from a tweet text.
    Returns deal dict or None if no deal detected.
    """
    deal = {

[622 more lines in file. Use offset=151 to continue.]


Let me check where the main scraping function is to see what it outputs.


0

Accounts

0

Deals Found

--

Last Scrape

➕ Add Account

                <input type="text" id="new-username" placeholder="@username (without @)">
                <input type="text" id="new-category" placeholder="Category (optional)" style="flex: 0.5;">
                <button onclick="addAccount()">Add Account</button>

📱 Monitored Accounts

No accounts configured

⚙ Settings

                Auto-import deals

                    <input type="checkbox" id="auto-import" checked>

                Priority boost (post first)

                    <input type="checkbox" id="priority-boost" checked>

                Scrape interval: 30 minutes

        <button class="run-btn" onclick="runScraper()">🚀 Run Scraper Now</button>


Let me find the main function that actually does the scraping.

grep -n "def scrape_" /home/crogers2287/openclaw-workspaces/ghc-marketing/scripts/x_deal_scraper.py | head -20


254:def scrape_account_tweets(username: str, limit: int = 20) -> List[Dict]:


def scrape_account_tweets(username: str, limit: int = 20) -> List[Dict]:
    """
    Scrape recent tweets from an X account using Maya VM Chrome/Playwright.
    Returns list of tweet dicts.
    """
    print(f"🔍 Scraping @{username}...")

    try:
        # Create a scraping script to run on Maya VM
        scrape_script = f'''
import asyncio
from playwright.async_api import async_playwright
import json

async def scrape():
    tweets = []
    async with async_playwright() as p:
        try:
            browser = await p.chromium.connect_over_cdp("http://localhost:9222")
            context = browser.contexts[0]
            page = await context.new_page()

            # Go to user's profile
            await page.goto(f"https://x.com/{username}", wait_until="networkidle", timeout=30000)
            await asyncio.sleep(3)

            # Extract tweets
            tweet_elements = await page.locator('article[data-testid="tweet"]').all()

            for tweet in tweet_elements[:{limit}]:
                try:
                    # Get tweet text
                    text_elem = tweet.locator('[data-testid="tweetText"]')
                    text = await text_elem.inner_text() if await text_elem.count() > 0 else ""

                    # Get timestamp
                    time_elem = tweet.locator('time')
                    timestamp = await time_elem.get_attribute('datetime') if await time_elem.count() > 0 else ""

                    if text:
                        tweets.append({{
                            "text": text,
                            "created_at": timestamp,
                            "media": []
                        }})
                except Exception as e:
                    pass

            await page.close()
        except Exception as e:
            print(f"Browser error: {{e}}")

    print(json.dumps(tweets, default=str))

asyncio.run(scrape())
'''

        # Save script to Maya VM
        script_path = f"/tmp/scrape_{username}.py"
        with open(script_path, 'w') as f:
            f.write(scrape_script)

        # Copy to Maya
        subprocess.run(['scp', '-q', script_path, f"{MAYA_HOST}:/tmp/scrape_{username}.py"], check=True)

        # Run the scraper on Maya
        result = subprocess.run(
            ['ssh', MAYA_HOST, f'python3 /tmp/scrape_{username}.py'],
            capture_output=True,
            text=True,
            timeout=60
        )

        if result.returncode != 0:
            print(f"⚠  Error scraping @{username}: {result.stderr}")
            return []

        # Parse output
        output = result.stdout.strip()
        if output:
            try:
                tweets = json.loads(output)
                print(f"✅ Found {len(tweets)} tweets from @{username}")
                return tweets
            except json.JSONDecodeError:
                print(f"⚠  Invalid JSON from @{username}")
                return []

        return []

    except subprocess.TimeoutExpired:
        print(f"⏱  Timeout scraping @{username}")
        return []
    except Exception as e:
        print(f"⚠  Error scraping @{username}: {e}")
        return []

def import_deal_to_ghc(deal: Dict, source_account: str) -> bool:
    """
    Import a scraped deal into the GottaHaveCoups system.
    Returns True if successful.
    """
    # For now, we'll create a draft deal that can be reviewed
    # In production, this would POST to the GHC API

    draft_deal = {
        "title": deal["title"],
        "description": f"Scraped from X @{source_account}",
        "price": deal["price"],
        "original_price": deal.get("original_price"),
        "discount_percentage": deal.get("discount_percentage"),
        "store_name": deal.get("store", "Unknown"),
        "category_id": deal.get("category", 1),
        "source": f"X:@{source_account}",
        "deal_url": deal.get("product_url", ""),
        "image_url": deal.get("image_url", ""),
        "quality_score": 50,  # Boost priority
        "is_hot": True,
        "is_featured": True,
        "scraped_at": datetime.now(timezone.utc).isoformat(),
        "needs_review": True  # Flag for manual review
    }

    # Save to scraped deals file
    scraped = load_scraped_deals()

    # Check for duplicates (by title + price)
    for existing in scraped:
        if (existing.get("title") == draft_deal["title"] and
            existing.get("price") == draft_deal["price"]):
            print(f"   ⚠  Duplicate deal skipped: {draft_deal['title'][:50]}...")
            return False

    scraped.append(draft_deal)
    save_scraped_deals(scraped)

    print(f"   ✅ Imported: {draft_deal['title'][:60]}...")
    return True

def run_scraper():
    """Main scraper routine."""
    config = load_config()

    if not config["accounts"]:
        print("📭 No accounts configured. Add accounts with --add @username")
        return

    print(f"\n🚀 Starting X Deal Scraper")

[369 more lines in file. Use offset=404 to continue.]


I see the issue! The scraper is trying to SSH to Maya VM and run Playwright scripts. The script prints status messages like "🔍 Scraping @username..." but if there are no accounts configured, it prints a message and returns early.

Let me check if there are accounts configured.


X (Twitter) Accounts to Scrape for Deals


Good! There are accounts configured. The last scrape was at 03:11:32 UTC (about 7 hours ago). The script should run and scrape these accounts. Let me check the process status again to see if it's still running or if it's completed.


Let me check the log to see if there's output now.


This is strange - the process has been running for a while with no output. Let me check if maybe SSH to Maya is hanging. Let me try to SSH to Maya separately to test connectivity.

ssh -o ConnectTimeout=5 maya@maya 'echo "Connected to Maya"'


ssh: connect to host maya port 22: Connection timed out

Command exited with code 255


AH! There's the problem. SSH to Maya VM is timing out. The script is stuck trying to connect to Maya VM via SSH, which is why there's no output and it just hangs.

The Maya VM is probably not reachable from Fred (this host). Let me check if Maya VM is up and what the proper way to reach it is.


100.124.155.26  maya                   tagged-devices          linux    active; relay "mia"; offline, last seen 6h ago, tx 7332 rx 0


Maya VM is offline! It was last seen 6 hours ago. That explains why SSH is timing out. The scraper script can't reach Maya VM to run the Playwright scraping.Let me kill the running process and report this issue.


Killed session dawn-lobster.


Now I need to report this in the group chat. The X scraper failed because the Maya VM (which handles the X/Twitter scraping via Playwright) is offline and has been for about 6 hours.

⚠ **X Scraper Failed - Maya VM Offline**

The X deal scraper couldn't run because the Maya VM is unreachable.

**Status:**
- Maya VM: Offline (last seen 6 hours ago)
- Tailscale: `100.124.155.26` - relay via Miami, but offline
- Monitored accounts: 3 (dealsfinderio, glitcheddeals, fatkiddeals)
- Last successful scrape: 7 hours ago (03:11 UTC)

**Issue:**
The scraper script needs SSH access to Maya VM to run Playwright/Chrome automation for scraping X accounts. Connection is timing out.

**Action needed:**
Maya VM needs to be brought back online or the X scraper needs to be reconfigured to run locally on Fred (would require Chrome/Playwright setup here).