[cron:541ac66e-23e1-40db-8995-866d12d10a0e X Account Scraper - Run Every 30 Min] Run X scraper to check monitored accounts for new deals:

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py

Log results to group chat. If new deals found, mention how many were imported.
Current time: Monday, February 16th, 2026 — 2:11 PM (UTC)

Return your summary as plain text; it will be delivered automatically. If the task explicitly calls for messaging a specific external recipient, note who/where it should go instead of sending it yourself.


This is a cron job that wants me to:
1. Run the X scraper script
2. Log results to group chat
3. Mention how many deals were imported if any were found

Let me execute the script and see what happens.

exec

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && python3 scripts/x_deal_scraper.py


Command still running (session young-ocean, pid 3262035). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


The script is still running. I should poll it to see the results.


(no new output)

Process still running.


Still running. Let me wait a bit more and check again.


Still running. Let me check the logs to see if there's any output yet.


(no output yet)


The process is still running but no output yet. Let me wait a bit longer since it's scraping X accounts which can take time.


Still running. Let me check again for any output.


Let me check the log one more time to see if there's any output.


The process is taking a while. Let me poll once more.


Still running. Let me wait a bit longer and check again.


Still no output. Let me poll again to see if it completes.


The script is taking a while. Let me check once more and then wait a bit.


Still no output. Let me poll one more time.


Let me try checking the log one more time.


The script is still running but producing no output. This could mean it's scraping accounts and hasn't printed anything yet. Let me poll one more time.


Let me try once more to see if there's any output or completion.


The process is still running. Let me check the log once more.


The script is still running without output. Let me poll one more time.


(no new output)

Process still running.


Still running. Let me try one more poll.


Let me poll once more to see if it completes.


This is taking a while. Let me check the log one more time and then wait.


(no output yet)


The process is still running with no output. Let me check once more and if still nothing, I may need to report that it's taking longer than expected. Let me poll once more.


Let me try one last time and see if anything happens.


The script is still running. Let me check the log one final time.


OK, the script has been running for quite a while now with no output. This is unusual. Let me check if maybe the script exists first or if there's an issue. Actually, let me try polling one more time to see if it completes.


The script is still running. Let me check the log once more.


The script has been running for quite some time now without any output. This could mean:
1. It's waiting for something (network, authentication)
2. It's buffering output
3. There's an issue with the script

Let me poll one more time to see if it completes.


(no output recorded)

Process exited with signal SIGKILL.


Oh! The process was killed (SIGKILL). This likely hit the timeout I set (120 seconds). The script was taking too long. Let me check what might have happened - maybe I should look at the script itself or check for any error logs. Let me see if there's a log file the script might have created.

ls -lah /home/crogers2287/openclaw-workspaces/ghc-marketing/scripts/ | grep -E "(log|error|x_deal)"


-rw-r--r--  1 crogers2287 crogers2287  714 Feb 11 17:48 check_pin_login.py
-rw-r--r--  1 crogers2287 crogers2287  819 Feb 11 17:48 fix_blog_format.py
-rw-r--r--  1 crogers2287 crogers2287 2.0K Feb 11 17:48 nuke_blog_pin.py
-rw-r--r--  1 crogers2287 crogers2287  883 Feb 11 17:48 nuke_blog_social.py
-rw-r--r--  1 crogers2287 crogers2287 4.4K Feb 12 14:09 post_blog_v2.py
-rw-r--r--  1 crogers2287 crogers2287 2.0K Feb 11 17:48 post_x_deal.py
-rw-r--r--  1 crogers2287 crogers2287 2.9K Feb 11 17:48 publish-blogs.ps1
-rw-r--r--  1 crogers2287 crogers2287 1.7K Feb 11 17:48 repost_blog_v2_pin.py
-rw-r--r--  1 crogers2287 crogers2287  684 Feb 11 17:48 repost_blog_v2_x.py
-rw-r--r--  1 crogers2287 crogers2287  749 Feb 11 17:48 republish_blog.py
-rw-r--r--  1 crogers2287 crogers2287  585 Feb 11 17:48 update_blog_slug.py
-rwxrwxr-x  1 crogers2287 crogers2287  27K Feb 15 04:12 x_deal_scraper.py


The script exists and is executable. Let me check if it has any dependencies or issues. Let me look at the script to understand what it does and why it might have timed out.


#!/usr/bin/env python3
"""
X (Twitter) Deal Scraper for GottaHaveCoups
============================================

Scrapes specified X accounts for deal posts and imports them into the GHC system.

Usage:
    python3 x_deal_scraper.py                    # Run scrape on all accounts
    python3 x_deal_scraper.py --add @account      # Add account to scrape list
    python3 x_deal_scraper.py --remove @account   # Remove account from list
    python3 x_deal_scraper.py --list              # List all monitored accounts
    python3 x_deal_scraper.py --dashboard         # Launch web dashboard
"""

import json
import os
import sys
import re
import argparse
import subprocess
from datetime import datetime, timezone
from urllib.request import Request, urlopen
from typing import List, Dict, Optional

# Config paths
CONFIG_DIR = "/home/crogers2287/openclaw-workspaces/ghc-marketing/config"
CONFIG_FILE = os.path.join(CONFIG_DIR, "x_scraper_accounts.json")
SCRAPED_DEALS_FILE = os.path.join(CONFIG_DIR, "x_scraped_deals.json")

# GHC API endpoints
GHC_API_BASE = "https://www.gottahavecoups.com/api/v1"
GHC_DEALS_API = f"{GHC_API_BASE}/deals"
GHC_IMPORT_API = f"{GHC_API_BASE}/admin/import-deal"  # Adjust based on actual API

# Maya VM connection for twikit
MAYA_HOST = "maya@maya"

def load_config() -> Dict:
    """Load scraper configuration."""
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, 'r') as f:
            return json.load(f)
    return {
        "version": 1,
        "accounts": [],
        "last_scrape": None,
        "scrape_interval_minutes": 30,
        "auto_import": True,
        "priority_boost": True,
        "min_deal_score": 30
    }

def save_config(config: Dict):
    """Save scraper configuration."""
    os.makedirs(CONFIG_DIR, exist_ok=True)
    with open(CONFIG_FILE, 'w') as f:
        json.dump(config, f, indent=2)

def load_scraped_deals() -> List[Dict]:
    """Load previously scraped deals to avoid duplicates."""
    if os.path.exists(SCRAPED_DEALS_FILE):
        with open(SCRAPED_DEALS_FILE, 'r') as f:
            return json.load(f)
    return []

def save_scraped_deals(deals: List[Dict]):
    """Save scraped deals history."""
    with open(SCRAPED_DEALS_FILE, 'w') as f:
        json.dump(deals, f, indent=2)

def add_account(username: str, category: str = "general", notes: str = ""):
    """Add an X account to the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    # Check if already exists
    for acc in config["accounts"]:
        if acc["username"].lower() == username:
            print(f"⚠  @{username} is already in the scrape list")
            return False

    config["accounts"].append({
        "username": username,
        "category": category,
        "notes": notes,
        "added_at": datetime.now(timezone.utc).isoformat(),
        "last_scraped": None,
        "active": True,
        "deals_found": 0
    })

    save_config(config)
    print(f"✅ Added @{username} to scrape list")
    return True

[672 more lines in file. Use offset=101 to continue.]


Let me continue reading to understand what the script does and check if there's any issue with the scraping process.


def remove_account(username: str):
    """Remove an X account from the scrape list."""
    config = load_config()
    username = username.lstrip('@').lower()

    original_count = len(config["accounts"])
    config["accounts"] = [a for a in config["accounts"] if a["username"].lower() != username]

    if len(config["accounts"]) < original_count:
        save_config(config)
        print(f"✅ Removed @{username} from scrape list")
        return True
    else:
        print(f"⚠  @{username} not found in scrape list")
        return False

def list_accounts():
    """List all accounts being scraped."""
    config = load_config()

    if not config["accounts"]:
        print("\n📭 No X accounts configured for scraping")
        print("\nAdd accounts with: python3 x_deal_scraper.py --add @username")
        return

    print(f"\n📱 X Accounts to Scrape ({len(config['accounts'])} total)\n")
    print(f"{'Username':<20} {'Category':<12} {'Status':<8} {'Deals':<6} {'Last Scraped'}")
    print("-" * 70)

    for acc in config["accounts"]:
        status = "🟢" if acc.get("active", True) else "🔴"
        last_scraped = acc.get("last_scraped", "Never")[:16] if acc.get("last_scraped") else "Never"
        deals = acc.get("deals_found", 0)
        print(f"@{acc['username']:<19} {acc.get('category', 'general'):<12} {status:<8} {deals:<6} {last_scraped}")

    print("\n" + "-" * 70)
    print(f"Auto-import: {'✅ ON' if config.get('auto_import') else '❌ OFF'}")
    print(f"Priority boost: {'✅ ON' if config.get('priority_boost') else '❌ OFF'}")
    print(f"Min deal score: {config.get('min_deal_score', 30)}")

def extract_deal_info(tweet_text: str, tweet_media: List[str] = None) -> Optional[Dict]:
    """
    Extract deal information from a tweet text.
    Returns deal dict or None if no deal detected.
    """
    deal = {
        "title": None,
        "price": None,
        "original_price": None,
        "discount_percentage": None,
        "store": None,
        "product_url": None,
        "image_url": None,
        "category": None,
        "source_tweet": None
    }

    # Price patterns
    price_patterns = [
        r'\$([\d,]+\.?\d*)',  # $XX.XX
        r'([\d,]+)\s*dollars',  # XX dollars
    ]

    # Look for prices
    prices = []
    for pattern in price_patterns:
        matches = re.findall(pattern, tweet_text, re.IGNORECASE)
        prices.extend([float(p.replace(',', '')) for p in matches])

    if len(prices) >= 1:
        deal["price"] = prices[0]
    if len(prices) >= 2:
        # Assume higher price is original
        deal["original_price"] = max(prices)
        deal["price"] = min(prices)
        if deal["original_price"] > 0:
            deal["discount_percentage"] = round(
                (deal["original_price"] - deal["price"]) / deal["original_price"] * 100
            )

    # Store detection
    store_patterns = {
        "Amazon": [r'amazon\.com', r'amzn\.to', r'@amazon'],
        "Walmart": [r'walmart\.com', r'w-mt\.co', r'@walmart'],
        "Target": [r'target\.com', r'@target'],
        "Best Buy": [r'bestbuy\.com', r'@bestbuy'],
        "Costco": [r'costco\.com', r'@costco'],
        "Home Depot": [r'homedepot\.com', r'@homedepot'],
    }

    for store, patterns in store_patterns.items():
        for pattern in patterns:
            if re.search(pattern, tweet_text, re.IGNORECASE):
                deal["store"] = store
                break
        if deal["store"]:

[572 more lines in file. Use offset=201 to continue.]


Let me check more of the script to see what's happening when it runs. I'm particularly interested in the main scraping function.


                    pass

            await page.close()
        except Exception as e:
            print(f"Browser error: {{e}}")

    print(json.dumps(tweets, default=str))

asyncio.run(scrape())
'''

        # Save script to Maya VM
        script_path = f"/tmp/scrape_{username}.py"
        with open(script_path, 'w') as f:
            f.write(scrape_script)

        # Copy to Maya
        subprocess.run(['scp', '-q', script_path, f"{MAYA_HOST}:/tmp/scrape_{username}.py"], check=True)

        # Run the scraper on Maya
        result = subprocess.run(
            ['ssh', MAYA_HOST, f'python3 /tmp/scrape_{username}.py'],
            capture_output=True,
            text=True,
            timeout=60
        )

        if result.returncode != 0:
            print(f"⚠  Error scraping @{username}: {result.stderr}")
            return []

        # Parse output
        output = result.stdout.strip()
        if output:
            try:
                tweets = json.loads(output)
                print(f"✅ Found {len(tweets)} tweets from @{username}")
                return tweets
            except json.JSONDecodeError:
                print(f"⚠  Invalid JSON from @{username}")
                return []

        return []

    except subprocess.TimeoutExpired:
        print(f"⏱  Timeout scraping @{username}")
        return []
    except Exception as e:
        print(f"⚠  Error scraping @{username}: {e}")
        return []

def import_deal_to_ghc(deal: Dict, source_account: str) -> bool:
    """
    Import a scraped deal into the GottaHaveCoups system.
    Returns True if successful.
    """
    # For now, we'll create a draft deal that can be reviewed
    # In production, this would POST to the GHC API

    draft_deal = {
        "title": deal["title"],
        "description": f"Scraped from X @{source_account}",
        "price": deal["price"],
        "original_price": deal.get("original_price"),
        "discount_percentage": deal.get("discount_percentage"),
        "store_name": deal.get("store", "Unknown"),
        "category_id": deal.get("category", 1),
        "source": f"X:@{source_account}",
        "deal_url": deal.get("product_url", ""),
        "image_url": deal.get("image_url", ""),
        "quality_score": 50,  # Boost priority
        "is_hot": True,
        "is_featured": True,
        "scraped_at": datetime.now(timezone.utc).isoformat(),
        "needs_review": True  # Flag for manual review
    }

    # Save to scraped deals file
    scraped = load_scraped_deals()

    # Check for duplicates (by title + price)
    for existing in scraped:
        if (existing.get("title") == draft_deal["title"] and
            existing.get("price") == draft_deal["price"]):
            print(f"   ⚠  Duplicate deal skipped: {draft_deal['title'][:50]}...")
            return False

    scraped.append(draft_deal)
    save_scraped_deals(scraped)

    print(f"   ✅ Imported: {draft_deal['title'][:60]}...")
    return True

def run_scraper():
    """Main scraper routine."""
    config = load_config()

    if not config["accounts"]:
        print("📭 No accounts configured. Add accounts with --add @username")
        return

    print(f"\n🚀 Starting X Deal Scraper")
    print(f"   Accounts: {len(config['accounts'])}")
    print(f"   Auto-import: {'ON' if config.get('auto_import') else 'OFF'}")
    print()

    total_deals = 0
    new_deals = 0

    for account in config["accounts"]:
        if not account.get("active", True):
            continue

        username = account["username"]
        tweets = scrape_account_tweets(username, limit=20)

        account_deals = 0
        for tweet in tweets:
            deal_info = extract_deal_info(tweet.get("text", ""), tweet.get("media", []))
            if deal_info:
                total_deals += 1
                if import_deal_to_ghc(deal_info, username):
                    new_deals += 1
                    account_deals += 1

        # Update account stats
        account["last_scraped"] = datetime.now(timezone.utc).isoformat()
        account["deals_found"] = account.get("deals_found", 0) + account_deals

        print()

    config["last_scrape"] = datetime.now(timezone.utc).isoformat()
    save_config(config)

    print(f"📊 Scrape Complete")
    print(f"   Total deals found: {total_deals}")
    print(f"   New deals imported: {new_deals}")
    print(f"   Next scrape: ~{config['scrape_interval_minutes']} minutes")

def launch_dashboard():
    """Launch the web dashboard for managing accounts."""
    dashboard_path = "/home/crogers2287/openclaw-workspaces/ghc-marketing/x-scraper-dashboard.html"
    print(f"🌐 Launching dashboard...")
    print(f"   File: {dashboard_path}")

    # Create dashboard HTML if it doesn't exist
    if not os.path.exists(dashboard_path):

[323 more lines in file. Use offset=450 to continue.]


I see the issue now! The script uses SSH to connect to Maya VM and runs a Playwright scraper there. The problem is:

1. The script tries to SSH to `maya@maya` which requires SSH keys to be set up
2. It runs Playwright which needs to be installed and configured
3. The timeout is 60 seconds per account which could accumulate

The script appears to be stuck trying to SSH to Maya or the Playwright scraper is hanging. Let me check if there are any accounts configured to scrape.


X (Twitter) Accounts to Scrape for Deals


I can see there are 3 accounts configured and they were last scraped just over 11 hours ago (at 03:11 UTC, and it's now 14:11 UTC). The scraper is set to run every 30 minutes, so it should run.

The issue is that the script timed out at 120 seconds, likely because:
1. It's trying to SSH to Maya VM
2. Running Playwright on each of the 3 accounts
3. Each account has a 60-second timeout

3 accounts × 60 seconds = 180 seconds, which exceeds my 120-second timeout.

Let me run the scraper again but without a timeout restriction, or I can check if there's an alternative way to run this or if there are any recent results I can report.

cd /home/crogers2287/openclaw-workspaces/ghc-marketing && timeout 300 python3 scripts/x_deal_scraper.py 2>&1


Command still running (session crisp-lagoon, pid 3312360). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


(no new output)

Process still running.


Let me check the log to see if there's any output yet.


(no output yet)


Let me wait a bit longer and poll again.


Still no output. Let me wait a bit longer as the script needs to SSH to Maya and run Playwright. Let me poll again.


This is taking a while. Let me test if the SSH connection to Maya is working, as that might be the issue.

timeout 10 ssh maya@maya "echo 'SSH test successful'" 2>&1


Ah ha! The SSH connection to Maya is timing out. This is why the script isn't working - it can't connect to the Maya VM. Let me kill the running process and check if Maya is reachable via Tailscale.


Killed session crisp-lagoon.


Let me check if Maya is accessible via Tailscale.


100.124.155.26  maya                   tagged-devices          linux    active; relay "mia"; offline, last seen 10h ago, tx 6084 rx 0


There's the problem! Maya VM has been offline for 10 hours. The scraper can't run because it needs to SSH to Maya to execute the Playwright scraper.

Now I need to report this to the group chat. The scraper failed because Maya VM is offline.

⚠ **X Deal Scraper - Connection Failed**

The scraper couldn't run because **Maya VM is offline** (last seen 10 hours ago).

**Configured Accounts:**
- @dealsfinderio
- @glitcheddeals
- @fatkiddeals

**Last successful scrape:** 03:11 UTC today (11 hours ago, found 0 new deals)

**To fix:** Maya VM needs to be brought back online for the scraper to work. The script uses SSH + Playwright on Maya to scrape X accounts.