┌────┬──────────────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name                     │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼──────────────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 17 │ acestep                  │ default     │ N/A     │ fork    │ 3441812  │ 6s     │ 12   │ online    │ 100%     │ 738.6mb  │ crogers… │ disabled │
│ 5  │ ash-compliance           │ default     │ N/A     │ fork    │ 48684    │ 10h    │ 0    │ online    │ 0%       │ 110.7mb  │ crogers… │ disabled │
│ 10 │ bankstatements           │ default     │ 1.0.0   │ fork    │ 48694    │ 10h    │ 0    │ online    │ 0%       │ 189.8mb  │ crogers… │ disabled │
│ 1  │ canonforge-backend       │ default     │ N/A     │ fork    │ 48604    │ 10h    │ 0    │ online    │ 0%       │ 12.7mb   │ crogers… │ disabled │
│ 14 │ cfrmanager-next          │ default     │ N/A     │ fork    │ 3443811  │ 0s     │ 9629 │ online    │ 0%       │ 55.1mb   │ crogers… │ disabled │
│ 13 │ cfrmanager-pocketbase    │ default     │ 0.1.0   │ fork    │ N/A      │ 0      │ 295… │ stopped   │ 0%       │ 0b       │ crogers… │ disabled │
│ 19 │ comfyui                  │ default     │ N/A     │ fork    │ 3388626  │ 2m     │ 0    │ online    │ 0%       │ 2.8mb    │ crogers… │ disabled │
│ 0  │ novel-os                 │ default     │ N/A     │ fork    │ 3443361  │ 1s     │ 172… │ online    │ 0%       │ 65.3mb   │ crogers… │ disabled │
│ 2  │ novel-os-dev             │ default     │ N/A     │ fork    │ N/A      │ 0      │ 430… │ stopped   │ 0%       │ 0b       │ crogers… │ disabled │
│ 16 │ omi-integration          │ default     │ N/A     │ fork    │ N/A      │ 0      │ 7    │ stopped   │ 0%       │ 0b       │ crogers… │ disabled │
│ 15 │ omi-stt                  │ default     │ N/A     │ fork    │ N/A      │ 0      │ 1075 │ stopped   │ 0%       │ 0b       │ crogers… │ disabled │
│ 3  │ ouroboros-api            │ default     │ N/A     │ fork    │ 48634    │ 10h    │ 0    │ online    │ 0%       │ 36.0mb   │ crogers… │ disabled │
│ 6  │ pasturelog-api           │ default     │ 1.0.0   │ fork    │ N/A      │ 0      │ 167… │ stopped   │ 0%       │ 0b       │ crogers… │ disabled │
│ 7  │ pasturelog-frontend      │ default     │ N/A     │ fork    │ 48686    │ 10h    │ 0    │ online    │ 0%       │ 52.8mb   │ crogers… │ disabled │
│ 8  │ permit-scheduler         │ default     │ 1.0.0   │ fork    │ 48687    │ 10h    │ 0    │ online    │ 0%       │ 46.7mb   │ crogers… │ disabled │
│ 9  │ permit-tracker           │ default     │ N/A     │ fork    │ 48688    │ 10h    │ 0    │ online    │ 0%       │ 40.5mb   │ crogers… │ disabled │
│ 12 │ quest-scheduler          │ default     │ N/A     │ fork    │ 48701    │ 10h    │ 0    │ online    │ 0%       │ 34.9mb   │ crogers… │ disabled │
│ 4  │ t5gemma-vision           │ default     │ N/A     │ fork    │ 48683    │ 10h    │ 0    │ online    │ 0%       │ 611.0mb  │ crogers… │ disabled │
│ 11 │ tmux-web                 │ default     │ N/A     │ fork    │ 3443709  │ 0s     │ 663… │ online    │ 100%     │ 36.6mb   │ crogers… │ disabled │
└────┴──────────────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘


ACE-Step is loading! It's at 738MB memory and 100% CPU (restart count 12, uptime 6 seconds). This is a good sign - it's loading the models. The memory is much lower than before, which means it's probably loading on GPU 1 now. Let me wait a bit longer for it to fully load and then check if port 7860 opens up.

ACE-Step is loading models (738MB, 100% CPU). Let me wait for it to finish and check the ports.

exec

sleep 30 && netstat -tuln | grep -E "7860|8188"


Command still running (session good-river, pid 3446134). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


The command is running with a 30 second sleep. Let me check the PM2 status now to see if ACE-Step has made progress.


┌────┬──────────────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name                     │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼──────────────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 17 │ acestep                  │ default     │ N/A     │ fork    │ 3446816  │ 12s    │ 13   │ online    │ 100%     │ 1.3gb    │ crogers… │ disabled │
│ 5  │ ash-compliance           │ default     │ N/A     │ fork    │ 48684    │ 10h    │ 0    │ online    │ 0%       │ 110.7mb  │ crogers… │ disabled │
│ 10 │ bankstatements           │ default     │ 1.0.0   │ fork    │ 48694    │ 10h    │ 0    │ online    │ 0%       │ 189.8mb  │ crogers… │ disabled │
│ 1  │ canonforge-backend       │ default     │ N/A     │ fork    │ 48604    │ 10h    │ 0    │ online    │ 0%       │ 12.7mb   │ crogers… │ disabled │
│ 14 │ cfrmanager-next          │ default     │ N/A     │ fork    │ 3449834  │ 1s     │ 9634 │ online    │ 0%       │ 65.8mb   │ crogers… │ disabled │
│ 13 │ cfrmanager-pocketbase    │ default     │ 0.1.0   │ fork    │ N/A      │ 0      │ 295… │ stopped   │ 0%       │ 0b       │ crogers… │ disabled │
│ 19 │ comfyui                  │ default     │ N/A     │ fork    │ 3388626  │ 3m     │ 0    │ online    │ 0%       │ 2.8mb    │ crogers… │ disabled │
│ 0  │ novel-os                 │ default     │ N/A     │ fork    │ 3449722  │ 1s     │ 172… │ online    │ 0%       │ 66.4mb   │ crogers… │ disabled │
│ 2  │ novel-os-dev             │ default     │ N/A     │ fork    │ N/A      │ 0      │ 430… │ stopped   │ 0%       │ 0b       │ crogers… │ disabled │
│ 16 │ omi-integration          │ default     │ N/A     │ fork    │ N/A      │ 0      │ 7    │ stopped   │ 0%       │ 0b       │ crogers… │ disabled │
│ 15 │ omi-stt                  │ default     │ N/A     │ fork    │ N/A      │ 0      │ 1075 │ stopped   │ 0%       │ 0b       │ crogers… │ disabled │
│ 3  │ ouroboros-api            │ default     │ N/A     │ fork    │ 48634    │ 10h    │ 0    │ online    │ 0%       │ 36.0mb   │ crogers… │ disabled │
│ 6  │ pasturelog-api           │ default     │ 1.0.0   │ fork    │ N/A      │ 0      │ 167… │ stopped   │ 0%       │ 0b       │ crogers… │ disabled │
│ 7  │ pasturelog-frontend      │ default     │ N/A     │ fork    │ 48686    │ 10h    │ 0    │ online    │ 0%       │ 52.8mb   │ crogers… │ disabled │
│ 8  │ permit-scheduler         │ default     │ 1.0.0   │ fork    │ 48687    │ 10h    │ 0    │ online    │ 0%       │ 46.7mb   │ crogers… │ disabled │
│ 9  │ permit-tracker           │ default     │ N/A     │ fork    │ 48688    │ 10h    │ 0    │ online    │ 0%       │ 40.5mb   │ crogers… │ disabled │
│ 12 │ quest-scheduler          │ default     │ N/A     │ fork    │ 48701    │ 10h    │ 0    │ online    │ 0%       │ 34.9mb   │ crogers… │ disabled │
│ 4  │ t5gemma-vision           │ default     │ N/A     │ fork    │ 48683    │ 10h    │ 0    │ online    │ 0%       │ 611.0mb  │ crogers… │ disabled │
│ 11 │ tmux-web                 │ default     │ N/A     │ fork    │ 3450135  │ 0s     │ 663… │ online    │ 200%     │ 27.0mb   │ crogers… │ disabled │
└────┴──────────────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘


Good progress! ACE-Step is now at 1.3GB memory (up from 738MB), 100% CPU, restart count 13, and has been running for 12 seconds. It's loading more models. Let me wait a bit more and then check the ports and final status.


Command still running (session glow-bloom, pid 3452081). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


Let me check the status again after the wait and see if the servers are both up.

sleep 15 && pm2 status | grep -E "acestep|comfyui"


Command still running (session quiet-nudibranch, pid 3456818). Use process (list/poll/log/write/kill/clear/remove) for follow-up.


Multiple commands running with sleep delays. Let me wait a moment and then check the ports directly to see which servers are up.

Let me check both servers' status now.

netstat -tuln 2>/dev/null | grep -E "7860|8188" && echo "---" && pm2 list | grep -E "acestep|comfyui"


tcp        0      0 0.0.0.0:8188            0.0.0.0:*               LISTEN
---
│ 17 │ acestep                  │ default     │ N/A     │ fork    │ 3460858  │ 1s     │ 16   │ online    │ 100%     │ 500.9mb  │ crogers… │ disabled │
│ 19 │ comfyui                  │ default     │ N/A     │ fork    │ 3388626  │ 3m     │ 0    │ online    │ 0%       │ 2.8mb    │ crogers… │ disabled │


ComfyUI is running on port 8188. ACE-Step is running (restart count 16 now, 1s uptime, 500MB memory) but port 7860 still isn't listening. This suggests that ACE-Step is still crashing before it can start the web server. Let me check the recent error logs to see what's happening.

ACE-Step keeps restarting (16 times). Let me check the latest error:

tail -25 ~/.pm2/logs/acestep-error-17.log | grep -A20 "Error\|Exception\|OutOfMemory"


2026-02-16T04:00:03: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.00 MiB. GPU 0 has a total capacity of 23.56 GiB of which 11.31 MiB is free. Process 2693 has 256.00 MiB memory in use. Process 2354 has 256.00 MiB memory in use. Process 843801 has 15.78 GiB memory in use. Process 3388627 has 256.00 MiB memory in use. Including non-PyTorch memory, this process has 6.92 GiB memory in use. Of the allocated memory 6.61 GiB is allocated by PyTorch, and 21.35 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2026-02-16T04:00:03:


It's still trying to load on GPU 0 and failing. The CUDA_VISIBLE_DEVICES environment variable didn't work as expected. Let me check what's using GPU 0 so heavily (process 843801 has 15.78 GiB).

Still loading on GPU 0. Let me check what process 843801 is (the one using 15.78 GiB):


crogers+  843801  0.2 14.9 135265948 19722880 ?  Sl   Feb15   1:30 /mnt/storage/llama.cpp-nemotron/build/bin/llama-server --model /mnt/storage/ollama/gguf-models/GLM-4.7-Flash-Q6_K.gguf --port 5803 --n-gpu-layers 999 --ctx-size 65536 --threads 8 --batch-size 2048 --flash-attn on --cont-batching --host 0.0.0.0 --jinja --split-mode layer --tensor-split 0.5,0.5 -ctk q8_0 -ctv q8_0 --reasoning-budget 0 --reasoning-format none


Ah! There's a llama-server process running GLM-4.7-Flash model using both GPUs (--tensor-split 0.5,0.5) and it's taking up most of the memory. This is why ACE-Step can't load. I have a few options:

1. Stop the llama-server temporarily
2. Kill ComfyUI temporarily to free GPU 0
3. Reconfigure llama-server to use only one GPU
4. Stop ACE-Step for now and only run ComfyUI

Since the user needs both servers, let me check if I can temporarily stop the llama-server or reconfigure it. First, let me notify the user about the situation and ask what they prefer.

Found the problem. A llama-server running GLM-4.7-Flash is using 15.78GB across both GPUs. That's why ACE-Step can't load.

**Options:**

1. **Stop llama-server temporarily** - Frees ~16GB VRAM for both ACE-Step + ComfyUI
2. **Reconfigure llama-server** - Use only GPU 0, free GPU 1 for ACE-Step
3. **ComfyUI only** - Stop ACE-Step; use ACE-Step nodes directly in ComfyUI workflows

Which do you prefer? The llama-server is running GLM-4.7-Flash on port 5803.