Remnawave node monitoring: what breaks and how to check

Everything below is checked against the source of Remnawave backend 3.4.10 and Remnawave node 3.4.5, the latest tags of remnawave/backend and remnawave/node, plus the install guide at docs.rw. I read the code because it answers questions the docs leave open, such as how often the panel checks a node and what it does when a check fails.

How does the Remnawave panel talk to its nodes?

The panel calls each node over a single HTTPS port with mutual TLS, and the node talks to its own Xray over a local socket. A node is a Docker container (remnawave/node, host networking) with two settings:

Setting	Example	What it does
NODE_PORT	2222	the port the node listens on for the panel's requests
SECRET_KEY	copied from the panel	carries the node's certificate and key, the CA certificate, and the panel's JWT public key

You get both by adding the node in the panel and clicking Copy docker-compose.yml. The node then accepts only clients whose certificate is signed by that CA (requestCert and rejectUnauthorized are both on), and checks the panel's JWT as well. The node's Xray API is a Unix socket inside the container, not a TCP port, so NODE_PORT is the only management port. The docs say to open it in the node's firewall for the panel's IP only.

The panel side works like this:

1. When the panel starts, it restarts every node once.
2. Starting a node means pushing an Xray config built from the node's config profile and the inbounds selected for that node, with the users who may use them.
3. Every 10 seconds, it asks each enabled node for system stats. The node gets those from its Xray through the API. Each request has a 15 second timeout, and the panel tries twice before it marks the node down.
4. If a node that was down answers again, the panel marks it connected and starts it again.

If the panel cannot reach a node directly, each node can have a proxyUrl, and the panel connects through that SOCKS proxy.

Notice what the health check covers: the management port and Xray's stats API. It never connects to an inbound.

What does each node status mean?

Each node has three flags, isConnected, isConnecting and isDisabled, plus lastStatusMessage and lastStatusChange. The dashboard's status comes from these:

Flags	Meaning
isDisabled true	an admin turned the node off, or the panel did because it has no config profile or no inbounds; the health check skips it
isConnecting true	the panel is pushing the config and starting Xray
isConnected true	the last health check passed
all false	the node is down; lastStatusMessage says why

The lastStatusMessage is where to look first. A few messages come from the panel itself:

• Required info is missing. Outdated version? The node answered but sent no Xray info, which usually means the node is older than the panel.
• Outdated version ... of Remnawave Node. Please upgrade to the latest version (>= 2.7.0). The panel refused to start a node that old.
• Failed to sync node plugins: ... The node is reachable, but a node plugin could not be set up.

Otherwise it holds the network or TLS error from the last failed request, or the error Xray returned when it failed to start.

The panel does alert on node problems, which is one big difference from Marzban. When a connected node fails its check, Remnawave emits node.connection_lost, and node.connection_restored when it comes back. With IS_TELEGRAM_NOTIFICATIONS_ENABLED=true, both go to the Telegram chat in TELEGRAM_NOTIFY_NODES (written as chat_id:thread_id). They also go to your webhook if WEBHOOK_ENABLED is on. Webhook payloads are signed with WEBHOOK_SECRET_HEADER.

Two limits to keep in mind. The panel sends that alert from inside your own infrastructure, so if the panel host goes down, nothing is sent. And there is no confirmation period beyond the two attempts in one check, so a node that drops for half a minute sends a lost and a restored message.

For a check that does not depend on the panel, read the node list from the API with an API token from the panel. The panel only accepts requests through a reverse proxy over HTTPS, so use the public address, not port 3000 on the host:

#!/usr/bin/env bash
# Ping a heartbeat when every enabled Remnawave node is connected, report /fail when one is not.
set -uo pipefail
PANEL="${PANEL:-https://panel.example.com}"

bad=$(curl -fsS -m 15 -H "Authorization: Bearer $RW_TOKEN" "$PANEL/api/nodes" |
  jq -r '.response[] | select((.isDisabled | not) and (.isConnected | not))
         | "\(.name) \(.lastStatusMessage // "")"') || exit 1

if [ -n "$bad" ]; then
  echo "$bad"
  curl -fsS -m 10 --retry 3 -o /dev/null "$HEARTBEAT_URL/fail"
else
  curl -fsS -m 10 --retry 3 -o /dev/null "$HEARTBEAT_URL"
fi

Run it every minute from a machine that is not the panel. If the panel does not answer, the script exits without calling the heartbeat, and the heartbeat alerts when its period runs out.

If you run Prometheus, the panel also serves metrics on METRICS_PORT (3001 in the sample config) at /metrics, behind basic auth with METRICS_USER and METRICS_PASS. node_status is 1 for a connected node and 0 otherwise, labelled with node_uuid, and node_basic_info maps each UUID to a node name. It is the same flag as the API, refreshed every 15 seconds.

Why can a node look online while users can't connect?

Because online only means the panel can reach NODE_PORT and Xray answers on its socket. Everything users touch is outside that check:

• Inbound ports blocked. The firewall allows the panel on 2222 but not 443 for everyone. The node is online and useless.
• The host address is wrong. In Remnawave, the address users get comes from the host, not the node. A host has its own address and port, so a typo there, or a DNS record that still points at an old server, breaks every link while the node stays green.
• The host points at an inbound the node doesn't run. A node runs only the inbounds selected for it in its config profile. A host tied to an inbound that this node does not run gives users a link that can never work.
• Offline nodes stay in the subscription. The subscription is built from the user's hosts. It does not check whether a node is connected, so users keep getting links to a node that has been down for hours.
• TLS and REALITY settings. An expired certificate on a TLS inbound, or a REALITY target that stopped answering, fails the handshake. Neither shows in the panel. See Xray REALITY not working for the REALITY side.
• Blocking. A node's IP can be blocked in one country and work everywhere else, including from your panel.

How do I check every node from outside?

Use a dedicated monitoring user's subscription, which already lists one link per host. Set the user up so its links only fail when a server does:

1. Create a user called monitor with no expiry and no traffic limit.
2. Add it to internal squads that cover every inbound you serve. A user only gets hosts whose inbounds are in its squads.
3. If the HWID device limit is on, set this user's device limit to 0, which the panel treats as no limit. Otherwise a checker that sends no HWID header gets a subscription with no links and the header x-hwid-not-supported: true.

Then fetch the subscription and count the links:

SUB='https://panel.example.com/api/sub/your-monitor-short-uuid'
curl -fsS "$SUB" | base64 -d | grep -c '://'

That count should equal the number of hosts the monitor user can see. To test each one, add /json to the URL. That returns Xray client configs built from the default template, each with a SOCKS inbound on 127.0.0.1:10808, so you can run each one with Xray and fetch a URL through it. The steps and a script are in How to test every server in a V2Ray subscription. Run the check from a network that is not your own.

Read the results by host remark. A failure on every host of one node points at that node. A failure on one inbound across every node points at the config profile.

How do I monitor the subscription Remnawave serves?

Request it like a client and check the status, the headers and the link count. The public address comes from SUB_PUBLIC_DOMAIN, which the sample config sets to panel.domain.com/api/sub, followed by the user's short UUID:

curl -sS -D - -o /dev/null 'https://panel.example.com/api/sub/your-monitor-short-uuid'

A healthy response is 200 with content-disposition and subscription-userinfo, plus profile-title, profile-update-interval and support-url when you set them. What the other answers mean:

Response	Cause
404	the short UUID doesn't match a user, for example after the user was deleted
403	no response rule matched the request, or a /json style path was used while path access is turned off
200 with no links	the HWID device limit refused the request; look for x-hwid-not-supported or x-hwid-max-devices-reached. The body is empty, or holds only notice entries if custom remarks are on
an HTML page	the request sent Accept: text/html, which the built-in Browser Subscription rule answers

The 403 needs care. Remnawave picks the format from response rules matched on request headers, and the default set ends with a Fallback Base64 rule marked as system critical. Delete or reorder it, and clients the other rules don't recognise get 403, your checker included. /api/sub/<short UUID>/info returns the user as JSON, which is handy for checking that the monitor user is still active.

Full disclosure: TunnelHQ covers this layer and the outside check above. Give it the monitor user's subscription URL and it tests every link behind it with a real handshake at each sync, every 5 minutes to every 24 hours, and pauses the monitors of hosts that leave the list. Subscription URLs need a paid plan: Starter includes 3, Pro 10, Business unlimited. See Monitoring V2Ray subscription URLs for what else can go wrong with one, and the VLESS monitoring page for what a check does.

A monitoring plan for a Remnawave fleet

What	How	How often	Alert when
Node status	/api/nodes script with a heartbeat, or node_status in Prometheus	1 minute	any enabled node not connected, or no answer
Panel alerts	TELEGRAM_NOTIFY_NODES or a webhook	as they happen	node.connection_lost
Every host on every node	real connection through the monitor user's links, from outside	1 to 5 minutes	the same link fails twice in a row
Subscription	status, headers, link count	5 to 15 minutes	not 200, an empty body, or fewer links
Certificates	openssl s_client against TLS inbounds	daily	less than 14 days left

Keep the panel's own alerts on, and treat them as the fast inside signal. The outside check is the one that tells you what users see.