Memory leak on "observed" node #3338

sakontwist · 2024-05-05T18:01:16Z

I turned on “observer” on April 30 and memory leak appeared on the nodes being "observed". The nodes under "observing" are using a primitive configuration, no DNS with one direct outbound.
RSS xray graph (green):

What debugging information needs to be attached?

Fangliding · 2024-05-05T18:15:34Z

Client config and server config just as the template required. Why ignore it

sakontwist · 2024-05-05T19:34:58Z

I use split and complex routing (homeland via vps balancer) on the client, it will be difficult to give full conifguration, I will give part of the config on the problem:

outbounds for observed node

    {
      "protocol": "shadowsocks",
      "tag": "vps6",
      "settings": {
	    "servers": [
            {
                "address": "x.x.x.x","port": xxxx,
                "method": "2022-blake3-chacha20-poly1305",
                "password": "xxxxxxxxxx",
                "uot":true, "UoTVersion":2
              }
	    ]
      },
      "mux": {
          "enabled": true,
          "concurrency": 8,
          "xudpConcurrency": 0,
          "xudpProxyUDP443": "allow"
      }
    }

observer config

"observatory": {
    "subjectSelector": [
      "vps6",
      "vps5",
      "vps2"
    ],
    "probeURL": "https://www.instagram.com",
    "probeInterval": "30s",
    "enableConcurrency": true
  }

balancer

	"balancers": [
		{
			"tag": "ss-bl",
			"selector": ["vps6","vps5","vps2"],
			"strategy": {
				"type": "leastPing"
			},
			"fallbackTag": "vps6"
		}
	]

"observed" nodes
inbound:

    {
      "tag":"in-ss",
      "listen": "x.x.x.x","port": xxx,
      "protocol": "shadowsocks",
      "settings": {
        "network":"tcp",
        "method": "2022-blake3-chacha20-poly1305",
        "password":"xxxxxxxxxx"
      },
      "sniffing": {
        "enabled": false,
        "destOverride":["http","tls","quic"],
        "routeOnly": true
      }
    }

routing:

  "routing": {
    "domainStrategy": "AsIs",
    "rules": [
      {
        "inboundTag": ["in_api"],
        "outboundTag": "api"
      },
      {
	"inboundTag": ["metrics-in"],
	"outboundTag": "metrics-out"
      }
    ]
  }

out:

  "outbounds": [
    {
      "tag": "direct",
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "AsIs"
      }
    }
  ]

I assume polls need to be done less frequently than the timeout policy.... (using default values for policy).
I have to disable observer for now so that Xray doesn't crash, but I can reproduce if necessary.

Fangliding · 2024-05-06T01:39:11Z

Attempting to remove unnecessary options and routes to provide a complete configuration, it is difficult to determine the problem if it cannot be reproduced
And if the observer is turned off, will the problem still be exists？

yuhan6665 · 2024-05-06T02:00:50Z

what if you change observatory to burstObservatory config?

sakontwist · 2024-05-06T07:39:59Z

Which of these do you consider “unnecessary options”?

-rw-r--r-- 1 root root  144 апр 27 09:27 00-log.json
-rw-r--r-- 1 root root  165 апр 27 00:26 05-api-stats-metrics.json
-rw-r--r-- 1 root root  672 апр 29 17:02 10-dns.json
-rw-r--r-- 1 root root 4467 мая  5 20:09 15-routing.json
-rw-r--r-- 1 root root  381 апр 27 00:26 19-in-api-metrics.json
-rw-r--r-- 1 root root  354 апр  7 17:47 20-policy.json
-rw-r--r-- 1 root root 1004 апр 27 09:27 23-in-lan.json
-rw-r--r-- 1 root root 1535 мая  5 22:11 25-in-spec.json
-rw-r--r-- 1 root root 1889 апр 24 12:45 35-in-ext.json
-rw-r--r-- 1 root root  254 апр 24 19:11 40-out-direct.json
-rw-r--r-- 1 root root 1033 апр 27 00:26 45-out-internal-tail.json
-rw-r--r-- 1 root root 2725 апр 24 21:41 50-out-ss-vps-tail.json
-rw-r--r-- 1 root root 4015 апр 25 01:08 60-out-rea-vps-tail.json

Turning off the observer completely eliminated the problem:
The documentation lags significantly behind the development, I did the "observer" customization by finding examples in the issues. I did not try burstObserver, because there is very little information on its configuration.

sakontwist · 2024-05-06T10:39:17Z

It seems the problem was that I have an entry "geosite:instagram" in the routing section:

  "routing": {
	"domainStrategy":"IPOnDemand",
	"rules": [
                ...
		{
			"inboundTag": ["in_tproxy","in_socks_lan","in_vless","in-ss"],
			"domain": [
				"geosite:instagram",
				"geosite:facebook",
                                ...
			],
			"balancerTag": "ss-bl"
		},

I have changed the ProbeURL to a different address that is not in the routing lists through balancer outbounds and the problem on the observed nodes is not reproduced.
It is not clear how "observer" requests through the routing and balancer is causing the problem on "observed" nodes.

yuhan6665 · 2024-05-06T14:53:41Z

@sakontwist interesting, sounds like the root cause is you created an infinite loop.
Ideally we should fix it.

Fangliding · 2024-05-06T15:02:19Z

Even if it leads to a loop, it should still occur on the client side. Therefore, this observers requests is different from normal requests, leading to memory leaks on the server side (such as mux sessions that may not close properly)

sakontwist · 2024-05-23T08:39:13Z

I've done some experiments and the conclusions are as follows:

using burstObserver does not fix memory problems
the problem is probably caused by the mux.cool module.

I disabled mux.cool on XRAY - XRAY trunks for testing (it's not very convenient, because RKN detects servers also by the number of sessions) and the problem doesn't appear until clients connect. I know that mostly clients use nekobox and v2rayng.

I've been testing v2rayNG and noticed in the server logs that even when using vless (rprx), mux.cool calls come in if a device makes an udp request (e.g. dns). I managed to forbid mux only after enabling mux in the client by setting “-1” (I apologize for writing here and not to v2ray developers - it's for example).

So far I've recommended to clients not to use mux if possible, but apparently someone is using it because the problem comes back periodically.

I'll try to pick a window so there is no client activity and confirm there is no problem on a clean xray to xray connection.

sakontwist · 2024-05-24T10:25:40Z

Yesterday I ran a simple scheme with two Xray 1.8.11 45ab4cb (go1.22.2 linux/amd64), with no extraneous clients. After enabling mux, the server started eating memory.

On another similar trunk, the result was the same:

server

{
  "inbounds": [
    {
      "tag":"in-ss",
      "listen": "xx.xx.xx.xx","port": 10051,
      "protocol": "shadowsocks",
      "settings": {
        "network":"tcp",
        "method": "2022-blake3-chacha20-poly1305",
        "password":"xxxxxxxxxxxxxxxxxxxxxx"
      },
      "sniffing": {
        "enabled": false,
        "destOverride":["http","tls","quic"],
        "routeOnly": true
      }
    }
  ],
  "routing": {
    "domainStrategy": "AsIs",
    "rules": [
    ]
  },
  "outbounds": [
    {
      "tag": "direct",
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "AsIs"
      }
    }
  ]
}

client

{
  "inbounds": [
    {
		"tag": "in-socks-vps",
		"listen": "192.168.88.2","port": 1094,
		"protocol": "socks",
		"settings": {
		  "udp": false
		}
	}
  ],
  "routing": {
	"domainStrategy":"IPOnDemand",
	"rules": [
		{
			"inboundTag": ["in-socks-vps"],
			"outboundTag": "ss-vps4"
		}
	]  
  },
  "outbounds": [ 
    {
      "protocol": "shadowsocks",
      "tag": "ss-vps4",
      "settings": {
        "servers": [
            {
                "address": "xx.xx.xx.xx","port": 10051,
                "method": "2022-blake3-chacha20-poly1305",
                "password": "xxxxxxxxxxxxxxxxxxxxxx",
                "uot": true,"UoTVersion": 2
              }
        ]
      },
      "mux": {
          "enabled": true,
          "concurrency": 8,
          "xudpConcurrency": 16,
          "xudpProxyUDP443": "allow"
      }
    }
  ]
}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory leak on "observed" node #3338

Memory leak on "observed" node #3338

sakontwist commented May 5, 2024 •

edited

Loading

Fangliding commented May 5, 2024 •

edited

Loading

sakontwist commented May 5, 2024 •

edited

Loading

Fangliding commented May 6, 2024

yuhan6665 commented May 6, 2024

sakontwist commented May 6, 2024

sakontwist commented May 6, 2024

yuhan6665 commented May 6, 2024

Fangliding commented May 6, 2024

sakontwist commented May 23, 2024 •

edited

Loading

sakontwist commented May 24, 2024 •

edited

Loading

Memory leak on "observed" node #3338

Memory leak on "observed" node #3338

Comments

sakontwist commented May 5, 2024 • edited Loading

Fangliding commented May 5, 2024 • edited Loading

sakontwist commented May 5, 2024 • edited Loading

Fangliding commented May 6, 2024

yuhan6665 commented May 6, 2024

sakontwist commented May 6, 2024

sakontwist commented May 6, 2024

yuhan6665 commented May 6, 2024

Fangliding commented May 6, 2024

sakontwist commented May 23, 2024 • edited Loading

sakontwist commented May 24, 2024 • edited Loading

sakontwist commented May 5, 2024 •

edited

Loading

Fangliding commented May 5, 2024 •

edited

Loading

sakontwist commented May 5, 2024 •

edited

Loading

sakontwist commented May 23, 2024 •

edited

Loading

sakontwist commented May 24, 2024 •

edited

Loading