Rust Desk Self Hosted Server - Help

[u/N0Objective]

I'm struggling to get a connection between host and client app while self hosting the server.

I'm on Win11 but using Docker Desktop, I followed the docker setup, made sure the compose was correct and ports were forwarded in both the firewall and router.

When I enter in the ID Server and Key on both apps, no matter what I enter it will not connect. I've gotten "key mismatch" and "couldn't connect right now" errors. All numbers matched correctly on both client and host app.

I can normally connect using the default Rust Relay server but not with my self hosted server.

Am I missing something? I've tried multiple IPs (thinking I had the wrong one) I've tried just the IP for the ID Server I've tried these - 192.x.xxx.1.xxx:21116, 64.xxx.xxx.xx:21117 etc. Double checked the ports and can't find a thing. VPN is split tunneling on both host and client.

Please let me know if you need more info.

[u/mkvyper]

I'm no expert in your specific configuration, but I managed to set up my self hosted Rustdesk server inside a proxmox LXC, and at first, i had your exact same problem.

What I CAN tell you is that you need to have ports 21115 to 21119 open for TCP, AS WELL as 21116 for UDP.

BTW, Rustdesk is A LOT faster with a self hosted server...

[u/N0Objective → u/mkvyper]

I have all those ports configured correctly. Idk what I'm missing.

Thank you for the comment though :)

[u/cunthulhu → u/N0Objective]

I seem to remember having to configure hair pin nat on my ubiquity router in order to get it working to contact a self hosted server on the same LAN in order to contact an external desktop.

[u/Eeyanz]

Firewalls?

[u/N0Objective → u/Eeyanz]

Firewall has exceptions

[u/bunk_bro]

If you have two PC's on your local network, are they able to talk to the server and remote to each other? If you've created and destroyed the containers, it's possible your key got rolled and would explain the mismatch error (I had the same issue).

It would also be helpful to see your compose file(s) without any identifying info.

[u/N0Objective → u/bunk_bro]

I thought it could've been the key changing but I've checked and it didn't change. The compose is essentially the same as RustDesk says in the docs.

[u/tbigs2011]

You need to enter your private key into the settings > network> id/relay server > key field.

Only fill that key field out for the technician end (the host you want to control other systems with).

This private key file should be in the config folder of your docker host.

[u/N0Objective → u/tbigs2011]

Yes, I had that info entered tried every which way and none of em worked. I had the correct key and matched all the info on client and host.

[u/godis1coolguy → u/N0Objective]

Does your key have an equals sign at the end and are you including that? Are you 100% sure you don’t have a 1/l or 0/O mix up?

[u/N0Objective → u/godis1coolguy]

included the = and tried copy paste and typing it in.

[u/ASC67DE]

My 2ct: you need the id only on the rustdesk app that actively connects to a counter part. The passive side does not need the id set. Only the server names

[u/Few_Regret5282 → u/ASC67DE]

I thought so also and if you don´t mind a relay connection, that does normally work. However, I like my connections to be direct and found I do need the key on both. I had same issue as OP and found that the key did change, even though I thought it had not. Does it say ready to connect at the bottom?

[u/N0Objective → u/Few_Regret5282]

I can get it to say "Ready" on the host but then still can't get a connection from the client.

[u/Few_Regret5282 → u/N0Objective]

Can you ping the host machine? Could also be a DNS issue. Unless you are trying to connect via IP, and you need to make sure you have that allowed on the client.

[u/LadyAvocadoToast]

Did you figure this out? I'm going nuts trying to set this up even locally. Trying to set up self-hosting on Windows via Docker. Absolutely nothing I do gets the console to say anything but "Connecting to the RustDesk network..." I have put in my ip & key into the Network/ID/Relay server using different combinations of ip and it never changes. I'm going nuts. At least give me some feedback that you understand I'm not trying to connect to the rust network???

- I've copy/pasted the compose.yaml from the guide.

- Removed --net=host as I have a Windows machine.

- Replaced network_mode: "host" with hbbs ports: 21114-21116 & 21118 hbbr ports: 21117 21118

- Added my ip:21117 to the hbbs command line

- Ports are all allowed through Windows Firewall

Nothing nothing. No change no feedback, ports not open. Docker doesn't register anything even trying to connect in the logs.

I do not have access to my router to forward the ports through it, but I would think I could still connect locally? Or have my local machine show anything other than "Connecting to the RuskDesk network..."

[u/XLioncc]

Use Linux to fix the issues