Remote management of VMs hosted in colocation
-
I'm taking a look at how I'm currently accessing and managing VMs on my colo lab server. What I currently have works, but I see some flaws I need to address. I figure this could turn into a discussion of methods of remote management for colo infrastructure. For me, my goal is to treat my lab as I would any business asset in colo as far as the practices I follow when doing
$tasks.This is currently how I access my lab -- using ZeroTier. I have ZeroTier installed on my laptop at home and my host. I run Virt Manager on my laptop at home to get console access to the host and the subsequent VMs.
The major problem I see with this:
If my laptop becomes compromised (think ransomware), then my host is pwn'd.Some options I've considered for handling this better
Create a VM for management and connect to it with a tool like ScreenConnect
I've experimented with this with some success. I've tried a Fedora VM with the ScreenConnect agent installed. Opening a terminal session and SSHing into the host was fine. I found that connecting to the host with VirtManager to be a pain (probably my GUI environment is to blame -- LXQT), and using a web browser (Falkon) was a slow, painful experience.
What I like about this approach is that it's using a jump box (the management VM). It's also using an on-demand connection to it, which I can access from anywhere with an Internet connection. It also reduces the risk of my pwn'd laptop pwning my server.
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
-
I thought virt-manager only used ssh to connect to the host. What about using key-based auth to the host and disable password auth? Seems it would be faster than going through ZT. I guess the laptop compromise still poses an issue in that scenario.
-
@brandon220 said in Remote management of VMs hosted in colocation:
I thought virt-manager only used ssh to connect to the host. What about using key-based auth to the host and disable password auth? Seems it would be faster than going through ZT. I guess the laptop compromise still poses an issue in that scenario.
Virt-manager does use SSH. I connect virt-manager to the host via SSH to its ZeroTier IP address.
-
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
SSH is an always encrypted connection. It doesn't even pass keys until an encrypted tunnel is setup, so using over VPN is.... 2 levels of encrypted communication.
-
@eddiejennings said in Remote management of VMs hosted in colocation:
The major problem I see with this:
If my laptop becomes compromised (think ransomware), then my host is pwn'd.Some other potential options.
You don't have to keep your laptop ZT permanently connected. Just enable it whenever you need to use it then disable it soon after.
You can also have VM (on you laptop) you use just for management.
-
@brandon220 said in Remote management of VMs hosted in colocation:
I thought virt-manager only used ssh to connect to the host. What about using key-based auth to the host and disable password auth? Seems it would be faster than going through ZT. I guess the laptop compromise still poses an issue in that scenario.
If you realize your laptop has been Pwned or stolen then kick it out of the ZT network.
-
@brandon220 said in Remote management of VMs hosted in colocation:
I thought virt-manager only used ssh to connect to the host. What about using key-based auth to the host and disable password auth? Seems it would be faster than going through ZT. I guess the laptop compromise still poses an issue in that scenario.
Keys or not, I will not open SSH to the public internet if I have a simple way around it.
That is what I like about using something like ZeroTier for. His KVM host has zero presence on the Public internet. It only uses an outbound SSL session to connect to the ZT network.
-
Perhaps if I wanted an extra thick tinfoil hat, I could have my managementVM connected to ZeroTier, setup
$remoteDesktopMechanismon the VM, and connect to the managementVM from my laptop via ZeroTier to manage the kvm host and all of the other VMs. -
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
-
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
Or do none of that and just use ZT. Way fewer things to fail.
-
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
I could see that in a situation where your office has a static IP. For me, I wouldn't be able to lock down the allowed IP, since there's a chance it'll change.
-
ZeroTier makes it super easy and safe. If you want, you could make a ZeroTier Bridge VM. I would still setup Fail2Ban and use key-based authentication.
-
@eddiejennings said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
I could see that in a situation where your office has a static IP. For me, I wouldn't be able to lock down the allowed IP, since there's a chance it'll change.
Not really an issue, use Salt to maintain it so not a problem even for dynamic.
-
@black3dynamite said in Remote management of VMs hosted in colocation:
ZeroTier makes it super easy and safe. If you want, you could make a ZeroTier Bridge VM. I would still setup Fail2Ban and use key-based authentication.
I'd argue that it doesn't really make it safer. ZT is just key based access, like SSH is. If you are concerned that SSH with keys isn't enough, ZT isn't either. ZT exposes even more should it be compromised, as well. In both cases, being compromised is pretty bad, but one is a lot more limited and able to be limited. If you feel someone can breach your keys with SSH, why not ZT just the same?
-
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
I could see that in a situation where your office has a static IP. For me, I wouldn't be able to lock down the allowed IP, since there's a chance it'll change.
Not really an issue, use Salt to maintain it so not a problem even for dynamic.
That would be another thread.
Spin up a Salt VM for maintaining stuff on my host and accessing it from afar. -
@scottalanmiller said in Remote management of VMs hosted in colocation:
@black3dynamite said in Remote management of VMs hosted in colocation:
ZeroTier makes it super easy and safe. If you want, you could make a ZeroTier Bridge VM. I would still setup Fail2Ban and use key-based authentication.
I'd argue that it doesn't really make it safer. ZT is just key based access, like SSH is. If you are concerned that SSH with keys isn't enough, ZT isn't either. ZT exposes even more should it be compromised, as well. In both cases, being compromised is pretty bad, but one is a lot more limited and able to be limited. If you feel someone can breach your keys with SSH, why not ZT just the same?
Not true.
- I never said I don't trust SSH key based auth. I said
@jaredbusch said in Remote management of VMs hosted in colocation:
Keys or not, I will not open SSH to the public internet if I have a simple way around it.
Because, while I may trust SSH, we live in reality where unknown vulnerabilities do exist. By not having it open, at all, to a public facing source, you drastically mitigate the attack surface.
ZT is access without a public inbound allowance. Someone would have to gain access to your ZT network before being able to then try and get your keys for the SSH session.
@scottalanmiller said in Remote management of VMs hosted in colocation:
ZT exposes even more should it be compromised, as well
This is a strawman. If you want to take up this argument, it would be a separate, yet related, discussion.
-
@jaredbusch said in Remote management of VMs hosted in colocation:
ZT is access without a public inbound allowance. Someone would have to gain access to your ZT network before being able to then try and get your keys for the SSH session.
that's my point. How is the access to your ZT network more secure than the keyed access to an SSH port? Both have the risk of vulnerabilities (but nothing has the eyes on it and reviews of OpenSSH, so if that's a concern that makes SSH the clean place to start), and both have access from the outside. Both rely on keys and both have known open ports.
-
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
ZT is access without a public inbound allowance. Someone would have to gain access to your ZT network before being able to then try and get your keys for the SSH session.
that's my point. How is the access to your ZT network more secure than the keyed access to an SSH port? Both have the risk of vulnerabilities (but nothing has the eyes on it and reviews of OpenSSH, so if that's a concern that makes SSH the clean place to start), and both have access from the outside. Both rely on keys and both have known open ports.
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
-
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
If you're using an infested machine - when you use the apps to make that remote session to the colo, be it SSH or SC, the malware can likely grab the needed information and provide it to the hackers.
I'm guessing that the Salt idea might give the greatest protection here when combined with the IP lock option that Scott mentioned - but then I ask - how do you securely manage the Saltmaster, where ever it lives?
-
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
