Solved Promox and VM replication
-
So I have played with Proxmox for a couple weeks.
Well the project needs to go live this weekend.
I either need to figure out how to replicate between two servers or switch to a different solution.
The only supported method requires that you not have hardware raid and use ZFS.
That defeats the purpose of blindswap, and shipping drive to the datacenter tech.
Not to mention that ZFS is not even the default setup when the install wizard runs.
-
I'll try and make a cleaner guide later..
- First, spin up your ProxmoxVE system.
- During the install wizard, I left the boot drive as LVM but did change it to XFS instead of ext4. You do not setup secondary drives during the install wizard.
- Once up, you need to create the ZFS store on each system, named the same, prior to joining the cluster.
- The replication process wants the ZFS pool to be named the same on both systems and you cannot name it the same (at least in the GUI) if it already exists anywhere in the cluster.
- The replication process wants the ZFS pool to be named the same on both systems and you cannot name it the same (at least in the GUI) if it already exists anywhere in the cluster.
- Then from the GUI, go to the disks of system 1.
- then click on ZFS and create the storage pool
- name it, single disk, compression off.
- name it, single disk, compression off.
- wait for it to show normal.
- then repeat the process on the second system.
- make sure to use the same name.
- Now create the cluster and join the second system to the cluster.
- You will not see the storage on ZFS storage on system 2 when it first loads up.
- But it does exist if you look.
- To make it available, go to storage under datacenter.
- edit the existing "data" and add pve2
- now it will show up and be available for use.
Is this perfect? No. but it is how I was able to get it setup to work.
The manual leave out quite a bit of specifics. on how to do things, but goes into detail on the technical of things.
- First, spin up your ProxmoxVE system.
-
Replicate regularly? Or one time for the migration?
-
@scottalanmiller said in Promox and VM replication:
Replicate regularly? Or one time for the migration?
Regularly.
This will be a manual failover device not a backup.
-
@scottalanmiller these servers are going in the rack on Saturday morning. The old switch will go back in my junk pile
Both will be on site in the client office initially.
But the second will be moved to colo after their IT support company (not us) figures out how to make their VPN handle the routing transparently in case of a failover.
-
@scottalanmiller so if I can break you away from your Zero trust LAN-less thread.
From what I have seen you post, I know you are using PM now, and it is using LVM.
I assume in production.Have you not done anything with replication at all? This is a pretty common use ofr virtualization platforms.
-
@JaredBusch said in Promox and VM replication:
@scottalanmiller so if I can break you away from your Zero trust LAN-less thread.
From what I have seen you post, I know you are using PM now, and it is using LVM.
I assume in production.Have you not done anything with replication at all? This is a pretty common use ofr virtualization platforms.
It is, but we do not. Most of our clients are small and don't want a second server. They just take backups and restore. The cost of a second server for outages that are generally less than twice a decade and typically quite minor just doesn't justify the cost. So we rarely see it.
When you need it, I think you have to stick to ZFS. Not that there is no possibility of doing it another way, but it's what ProxMox expects at this point.
-
There should be no issue using ZFS on top of hardware RAID so that you get blind swap AND replication.
-
@scottalanmiller said in Promox and VM replication:
There should be no issue using ZFS on top of hardware RAID so that you get blind swap AND replication.
Yeah, one test I just set it up as ZFS, single disk.
It should have no clue what is underneath.
No matter what the fanboys say about ZFS
-
@JaredBusch exactly
-
@scottalanmiller said in Promox and VM replication:
The cost of a second server for outages that are generally less than twice a decade and typically quite minor just doesn't justify the cost.
For this client, they can't take a full 24 hours of downtime before they begin to lose data. They need a replica.
It is the far remote sites that send data into the office that drop the data after 24 hours. They were never designed to hold data forever. Machines (pumps actually) with fixed capacities, etc.
There are local printouts that can be manually keyed in later in case of a real large scale disaster at the office location. But an extra $4k for a server that will soon be regionally separated makes it worth it to them.
-
@JaredBusch said in Promox and VM replication:
For this client, they can't take a full 24 hours of downtime before they begin to lose data. They need a replica.
Most of ours can restore from a backup in under an hour to limp along on alternative hardware. It's a pain, but they can operate a day or two that way.
-
In my testing I couldn't figure out a way to have hardware raid and ZFS, at least not with one large array.
The only way I could get it to work was to create an R1, leaving unconfigured drives to create the ZFS with so I could then create the ZFS pool.
-
@DustinB3403 said in Promox and VM replication:
In my testing I couldn't figure out a way to have hardware raid and ZFS, at least not with one large array.
The only way I could get it to work was to create an R1, leaving unconfigured drives to create the ZFS with so I could then create the ZFS pool.
I need to spin one of these up... are you saying you can't install Proxmox to to a single disk and use ZFS? i.e. one physical drive?
If you can.. .then the hardware RAID should just look like a single presented drive, and use ZFS on it -
@Dashrender said in Promox and VM replication:
@DustinB3403 said in Promox and VM replication:
In my testing I couldn't figure out a way to have hardware raid and ZFS, at least not with one large array.
The only way I could get it to work was to create an R1, leaving unconfigured drives to create the ZFS with so I could then create the ZFS pool.
I need to spin one of these up... are you saying you can't install Proxmox to to a single disk and use ZFS? i.e. one physical drive?
If you can.. .then the hardware RAID should just look like a single presented drive, and use ZFS on itYou'd think, but Proxmox goes to extreme lengths to make sure you're using storage how THEY want you to. I haven't messed with it since the 5.xx branch, which has been a long time.
-
@DustinB3403 said in Promox and VM replication:
In my testing I couldn't figure out a way to have hardware raid and ZFS, at least not with one large array.
The only way I could get it to work was to create an R1, leaving unconfigured drives to create the ZFS with so I could then create the ZFS pool.
See screenshot..... It was not like hard.
-
@Dashrender said in Promox and VM replication:
I need to spin one of these up... are you saying you can't install Proxmox to to a single disk and use ZFS? i.e. one physical drive?
If you can.. .then the hardware RAID should just look like a single presented drive, and use ZFS on it@travisdh1 said in Promox and VM replication:
You'd think, but Proxmox goes to extreme lengths to make sure you're using storage how THEY want you to. I haven't messed with it since the 5.xx branch, which has been a long time.
Look above.
- I said it is possible I did it.
- I posted a damned screenshot...
-
@JaredBusch said in Promox and VM replication:
@Dashrender said in Promox and VM replication:
I need to spin one of these up... are you saying you can't install Proxmox to to a single disk and use ZFS? i.e. one physical drive?
If you can.. .then the hardware RAID should just look like a single presented drive, and use ZFS on it@travisdh1 said in Promox and VM replication:
You'd think, but Proxmox goes to extreme lengths to make sure you're using storage how THEY want you to. I haven't messed with it since the 5.xx branch, which has been a long time.
Look above.
- I said it is possible I did it.
- I posted a damned screenshot...
I had forgotten that by the time I replied, obviously. I think it's time for me to load up Proxmox in the home lab to check out.
-
Setup and tested.
-
@scottalanmiller said in Promox and VM replication:
When you need it, I think you have to stick to ZFS. Not that there is no possibility of doing it another way, but it's what ProxMox expects at this point.
Sucks, but working.
-
@JaredBusch 5 minutes later.
