KVM and Back Ups
-
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
However, I'd like to explore the possibility of using KVM in a production scenario. Backups are obviously key here. Automated, live, nightly snapshots (entire VM) is what I am looking for.
What would be the best way to achieve this? Assuming I am running a single KVM server with local storage.You come back to the fundamental question of... why are you approaching backups this way?
Backups have never been something that we think of at the hypervisor level for decades, all of a sudden its the hot new "everyone has to do it" way. But it has all kinds of limitations and should only be a consideration for certain workloads, and only one of the options. Backups are always by workload, never by platform. It's fundamentally a flawed way to think of backups because it hides the scope and complexity of getting reliable backups and causes all kinds of problems.
Under no circumstances can anyone answer how you should take backups based on knowing your platform - no matter what hypervisor or OS you use, your backups have to be a decision based on your workloads.
So in this case, I’d have a PBX, a Wordpress site and eventually some windows server workloads. All of them are individually backed up via scripts at the OS level.
What I’m after is an “instant” type restore. For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
-
@fuznutz04 said in KVM and Back Ups:
So in this case, I’d have a PBX, a Wordpress site and eventually some windows server workloads. All of them are individually backed up via scripts at the OS level.
That's all that you want. Just the OS level backups.
-
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
-
For most workloads, like the ones that you have, you can just shut down the VM, and take a totally safe backup. Then you only need to take application level backups and do a restore from them for the latest data. Even ProxMox for their "safe" backups powers down your VMs first, that's how they handle it.
-
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
-
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
If you want the Vultr style (or ProxMox risky style), you can do that right from the storage layer. So first determine the storage that you are going to use. ZFS, BtrFS, XFS, LVM, etc. Then you use the native tools (if you want) to snap it. Everything except the scheduling is just built in.
-
To answer this, what I did for my lab is setup UrBackup and just installed the agent into each of my VM's.
It works and is simple enough to restore from.
-
It's for this kind of stuff that ZFS and BtrFS has been made popular, because they handle this very well.
-
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
If you want the Vultr style (or ProxMox risky style), you can do that right from the storage layer. So first determine the storage that you are going to use. ZFS, BtrFS, XFS, LVM, etc. Then you use the native tools (if you want) to snap it. Everything except the scheduling is just built in.
What is the latest recommendation for storage now? LVM?
-
@DustinB3403 said in KVM and Back Ups:
To answer this, what I did for my lab is setup UrBackup and just installed the agent into each of my VM's.
It works and is simple enough to restore from.
Right, that's often the better option. It's safer than hypervisor level backups, and if set up well, is just as fast and more flexible.
-
@DustinB3403 said in KVM and Back Ups:
To answer this, what I did for my lab is setup UrBackup and just installed the agent into each of my VM's.
It works and is simple enough to restore from.
I have to check that out tonight in my lab
-
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
If you want the Vultr style (or ProxMox risky style), you can do that right from the storage layer. So first determine the storage that you are going to use. ZFS, BtrFS, XFS, LVM, etc. Then you use the native tools (if you want) to snap it. Everything except the scheduling is just built in.
What is the latest recommendation for storage now? LVM?
LVM, ZFS, BtrFS are all fine. I've not used this but here is a script to do LVM backups...
-
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
If you want the Vultr style (or ProxMox risky style), you can do that right from the storage layer. So first determine the storage that you are going to use. ZFS, BtrFS, XFS, LVM, etc. Then you use the native tools (if you want) to snap it. Everything except the scheduling is just built in.
What is the latest recommendation for storage now? LVM?
LVM, ZFS, BtrFS are all fine. I've not used this but here is a script to do LVM backups...
Awesome. I’ve used ZFS in the past, but it was on a freeNas box I was testing. Seemed pretty good at the time (zfs)
-
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
QEMU has both internal and external snapshots. Internal are inside of the qcow2 file, external are redirect on write snapshots. The external are the more robust since they don't do full COW like the internal ones.
-
@stacksofplates said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
QEMU has both internal and external snapshots. Internal are inside of the qcow2 file, external are redirect on write snapshots. The external are the more robust since they don't do full COW like the internal ones.
-
@scottalanmiller said in KVM and Back Ups:
@stacksofplates said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
QEMU has both internal and external snapshots. Internal are inside of the qcow2 file, external are redirect on write snapshots. The external are the more robust since they don't do full COW like the internal ones.
I've got one in here somewhere also. You just put in the location you want the backup sent to and it copies the snapshot there.
-
@stacksofplates
Lots of testing to do later tonight. -
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
So in this case, I’d have a PBX, a Wordpress site and eventually some windows server workloads. All of them are individually backed up via scripts at the OS level.
That's all that you want. Just the OS level backups.
No, that is all you want. The rest of us want VM level backups.
-
@JaredBusch said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
So in this case, I’d have a PBX, a Wordpress site and eventually some windows server workloads. All of them are individually backed up via scripts at the OS level.
That's all that you want. Just the OS level backups.
No, that is all you want. The rest of us want VM level backups.
I want what is reliable and good for restoring the environment, not what's pushed by marketing companies.
I'm focused on the goal: working backups. Or even better: Disaster protection.
I'm not being distracted by the means. When anyone in IT talks about wanting a hypervisor level backup, that's a "means" distraction caused by forgetting to stay goal focused.
-
I think it's wise to consider what kind of failure you are trying to protect yourself from and how you are going to recover.
If I want to restore a VM that doesn't work as it should or the host crashed, I'd want a VM backup, taken in a known good state, because that is the fastest way to get something working again, perhaps on another host. To me this is an infrastructure backup. Our infrastructure is broken and we need to recover from that. Which also means we need a backup of the VM host of course, and everything else that could fail, including documentation and procedures how to get it back up.
If a user deleted some files and want them back, I'd want a file level backup. That to me is a backup of business data, not infrastructure, and I'd want that backed up on a schedule that fits the data.
And if I have to restore a database or a table within the database, I'd want a consistent database backup, not the database files and absolutely not the VM level backup. This to me is a different kind of business data.
