Hi folks,

I think my ASA is on the way out. Any ideas on what to check to make sure? I've contacted Cisco already, but that response could be slow, so just asking for troubleshooting tips...

So, no network changes for months, and the device has been in use in the rack since last November. Overnight around 3am, all office locations globally lose access to London.

I get a call at about 4am, head in at 5am figuring the office power loss or something. But nope, all power on fine.

I check the ASA and the SSD light on the far left is not green. All other lights green. I also cannot contact the ASA locally ping, use ADSM, or connect by putty. So, I hard reboot by pulling the power.

Within 5 minutes, all services back online. But, the SSD light is flickering green. Not solid.

Then, around 8am (few hours maybe), total loss again. SSD light off. Only option to hard boot.

I have done this several times now and failure looks to happen between 1 - 3 hours of last reboot. No errors in logs. Just instant failure.

I figure based on the light that the SSD in the back needs to be replaced and have contacted Cisco. Any other ideas on what to check in case im wrong?

I have a backup of the configuration so thats fine. But, no reliable device...

Best,
Jim

@PhlipElder said in Why IT certifications are worth more than you think...:

Parenthood 500

Ha!

@PhlipElder said in Why IT certifications are worth more than you think...:

@Jimmy9008 said in Why IT certifications are worth more than you think...:

What certification would you suggest for IT management in terms of people and projects?

For project work, PMP. Project Management Professional. From a reputable source of course.

A thorough understanding of project work by experience would also be an asset.

Experience with the products brings a type of knowledge, understanding, and wisdom to bring about a successful conclusion to a project whether $25K or $25M.

Boots on the ground experience.

Oh, and a sense of tact. I don't have that so can be difficult in larger projects. Plus, I'm the humblest guy I know.

Ok, so how about on purely people management side? IT folks can usually be technical, but not good managers. What course or program can core management competencies be gained? Other than of course time and experience?

What certification would you suggest for IT management in terms of people and projects?

In the UK i've waited over one month for 2 x laptops recently. One with UK keyboard, one Arabic. After all that time... both were Arabic! So yes, they are having some issues!

Interesting read. I am from the UK, so it totally doesn't actually matter - but I would agree with Scott here.

If the police say: "We think there is child porn on your laptop, please give us the password."

Then the person being interviewed says: "You know whats on there."

... regardless of password, or whatever else, you now know what is on there. They hung themselves.

If they said "No.", sure - no issue. But they didn't.

Guilty!

Spanish or English would be fine. But, needs to be local to Barcelona. The only reason we are looking is for the physical side we cannot do from London or Calgary.

Hi folks,

Does anybody know of reputable IT Support services in Barcelona? We are mostly looking for a company that can travel to our Barcelona office for physical type work and low level user support. Perhaps a standard fee per call out or something like that...

Best,
Jim

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

Hi folks,

I am having VMQ issues on my 3 x Windows Server 2019 host. I am using a team of Intel X550 NIC (3 x 10 GbE interfaces to 3 x Stacked switch). The Team has been configured using Switch Independent mode.

Now, VMs are on the LAN perfectly and communicate. Rock solid.

Now to the issue. When I use Veeam with Application Aware turned on, the hosts get stuck creating checkpoints for VMs at 9%. The only option is to kill the host, restart and turn the VMs back on. It happens every time. (Strangely, checkpoints run fine native).

Veeam have looked at the logs and have said its an issue they have seen before, and to turn off VMQ initially on NICs and VMs as that often solves the issue. No worries. I do that, and the backups then run fine - perfect!

However, VMs over time then drop off of the network. I can connect to them in Hyper-V, but nothing I do will bring them back on the network. Initially they are on the network, just at some point in time many drop off, whilst others stay on.

The only resolution is to turn VMQ back on and reboot. I cant really keep testing this either as it causes much downtime! Not good. Of course, when VMQ is back on... Application Aware backups then fail and kill the host like I said at the start

Any idea why some VMs drop off of the network with VMQ disabled on the VM and NIC? Host, fully patched. NIC, latest firmware. I thought you didnt have to use VMQ...

Best,
Jim

Is the the same VMs that drop off the network every time?

What OS are the VMs that get dropped off the network?

What kind of switches are you running?

1. Uncertain. I've noticed a few are the same, but I've had no time to really look in to it. Too many users. I'm trying to get ideas that could be the issue to do proper testing with downtime around Dec 20th when a lot are on holiday.

2. I think 2012 R2. Possibly 2016 too. Can't recall any 2019.

3. Dell N4064 Stack.

Try the ArpRetryCount trick. You'll have to reboot each VM or host after you make the change.

Ok, I can try that. So:

1. disable VMQ physical
2. disable VMQ on all VM NIC
3. change registry
4. reboot

Sound about right?

3. change the registry on any machine that is down.

But yeah, that sounds about right.

Do I need to change the reg on the host too?

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

Hi folks,

I am having VMQ issues on my 3 x Windows Server 2019 host. I am using a team of Intel X550 NIC (3 x 10 GbE interfaces to 3 x Stacked switch). The Team has been configured using Switch Independent mode.

Now, VMs are on the LAN perfectly and communicate. Rock solid.

Now to the issue. When I use Veeam with Application Aware turned on, the hosts get stuck creating checkpoints for VMs at 9%. The only option is to kill the host, restart and turn the VMs back on. It happens every time. (Strangely, checkpoints run fine native).

Veeam have looked at the logs and have said its an issue they have seen before, and to turn off VMQ initially on NICs and VMs as that often solves the issue. No worries. I do that, and the backups then run fine - perfect!

However, VMs over time then drop off of the network. I can connect to them in Hyper-V, but nothing I do will bring them back on the network. Initially they are on the network, just at some point in time many drop off, whilst others stay on.

The only resolution is to turn VMQ back on and reboot. I cant really keep testing this either as it causes much downtime! Not good. Of course, when VMQ is back on... Application Aware backups then fail and kill the host like I said at the start

Any idea why some VMs drop off of the network with VMQ disabled on the VM and NIC? Host, fully patched. NIC, latest firmware. I thought you didnt have to use VMQ...

Best,
Jim

Is the the same VMs that drop off the network every time?

What OS are the VMs that get dropped off the network?

What kind of switches are you running?

For Windows systems (Even sometimes a Physical Server, I usually have to do this in regedit:

I haven't run into anything like that with our Linux systems. You may need to do it at the Host level on HyperV as well.

That's something I can try in December. So do that to the VMs and hosts?

Yeah. Or you can try it the next time a VM goes down. That's what we did. Make the change, reboot the VM, and it's usually happy again.

That's the thing. The VM are rock solid with VMQ and Veeam, just only when no App Aware backups are ran.

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

Hi folks,

I am having VMQ issues on my 3 x Windows Server 2019 host. I am using a team of Intel X550 NIC (3 x 10 GbE interfaces to 3 x Stacked switch). The Team has been configured using Switch Independent mode.

Now, VMs are on the LAN perfectly and communicate. Rock solid.

Now to the issue. When I use Veeam with Application Aware turned on, the hosts get stuck creating checkpoints for VMs at 9%. The only option is to kill the host, restart and turn the VMs back on. It happens every time. (Strangely, checkpoints run fine native).

Veeam have looked at the logs and have said its an issue they have seen before, and to turn off VMQ initially on NICs and VMs as that often solves the issue. No worries. I do that, and the backups then run fine - perfect!

However, VMs over time then drop off of the network. I can connect to them in Hyper-V, but nothing I do will bring them back on the network. Initially they are on the network, just at some point in time many drop off, whilst others stay on.

The only resolution is to turn VMQ back on and reboot. I cant really keep testing this either as it causes much downtime! Not good. Of course, when VMQ is back on... Application Aware backups then fail and kill the host like I said at the start

Any idea why some VMs drop off of the network with VMQ disabled on the VM and NIC? Host, fully patched. NIC, latest firmware. I thought you didnt have to use VMQ...

Best,
Jim

Is the the same VMs that drop off the network every time?

What OS are the VMs that get dropped off the network?

What kind of switches are you running?

1. Uncertain. I've noticed a few are the same, but I've had no time to really look in to it. Too many users. I'm trying to get ideas that could be the issue to do proper testing with downtime around Dec 20th when a lot are on holiday.

2. I think 2012 R2. Possibly 2016 too. Can't recall any 2019.

3. Dell N4064 Stack.

Try the ArpRetryCount trick. You'll have to reboot each VM or host after you make the change.

Ok, I can try that. So:

1. disable VMQ physical
2. disable VMQ on all VM NIC
3. change registry
4. reboot

Sound about right?

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

Hi folks,

I am having VMQ issues on my 3 x Windows Server 2019 host. I am using a team of Intel X550 NIC (3 x 10 GbE interfaces to 3 x Stacked switch). The Team has been configured using Switch Independent mode.

Now, VMs are on the LAN perfectly and communicate. Rock solid.

Now to the issue. When I use Veeam with Application Aware turned on, the hosts get stuck creating checkpoints for VMs at 9%. The only option is to kill the host, restart and turn the VMs back on. It happens every time. (Strangely, checkpoints run fine native).

Veeam have looked at the logs and have said its an issue they have seen before, and to turn off VMQ initially on NICs and VMs as that often solves the issue. No worries. I do that, and the backups then run fine - perfect!

However, VMs over time then drop off of the network. I can connect to them in Hyper-V, but nothing I do will bring them back on the network. Initially they are on the network, just at some point in time many drop off, whilst others stay on.

The only resolution is to turn VMQ back on and reboot. I cant really keep testing this either as it causes much downtime! Not good. Of course, when VMQ is back on... Application Aware backups then fail and kill the host like I said at the start

Any idea why some VMs drop off of the network with VMQ disabled on the VM and NIC? Host, fully patched. NIC, latest firmware. I thought you didnt have to use VMQ...

Best,
Jim

Is the the same VMs that drop off the network every time?

What OS are the VMs that get dropped off the network?

What kind of switches are you running?

For Windows systems (Even sometimes a Physical Server, I usually have to do this in regedit:

I haven't run into anything like that with our Linux systems. You may need to do it at the Host level on HyperV as well.

That's something I can try in December. So do that to the VMs and hosts?

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

Hi folks,

I am having VMQ issues on my 3 x Windows Server 2019 host. I am using a team of Intel X550 NIC (3 x 10 GbE interfaces to 3 x Stacked switch). The Team has been configured using Switch Independent mode.

Now, VMs are on the LAN perfectly and communicate. Rock solid.

Now to the issue. When I use Veeam with Application Aware turned on, the hosts get stuck creating checkpoints for VMs at 9%. The only option is to kill the host, restart and turn the VMs back on. It happens every time. (Strangely, checkpoints run fine native).

Veeam have looked at the logs and have said its an issue they have seen before, and to turn off VMQ initially on NICs and VMs as that often solves the issue. No worries. I do that, and the backups then run fine - perfect!

However, VMs over time then drop off of the network. I can connect to them in Hyper-V, but nothing I do will bring them back on the network. Initially they are on the network, just at some point in time many drop off, whilst others stay on.

The only resolution is to turn VMQ back on and reboot. I cant really keep testing this either as it causes much downtime! Not good. Of course, when VMQ is back on... Application Aware backups then fail and kill the host like I said at the start

Any idea why some VMs drop off of the network with VMQ disabled on the VM and NIC? Host, fully patched. NIC, latest firmware. I thought you didnt have to use VMQ...

Best,
Jim

Is the the same VMs that drop off the network every time?

What OS are the VMs that get dropped off the network?

What kind of switches are you running?

1. Uncertain. I've noticed a few are the same, but I've had no time to really look in to it. Too many users. I'm trying to get ideas that could be the issue to do proper testing with downtime around Dec 20th when a lot are on holiday.

2. I think 2012 R2. Possibly 2016 too. Can't recall any 2019.

3. Dell N4064 Stack.

@Dashrender said in Veeam Active Full or Synthetic Full?:

@DustinB3403 said in Veeam Active Full or Synthetic Full?:

& Replication accesses the previous full backup file and a chain of subsequent incremental backup files on the backup

Reverse it on him - ask him how active full is safer?

If you don't trust the data in the incrementals and the previous full backup, then why are you using the product?

Putting load on production shortens its lifespan - maybe not noticeably, but it still does because of work being done.

I like this idea. That could work well - thank you.

Are they safer? If not, why do active full even exist?

@DustinB3403 said in Veeam Active Full or Synthetic Full?:

@Jimmy9008 said in Veeam Active Full or Synthetic Full?:

@DustinB3403 said in Veeam Active Full or Synthetic Full?:

So to your coworkers point, I would say, no there is no need for a separate full backup weekly.

He is daying npt to use synthetics on a saturday. Just do the incremental and an active full

I'm going to take some guesses fixing this.

He is saying not to use synthetics on Saturday, just use incrementals and a separate active full

Is that correct?

Yes.

@DustinB3403 said in Veeam Active Full or Synthetic Full?:

So to your coworkers point, I would say, no there is no need for a separate full backup weekly.

He is daying npt to use synthetics on a saturday. Just do the incremental and an active full

@DustinB3403 Yeah, I read that already but it dodnt help clarify on those questions I had.

Hi folks,

I am having VMQ issues on my 3 x Windows Server 2019 host. I am using a team of Intel X550 NIC (3 x 10 GbE interfaces to 3 x Stacked switch). The Team has been configured using Switch Independent mode.

Now, VMs are on the LAN perfectly and communicate. Rock solid.

Now to the issue. When I use Veeam with Application Aware turned on, the hosts get stuck creating checkpoints for VMs at 9%. The only option is to kill the host, restart and turn the VMs back on. It happens every time. (Strangely, checkpoints run fine native).

Veeam have looked at the logs and have said its an issue they have seen before, and to turn off VMQ initially on NICs and VMs as that often solves the issue. No worries. I do that, and the backups then run fine - perfect!

However, VMs over time then drop off of the network. I can connect to them in Hyper-V, but nothing I do will bring them back on the network. Initially they are on the network, just at some point in time many drop off, whilst others stay on.

The only resolution is to turn VMQ back on and reboot. I cant really keep testing this either as it causes much downtime! Not good. Of course, when VMQ is back on... Application Aware backups then fail and kill the host like I said at the start

Any idea why some VMs drop off of the network with VMQ disabled on the VM and NIC? Host, fully patched. NIC, latest firmware. I thought you didnt have to use VMQ...

Best,
Jim

Hi folks,

I am trying to think through some backup settings and would like to know what you think.

So, I use Veeam B&R 9.5 and have a few scale out backups to Azure. When a chain becomes inactive, that is when the backup chain is moved to Azure.

A backup chain becomes inactive when either an Active Full is taken, or when a Synthetic full is taken.

Sunday to Friday are incremental, and Saturday is a full. But, should I choose Active Full, or Synthetic full?

My colleague wants to use Active Full as 'its safer and our systems can handle it', but is that the right choice? I am not sure Active Full is actually safer than synthetic full, and cant really find anything anywhere to suggest so. Any advice?

Either way, our Veeam server, production storage and network can handle the weekly active full, but does it make sense to put that extra load on the host weekly for no actual benefit? Unless Active Fulls are, in fact, safer?

Im thinking, sure... I can set it. But, I have to get my hosts to do more work once a week. When, the synthetic full will give me the same and leave most of the load directly on the Veeam server... so why do it?

What do you think?

Best,
Jim