Exchange 2010 Hard Drive Space Issues

JaredBusch

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

The logs are going to the same box? Why not send logs to a central service?

WTF? This is not how Exchange is designed. These are DB rollback logs. Not usage logs. They do not get shipped out for monitoring.

Dashrender

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I'm talking to Microsoft right now and they basically just told me I have two options in this situation, after reading my logs. 1. Turn on circular logging which will increase my CPU utilization significantly (this server sits at 70%+ utilization consistently) which would most likely seize it or do what I did. I considered each option and spoke to my boss. We are running out of space rapidly and he said at this point we unfortunately are going to have to take the risk. This realistically should have been fixed a year ago, but from what I understand they were running around putting out fires.

The logs are going to the same box? Why not send logs to a central service?

Does exchange logging work that way?

https://www.splunk.com/en_us/solutions/solution-areas/it-operations-management/microsoft-infrastructure-monitoring/splunk-app-for-microsoft-exchange.html

OK makes sense for tracking, etc... but what about recovery?

Recovery of the logs, or recovery of email?

recovery of email in case of a failure - it's my understanding that you restore you IS, then replay the logs to get the newer data back.

scottalanmiller

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

The logs are going to the same box? Why not send logs to a central service?

WTF? This is not how Exchange is designed. These are DB rollback logs. Not usage logs. They do not get shipped out for monitoring.

Oh, sorry, didn't realize what logs we were discussing.

scottalanmiller

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

@scottalanmiller said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I'm talking to Microsoft right now and they basically just told me I have two options in this situation, after reading my logs. 1. Turn on circular logging which will increase my CPU utilization significantly (this server sits at 70%+ utilization consistently) which would most likely seize it or do what I did. I considered each option and spoke to my boss. We are running out of space rapidly and he said at this point we unfortunately are going to have to take the risk. This realistically should have been fixed a year ago, but from what I understand they were running around putting out fires.

The logs are going to the same box? Why not send logs to a central service?

Does exchange logging work that way?

https://www.splunk.com/en_us/solutions/solution-areas/it-operations-management/microsoft-infrastructure-monitoring/splunk-app-for-microsoft-exchange.html

OK makes sense for tracking, etc... but what about recovery?

Recovery of the logs, or recovery of email?

recovery of email in case of a failure - it's my understanding that you restore you IS, then replay the logs to get the newer data back.

Right, now I see, these are the DB logs, not the application logs.

wirestyle22

@JaredBusch Can you explain some of the risks associated with what I did just for my own knowledge? I did verify clean shutdown in powershell.

JaredBusch

@wirestyle22 you could have potentially been unable to mount the database again.

momurda

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I am confused here, how did this solve the problem? Youre just cutting out all those old log files and starting over? Since when can you do that in Exchange? Last time i did that(years ago) very bad things happened, like total loss of mail db. If these log files arent actually needed for Exchange to serve mail wtf are they for?

JaredBusch

@momurda said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

My thought is I can dismount my datastore, change the name of the log folder from Mailbox - Domain to Mailbox - Domain.old and then create a new folder named Mailbox - Domain. If the logs populate correctly I should be able to delete the old logs and do a fast full backup with the logs truncating properly.

Note: I just did this and it seems to be generating logs properly.

This was horribly risky, but is something I have done before too. I would certainly have tried other means first, but you seem to have some really odd constraints.

I am confused here, how did this solve the problem? Youre just cutting out all those old log files and starting over? Since when can you do that in Exchange? Last time i did that(years ago) very bad things happened, like total loss of mail db. If these log files arent actually needed for Exchange to serve mail wtf are they for?

You have always been able to do that in exchange. These log files are not needed once the data in the log is wrote to the Exchange DB.

Exchange writes first to the log file and then to the DB.

In theory, the files could even be deleted with everything running because they are no longer needed once the DB is updated.

The system is designed to handle it all behind the scenes though. and not with manual interaction.

Their problem was never using the correct backup method that would cause the system to auto remove the files.

wirestyle22

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

In theory, the files could even be deleted with everything running because they are no longer needed once the DB is updated.

Everything up to the most recent log file as that would be in use

jt1001001

Now that you got it at least cleaned up, have you run a new backup yet?

wirestyle22

@jt1001001 said in Exchange 2010 Hard Drive Space Issues:

Now that you got it at least cleaned up, have you run a new backup yet?

I'm running it

JaredBusch

@wirestyle22 said in Exchange 2010 Hard Drive Space Issues:

@JaredBusch said in Exchange 2010 Hard Drive Space Issues:

In theory, the files could even be deleted with everything running because they are no longer needed once the DB is updated.

Everything up to the most recent log file as that would be in use

Correct.

wirestyle22

Thanks for all of your help guys. I really appreciate it

Dashrender

well, since you have the IS back online and the users aren't screaming.. sounds like you found a solution and things are looking good.

I'd still run a chkdsk on the system when you can take it all offline (I think that is what was wrong with your previous backups). Don't want to leave a possible funky disk out there if you can help it.

Let us know when the backups are running again, and trunking logs.

wirestyle22

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

well, since you have the IS back online and the users aren't screaming.. sounds like you found a solution and things are looking good.

I'd still run a chkdsk on the system when you can take it all offline (I think that is what was wrong with your previous backups). Don't want to leave a possible funky disk out there if you can help it.

Let us know when the backups are running again, and trunking logs.

I'll know tomorrow

Dashrender

Are they going to work toward having a real DAG now too? or migrate to O365?

wirestyle22

@Dashrender said in Exchange 2010 Hard Drive Space Issues:

Are they going to work toward having a real DAG now too? or migrate to O365?

O365 is the plan. WHEN that's going to happen I don't know. It may be a year. They'd rather spend money on fun stuff it seems.

wirestyle22

Ran VSS Full Backup. Checked for ESE Event ID 224 in the Event Viewer and I see:

Information Store (3148) Public Folder - Domain: Deleting log files D:\Data\Public Folders\Domain\Logs\Public Folder-Domain.edb\E0100000001.log to D:\Data\Public Folders\Domain\Logs\Public Folder-Domain.edb\E010000043D.log.

vssadmin list writers lists no errors.

The question is, I ran the backup the 22nd and it finished the 25th. Should the logs be clearing all the way up to the 25th or only whatever was generated for the 22nd before I started the full backup? The event log seems to be showing it as successfully completing.

momurda

The backup only works from when it is started, so the logs will be truncated up to that point if it worked. That event you linked looks like it is your public folder db which is probably very small compared to the mailbox db. There should be similar entries for that db and others if they exist.
You should also have a lot more free space if it worked.

wirestyle22

@momurda Thanks