XenServer - Crash post mortem
-
@Dashrender said:
When Windows crashes, it still has the ability to write things back to the disk during the crash process - does nix not do the same?
But Linux didn't crash here, right? It didn't boot up at all?
-
@JaredBusch said:
For Windows I use ImageUSB from PassMark
And I use Winimage.
Will ImageUSB read things that Windows itself can't?
-
@Dashrender said:
Assuming the boot did work, are you suggesting that the system would retain in memory some information about the crash that could then have been written to the logs after rebooting - This would defy everything I know about rebooting, so I'm sure I'm just misunderstanding you.
I thought that we knew that the issue was that the storage disconnected. So Windows, Linux or otherwise here are the issues with logging...
- The device to which to log isn't writeable so no OS capability will fix that. There is nowhere to log (this is why logs should always go to an external collector like ELK, ELG, Logg.ly, etc.
- When the system was having issues, it was unable to boot into Xen or Linux in any way, so the logging mechanisms would not be there anyway.
- The issue you are dealing with is with the hardware, not with the software, so software logging doesn't sound very important here.
-
@scottalanmiller said:
@Dashrender said:
When Windows crashes, it still has the ability to write things back to the disk during the crash process - does nix not do the same?
But Linux didn't crash here, right? It didn't boot up at all?
XS crashed - well at least I'm assuming it did. I left work, the server was running, XS was running a copy process over the network was in happening.
When I arrived in the morning, the server was stuck in a boot loop - most likely because it couldn't read the SD card.
The question is, why did the server reboot - I'm assuming because XS crashed and auto restarted.
-
@Dashrender said:
@JaredBusch said:
For Windows I use ImageUSB from PassMark
And I use Winimage.
Will ImageUSB read things that Windows itself can't?
By definition, an image reads the drive, not the filesystem. Windows ability to mount the filesystem is unrelated. Imaging is imaging, there aren't Windows specific imaging capabilities.
-
@scottalanmiller said:
@Dashrender said:
@JaredBusch said:
For Windows I use ImageUSB from PassMark
And I use Winimage.
Will ImageUSB read things that Windows itself can't?
By definition, an image reads the drive, not the filesystem. Windows ability to mount the filesystem is unrelated. Imaging is imaging, there aren't Windows specific imaging capabilities.
yeah, I assumed as much - but needed to make sure.
-
@Dashrender said:
XS crashed - well at least I'm assuming it did. I left work, the server was running, XS was running a copy process over the network was in happening.
Sure, we would assume this would happen if the storage failed. What we know after the crash is that the storage was having issues at the hardware level. Is it possible that XS crashed from a software error and then, totally coincidence, the hardware failed at the same time, in such a way that it would have caused the original crash but didn't? Sure. But did it really? No, you know what the issue is here with 99.9% certainty.
-
@Dashrender said:
The question is, why did the server reboot - I'm assuming because XS crashed and auto restarted because the filesystem became unavailable causing there to be nowhere to write logs.
-
On a somewhat unrelated note, thanks for getting me to look at the /var/log directory on the XenServer here. Had a bunch of old logs clogging things up (500MB on the tiny little root partition.) Moved those off to my workstation for now, probably need to take some time and see what's going on.
-
@scottalanmiller said:
@Dashrender said:
When Windows crashes, it still has the ability to write things back to the disk during the crash process - does nix not do the same?
But Linux didn't crash here, right? It didn't boot up at all?
I guess I mistook this to be you saying that Linux didn't crash - it did, but not because of software, but because of hardware, so it's not Linux's fault, it's hardware's fault.
OK I gotcha!
So now I need to get another SD card and create images of it.
-
@travisdh1 said:
On a somewhat unrelated note, thanks for getting me to look at the /var/log directory on the XenServer here. Had a bunch of old logs clogging things up (500MB on the tiny little root partition.) Moved those off to my workstation for now, probably need to take some time and see what's going on.
Glad my freak out could help ya out
