Burned by Eschewing Best Practices
-
@scottalanmiller said:
@Dashrender said:
@scottalanmiller said:
So with Exchange, you can kind of get this "but our users can't use it" perspective. Let's change that up and say your company website. If your ISP goes down and internal users cannot access the company website, do you say that your website is down or offline or has an outage? Of course not. If your customers looks, the site is up and normal. We all know that calling that an outage would be ridiculous and just being obnoxious.
But how would hosted web and hosted Exchange really be different in this case? What makes it okay to call Exchange down even when it is sending and receiving emails from the outside and not okay to call the website down under identical circumstances?
You've lost me here.
If hosted Exchange is down then it's down. If my local ISP is down, I would never tell me users we were having an Exchange outage.But the whole question is... how do we define that the hosted Exchange is down?
How do I define it? how do 'we' define it?
Great question - I'm guessing that MS doesn't tell us the lowly users where the actual fault is, therefore the only thing we can say is... it's an Exchange outage if we can't use the service.
IF MS says.. hey our ISP is out, but Exchange itself is fine... OK great, I'm personally no longer pissed at MS (other than why don't you have 2+ ISPs with auto failover), I'm now pissed at their ISP.
Same goes if the outage is caused by Azure... I'm happy to put the blame where it belongs.. but rarely do cloud service providers tell us that information.
-
@Dashrender said:
Great question - I'm guessing that MS doesn't tell us the lowly users where the actual fault is, therefore the only thing we can say is... it's an Exchange outage if we can't use the service.
That's the issue, though, do we define it as "when Exchange is down"? Or "when the vendor encapsulated service is unavailable to everyone?" Or "when the vendor service is unavailable to an account, region, country, product?" Or "when it is down in a specific place?"
To me, you can call it an Office 365 outage when the customer does not have an option of a workaround. Which is why our Azure outage, to me, was as clear as any outage can be to being an outage. The fault was Azure, Azure was unavailable to people with whom there was on possible workaround. Only the Azure support team had any ability to bring it back up.
-
@Dashrender said:
Same goes if the outage is caused by Azure... I'm happy to put the blame where it belongs.. but rarely do cloud service providers tell us that information.
But we know when we can work around or when we cannot. At least generally speaking.
-
@scottalanmiller said:
@Dashrender said:
I did this morning ask the question, why not just go to one single super awesome box... the answer 'HA.'
Pretty sure that HA was listed as a requirement. We definitely covered what I said above several times in stating that the NAS was not delivering any benefit and not meeting goals which is all it takes to get all that is stated above. Once the NAS is known to not have delivered HA, we know that all of the money around that was wasted.
HA was listed by the OP, not necessarily by his boss. I say this because the OP indicated that his boss gets pissy every time there is an outage. Of course we can't just go guessing what's really needed, we do have to take some things at face value until we are shown that they were wrong, then change direction based on new information.
-
@scottalanmiller said:
@Dashrender said:
Same goes if the outage is caused by Azure... I'm happy to put the blame where it belongs.. but rarely do cloud service providers tell us that information.
But we know when we can work around or when we cannot. At least generally speaking.
In a cloud solution like O365, you do?
-
@Dashrender said:
@scottalanmiller said:
@Dashrender said:
Same goes if the outage is caused by Azure... I'm happy to put the blame where it belongs.. but rarely do cloud service providers tell us that information.
But we know when we can work around or when we cannot. At least generally speaking.
In a cloud solution like O365, you do?
Generally. While it is possible to not be the case, can you think of any circumstance where a cloud provider is not down but you cannot reach them to verify while also not knowing that you have a more significant outage?
-
You lost me again.
I think you're blending Cloud provider outages with my local onsite outages.
If O365 is having an outage - other than I can't access their services, it has nothing to do with my local services.
-
Perhaps you saying... I'm in the local helpdesk and I get a call - hey Exchange isn't responding.. is it down?
I tell them I have to look into it and get back to them.
I dig around and discover that my local ISP is down.
When I call the user back I'm not going to say, oh yeah. Exchange is down.. oh and yeah, everything else on the internet is also...
nah - instead I'm going to say - hey our connection to the internet is down. Anything we access through online services is inaccessible because of that.
-
@Dashrender said:
If O365 is having an outage - other than I can't access their services, it has nothing to do with my local services.
right, but you asked if I knew that or not. And I do, in nearly all cases.
-
@Dashrender said:
Perhaps you saying... I'm in the local helpdesk and I get a call - hey Exchange isn't responding.. is it down?
I tell them I have to look into it and get back to them.
I dig around and discover that my local ISP is down.
When I call the user back I'm not going to say, oh yeah. Exchange is down.. oh and yeah, everything else on the internet is also...
nah - instead I'm going to say - hey our connection to the internet is down. Anything we access through online services is inaccessible because of that.
Exactly. In a case like that you would know that you have no reason to suspect that Exchange is down. You can also hop on your phone and determine, almost certainly, that it is up.
-
@scottalanmiller said:
@Dashrender said:
Perhaps you saying... I'm in the local helpdesk and I get a call - hey Exchange isn't responding.. is it down?
I tell them I have to look into it and get back to them.
I dig around and discover that my local ISP is down.
When I call the user back I'm not going to say, oh yeah. Exchange is down.. oh and yeah, everything else on the internet is also...
nah - instead I'm going to say - hey our connection to the internet is down. Anything we access through online services is inaccessible because of that.
Exactly. In a case like that you would know that you have no reason to suspect that Exchange is down. You can also hop on your phone and determine, almost certainly, that it is up.
But why does this matter? No one is blaming Exchange for being down in this case. Can you restate the question you think I'm asking?
-
@Dashrender said:
@scottalanmiller said:
@Dashrender said:
Perhaps you saying... I'm in the local helpdesk and I get a call - hey Exchange isn't responding.. is it down?
I tell them I have to look into it and get back to them.
I dig around and discover that my local ISP is down.
When I call the user back I'm not going to say, oh yeah. Exchange is down.. oh and yeah, everything else on the internet is also...
nah - instead I'm going to say - hey our connection to the internet is down. Anything we access through online services is inaccessible because of that.
Exactly. In a case like that you would know that you have no reason to suspect that Exchange is down. You can also hop on your phone and determine, almost certainly, that it is up.
But why does this matter? No one is blaming Exchange for being down in this case. Can you restate the question you think I'm asking?
In most cases where people state the risk of Exchange being down, it's in a position where "some people can still access it" and we are not clear where the momentary outage is.
-
@scottalanmiller said:
In most cases where people state the risk of Exchange being down, it's in a position where "some people can still access it" and we are not clear where the momentary outage is.
aww OK.
Well from what I've seen around here, if MS is having an outage in O365, it normally affects an entire small company (large businesses due to large geo-diversity might have some parts be up while others are down), not just a few users.
-
@Dashrender said:
@scottalanmiller said:
In most cases where people state the risk of Exchange being down, it's in a position where "some people can still access it" and we are not clear where the momentary outage is.
aww OK.
Well from what I've seen around here, if MS is having an outage in O365, it normally affects an entire small company (large businesses due to large geo-diversity might have some parts be up while others are down), not just a few users.
Oh yes, typically it is large. But it can be hard to tell. Outages can be account, datacenter, region, ISP, total, etc. Total has never, TTBOMK, happened. Lots of outages have happened that are MS' fault. But determining when it is can get to be a little complicated.
-
@scottalanmiller said:
@Dashrender said:
@scottalanmiller said:
In most cases where people state the risk of Exchange being down, it's in a position where "some people can still access it" and we are not clear where the momentary outage is.
aww OK.
Well from what I've seen around here, if MS is having an outage in O365, it normally affects an entire small company (large businesses due to large geo-diversity might have some parts be up while others are down), not just a few users.
Oh yes, typically it is large. But it can be hard to tell. Outages can be account, datacenter, region, ISP, total, etc. Total has never, TTBOMK, happened. Lots of outages have happened that are MS' fault. But determining when it is can get to be a little complicated.
But those outages are all Exchange outages because from the outside, we have no idea what caused the outage - unless there are reports telling us (there might be, I've never looked) what happened.
-
@Dashrender said:
@scottalanmiller said:
@Dashrender said:
@scottalanmiller said:
In most cases where people state the risk of Exchange being down, it's in a position where "some people can still access it" and we are not clear where the momentary outage is.
aww OK.
Well from what I've seen around here, if MS is having an outage in O365, it normally affects an entire small company (large businesses due to large geo-diversity might have some parts be up while others are down), not just a few users.
Oh yes, typically it is large. But it can be hard to tell. Outages can be account, datacenter, region, ISP, total, etc. Total has never, TTBOMK, happened. Lots of outages have happened that are MS' fault. But determining when it is can get to be a little complicated.
But those outages are all Exchange outages because from the outside, we have no idea what caused the outage - unless there are reports telling us (there might be, I've never looked) what happened.
I agree. But I've been yelled at for calling some of them outages here on the community before, you see.
-
@scottalanmiller said:
@Dashrender said:
@scottalanmiller said:
@Dashrender said:
@scottalanmiller said:
In most cases where people state the risk of Exchange being down, it's in a position where "some people can still access it" and we are not clear where the momentary outage is.
aww OK.
Well from what I've seen around here, if MS is having an outage in O365, it normally affects an entire small company (large businesses due to large geo-diversity might have some parts be up while others are down), not just a few users.
Oh yes, typically it is large. But it can be hard to tell. Outages can be account, datacenter, region, ISP, total, etc. Total has never, TTBOMK, happened. Lots of outages have happened that are MS' fault. But determining when it is can get to be a little complicated.
But those outages are all Exchange outages because from the outside, we have no idea what caused the outage - unless there are reports telling us (there might be, I've never looked) what happened.
I agree. But I've been yelled at for calling some of them outages here on the community before, you see.
Really? I'd ask by whom, but that's not important -
Ohh.. they were saying, we'll I'm not down.. so you're just crazy Scott. gotcha.
yeah I'm on your side here.
The really tricky part comes when it's just a single user - is it an outage if a single user can't access, but everyone else can? I'd say no. -
@Dashrender said:
@scottalanmiller said:
@Dashrender said:
@scottalanmiller said:
@Dashrender said:
@scottalanmiller said:
In most cases where people state the risk of Exchange being down, it's in a position where "some people can still access it" and we are not clear where the momentary outage is.
aww OK.
Well from what I've seen around here, if MS is having an outage in O365, it normally affects an entire small company (large businesses due to large geo-diversity might have some parts be up while others are down), not just a few users.
Oh yes, typically it is large. But it can be hard to tell. Outages can be account, datacenter, region, ISP, total, etc. Total has never, TTBOMK, happened. Lots of outages have happened that are MS' fault. But determining when it is can get to be a little complicated.
But those outages are all Exchange outages because from the outside, we have no idea what caused the outage - unless there are reports telling us (there might be, I've never looked) what happened.
I agree. But I've been yelled at for calling some of them outages here on the community before, you see.
Really? I'd ask by whom, but that's not important -
Ohh.. they were saying, we'll I'm not down.. so you're just crazy Scott. gotcha.
yeah I'm on your side here.
The really tricky part comes when it's just a single user - is it an outage if a single user can't access, but everyone else can? I'd say no.Yeah, it was that the outage, while being from the provider and no means of working around and a REAL outage with services 100% gone, that because it was isolated by account that I couldn't say that they had an outage. But the services were gone and they could not even bring them back themselves.
It wasn't a single user, not even a single company, but a single class of companies.
-
@Dashrender by me of course.
Because the way it was phrased was click baity and inferring more than it was.
I never said it wasn't an outage. I only said it wasn't an Exchange outage. it was an account outage. Semantics. But important to be clear on exactly where the outage occurred.
Something like that can easily be called an Exchange outage initially. but once things are known to be an account outage, then it needs specified.
On the other hand I have also argued with vendors that say they do not have an outage because their upstream provider has an outage. To me the user, it is my vendor's outage. I am not a client of the upstream provider.
This is different because it is unrelated to my account in any way. Unlike the instance @scottalanmiller was referring to.
Also, I want my cake and will eat it too.
-
@JaredBusch said:
@Dashrender by me of course.
Because the way it was phrased was click baity and inferring more than it was.
I never said it wasn't an outage. I only said it wasn't an Exchange outage. it was an account outage. Semantics. But important to be clear on exactly where the outage occurred.
Something like that can easily be called an Exchange outage initially. but once things are known to be an account outage, then it needs specified.
On the other hand I have also argued with vendors that say they do not have an outage because their upstream provider has an outage. To me the user, it is my vendor's outage. I am not a client of the upstream provider.
This is different because it is unrelated to my account in any way. Unlike the instance @scottalanmiller was referring to.
Also, I want my cake and will eat it too.
I mostly agree here, but the issue, to us, was that while there was an account outage, that triggered an account-localized Exchange and Azure outage. Exchange and Azure services were unavailable to a group of customers (we know of several others affected too, not just our account). So by any normal reasoning yes, there was an account outage, but that's a little like the vendor saying that their ISP failed. To us, as a customer, Exchange and Azure had failed and it something technical on the vendor's side (account is not technical, that they could not fix the account was technical) that they took days in one instance to fix and are still on months trying to fix in the other.
If we use the criteria of "are the services no longer available to the customer(s)" then the answer was yes. The cause was "technical glitches in the account management system" rather than "vendor's ISP failed" , but that's after the point that Exchange and Azure had outages. The end result was Exchange and Azure being down to customer(s).
