Proposed server purchase for GitLab.com
-
Gitlab leaving cloud and getting their own servers.
https://about.gitlab.com/2016/12/11/proposed-server-purchase-for-gitlab-com/And their colocation hardware details published on Google Docs
https://docs.google.com/spreadsheets/d/1XG9VXdDxNd8ipgPlEr7Nb7Eg22twXPuzgDwsOhtdYKQ/edit#gid=894825456 -
CephFS? That's kind of concerning.
-
@stacksofplates said in Proposed server purchase for GitLab.com:
CephFS? That's kind of concerning.
Huh. I wonder why that is.
Important: CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters
-
@aidan_walsh said in Proposed server purchase for GitLab.com:
@stacksofplates said in Proposed server purchase for GitLab.com:
CephFS? That's kind of concerning.
Huh. I wonder why that is.
Important: CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters
Ya it's not production ready yet. I can understand it for block and object but if you're going to use a DFS use a mature one.
-
https://about.gitlab.com/2016/11/10/why-choose-bare-metal/
So this whole move was because they chose to use CephFS. This seems way more complex than it needs to be. Currently they should be able to scale out with everything as much as needed. I don't see how this move is better.
Edit: so they can scale out as much as needed but "There is a threshold of performance on the cloud and if you need more, you will have to pay a lot more". This was news to someone? I'd bet dollars to donuts that scaling out with a more performant server as needed and then scaling back when not is much much cheaper than what they are planning on doing.
-
@stacksofplates said in Proposed server purchase for GitLab.com:
CephFS? That's kind of concerning.
Why?
-
@stacksofplates said in Proposed server purchase for GitLab.com:
@aidan_walsh said in Proposed server purchase for GitLab.com:
@stacksofplates said in Proposed server purchase for GitLab.com:
CephFS? That's kind of concerning.
Huh. I wonder why that is.
Important: CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters
Ya it's not production ready yet. I can understand it for block and object but if you're going to use a DFS use a mature one.
OIC
-
Performance Issues on the Cloud
"By choosing to use the cloud, we are by default sharing infrastructure with a lot of other people. The cloud is timesharing, i.e. you share the machine with others on the providers resources. As such, the provider has to ensure that everyone gets a fair slice of the time share. To do this, providers place performance limits and thresholds on the services they provide."
Basically they heard some buzzwords and made the typical knee jerk "we don't listen to IT and we know best" management decisions. They didn't understand what cloud computing was and are having an emotional reaction. This is the problem with people thinking that software engineers are IT, very different skill sets.
-
I also don't get why the jump from 96 TB to over 400. The whole point of distributed file systems is linear growth. If you don't have that much data right now, why are you paying for that much?
-
@stacksofplates said in Proposed server purchase for GitLab.com:
I also don't get why the jump from 96 TB to over 400. The whole point of distributed file systems is linear growth. If you don't have that much data right now, why are you paying for that much?
Yeah, and with that much cpu/ram for only 3 drives per node? Using blade servers, but sound like they think it's not blade servers in the document. Not doing math in my head well at the moment, is it possible a single blade chassis going down could take the entire cluster down?
-
@stacksofplates said in Proposed server purchase for GitLab.com:
I also don't get why the jump from 96 TB to over 400. The whole point of distributed file systems is linear growth. If you don't have that much data right now, why are you paying for that much?
Good point as well.
-
@travisdh1 said in Proposed server purchase for GitLab.com:
@stacksofplates said in Proposed server purchase for GitLab.com:
I also don't get why the jump from 96 TB to over 400. The whole point of distributed file systems is linear growth. If you don't have that much data right now, why are you paying for that much?
Yeah, and with that much cpu/ram for only 3 drives per node? Using blade servers, but sound like they think it's not blade servers in the document. Not doing math in my head well at the moment, is it possible a single blade chassis going down could take the entire cluster down?
Could yeah, depending on a lot of things, but losing 8-16 nodes at a single go can easily kill a cluster.
-
@scottalanmiller said in Proposed server purchase for GitLab.com:
@travisdh1 said in Proposed server purchase for GitLab.com:
@stacksofplates said in Proposed server purchase for GitLab.com:
I also don't get why the jump from 96 TB to over 400. The whole point of distributed file systems is linear growth. If you don't have that much data right now, why are you paying for that much?
Yeah, and with that much cpu/ram for only 3 drives per node? Using blade servers, but sound like they think it's not blade servers in the document. Not doing math in my head well at the moment, is it possible a single blade chassis going down could take the entire cluster down?
Could yeah, depending on a lot of things, but losing 8-16 nodes at a single go can easily kill a cluster.
Well, at least these are only 2U, 4 node boxes, but still, yuck.
-
@travisdh1 said in Proposed server purchase for GitLab.com:
@scottalanmiller said in Proposed server purchase for GitLab.com:
@travisdh1 said in Proposed server purchase for GitLab.com:
@stacksofplates said in Proposed server purchase for GitLab.com:
I also don't get why the jump from 96 TB to over 400. The whole point of distributed file systems is linear growth. If you don't have that much data right now, why are you paying for that much?
Yeah, and with that much cpu/ram for only 3 drives per node? Using blade servers, but sound like they think it's not blade servers in the document. Not doing math in my head well at the moment, is it possible a single blade chassis going down could take the entire cluster down?
Could yeah, depending on a lot of things, but losing 8-16 nodes at a single go can easily kill a cluster.
Well, at least these are only 2U, 4 node boxes, but still, yuck.
Like Dell FX?
-
GitLab is sounding like a very silly endeavor, indeed.
-
@scottalanmiller said in Proposed server purchase for GitLab.com:
GitLab is sounding like a very silly endeavor, indeed.
Ya it sucks because I use their stuff. Hopefully they don't lose everything.
-
@stacksofplates said in Proposed server purchase for GitLab.com:
@scottalanmiller said in Proposed server purchase for GitLab.com:
GitLab is sounding like a very silly endeavor, indeed.
Ya it sucks because I use their stuff. Hopefully they don't lose everything.
You can be the best software engineer and know nothing about IT.
-
@scottalanmiller said in Proposed server purchase for GitLab.com:
@travisdh1 said in Proposed server purchase for GitLab.com:
@scottalanmiller said in Proposed server purchase for GitLab.com:
@travisdh1 said in Proposed server purchase for GitLab.com:
@stacksofplates said in Proposed server purchase for GitLab.com:
I also don't get why the jump from 96 TB to over 400. The whole point of distributed file systems is linear growth. If you don't have that much data right now, why are you paying for that much?
Yeah, and with that much cpu/ram for only 3 drives per node? Using blade servers, but sound like they think it's not blade servers in the document. Not doing math in my head well at the moment, is it possible a single blade chassis going down could take the entire cluster down?
Could yeah, depending on a lot of things, but losing 8-16 nodes at a single go can easily kill a cluster.
Well, at least these are only 2U, 4 node boxes, but still, yuck.
Like Dell FX?
Not quite, they're using Supermicro. The only shared piece in the chassis is the power supply. 3 3.5" drive bays per node. They're also talking about having a DOM for the OS and PCIE storage card in each. I'm not familiar with the Dell FX line, so it could be the same sort of thing.
-
@scottalanmiller said in Proposed server purchase for GitLab.com:
@stacksofplates said in Proposed server purchase for GitLab.com:
@scottalanmiller said in Proposed server purchase for GitLab.com:
GitLab is sounding like a very silly endeavor, indeed.
Ya it sucks because I use their stuff. Hopefully they don't lose everything.
You can be the best software engineer and know nothing about IT.
Github seems insistent on proving this for you in grand scale and fashion.
-
@Ambarishrh said in Proposed server purchase for GitLab.com:
Gitlab leaving cloud and getting their own servers.
https://about.gitlab.com/2016/12/11/proposed-server-purchase-for-gitlab-com/And... disaster: https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/
