GitLab in Massive Data Loss with No Working Backups
-
This is worth discussing as this is a key factor, we assume, in how GitLab got to where they were. Pablo, it would seem, is probably a great guy that was thrown in way, way over his head. Imagine if Pablo was to pop into SW and start posting ideas about how he was going to leave the cloud, eschew virtualization and go all physical and was going to use scripts that his team would make for backups rather than using tried and true products.
We'd say he was a crazy noob who isn't special and needs to not reinvent the wheel and to use common and best practices, common sense and to learn something about IT before doing nutty things thinking that the general good decision making in IT doesn't apply to him like it does to everyone else. We'd not be surprised at all to learn that he had just taken not only a new job, but his very first IT job and was only a few months into it (even today, long after the switch to physical is complete, Pablo has only been in IT for less than eleven months, he's not even finished his first year yet.) We'd laugh about noobs and crazy ideas and the kinds of people that show up on SW and talk about fake titles given out to sound good so that even first day newbies tend to get senior management titles and very strongly correct his crazy ideas.
This is just a very public company with a lot to lose that let this happen. Here is PC's LI profile so that we can understand the background. Notice his last few years were just managing change management, not even really very technical. He's got what looks like decent programming background, but literally nothing that would suggest he was ready for any kind of IT role with oversight. He previous roles don't appear to have anything IT related, nor anything that would obviously signal the kind of business experience necessary to understand how IT technical decisions fit into the bigger picture. What did GitLab look at to even consider him for this role? If I was interviewing him in IT, his resume says L1 helpdesk or maybe entry level system administrator to "see if he would have the necessary operational mindset" before committing to keeping him on. Nothing in his experience points to operational mindset, the top skill for infrastructure roles, so how did he get on someone's radar in the first place?
My guess, and this is only a guess, is that there is an endemic issue with management at GitLab. Maybe someone sees IT as a joke and things programmers are just high end IT? Maybe someone in management is just very foolish and doesn't think that they, as a hosting company, need good IT? Maybe they are just a normal, clueless SMB? Maybe a VC forced a nephew on them as part of a funding deal? Who knows, but root cause analysis would be a good idea for GitLab to do. Yes, Pablo did some insanely reckless things that even newbies on SW were be mocked for mentioning they wanted to do and should be held accountable for that. But there is no reasonable way that Pablo even got an interview for such a position or the power to do this kind of damage without something being very, very wrong much higher up in the organization.
-
Given the time and scope of a move like this and the time frame that Pablo only began the job in February, and the very likely wrong assumption that Pablo initiated the change rather than just speaking on behalf of it, and the time till the switch over... it seems likely that the project of tearing the infrastructure apart in an epic rip and replace disaster must have started pretty much the moment he joined the company. A pretty bold move for a newbie in their first days in a new career. Imagine being hired fresh into automotive engineering for Ford with no experience and on day one announcing that wheels were legacy and you were going to make all new cars used tank treads and skis in the front instead.
-
Too easy to make a joke here. Are they hiring because clearly they forgot to hire IT people? Or is it funny that they are so adamant about hiring when, likely, they will be out of business in a few days.
-
-
@scottalanmiller said in GitLab in Massive Data Loss with No Working Backups:
Too easy to make a joke here. Are they hiring because clearly they forgot to hire IT people? Or is it funny that they are so adamant about hiring when, likely, they will be out of business in a few days.
So long as that contract is guaranteed pay... regardless of time worked / business viability.
-
Looks like recovery is on track at roughly 5% per hour.
-
"That's one of the things we want to start doing or looking at".
In reference to creating backups... . . .
-
@DustinB3403 said in GitLab in Massive Data Loss with No Working Backups:
"That's one of the things we want to start doing or looking at".
In reference to creating backups... . . .
Is that a real quote? LOL
-
@scottalanmiller said in GitLab in Massive Data Loss with No Working Backups:
@DustinB3403 said in GitLab in Massive Data Loss with No Working Backups:
"That's one of the things we want to start doing or looking at".
In reference to creating backups... . . .
Is that a real quote? LOL
Yeah. not even joking.
-
"Could use time machine right?" ummm wtf...
-
If ever there was an RGE*, this would be it.
*Resume Generating Event
