@scottalanmiller said in Testing oVirt...:

And vice versa. Having worked at the vendor in question, it feels like you are seeing this through the eyes of the sales team, and not thinking about it from the perspective of the customers needing to actually run the product. Where is your conviction coming from, other than it being the sales mantra of the vendor?

Never worked in sales, but I have worked for other enterprise vendors and their customers quite a bit. From government to automotive and aerospace, to oil and gas to hipster-ish startups. No Fedora anywhere but desktops sometimes. Ever.

Having used both in the real world, what problems do you see with Fedora and what old code are you running that EL always has what you need?

I can install EL, keep it updated without moving versions, and be certain everything I build on top of it will keep working throughout the OS lifecycle, which is very long. That means I can concentrate on developing my software instead of wasting time on keeping up with what the underlying OS is doing, introducing bugs into my code which used to work before the OS update. Without losing the safety of important updates coming in on time. Yes, some of the prepackaged stuff is outdated, but I can get the newer code, if I need it, from repositories that are code-specific (in my case - mostly pypi). I've come into a place once, where Fedora was installed on several hundred servers which were rendering CGI. They had two people there just making sure the cluster task dispatcher was able to work after every update. Two expensive coders, doing nothing but test fedora releases versus their task scheduler. A test cluster on CentOS turned out to be able to do exactly the same job, but the guys dealing with fedora quirks became free to develop the task scheduler, which boosted their bottom line productivity after a few months.

Are you really seeing Fedora instabilities that the rest of us are not? Are you really not running any modern code that benefits from current libraries and packages?

Of course I am. pip, cpan, (whatever php has for that same purpose) etc do the job perfectly. There isn't much I cannot do on an EL distribution, one way or another

@scottalanmiller said in Testing oVirt...:

Yes, but I would never pay for EL support or do business with RH on ethics grounds.

Care to elaborate?

The "support" is how much they support the OS and make sure that it works,

Oh no, you clearly never worked with RH proper. I've spent 4 years in RH support, and there is so much more there

and they do a pretty amazing job with Fedora. Bottom line, Fedora gets "support" to make it work in the real world, EL gets "support" that you pay for to help you when you don't know what you are doing.

Nope. Fedora gets built by professionals to be an OS that has all the latest code that is capable of working at least somehow. Not well, not in a stable manner, but work. Working well and stabilization is all the work that goes into EL afterwards.

Support for EL isn't about handholding, it's about making integrations work, about providing customers with very quick fixes to their specific problems at the code level sometimes, and about working with the customers on what they need in their distribution. I've gone through T4 support to product management and seen exactly that all the way. Simple handholding is something you get from ISP support with a bunch of kids who read from a script. Don't get confused between the two.

In the real world, Fedora is supported and works. EL does not for production workloads I deal with. When I have the choice between the two, Fedora every time.

In the real world, especially at scale, nobody has the time to deal with the bugs you get in the new untested code. Not everything can be solved by restarting a microservice, and not many organizations keep talent onboard, who can fix a kernel bug on the fly. This is why people rely on EL and other enterprise grade software.

@scottalanmiller said in Testing oVirt...:

Remember no one wants bleeding edge. Current stable and bleeding edge are worlds apart.

Long Term Release < Current Stable < Cutting Edge < Bleeding Edge

Fedora is very production ready, very stable. It's very, very far away from bleeding edge. Even Tumbleweed is only cutting edge.

I really don't know where you are getting this from. I've never seen such an opinion expressed from anyone in the business, neither in my 10 years within Red Hat or in my over 20 years in IT in general. Fedora is supposed to be as close to the development code as possible, there is no QA there, just basic testing to make sure stuff can be compiled/built/installed, nothing else. Even the layered upstream products get way more testing on EL than on Fedora (hence the dumping of Fedora for oVirt, and IIRC RDO too).

I really wonder how you managed to come up with this conviction of yours and what makes you believe it besides your own experiences

@scottalanmiller said in Testing oVirt...:

@dyasny said in Testing oVirt...:

@black3dynamite said in Testing oVirt...:

Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.

Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you. Why use Fedora though, when you can use something more lightweight, like Alpine, in a container?

Support. Fedora has insanely broad vendor (meaning RH) and third party (the software makers) support. Possibly the broadest in the industry, or maybe second after Ubuntu. But Ubuntu support leans towards the unsupported LTS releases making Ubuntu products questionably supported at all (since Ubuntu's official stance is that if you need LTS support beyond consulting, meaning actual fixes, you might have to leave LTS and go to Current and if your software vendor is LTS only, the resulting product is unsupported.)

Alpine is great, but not many vendors test against it.

Actual vendor support for Fedora is tiny compared to EL, you are really describing an alternate reality here. I have never seen anyone doing any production work on Fedora, beyond development and desktop computing.

@flaxking said in Diving into a completely new tech stack:

@dyasny So far my only complaint is that they are lacking in kubernetes related courses

There are WAY too many k8s related resources out there. Openshift is harder to come by, but only marginally

@jmoore said in Testing oVirt...:

@dyasny Lol I agree with that. People in many industries are constantly renaming things to make it sound new and raise the hype.

When I was working as an Openstack integration engineer, I had a little framed note on my desk. It read "there is no cloud, it's just someone else's computer"

@black3dynamite said in Testing oVirt...:

Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.

Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you. Why use Fedora though, when you can use something more lightweight, like Alpine, in a container?

@scottalanmiller said in Testing oVirt...:

Right, Fedora has been faster and more stable for us. CentOS was much slower, lacked solid features, and had support issues (because it was unable to continue to support living software that was still updating while the OS had stagnated.)

What exactly was CentOS slower at? What features were lacking? How exactly it could not support "living software"?

@scottalanmiller said in Testing oVirt...:

Big business tends to list requirements that they sense as trends, long before they use them internally.

Maybe, but the interviews were with the guys already implementing the tech, and they were quite happy to describe what is already done and why they wanted me to join up

@scottalanmiller said in Testing oVirt...:

Yeah, DevOps in finance is old hat. They've been doing that for quite a while.

devops, config management, containers, kubernetes, a bunch of various big-data tech. When I see that mentioned, I can easily imagine what the structure of their currently developed software is - microservices all the way, no legacy involved.

And if anyone but us two is reading this - DevOps isn't new, it's as ancient as companies like Ford and Toyota, ask any business major (think of that over your next smoothie, young hipsters)

@scottalanmiller said in Testing oVirt...:

Tell that to the financial sector

Actually, the latest few calls, all about the very shiny and new devopsy stack involved, were from financial companies - old prominent banks and a couple of hedge funds. As Wall-street as they ever come.

Developers get big bucks doing new work. IT gets big bucks supporting bad development.

Or good development, it's an ongoing process after all, bugs get fixed, features get introduced, more bugs come up etc etc etc

@scottalanmiller said in Testing oVirt...:

No one cared that it was FreeBSD, it was 100% about ZFS. In fact, companies packaged FreeBSD to hide it and touted only ZFS as the reason to use their stuff.

There were a few companies that managed to sell some ZFS based stuff, but I really wouldn't call it a craze. And all the major SAN vendors caught up and produced their own stuff with the same featureset, only stable

@scottalanmiller said in Testing oVirt...:

Yes, but there is a lot of legacy stuff that isn't going anywhere. Most people have to deal with legacy stuff indefinitely.

I get recruiter calls all the time, and they all want the new shiny tech, not old legacy knowledge. At least all the recruiters who have a decent offer on hand. The ones who want old school sysadmins to work on old systems that aren't going anywhere, are offering miniscule wages.

And like I mentioned above - there are means of dealing with legacy stuff in containers, just like when vmware was starting to become prominent, a lot of effort was invested in supporting older OS inside a VM, so that people would be able to move away from old hardware

@scottalanmiller said in Testing oVirt...:

ZoL isn't where the frenzy was.

I missed the FBSD frenzy, in fact, I haven't seen anything resembling a frenzy around that old thing for about 10-12 years now. I wish there was one - moving companies to Linux from a pre-existing Unix setup is the easiest sell ever

@scottalanmiller said in Testing oVirt...:

Yes, if you have microservices, which is getting traction but will be a long time before most workloads are that way, it can be very good to have minuscule containers to handle them individually.

It's pretty much the default to all new software that gets developed. New version to existing legacy stuff is not included of course.

@scottalanmiller said in Testing oVirt...:

They were. You are thinking x86 commodity space. But in the enterprise, we were using them heavily for a very, very long time.

Like I said, LPARs and similar tech from other vendors (don't even remember the names now) were much closer to containers than to proper VMs.

@scottalanmiller said in Testing oVirt...:

Yeah, but the frenzy around it was crazy. Seriously nuts. People were out of their minds in love with ZFS to the point that they based whole infrastructure decisions around getting it (and on FreeBSD no less.)

I've seen ZoL break way too many times to even consider it

@scottalanmiller said in Testing oVirt...:

I like this take on Docker from an important database vendor: "Running Scylla in Docker is the simplest way to experiment with Scylla and we highly recommend it. However, running stateful containers is complex and tuning is needed to maximize the performance. We recommend that you use packages..."

And yet, it is now possible to run a stateful database which is very close to the metal, in docker, with no performance losses. Moreover, if the container dies, you do not reinstall, you simply respawn the container, and if the storage survived - simply attach it when you spawn.

I agree with this. Docker is great for testing, absolutely excellent. And some workloads, it's great for deploying (especially when it is internal code that you control and know it will be compatible.)

Microservices. When all components are independent daemons, talking over a common message bus or API, keeping them containerized (note how I don't mention docker specifically) makes keeping the system up very easy.

There's a good reason even a monster like Openstack is moving towards containerizing all the various services it is running

@scottalanmiller said in Testing oVirt...:

Well no, VMs have been the enterprise standard since 1964. Quite different. And containers aren't new, treating them like magic is the new fad.

Actually, mainframe partitions are much closer to containers than to VMs. Containers became possible on x86 only with the feature completeness of cgroups and kernel namespaces, before that, OVZ wasn't too bad (but Parallels was and is). Besides, scaling that entire kitchen was a problem without the advent of SDN.

Like ZFS. We had it a decade, then it became a fad that no one could live without, now no one remembers it.

No, ZoL, IMO is still a piece of dung. And Solaris is a no-go OS nowadays, so I just keep away. VDO is nice if you need dedupe, for all the other features, there are solutions available too.

VMs are tried and true, as are true containers. But the Docker craze... that's a fad.

VMs were not tried and true before ~2010-ish, when the tech became more or less commonplace and "boring".

Docker itself isn't great, it was just the first of the emerging systems utilising cgroups and namespaces properly. I used CRI-O for a few months and it was much faster and more stable. The point here is, containers aren't a fad, just like VMs, they are here to stay and get used everywhere. And just like some VM technologies, some types will go away (like Xen in the enterprise and vbox becoming desktop niche) and some becoming the default choice, like VMWare and KVM (well, some Hyper-V for the folks who are stuck on Windows of course), docker might not stick around in the end, but containerization will.

@scottalanmiller said in Testing oVirt...:

Containers != Enterprise software deployments. And it's a fad.

Yeah, only 10 years ago people said that about VMs

Fedora is rock solid on containers too, but with later tech. If we don't care about the packages that come with the OS, and only the most basic pieces, Fedora blows CentOS out of the water.

And again, you say "rock solid" but provide no proof. Can you show any research, benchmarks, stats, anything that shows Fedora is actually better and more stable than an EL distribution? And if you cannot, how about a man-hour comparison of engineering and QA effort that went into either? You know full well Fedora and any other non-enterprise distro can't compare, not even close.

If containers are the current standard for "enterprise", then I'm again, in the Fedora camp. I've seen the problems and instability with containers (presuming you mean Docker, not LXC) and yeah, that's what the kids trying to get jobs based on resume words do, but for enterprise workloads that actually matter, that's anything but the norm.

That hasn't been my experience. Just like before clouds became a thing and VMs were new, everyone was after talent who knew how to build virtualized DCs, it is all about containers now. And containers are the craze because they are so easy to automate. And guess which kind of distro is easier to automate and keep automated - one with stable APIs and handles, or one which changes things on the fly without really caring what your particular code does

Containers are great for testing things, and sometimes great for internally controlled software. But for deploying other peoples' code... see all the threads where we've discussed how they aren't reliable because Docker just doesn't address compatibility issues in the real world.

Plenty of problems there, and of course containers don't fit any workload pattern (though there are advances there, for example persistent storage and kubevirt to name a couple), but this is where the industry is not just going, but has been at for a while now. It's a large industry, and I know in some parts of it things are still in the dark ages (especially in SMBs, who are still running Windows SBS 2011 and don't really need anything else), but if you look at where the large corporations, containers are in production everywhere.

They put on a good marketing blitz, but it doesn't hold up in practice. Maybe someday, but "someday" comes several years earlier on Fedora than on CentOS / RHEL.

This "someday" is already here, and has been for a while. And anything that becomes interesting for the enterprise, EL (and the product portfolio based on it) is on exactly the same page as Fedora, that's how RHT became a multi-billion dollar open-source company.