Thanks for your replies!

It's tricky for me to change to the same OS for the three clusters, since they are all in production... I barely managed to get the users to do some benchmarking before they started using the newest cluster full time... I guess I can take one node from each cluster and install CentOS 7 and test again, but then I will need another program to test with. Any ideas? Also, I cannot use CentOS 6, since I guess the newest CPUs will not be recognized.

As for BIOS settings, I have followed the vendor's recommendations. When it comes to help from Supermicro, it has been very little so far. They told me to check out the tips given in a paper made by Intel and Ansys, but that did not give me much insight. It's more like a promotional paper; try searching for "Higher Performance Across a Wide Range of ANSYS Fluent Simulations with the Intel Xeon Scalable Gold 6148 Processor". The only comment that I found useful was "The improved vectorization, which is available as a runtime option in Fluent 18.1, was used in these benchmarks." I am pretty sure that version 19.2 of Fluent is aware of the Scalable processor features.

Anyway, since the core speed of 6146 and E5-2667 v4 is the same, I would think that it should at lest be equally fast. The faster memory (and we have populated all channels, as recommended) is supposed to be very important for Fluent, and this should make the 6146 cluster faster...

Do you know of any test program I can use to test single core performance, and also multicore performance, in a simple way from the command line? I can run this on one node on all clusters and compare. And if you know of any contact person or division in Supermicro that I can contact bout this matter?

Hi!

Is Intel Xeon Scalable really better than the E5? In real life?

I manage three small HPC clusters. We use them to run Ansys Fluent 19.2 simulations. Rough specs for the three clusters are as follows:

• #1 - Purchased in 2013/2014, 128 cores, with nodes having two Intel Xeon E5-2643 v1 CPUs each
• #2 - Purchased in 2016, 256 cores, with nodes having two Intel Xeon E5-2667 v4 CPUs each
• #3 - Purchased March 2019, 544 cores, with nodes having two Intel Xeon Scalable 6146 v1 CPUs each

All systems use Supermicro hardware.

All three clusters have InfiniBand network between the nodes and head node, and scale very well. By scale I mean that the performance of a job almost linearly scale up as we use more cores. Fast disk access is supposedly not a big concern for Ansys Fluent, more so speed of cores and memory. All nodes have local SSD drives. All resulting data is written over NFS via ordinary Gb ethernet to head node, that has a RAID6 with SSD cache. I have never been able to see that disk access is a bottleneck for the calculations. The only times I notice that the system is spending time on "system" rather than "user", is when someone accidentally starts a job using Gb interconnect rather than InfiniBand. So, it seems to me that the only bottle neck is the speed of the cores.

When we purchase a new cluster we have of course compared the performance of the new cluster against the old cluster, using jobs that fit on both of them, e.g. various jobs up to 128 core jobs, since the oldest cluster has 128 cores. In this way we feel that we have been comparing them in a "fair" way.

Cluster #2 is about 20% faster than cluster #1. The 2643s CPUs have a core speed of 3.3 GHz versus 3.2 GHz for the newer 2667s. Now, I have explained to the users that the newer cluster (#2) is faster than the oldest one (#1) because of faster memory (1600 MHz for #1 and 2400 MHz for #2) and generally better and faster architecture in the newer cluster. The InfiniBand interconnect is also faster (40 versus 56 Gb), but I think this does not matter. So, even if the oldest cluster has the fastest cores, everything else is faster.

Our newest cluster have CPUs with core speed 3.2 GHz, has a generally newer and better/faster architecture and has faster memory (2666 MHz). Now, based on the difference in speed between #1 and #2, I made a guess that the newest cluster (#3) should be at least 20% faster than cluster #2, based on the reasoning above about being newer and better, and having faster memory. Faster, even if the clock speed remains like #2 (#2 has lower clock speed but is still faster than #1). And the InfiniBand for #3 being 100 Gb might also contribute.

But, no! Cluster #3 is actually slower than cluster #2... Cluster #2 is about 10% faster than than the newer and presumably better cluster #3...

So, I wonder... Have anybody else seen this? Does anybody else have real world examples and cases like ours? Is Intel Scalable generally faster than E5 v4 in the real world? We find that our E5 v4 cluster is actually fastest...

If I have set something up the wrong way I would be delighted if someone can point this out for me. How can I check if something is wrong? All three systems run the Rocks cluster distribution (CentOS with extras), #1 version 6.1, #2 version 6.2 and #3 version 7.0.

I am both puzzled and disappointed right now... I hope you gurus can help us out!

Well, it's not so simple for us. We usually have no remote support, and it's up to the customer's IT dept. to take care of things once the system leaves our premises. After commissioning we often don't even have admin rights. It's really puzzling, but we often feel that we abandon the systems once the customer takes over. Our company do have service personnel on the customer's site regularly, but that is for other tasks, and not to service our system. We have had cases where we have been told that "the machine beeps, it's been like that for 3 months"... I guess you could say that someone, somewhere has a job to do. I will try to investigate this when we get our next project. We are often a sub contractor, and we rarely speak to the end user or the IT staff supporting him in the early stages of a project.

@scottalanmiller I will try to investigate this and try to get in touch with our customer's IT dept. early in the process for our next project. It will be interesting to see what will happen. Already it is quite puzzling that most customers want us to deliver the hardware, rather than just providing a server to us, or ask us for a VM. I guess that should be an indication the the instrumentation part of offshore business is a little "special" when it comes to these things.

Well, RAID is often a requirement, and will be specified in the functional description. The idea is of course that it adds to the uptime, and that a failed drive does not make the "instrument" usable. Then again, when a drive fails it will most probably go unnoticed (the server is isolated and out of our reach), and it might perhaps be noticed if service personnel from our company is on the premises for other reasons. We are quite appalled by how our customer treats some of this equipment (some never take back-ups), but it's difficult for us to change this. Our servers are often small add-ons to tons of steel that make up the rest of the multi-million dollar deliveries. The value of our systems (production monitoring) is often not realized until after the field is in production.

But, seriously, how would you go about installing the hypervisor. On an internal SATA-DOM or USB, and who will administer the hypervisor. We are often only allotted an IP-address. I can see that introducing the hypervisor with an IP in addition will make the IT guyes freak out. Then again, perhaps not. They might be all for it. I will try to investigate this on our next project.

OK, bad choice of words, then. Perhaps backwards would have been better ;-). Anyway, the VM layer will introduce a layer that will need a little bit of training, and it might seem more complicated to our customers. Perhaps I will try to suggest we go virtual on our next project. I guess there will be Hyper-V available for free with the WS 2012 OS, or am I wrong?

@scottalanmiller
I agree ;-). But, we are in the offshore industry, and our machine(s) are regarded as instrumentation, and our segment of the industry is ... slow changing. I think that what you say might one day be the case, though. We have other customers that are moving to VMs these days, but that is for our software doing calculations only. In this particular instance we are also in the area of collecting data, and this is the subsea segment, not topside. The cusomers are more conservative on the subsea side.

Sounds good! I am thinking it would be like this, but thought it was better to hear if anybody have experienced problems before promising to help the other guys out. Yes, it is Windows, but I assume the other company will have the necessary info to re-activate the OS on their machine. Thank you for your reply!

Dear community!

We have a customer that have purchased our software. We deliver our software on a HP DL380 Gen9 that I have set up and installed OS and software on.

The customer have another supplier located close to us that also have supplied them with a piece of software on an identical type of server. This servers has been sent to Singapore for a test. When it arrived at the destination, all the 8 hard drives had disappeared... Now, they can of course have the machine returned to them and put in new drives and reinstall everything and ship it to the test site again. This will add weeks to the time spent and will be bad for our customer.

But, our customer suggested that this other supplier come to our premises with a set of drives and borrow our server in order to create RAID on the drives, install the OS and put in their software. Then they can ship only the drives to Singapore.

We have no problem with this solution seen from an administrative view. This will be helpful to our customer and this other supplier is no competitor of ours.

Now, what I am wondering about is this; if I power down our server and remove all the drives from the machine, am I risking running into problems of any kind? When this other party is done borrowing our server (or more preciely, our customer's machine) can I simply just put back our drives in the machine and everything will be like before? Will our RAID controller be confused about having the other new drives in the bays for a few hours? Will the drives that are shipped to Singapore actually work in their server there?

If we don't disrupt our server doing the above, this will be a solution that is a win-win-win situation. No problem for us, very good for our customer and very good for the other supplier

Well, the main application itself, is a "server" that is started automatically as a service. It gathers data and performs calculations based on the input. Data may be exported, but not always, but stored in proprietary databases. The user interface comes up by way of Interactive Services Detection, and is a bit of a pain... The application is being rewritten as we speak and will use HTML and a browser for GUI in the next version. BUT, the customer is only allowing RDP traffic to the server, not http, so...

@scottalanmiller Customer education is not so easy... We want them to choose our product, and like I said, the users were all afraid of Unix/Linux. In many ways some of us working with the software have always looked at Windows as being inferior. After many years I guess it is getting quite usable, but I am surprised about all the licenses we require and the complication it all leads to...

Thanks for all the answers!

I'll order the necessary RDS CALs and get this sorted out right.

@scottalanmiller Well, back in the late ninties we used to be a Unix shop, since our software required laaarge servers and lots of processing powers. The customers really wanted us to change to Windows, and over the years we even started using Java rather than C++/Motif, so now we are stuck in Windows-land for most projects. Since we are using Java, I guess I will inform the project leaders that Windows licenses actually might be a bigger cost than we originally thought... Only one of our customers still use Linux, but it's their system that we never have any problems with

@scottalanmiller Hmm... I get a creeping feeling that there might be a server here and there that might be running without the striclty required number/type of licenses. My present project just got a bit more expensive

@scottalanmiller So, what exactly can one use Windows Server 2012 R2 Standard for right after installing it and just applying the license that comes along with it? I'm trying to understand what is possible without additional licenses, as I'm unsure right now...

@scottalanmiller OK, I am beginning to understand now. But, what can one use the 5 CALs for if you in addition need the 5 RDS licenses? What is their intended use?

@Dashrender Hm. I am confused now. We have several servers i n our lab, and both the administrator and at least one other user is able to use RDP, but I (think) only two sessions at any one time.

@brianlittlejohn So, I can't use the 5 CALs for this?

@dafyre Thanks, I will read it and give it a try!

Dear community!

I have the following scenario; We are supplying to one of our customers our own software installed on a HP server running Windows Server 2012 R2 Standard. This machine will be in a industrial (clean) production environment without a domain controller, and only have local users.

Now, several customer employees need to access this server via Remote Desktop, and they can have two roles; operator (read/write) and observer (read only). We as supplier will use the administrator account and also have the role as engineer.

I created three user groups; engineer, operator and observer. Then, it is the idea that for each person/user an account is created in one or more of the appropriate groups. I guess we need a total of 3 different users from our company and maybe 4 from the Customer. And each person will have from one to three users each. Our personnel will have one user in each user group, whereas the Customer users will have either two (observer and operator) or just one (observer). We also purchased a 5 CAL license.

Now, what do I need to set up in order for this to work? I assume that without doing anything a total of two concurrent RDP sessions is supported/allowed, and for more users or sessions I will have to perform additional setup. Are the 5 CAL licenses sufficient?

I have seen somewhere that I need to set up Remote Desktop Services, but then I also read that in order for that to be possible the server had to be on a domain. And that is not the case. And there were so many different services that could be installed; Gateway, Connection Broker, Session Host, etc -- which ones do I need?

I have very little knowledge on this topic, and would be very grateful for a quick and easy guide to what we need and how to set it up.

Yes. At the moment we are having two different Essentials Plus environments, and I must say I am at times very frustrated about the limitations. I mean, why not 5 hosts instead of just three (6 CPUs)?? We are a rather small SMB-type department in a large organization, and I see that XenServer will give us things like live storage vmotion for free, whereas for vSphere I have to shut down the VMs... I am also playing with View, and thinking about virtualizing a few workstations that have powerful GPUs -- for that to work we need full licenses in order for vGPU to work, and this will be prohibitively expensive when we are just talking about 5 users... I feel sometimes that we are cought in the middle. We are a business, but have a small budget. Sigh...