Small colo infrastructure - rack layout feedback
-
I need some feedback on how to place servers in a colo rack and how to wire them up so it is practical.
14U (1/3 rack) is what I have. The real rack cabinet is of course full height but I drawn it as a 14U rack to show the available space.
I've placed some 1U cable management but I'm not sure I need it. I have about 20 CAT6 connections per switch to hook up. And stacking cables between them. They are 48 port switches.
-
Servers are supermicro twins so it should be possible to pull out the nodes backwards. Power supplies are in the middle. The front has only drive bays.
-
@pete-s
No in rack UPS? -
@jimmy9008 said in Small colo infrastructure - rack layout feedback:
@pete-s
No in rack UPS?not really needed in a Colo - as it will have (well it should anyway) Dual sourced feeds, and full facility UPS / Isolation / Generator.
-
@gjacobse said in Small colo infrastructure - rack layout feedback:
@jimmy9008 said in Small colo infrastructure - rack layout feedback:
@pete-s
No in rack UPS?not really needed in a Colo - as it will have (well it should anyway) Dual sourced feeds, and full facility UPS / Isolation / Generator.
I missed the Colo part so that makes sense. Just a thought anyway.
-
@jimmy9008 said in Small colo infrastructure - rack layout feedback:
@gjacobse said in Small colo infrastructure - rack layout feedback:
@jimmy9008 said in Small colo infrastructure - rack layout feedback:
@pete-s
No in rack UPS?not really needed in a Colo - as it will have (well it should anyway) Dual sourced feeds, and full facility UPS / Isolation / Generator.
I missed the Colo part so that makes sense. Just a thought anyway.
I agree. if it were not in a Colo - def need a UPS
-
@jimmy9008 @gjacobse
Yes, the datacenter is fully redundant UPS and generators. And also on internet connections, network infrastructure, cooling etc. Good point though!After some thought, this is what I'm thinking on cabling.
- First remove cable management so no space is unused.
- Place and connect servers from the top down. Should be easier to add equipment that way.
- Make a bundle for each switch and use colors for it.
- Wire the nodes from each side and let the power cables also go to each side. I don't know if the racks have PDUs on each side though.
-
@pete-s
Interesting, is this Visio? secondly, what is the reasoning for the DB host 1+2, do you mean database or data backup?
-
@emad-r said in Small colo infrastructure - rack layout feedback:
@pete-s
Interesting, is this Visio?
Yes, Visio Pro.
secondly, what is the reasoning for the DB host 1+2, do you mean database or data backup?
No, I mean database servers. Backup will be on the fileservers and off site.
The reasoning was to set up the databases with replication / cluster between the DB server 1 & 2 - and just do it once. This would also give good performance and be scalable to more servers. The VMs running the application code are on the other servers.
DB servers will maybe run on bare metal. They have NVMe drives and both Xen and KVM causes severe performance drops on iops and throughput. I'll test both bare metal and virtualized when I have everything setup to see how much difference it is in real life.
-
@pete-s said in Small colo infrastructure - rack layout feedback:
DB servers will maybe run on bare metal. They have NVMe drives and both Xen and KVM causes severe performance drops on iops and throughput.
LXC would be the more obvious choice.
-
@scottalanmiller said in Small colo infrastructure - rack layout feedback:
@pete-s said in Small colo infrastructure - rack layout feedback:
DB servers will maybe run on bare metal. They have NVMe drives and both Xen and KVM causes severe performance drops on iops and throughput.
LXC would be the more obvious choice.
Thanks, I'll look into LXC. Did you mean LXC specifically or containers in general?
Right now I'm mostly focused on getting the hardware set up in a practical and standardized way. Then it will be configuration of switches and firewalls. Then I can start setting up the servers with hypervisors, raid arrays etc. And then start the work on setting up VMs, containers maybe, installing OS, software etc, etc.
-
I think you have the physical design down. Only question really is "switch on top" or "switch in the middle". And I think at 14U, on top is better.
-
@scottalanmiller said in Small colo infrastructure - rack layout feedback:
I think you have the physical design down. Only question really is "switch on top" or "switch in the middle". And I think at 14U, on top is better.
If it was a full sized rack with say twice as many servers. Would it still be switch on top?
-
@pete-s said in Small colo infrastructure - rack layout feedback:
@scottalanmiller said in Small colo infrastructure - rack layout feedback:
I think you have the physical design down. Only question really is "switch on top" or "switch in the middle". And I think at 14U, on top is better.
If it was a full sized rack with say twice as many servers. Would it still be switch on top?
For weight distribution, yes. But if you were solidly anchored, no, switches in the middle to make cabling simplified.
-
@pete-s If the runs are not fibre look into 10GbE certified ultra-thin patch cables. We've started using them for all of our data centre deployments as they save a huge amount of space. There's some really good but expensive VELCRO rolls for tying things up. We've picked up a box or two of VELCRO thin and wide plant ties each. Same stuff as the computer ones in black but a tenth of the price. So what if they're green. ;0)
PDU cables rated for 240V are freaking huge and a bear to manage. I'd bundle and run them straight down the middle then to the sizes and up to their position on the PDUs L/R. That's a bit more cabling to deal with, but it would keep the sides clear for the nodes to be pulled without messing around with getting the PDU cables out of the way. Think W for the cable bundles one left and one right.
EDIT: Make sure the PDU cables support a native locking mechanism at the PDU side at the very least.
-
@phlipelder said in Small colo infrastructure - rack layout feedback:
@pete-s If the runs are not fibre look into 10GbE certified ultra-thin patch cables. We've started using them for all of our data centre deployments as they save a huge amount of space. There's some really good but expensive VELCRO rolls for tying things up. We've picked up a box or two of VELCRO thin and wide plant ties each. Same stuff as the computer ones in black but a tenth of the price. So what if they're green. ;0)
PDU cables rated for 240V are freaking huge and a bear to manage. I'd bundle and run them straight down the middle then to the sizes and up to their position on the PDUs L/R. That's a bit more cabling to deal with, but it would keep the sides clear for the nodes to be pulled without messing around with getting the PDU cables out of the way. Think W for the cable bundles one left and one right.
EDIT: Make sure the PDU cables support a native locking mechanism at the PDU side at the very least.
Great info! I didn't know about the thin patch cables but they look great.
I'll think about what to do with the PDU cables though. Power supply is actually redundant for both nodes so it's not a big problem to unplug one cable to pull out a node and then put it back.
-
@gjacobse said in Small colo infrastructure - rack layout feedback:
@jimmy9008 said in Small colo infrastructure - rack layout feedback:
@pete-s
No in rack UPS?not really needed in a Colo - as it will have (well it should anyway) Dual sourced feeds, and full facility UPS / Isolation / Generator.
Had an issue a couple times, several years ago, at a colo in Los Angeles. Redundant power systems failed and the generators didn't work. Yay!! So, I guess it depends on your budget and how much up-time you really need.
-
@wrx7m said in Small colo infrastructure - rack layout feedback:
@gjacobse said in Small colo infrastructure - rack layout feedback:
@jimmy9008 said in Small colo infrastructure - rack layout feedback:
@pete-s
No in rack UPS?not really needed in a Colo - as it will have (well it should anyway) Dual sourced feeds, and full facility UPS / Isolation / Generator.
Had an issue a couple times, several years ago, at a colo in Los Angeles. Redundant power systems failed and the generators didn't work. Yay!! So, I guess it depends on your budget and how much up-time you really need.
Was it a Tier IV?
-
@pete-s said in Small colo infrastructure - rack layout feedback:
@phlipelder said in Small colo infrastructure - rack layout feedback:
@pete-s If the runs are not fibre look into 10GbE certified ultra-thin patch cables. We've started using them for all of our data centre deployments as they save a huge amount of space. There's some really good but expensive VELCRO rolls for tying things up. We've picked up a box or two of VELCRO thin and wide plant ties each. Same stuff as the computer ones in black but a tenth of the price. So what if they're green. ;0)
PDU cables rated for 240V are freaking huge and a bear to manage. I'd bundle and run them straight down the middle then to the sizes and up to their position on the PDUs L/R. That's a bit more cabling to deal with, but it would keep the sides clear for the nodes to be pulled without messing around with getting the PDU cables out of the way. Think W for the cable bundles one left and one right.
EDIT: Make sure the PDU cables support a native locking mechanism at the PDU side at the very least.
Great info! I didn't know about the thin patch cables but they look great.
I'll think about what to do with the PDU cables though. Power supply is actually redundant for both nodes so it's not a big problem to unplug one cable to pull out a node and then put it back.
I'm of the mind that if I can do something so as to not reduce redundancy on a setup I'll take that route.
One nice thing about the heavy gauge PDU cables is they tend to hold the bend. So, I'd L bend the cable at the PSU leaving the run about 2" to 2.5" away from the PSU to allow for that time when one would need to be replaced.
I've had to work on rack setups where the folks that wired it up didn't really think things through including myself when setting them up for the first few run throughs.
Simplicity and ease of access are the two primary guiding principles. Oh, and a nice tidy set of runs too.
-
@wrx7m said in Small colo infrastructure - rack layout feedback:
@gjacobse said in Small colo infrastructure - rack layout feedback:
@jimmy9008 said in Small colo infrastructure - rack layout feedback:
@pete-s
No in rack UPS?not really needed in a Colo - as it will have (well it should anyway) Dual sourced feeds, and full facility UPS / Isolation / Generator.
Had an issue a couple times, several years ago, at a colo in Los Angeles. Redundant power systems failed and the generators didn't work. Yay!! So, I guess it depends on your budget and how much up-time you really need.
I find that data centre setups tend to skimp on the power side. We find out really quick that there's a problem when there's a cascading failure that eventually wipes the whole DC out.
Such as an ATS (Automatic Transfer Switch) failure and literally blows out taking the entire DC with it.
Another was the recent Azure outage caused by lightening striking a circuit that must not have been properly protected that in turn took out the A/C then boom. This one is actually a lot more common if one pays attention to outage news.
SuperNap (Switch) in Vegas is Tier IV Platinum with true A/B/C circuits throughout the facility. All three circuits are in different parts of the DC building they feed.
We have a Tier III here in the Edmonton region that can't get Tier IV because we only have one power source in this province. They have a similar setup to Switch though only A/B with isolated circuits all the way through to the multiple N+1 generators outside.
If it's too good to be true price wise there is a very good reason why that is so.
