Updated Ceph Recommendations
Periodically, it makes sense to revisit and update recommendations based on what we see in the field and what is happening with a project’s code base. This is one of those updates.
In the past, we have recommended the use of two physically separate network fabrics to help better manage the I/O paths and allow for the back end traffic to have a dedicated link without other contention. We are now changing that recommendation to reduce complexity and improve general availability. The new recommendation is as follows:
- Stack Top of Rack Network Switches
- Create 802.3ad (LACP) bonding groups across the switches to protect against switch failure
- Create a single bonded interface on the SUSE Enterprise Storage node
- Use mode 4 (802.3ad)
- Use VLANs to logically segment traffic
It is important to remember to architect the environment with more bandwidth for the cluster than the outbound links to the clients. This will help ensure the cluster has the bandwidth resources needed for replication, recovery, and other back end actions. If your switches support it, it might also be worth using QoS to ensure that the backend VLAN has higher performance priority than the client-facing VLAN.
You should still continue to utilize jumbo frames for the back-end, private network, and if possible, the public network. These larger frame sizes provide a significant boost in data payload and potential throughput to the storage. It is important to note that you should NOT mix frame sizes on the same VLAN or try to have a routing device convert from standard to jumbo MTUs, as this will quickly overrun the capability of processors on most switching infrastructure.
From a security perspective, the VLANs will control where the data goes, thus keeping those back end flows isolated to the storage networking switches.
This change will be reflected in the documentation for our next product release and is present in our current reference architectures that I discussed in a previous blog. For your reference, the updated papers are linked below.