Abdulkadir Celik, Abdelbaset Hamza, Basem Shihada, and Mohamed-Slim Alouini
Published: 25 Feb 2020
CTN Issue: February 2020
A note from the editor:
Data centers play an essential role in our modern life and the rise of Artificial Intelligence/Machine Learning and other emerging services and applications will drive data centers to perform at a new level. In this issue, Abdulkadir, Abdelbaset, Basem and Mohamed-Slim argue that wireless communications inside of the data center will be a critical new technology to allow data centers to scale. They present an overview of the wireless technologies being proposed, their integration with data centers using new topologies (herringbones and disco balls amongst others!!), and end with some key challenges. Enjoy this article and as usual let us have your comments or feedback.
Can Wireless Technologies Flex Today’s Data Centers?
Data Centers (DCs) are essential infrastructures to process and store massive amounts of data generated by emerging technological trends such as fifth generation (5G) networks, Internet of things, big data, cloud services, artificial intelligence, etc. The diverse quality of service (QoS) demands of these can be fulfilled, either by centralized mega DCs scaling up to hundreds of servers, or by a network of distributed micro DCs. In this respect, there is a dire need for novel networking solutions, architectural approaches, and communication technologies to provide high-speed low-latency interconnections within and between DCs. Therefore, a scalable, resilient, and sustainable DC network (DCN) architecture design is of the utmost importance to utilize network resources efficiently, minimize capital and operational expenditures, and adapt network topology to dynamically changing traffic patterns .
1. Chronicle DCN Problems: Oversubscriptions, Hotspots, and Cabling Complexity
Among many other architectures, technology giants construct DCNs in a hierarchical tree topology where servers are arranged in racks as shown in Figure 1. While intra-rack communication is realized by top-of-rack edge switches (ESs) in the lower layer, inter-rack communication is established by linking ESs with aggregate switches (ASs) in the middle layer and then connecting ASs with the core switches (CSs) at the top layer. All these connections are established via uniform and fixed capacity wires/fibers. However, this hierarchical topology yields a multi-root tree where more powerful links and switches are required in the branches nearer the top layer, which makes CSs the DCN bottleneck in heavy traffic conditions. Therefore, inter-rack communication observes a throughput much lower than the actual available bandwidth , a.k.a. oversubscription. Moreover, measurements and analysis of real-life DCN traffic characteristics reveal that some applications generate unpredictable traffic patterns and asymmetrical traffic distributions. This unbalanced traffic is mainly caused by hotspots that contain common data required by many ongoing jobs in several DCN entities. The DCN traces also show that only 60% of the edge and core links are active at a time and utilization of 95th percentile of aggregation links is below 10% . It is challenging for wired DCNs to conform with this unpredictable and unbalanced traffic because of the fixed hierarchical topology and the inflexible links with uniform capacity. Unfortunately, there is no practical way of preventing the performance degradation caused by these stochastic and unbalanced traffic conditions. Moreover, the cabling cost and complexity increase as the size of DCN scales up.
Wireless Data Centers: Towards Reconfigurable DCN Architectures
As a remedy, Wireless DCNs (WDCNs) have recently received attention to augment the aforementioned limitations of the wired DCs. Motivated by the fact that more than 95% of the entire DCN traffic is carried out by the top 10% largest flows, Skandula et. al. was first to propose mitigating hotspots by establishing wireless on-demand links in DCNs oversubscribed with such large flows . Wireless technologies can provide unique DCN attributes which is summarized as follows:
- Wiring a huge number of servers is a quite demanding engineering task which exacerbates as the DCN size scales up. Since wire deployment is usually an error-prone task, necessary maintenance and modifications also consumes considerable time and efforts. On the other hand, convenient plug-and-play wireless modules can alleviate wiring cost and complexity to a great extent.
- Wireless techniques may yield a more energy and bandwidth efficient DCN by eliminating the need for switches which are typically power-hungry, error-prone, and bandwidth limited.
- WDCNs can handle oversubscription and hotspots thanks to its flexibility and reconfigurability. Given an effective hardware topology, adapting the virtual topology as per the QoS demands and traffic loads also provides a higher throughput and efficient bandwidth utilization.
Nonetheless, there exist formidable challenges in design and provisioning of WDCNs, which varies with distinct characteristics of the underlying communication technology.
2. Proof-of-Concept Studies
With wireless data centers, the first thing come into mind is RF technologies, e.g., cellular, WIFI, Bluetooth, etc. As their interference footprint would scale up with the DC size, RF solutions with limited bandwidth cannot tackle the traffic volume and QoS requirements of DCNs. The solution of this problem is hidden in higher frequency directed communication techniques such as mmWave and free-space-optics (FSO). However, these technologies require a line-of-sight (LoS) link which necessitate a paradigm shift in current grid-based DCN hardware topology.
A. mmWave Concepts: How the oversubscription problem can be mitigated by on-demand wireless links, a.k.a Flyways, was first shown in . To this aim, authors deployed both 60 GHz and 802.11b/g/n modems with 7 GHz and 80 MHz bandwidths, respectively. Unlike the 802.11b/g/n, 60 GHz flyways were able to improve overall performance by 50%, depending on the number of flyways and their deployment strategy. 3DBeam concept took flyways a step further by using three-dimensional mmWave beamforming in  where the concept is realized by three hardware components: beamforming radios, ceiling reflectors to establish non-line-of-sight links, and electromagnetic absorbers to prevent local reflection and scatterings. Results showed that 3DBeam greatly expands the reach and capacity of 60 GHz links, making them feasible as flexible and reconfigurable alternatives to wired cabling.
B. FSO Concepts: The first proof-of-concept FSO based DC is presented in  where authors build a small form factor steerable FSO transceivers by using commodity components. Employing these FSO transceivers, they develop an inter-rack network solution, namely FireFly, where a ceiling mirror is preferred to establish line-of-sight FSO links between the racks. Noting that available outdoor FSO modules are bulky and expensive, authors were able to build FireFly system with low-cost commodity optical devices with high-precision and low-latency steering techniques. The experimental results show that FireFly can deliver desirable throughput and flow completion time performance comparable to Fat-Tree DCNs. Another interesting wireless DC, namely ProjecToR [c.f. Figure 2], is presented in  where authors use a combination of arrays of digital micromirror devices and a disco-ball shaped mirror assembly. ProjecToR is shown to provide seamless reconfigurability with high agility and fanout in comparison with Flyways, 3DBeam, and FireFly.
3. Recent Advances and Open Research Challenges
Physical Topology (PhyTop) Design: Although the grid-based rack arrangements of wired DCNs use the space efficiently, wireless DCN architecture must be designed to avoid LoS obstructions among the racks. Therefore, early works on wireless DCNs mostly focus on designing proper physical topologies. Shin et. al. proposed a novel Cayley DC (CDC) concept in  where cylindrical racks have containers each of which encapsulates a commodity blade server [c.f. Figure 3]. Each server is equipped with two 60-GHz transceivers; one for intra-rack and the other for inter-rack communications. Inspired by , the work in  use spherical racks for the sake of a higher fault tolerance of the intra-rack communications. However, it is expected to have practical ramifications in implementing such irregular shapes. Diamond DC (DDC) is another interesting PhyTop design which hybridizes the wired and 60-GHz wireless DCNs . DDC arranges and wires the racks in a herringbone pattern [c.f. Figure 3]. To prevent blocking and interference, the gap between the racks is used as a ring reflection space (RRS) by equipping the edges with the reflectors at each layer. Alternatively, OWCell uses the cell of racks as a building block to create mega DCNs . In order to facilitate LoS links, racks within a cell are arranged in regular polygonal shapes and fully connected to each other. Although OWCell has desirable scalability and connectivity, its space utilization is not as efficient as in  and . Even though these works propose interesting solutions to establish LoS links, they are not implemented in real-life to evaluate overall performance.
Virtual Topology (VirTop) Design: VirTop design is essential for leveraging the reconfigurability and flexibility of wireless DCNs to meet the dynamically changing QoS demands of network entities. In this respect, VirTop design involves many interesting and challenging open research areas such as resource allocation, scheduling, power control, beamforming, interference management, routing, etc. QoS provisioning is a critical requirement that must be guaranteed for highly diversified types of DCN traffic. Therefore, it is important to distinguish flows based on size, completion time requests, priority, etc. DC flows are typically classified as bandwidth-hungry elephant flows (EFs) or delay-sensitive mice flows (MFs). While the majority of the traffic volume is carried out by EFs, the majority of flows are typically MFs. Among the MFs, there might be higher priority critical flows with ultra-reliable and low-latency service demands. Thus, VirTop should be designed to distinguish and treat different types of flows based on their needs. For instance, MFs may experience intolerable delays if they are routed along with EFs on the same path. Hence, flow detection mechanism should be rapid, precise, and low-overhead to prevent such performance degradations.
To this end, a distributed fast, lightweight, and accurate flow detection mechanism, namely LightFD, is developed in . By leveraging the TCP behavior, LightFD is designed as a module installed in the virtual-switches. However, distributed solutions are only suitable for modifiable switches with location privileges. Therefore, an alternative centralized scheme is also proposed such that a central unit reconfigures the virtual-switches by remotely communicating with switches via OpenFlow protocol. Extensive emulation results show that the accuracy of LightFD schemes outperforms the traditional methods. Moreover, the detection speed of the distributed and centralized schemes are in the order of milliseconds and ten milliseconds, respectively.
The complexity of VirTop design increases not only with the size but also with the massive and diverse amount of traffic generated across the DCN. By bundling and treating the same class of flows destined to a common destination, flow grooming (FG) can increase the bandwidth utilization and relax the complexity of the network management. In this direction, a three- step FG (3SFG) policy is developed for a two-layer wireless DCN topology in  where leaf and spine layers respectively comprise of ESs and CSs, which are interconnected via WDM-FSO links. In 3SFG, servers first groom MFs destined to the same server. Then, servers re-groom the flows in the first step intended to the same rack. Finally, ESs performs a final grooming to create rack-to-rack (R2R) jumbo flows. Since MFs forms the majority of the generated flows, 3SFG calculates predetermined R2R routing paths whose capacity is calculated based on the flow arrival statistics, flow size, and required flow completion duration. In this manner, a significant portion of flows is handled with a very low complexity by ensuring the QoS demands. The remaining wavelengths (i.e., channels or RBs) are exploited by EFs which are transferred over server-to-server (S2S) express routes. By routing MFs and EFs on disjoint paths, 3SFG prevents highly probable congestion occasions as delay sensitive MFs can be affected by bandwidth-hungry EFs easily. On top of this, 3SDG immediately serves mission critical flows over available S2S routes by interrupting the ongoing EFs. Noting the small size of the MFs and available large capacity, such an impulsive interruption does not harm the EF performance distinguishably. Numerical results show that 3SFG can significantly improve the throughput and flow completion time.
Security and Privacy: Nowadays it is a common practice for companies to rent a portion of mega DCNs in order to prevent maintaining and upgrading costs of a self-owned enterprise network. In this case, it is critical to isolate wirelessly transmitted data from unintended nodes in order to prevent security and privacy risks. Fortunately, the directivity and limited penetration of high-speed wireless links make it possible to partition the DCN via physical obstacles. Such an approach may enhance the immunity to eavesdropping, allow to develop low-overhead security protocols, and hence increase overall networking performance. Security and privacy are in the center of skepticism of the cloud service providers toward the WDCNs as they are legally responsible for protecting the sensitive information.
In this newsletter, we provide an overview of wireless DCNs for embracing the emerging information and communication technologies that require storage and processing of the highly diverse massive amount of data. After briefing and comparing the virtues and drawbacks of potential high-speed wireless technologies, we have presented advances and challenges of implementing WDCNs from both theoretical and practical points of views. Following the presentation of ongoing research efforts focused on PhyTop design, we also discussed overlooked aspects such as VirTop design, flow detection, flow grooming, and load balancing.
- A. Celik, B. Shihada, and M.-S. Alouini, “Wireless data center networks: Advances, challenges, and opportunities,” arXiv preprint arXiv:1811.11717, 2018. [Online]. Available: https://arxiv.org/abs/1811.11717
- S. Kandula, J. Padhye, V. Bahl, “Flyways to de-congest data center networks,” in Proc. ACM Hotnets-VIII, 2009, pp. 202–208.
- S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken, “The nature of data center traffic: measurements & analysis,” in Proc. ACM SIGCOMM, 2009.
- X. Zhou, Z. Zhang, Y. Zhu , Y. Li, S. Kumar , A. Vahdat, B.Y. Zhao, and H. Zheng, “Mirror mirror on the ceiling: Flexible wireless links for data centers,” in Proc. ACM SIGCOMM, 2012.
- N. Hamedazimi, Z. Qazi, H. Gupta, V. Sekar, S.R. Das, J.P. Longtin, H. Shah and A. Tanwer, "Firefly: A reconfigurable wireless data center fabric using free-space optics." in Proc. ACM SIGCOMM, 2014.
- M. Ghobadi, R. Mahajan, A. Phanishayee, N. Devanur, J. Kulkarni, G. Ranade, P.A. Blanche, H. Rastegarfar, M. Glick, and D. Kilper, "Projector: Agile reconfigurable data center interconnect." In Proc. ACM SIGCOMM, 2016.
- J.Y. Shin, E.G. Sirer, H. Weatherspoon and D. Kirovski, “On the feasibility of completely wireless datacenters,” IEEE Trans. Netw., vol. 21, no. 5, Oct. 2013.
- K. Suto, H. Nishiyama, N. Kato, T. Nakachi, T. Sakano, and A. Takahara, “A failure-tolerant and spectrum-efficient wireless data center network design for improving performance of big data mining,” in Proc. IEEE 81st Veh. Technol. Conf. (VTC), May. 2015, pp. 1–5.
- Y. Cui, S Xiao, X. Wang, Z. Yang, S. Yan, C. Zhu, X.Y. Li, and N. Ge, “Diamond: Nesting the data center network with wireless rings in 3-D space,” IEEE Trans. Netw., vol. 26, no. 1, pp. 145–160, Feb. 2018.
- A. S. Hamza, S. Yadav, S. Ketan, J.S. Deogun, and D.R. Alexander, “OWCell: Optical wireless cellular data center network architecture,” in Proc. IEEE Intl. Conf. Commun. (ICC), May 2017.
- A. Al-Ghadhban, A. Celik, B. Shihada, and M.-S. Alouini, ''LightFDG: An Integrated Flow Detection and Grooming Approach for Optical Wireless Data Centers,'' IEEE Trans. Netw. Service Manag., in press.
- A. Celik, A. Al-Ghadhban, B. Shihada, and M.-S. Alouini, "Design and Provision of Traffic Grooming for Optical Wireless Data Center Networks", IEEE Trans. Commun., vol. 67, no. 3, pp. 2245-2259, Mar. 2019.
Statements and opinions given in a work published by the IEEE or the IEEE Communications Society are the expressions of the author(s). Responsibility for the content of published articles rests upon the authors(s), not IEEE nor the IEEE Communications Society.