1.
아주 오랜 동안 네트워크카드에 대한 생각은 변함이 없었습니다. Chelsio와 Solarplare중 하나를 선택하는 문제였고 Jitter로 얻는 이익이 Jitter를 버리고 Latency를 얻는 것보다 좋다고 생각했습니다. 그러다 최근 다른 선택을 했습니다. 물론 고객의 요청이었지만 오랜만에 자료를 보면서 생각을 바꿨습니다. 우선 Chelsio가 판매하는 제품과 공급하는 소프트웨어 드라이버의 버전이 T6인데 연식이 좀 되었습니다. Solarplare를 보니까 그 사이 Xilinx와 M&A를 했네요. OpenOnLoad도 그대로입니다. 오버클락서버를 도입할 때 검토를 했지만 요구하는 Core를 할당할 수 없어서 다른 선택을 했습니다. 이번에 선택한 제품은 Mellanox입니다. Intel과 함게 10G Ethernet Card를 판매하지만 별다른 관심을 가지지 않았는데 다른 선택을 하는 계기가 있었습니다. Mellanox Messaging Accelerator (VMA)때문입니다. Mellanox는 Nvidia가 인수를 하였네요.
TCP Offload와 관련하여 다양한 기술이 있습니다. Intel의 DPDK,Solarflare의 Openonload 그리고 Mellanox의 VMA입니다. 시간을 조금더 올라가면 Myricom도 있습니다.
To better understand what Google acquired, we need to roll back to 2009 when Myricom had ported its MX communications stack to Ethernet. Mx over Ethernet (MXoE) was originally crafted for the High-Performance Computing (HPC) market, to create a very limited, but extremely low latency, stack for their then two-year-old 10GbE NICs. MXoE was reformulated using UDP instead of MX, and this new low latency driver was then named DBL. It was then engineered to service the High-Frequency Traders (HFT) on Wall Street. Throughout 2010 Myricom had evolved this stack by further adding limited TCP functionality. At that time, half round trip performance (one send plus one receive) over 10GbE was averaging 10-15 microseconds, DBL did it 4 microseconds. Today (2015) by comparison Solarflare, the leader in ultra-low latency network adapters, does this in well under 2 microseconds. So in 2012, Google was searching for a method to apply ultra-low latency networking tricks to a new technology called Memcache. In the fall of 2012 Google invited several Myricom engineers to discuss how DBL might be altered to service Memcache directly. Also they were interested in known if Myricom’s new silicon, due out in early 2013, might also be a good fit for this application. By March of 2013 Google had tendered an offer to Myricom to acquire their latest 10/40GbE chip, which was about to go into production, along with 12 of their PhDs who handled both the hardware & software architecture. Little is publicly known if that chip or the underlying accelerated driver layer for Memcache has ever made it into production at Google.
Fast forward to today (2015), and earlier this week, Solarflare released a new whitepaper highlighting how they’ve accelerated the publicly available free open-source layer called Memcached using their ultra-low latency driver layer called OpenOnload. In this whitepaper, Solarflare demonstrated performance gains of 2-3 times that of a similar Intel 10GbE adapter. Imagine Google’s Memcache farm being only 1/3 the size it is today? We’re talking serious performance gains here. For example, leveraging a 20 core server, the Intel dual 10GbE adapter supported 7.4 million multiple get operations while Solarflare provided 21.9 million, nearly a 200% increase in the number of requests. If we look at mixed throughput (get/set in a ratio of 9:1), Intel delivered 6.3 million operations per second while Solarflare delivered 13.3 million, a 110% gain. That’s throughput, how about latency performance? Using all 20 cores, and batches of 48 get requests Solarflare clocked in at 2,000 microseconds, and Intel at 6,000 microseconds. Across all latency tests, Solarflare reduced network latency by, on average, 69.4% (the lowest reduction was 50%, and the highest 85%). Here is a link to the 10-page Solarflare whitepaper with all the details.
Google, Memcache, and How Solarflare May Have Come Out on Top중에서
Solarflare와 VMA를 비교하면 어떤 제품이 좀더 좋은 성능을 낼까요? 보고서마다 다른 결과를 보여주비만 828ns – A Legacy of Low Latency의 결과는 아래입니다. 숫자가 조금 나쁘지만 Mellanox를 선택한 이유는 외국인투자자의 조언이었습니다. 물론 고객이 전한 내용입니다.
About five years ago, Solarflare saw an opportunity to revisit TCP/UDP networking stacks within Onload and determined that it is possible to squeeze another 35-50% in performance gains if developers were willing to use a new C language application programming interface (API). This new API was built from the ground up focused on performance, and it implements only a subset of the complete BSD Sockets API. Every API call has been highly tuned to deliver optimum performance. On the road to formulating this API Solarflare has patented several new innovations, and in 2016 it leaped forward again by introducing this API and branding it TCPDirect. Initially, TCPDirect improved latency on Solarflare’s SFN8522 adapter by an astonishing 38%!
Recently TCPDirect was tested with the Solarflare’s latest X2522 cards, and it delivered an improved 48% latency reduction over Onload on the same adapter (click on the graph below). Today TCPDirect with the X2522 provides an amazing 828ns of latency with TCP. So how does this compare with Mellanox? The X2522 with TCPDirect is 39% faster than the Mellanox ConnectX-5 with VMA and Exasock! This gain is shown in the graph below. It should be noted that this testing was done using an older more performant Intel Skylake processor with a 3.6Ghz clock. Intel’s newest Cascade Lake processors burst up to 4.4Ghz, but they were not available at the time of this testing. Recent testing indicates that they should produce even more impressive results.
828ns – A Legacy of Low Latency중에서
2.
Mellanox Ethenet Card를 설치하고 VMA를 설정하는 부분은 다른 글에서 정리하겠습니다. 오늘은 10G 카드를 사용할 때 시간동기화를 위해 설치하는 PTP와 관련한 내용만 다루려고 합니다. PTP 설정과 관련한 자료를 넘칩니다. 어떤 자료를 읽더라도 큰 차이가 없습니다. 기본적인 줄기는 아래를 참고하였습니다.
Chapter 20. Configuring PTP Using ptp4l
문서를 읽어보면 제일 먼저 PTP를 지원하는 네트워크카드인지 확인하여야 합니다.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
[root@am0n mellanox]# ethtool -T enp4s0f0np0 Time stamping parameters for enp4s0f0np0: Capabilities: hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE) hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE) hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE) PTP Hardware Clock: 1 Hardware Transmit Timestamp Modes: off (HWTSTAMP_TX_OFF) on (HWTSTAMP_TX_ON) Hardware Receive Filter Modes: none (HWTSTAMP_FILTER_NONE) all (HWTSTAMP_FILTER_ALL) [root@am0n mellanox]# ethtool -T enp12s0f1np1 Time stamping parameters for enp12s0f1np1: Capabilities: hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE) hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE) hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE) PTP Hardware Clock: 4 Hardware Transmit Timestamp Modes: off (HWTSTAMP_TX_OFF) on (HWTSTAMP_TX_ON) Hardware Receive Filter Modes: none (HWTSTAMP_FILTER_NONE) all (HWTSTAMP_FILTER_ALL) |
다음은 ptp4l을 설치합니다. 인터넷에 접속하기 힘든 조건이라고 하면 rpm 파일을 받아서 설치해도 무방합니다. 보통 Dependecy 때문에 설치에 어려움을 겪지만 ptp4l를 그런 문제가 없었습니다. ptp4l와 phc2sys을 설치합니다. 이제 ptp4l을 이용하여 clock source와 통신
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
root@am0n mellanox]# ptp4l -i enp12s0f1np1 --step_threshold=0.00002 -m ptp4l[648037.105]: selected /dev/ptp4 as PTP clock ptp4l[648037.105]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[648037.105]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[648038.552]: port 1: new foreign master 001c73.ffff.1f6520-14 ptp4l[648042.552]: selected best master clock 00b0ae.fffe.0422b5 ptp4l[648042.552]: updating UTC offset to 37 ptp4l[648042.552]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[648043.277]: port 1: minimum delay request interval 2^5 ptp4l[648069.468]: master offset -14431607809186 s0 freq +0 path delay -484 ptp4l[648070.468]: master offset -14431607807013 s1 freq +2173 path delay -484 ptp4l[648071.468]: master offset 65 s2 freq +2238 path delay -484 ptp4l[648071.468]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[648072.468]: master offset -100 s2 freq +2092 path delay -484 ptp4l[648073.468]: master offset -59 s2 freq +2103 path delay -484 ptp4l[648074.468]: master offset 40 s2 freq +2185 path delay -484 ptp4l[648075.469]: master offset 20 s2 freq +2177 path delay -484 ptp4l[648076.468]: master offset 19 s2 freq +2182 path delay -484 ptp4l[648077.468]: master offset 6 s2 freq +2174 path delay -484 ptp4l[648078.468]: master offset 5 s2 freq +2175 path delay -484 ptp4l[648079.468]: master offset 3 s2 freq +2175 path delay -484 ptp4l[648080.468]: master offset 53 s2 freq +2226 path delay -484 ptp4l[648081.468]: master offset -29 s2 freq +2160 path delay -484 ptp4l[648082.468]: master offset 7 s2 freq +2187 path delay -484 ptp4l[648083.468]: master offset 39 s2 freq +2221 path delay -484 |
또다른 포트의 결과입니다.
|
1 2 3 4 5 6 7 8 |
[root@am0n ~]# ptp4l -i enp12s0f0np0 --step_threshold=0.00002 -m ptp4l[757896.605]: selected /dev/ptp3 as PTP clock ptp4l[757896.606]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[757896.606]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[757896.606]: port 1: link down ptp4l[757896.606]: port 1: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED) ptp4l[757896.606]: selected local clock 1c34da.fffe.656094 as best master ptp4l[757896.606]: assuming the grand master role |
같은 서버에서 다른 포트를 대상으로 실행한 결과입니다. 다릅니다. 다른 부분을 보시면..
selected best master clock 00b0ae.fffe.0422b5
selected local clock 1c34da.fffe.656094 as best master
절대 시간 정보를 제공하는 그랜드 마스터 클럭이 있을 경우 Master로 동작하게 되고, 만약 그랜드 마스터 클럭이 없을 경우에는 마스터가 자동으로 선정합니다. 이를 BMC(Best Master Clock 알고리즘)라고 하네요. 이 때문에 서로 다른 결과가 나왔습니다. 첫번째 경우처럼 결과가 나오면 행복합니다. 어떤 경우 몇 일동안 이런 저런 시험을 했는데 두번째 결과만 보인 경우도 있습니다. 분명히 스위치 장비에서 Master Clock이 있다고 하였는데도…
3.
이제 시간을 동기화하죠. 문서로 보면 ‘Synchronizing the Clocks’라는 주제입니다. 문서를 보면 phc2sys라는 명령어를 사용합니다.
The phc2sys program is used to synchronize the system clock to the PTP hardware clock (PHC) on the NIC. The phc2sys service is configured in the /etc/sysconfig/phc2sys configuration file.
저도 같은 방법으로 설정을 했습니다. 그런데 이상하게 시간이 지나면서 표준시간과 비교하여 벌어집니다. 무슨 일일지 이것저것 찾아보았습니다. 해답을 아래에서 찾았습니다.
위의 그림을 이렇게 이해하여야 합니다. 시간을 동기화하려면 두가지 방법이 가능합니다.
1. HW time stamping with PHC
2. SW time stamping with the system clock (CLOCK_REALTIME)
phc2sys를 사용하는 때는 1번인 경우입니다. ptp4l의 옵션을 보면 Hardware timt Stamping과 Software Time Stamping이 가능합니다. 저는 두번째를 사용하였고 이중으로 시간을 동기화하면서 문제가 발생한 것이었습니다. 이상은 Chelsio를 이용하여 설정할 때 경험입니다. 이 때 명령어는 아래입니다. Mellanox와 비교하면 -S 이 빠졌습니다. -S는 Software Time Stamping을 말합니다.
ptp4l -i enp5s0f4 -S –step_threshold=0.00002 -m
그러면 step_threshold는 무슨 의미일까요?
Time Jumps
When a jump in time occurs in the gPTP domain, the System clock can take a considerable amount of time to converge to the new time. This happens because the clock is synchronized by adjusting the frequency. For example, if the PHC time jumps by 1 second, empirical tests have shown that System clock can take up to 30 seconds to synchronize (considering offset less than 100 ns). The phc2sys daemon provides the –step_threshold=n option which sets a threshold. If the time jump is greater than n seconds, time is adjusted by stepping the clock (that means to adjust current time) instead of changing the frequency.
However, stepping the clock has its own downsides as well. All timers set to expire between the current time and the new time expire once the time is set. This can affect the real-time behavior of the systems. So, use clock stepping carefully.
Synchronizing Time with Linux* PTP중에서
