Clear Linux와 Mellanox

1.
Chelsio와 Solarflare와 비교할 때 Mellanox가 TCP Offload를 구현하는 방식은 다릅니다. RDMA방식의 TCP Bypass인 Mellanox VMA 설치와 설정에서 소개한 바와 같이 RDMA입니다. Chelsio은 커널모듈을 이용하여 TOE를 구현하지만 리눅스커널에 기본으로 들어가는 모듈이 아닙니다. 비표준적이라는 의미입니다. Solarflare의 소스코드를 세밀히 살피지 않았지만 Solarflare도 역시 OpenOnload를 이용하지만 비표준적인 방식입니다. 직접 시험을 해보지 않았지만 Chelsio와 비슷할 듯 합니다.

Support for SolarFlare Network Interface Cards이 소개한 결과를 보면 예상대로입니다.

여기서 중요한 차이가 발생합니다. Melaanox가 구현한 RDMA는 표준으로 OFED (Open Fabric Enterprise Distirbution)가 커널모듈입니다. OFED in Linux는 다음과 같이 소개합니다.

OFED™, (OpenFabrics Enterprise Distribution) is open-source software for RDMA and kernel bypass applications. OFED can be used in business, research and scientific environments that require highly efficient networks, storage connectivity and parallel computing. It is an OpenFabrics distribution of the RDMA/Advanced Networks code base. The majority of the code is pulled from https://github.com/linux-rdma and https://git.kernel.org. That code is then sometimes enhanced with new modifications being tested by vendors, and/or back ported patches.

2.
이상과 같이 판단에 따라 Mellanox 카드를 설치하기 위해 사용한 Clear Linux는 최신버전입니다. 별도의 작업이 필요없을 것이라 판단하였습니다. Clear Linux를 설치한 후 아래와 같은 명령어를 실행하였습니다.

smallake@as11~ $ lspci

01:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
01:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
02:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
02:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

어떤 드라이버가 설치되었는지를 확인하였습니다. mlx5_core가 Infiniband와 Ethernet를 지원하기 위한 드라이버입니다. 별도의 설치작업없이 OS설치만으로 동작합니다.

smallake@as11~ $ lsmod | grep mlx
mlx5_ib 360448 0
ib_uverbs 151552 2 rdma_ucm,mlx5_ib
ib_core 368640 10 rdma_cm,ib_ipoib,rpcrdma,iw_cm,ib_iser,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
mlx5_core 1482752 1 mlx5_ib
mlxfw 32768 1 mlx5_core
psample 20480 1 mlx5_core
tls 106496 1 mlx5_core
pci_hyperv_intf 16384 1 mlx5_core

이제 가속기능입니다. Chelsio는 T4_TOM이 가속기능을 담당하고 Solarflare는 OpenOnload입니다. Mellanos는 VMA입니다. 설치과정은 무척이나 단순합니다. 설치가이드에 나온 내용대로 합니다.

####Build binary/shared-object####

./autogen.sh
./configure –with-ofed=/usr –prefix=/usr –libdir=/usr/lib64 –includedir=/usr/include –docdir=/usr/share/doc/libvma –sysconfdir=/etc
make
sudo make install

설치를 한 후 시험을 하기 위하여 sockperf를 설치한 후 아래와 같이 명령어를 실행한 결과입니다.

zeroaos@as11~/sockperf (sockperf_v2) $ LD_PRELOAD=libvma.so sockperf sr -i 121.166.192.77 --tcp
 VMA INFO: ---------------------------------------------------------------------------
 VMA INFO: VMA_VERSION: 9.5.0-0 Development Snapshot built on Mar  8 2022 17:17:42
 VMA INFO: Git: 857ccb6abcea64a1272c4347bcb6922541ebbcc9
 VMA INFO: Cmd Line: sockperf sr -i 121.166.192.77 --tcp
 VMA INFO: Current Time: Thu Mar 10 10:01:54 2022
 VMA INFO: Pid:  7513
 VMA INFO: Architecture: x86_64
 VMA INFO: Node: as11
 VMA INFO: ---------------------------------------------------------------------------
 VMA INFO: Log Level                      INFO                       [VMA_TRACELEVEL]
 VMA INFO: ---------------------------------------------------------------------------
 VMA WARNING: ************************************************************************
 VMA WARNING: Your current max locked memory is: 8388608. Please change it to unlimited.
 VMA WARNING: Set this user's default to `ulimit -l unlimited`.
 VMA WARNING: Read more about this topic in the VMA's User Manual.
 VMA WARNING: ************************************************************************
 VMA WARNING: **************************************************************
 VMA WARNING: * NO IMMEDIATE ACTION NEEDED!                                 
 VMA WARNING: * Not enough hugepage resources for VMA memory allocation.    
 VMA WARNING: * VMA will continue working with regular memory allocation.   
 VMA INFO:    * Optional:                                                   
 VMA INFO:    *   1. Switch to a different memory allocation type           
 VMA INFO:    *      (VMA_MEM_ALLOC_TYPE!= 2)                                              
 VMA INFO:    *   2. Restart process after increasing the number of         
 VMA INFO:    *      hugepages resources in the system:                     
 VMA INFO:    *      "echo 1000000000 > /proc/sys/kernel/shmmax"          
 VMA INFO:    *      "echo 800 > /proc/sys/vm/nr_hugepages"               
 VMA WARNING: * Please refer to the memory allocation section in the VMA's  
 VMA WARNING: * User Manual for more information                            
 VMA WARNING: **************************************************************
 VMA ERROR: ibch421:mem_reg() failed registering a memory region (errno=12 Cannot allocate memory)
 VMA WARNING: allocator[0x55d8a3f7d2c0]:300:register_memory() Failure during memory registration on dev: rocep2s0f1 addr=0x7fc6cbfe9000 length=67200063
 VMA WARNING: allocator[0x55d8a3f7d2c0]:337:register_memory() Failed registering memory, This might happen due to low MTT entries. Please refer to README.txt for more info
 VMA ERROR: socket_internal vma failed to start errno: Cannot allocate memory
sockperf: ERROR: socket(AF_INET4/6, SOCK_x) (errno=12 Cannot allocate memory)
sockperf: ERROR: program exits because of an error. (errno=12 Cannot allocate memory)

zeroaos@as11~/sockperf (sockperf_v2) $ LD_PRELOAD=libvma.so sockperf sr -i 121.166.192.77 --tcp

VMA INFO: ---------------------------------------------------------------------------

VMA INFO: VMA_VERSION: 9.5.0-0 Development Snapshot built on Mar 8 2022 17:17:42

VMA INFO: Git: 857ccb6abcea64a1272c4347bcb6922541ebbcc9

VMA INFO: Cmd Line: sockperf sr -i 121.166.192.77 --tcp

VMA INFO: Current Time: Thu Mar 10 10:01:54 2022

VMA INFO: Pid: 7513

VMA INFO: Architecture: x86_64

VMA INFO: Node: as11

VMA INFO: ---------------------------------------------------------------------------

VMA INFO: Log Level INFO [VMA_TRACELEVEL]

VMA INFO: ---------------------------------------------------------------------------

VMA WARNING: ************************************************************************

VMA WARNING: Your current max locked memory is: 8388608. Please change it to unlimited.

VMA WARNING: Set this user's default to `ulimit -l unlimited`.

VMA WARNING: Read more about this topic in the VMA's User Manual.

VMA WARNING: ************************************************************************

VMA WARNING: **************************************************************

VMA WARNING: * NO IMMEDIATE ACTION NEEDED!

VMA WARNING: * Not enough hugepage resources for VMA memory allocation.

VMA WARNING: * VMA will continue working with regular memory allocation.

VMA INFO: * Optional:

VMA INFO: * 1. Switch to a different memory allocation type

VMA INFO: * (VMA_MEM_ALLOC_TYPE!= 2)

VMA INFO: * 2. Restart process after increasing the number of

VMA INFO: * hugepages resources in the system:

VMA INFO: * "echo 1000000000 > /proc/sys/kernel/shmmax"

VMA INFO: * "echo 800 > /proc/sys/vm/nr_hugepages"

VMA WARNING: * Please refer to the memory allocation section in the VMA's

VMA WARNING: * User Manual for more information

VMA WARNING: **************************************************************

VMA ERROR: ibch421:mem_reg() failed registering a memory region (errno=12 Cannot allocate memory)

VMA WARNING: allocator[0x55d8a3f7d2c0]:300:register_memory() Failure during memory registration on dev: rocep2s0f1 addr=0x7fc6cbfe9000 length=67200063

VMA WARNING: allocator[0x55d8a3f7d2c0]:337:register_memory() Failed registering memory, This might happen due to low MTT entries. Please refer to README.txt for more info

VMA ERROR: socket_internal vma failed to start errno: Cannot allocate memory

sockperf: ERROR: socket(AF_INET4/6, SOCK_x) (errno=12 Cannot allocate memory)

sockperf: ERROR: program exits because of an error. (errno=12 Cannot allocate memory)

무언가 심각해 보입니다만 Warning을 자세히 보면 해결책을 안내해줍니다.

ulimit -l unlimited

위 명령을 실행한 후 결과입니다.

root@as11/home/zeroaos # LD_PRELOAD=libvma.so sockperf sr -i 21.166.192.77 --tcp
 VMA INFO: ---------------------------------------------------------------------------
 VMA INFO: VMA_VERSION: 9.5.0-0 Development Snapshot built on Mar  8 2022 17:17:42
 VMA INFO: Git: 857ccb6abcea64a1272c4347bcb6922541ebbcc9
 VMA INFO: Cmd Line: sockperf sr -i 21.166.192.77 --tcp
 VMA INFO: Current Time: Thu Mar 10 10:17:38 2022
 VMA INFO: Pid:   658
 VMA INFO: Architecture: x86_64
 VMA INFO: Node: as11
 VMA INFO: ---------------------------------------------------------------------------
 VMA INFO: Log Level                      INFO                       [VMA_TRACELEVEL]
 VMA INFO: ---------------------------------------------------------------------------
 VMA WARNING: **************************************************************
 VMA WARNING: * NO IMMEDIATE ACTION NEEDED!                                 
 VMA WARNING: * Not enough hugepage resources for VMA memory allocation.    
 VMA WARNING: * VMA will continue working with regular memory allocation.   
 VMA INFO:    * Optional:                                                   
 VMA INFO:    *   1. Switch to a different memory allocation type           
 VMA INFO:    *      (VMA_MEM_ALLOC_TYPE!= 2)                                              
 VMA INFO:    *   2. Restart process after increasing the number of         
 VMA INFO:    *      hugepages resources in the system:                     
 VMA INFO:    *      "echo 1000000000 > /proc/sys/kernel/shmmax"          
 VMA INFO:    *      "echo 800 > /proc/sys/vm/nr_hugepages"               
 VMA WARNING: * Please refer to the memory allocation section in the VMA's  
 VMA WARNING: * User Manual for more information                            
 VMA WARNING: **************************************************************
sockperf: == version #3.7-27.gitcfe4900d3847 == 
sockperf: [SERVER] listen on:
[ 0] IP = 121.166.192.77  PORT = 11111 # TCP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: [tid 682] using recvfrom() to block on socket(s)

root@as11/home/zeroaos # LD_PRELOAD=libvma.so sockperf sr -i 21.166.192.77 --tcp

VMA INFO: ---------------------------------------------------------------------------

VMA INFO: VMA_VERSION: 9.5.0-0 Development Snapshot built on Mar 8 2022 17:17:42

VMA INFO: Git: 857ccb6abcea64a1272c4347bcb6922541ebbcc9

VMA INFO: Cmd Line: sockperf sr -i 21.166.192.77 --tcp

VMA INFO: Current Time: Thu Mar 10 10:17:38 2022

VMA INFO: Pid: 658

VMA INFO: Architecture: x86_64

VMA INFO: Node: as11

VMA INFO: ---------------------------------------------------------------------------

VMA INFO: Log Level INFO [VMA_TRACELEVEL]

VMA INFO: ---------------------------------------------------------------------------

VMA WARNING: **************************************************************

VMA WARNING: * NO IMMEDIATE ACTION NEEDED!

VMA WARNING: * Not enough hugepage resources for VMA memory allocation.

VMA WARNING: * VMA will continue working with regular memory allocation.

VMA INFO: * Optional:

VMA INFO: * 1. Switch to a different memory allocation type

VMA INFO: * (VMA_MEM_ALLOC_TYPE!= 2)

VMA INFO: * 2. Restart process after increasing the number of

VMA INFO: * hugepages resources in the system:

VMA INFO: * "echo 1000000000 > /proc/sys/kernel/shmmax"

VMA INFO: * "echo 800 > /proc/sys/vm/nr_hugepages"

VMA WARNING: * Please refer to the memory allocation section in the VMA's

VMA WARNING: * User Manual for more information

VMA WARNING: **************************************************************

sockperf: == version #3.7-27.gitcfe4900d3847 ==

sockperf: [SERVER] listen on:

[ 0] IP = 121.166.192.77 PORT = 11111 # TCP

sockperf: Warmup stage (sending a few dummy messages)...

sockperf: [tid 682] using recvfrom() to block on socket(s)

이제 남은 일은 시험입니다. Intel CPU에 최적화한 OS 환경에서 Solarflare와 Chelsio에 비해 어떤 성능을 보일지를 확인하기 위함입니다.

이 글 공유하기:

Leave a Comment 응답 취소