Clear Linux와 Chelsio 하나

1.
아마 Linux계열의 OS를 사용하는 트레이더들이라고 하면 대부분 RHES 혹은 Centos를 사용할 듯 합니다. 현재 ZeroAOS를 운용하는 OS도 Centos입니다. 물론 고객이 원하는 RedHat를 설치하기도 합니다. 속도에 민감한 전략을 운용하는 고객들의 특성상 “가능하면 빠른” 시스템을 원합니다. 소프트웨어 및 하드웨어 모두를 포함하여 그렇습니다. 이 때문에 이제는 오버클락서버가 일반적입니다. 재무적인 여력이 있으면 해외에서 수입한 서버를, 없으면 개별적으로 오버클락을 한 서버를 이용합니다.

저의 고민은 “하드웨어가 아닌 영역에서 할 수 있는 일이 무엇일까”에서 출발하였습니다. 필요에 의해 어플리케이션 구조를 변경하는 일을 합니다. 여기에 더하여 무엇을 해보자, 이런 고민에서 출발한 것이 Clear Linux입니다. Intel CPU에 최적화한 어플리케이션을 위해 Intel Compiler를 고민하다가 아예 OS까지 바꾸자는 취지였습니다.Clear Linux는 Intel이 배포하는 리눅스입니다. 여러가지 강조를 하지만 가장 눈에 들어온 부분은

Optimizated throughout the OS software stack.
Performance tuned for Intel Architecture.

어떤 분이 올린 글을 보면 꼭 Intel이 아니고 AMD라고 하더라도 좋은 성능을 보인다 합니다만 검증을 해보지 않았습니다.

Download (PDF, 9.02MB)

간접적으로 성능을 측정하여 올린 글들이 있습니다.

Clear Linux Squeezed Out More Open-Source Performance In 2020
Intel’s Clear Linux Still Outperforming Other Distributions For Mid-2020

위 글에 올라온 결과중 하나입니다.

또다른 성능자료는 Clear Linux 라이젠 서버에 설치해봤는데 성능이 거의 치트급이네요입니다. 라이제에 설치하여 Redis시험을 하였네요.

2.
우선 Clear Linux 배포판을 받습니다. 이 때 가장 고민하여야 하는 부분은 Clear Linux의 배포판은 숫자로 되어 있습니다. 최초 버전이 300이고 현재 버전은 35920입니다. 저도 처음에 최신판을 받아서 설치하였습니다. 그런데 문제가 발생하였습니다. 매매와 관련한 어플리케이션은 최신 커널에서도 동작하도록 관리하고 있지만 제3자가 제공하는 소프트웨어는 다릅니다. Low Latency를 고려하는 대부분 시스템은 TOE 혹은 가속기능을 가진 10G RNIC을 사용합니다. Centos나 OpenSUSE 혹은 Ubuntu의 경우 드라이버나 프로그램을 제공하지만 그렇지 않은 경우에는 소스를 이용하여 컴파일을 합니다. Solarflare, Mellanox VMA 혹은 Chelsio의 경우 소스를 컴파일을 할 때 커널 버전을 확인하여 관련한 Clear Linux을 받아야 합니다. 나중에 알았지만 최신 버전의 Clear Linux를 설치한 후 관련한 Kernel을 받아서 설치하는 방식도 가능합니다.

하여튼 약간의 노력을 한 후 Clear Linux를 설치하였습니다. 설치하는 과정은 무척 단순합니다. Linux를 설치해보신 분이면 생소하지만 익숙한 기능들입니다.

clear Linux를 설치한 후 로그인한 화면입니다. Server가 아니라 Desktop입니다.

Clear Linux는 패키지관리를 위한 명령어가 swupd입니다. 이를 이용하여 여러가지를 할 수 있습니다. 앞서 언급한 커널관리 및 설치와 관련하여 swupd를 이용하여 아래과 같이 설치하거나 설치가능한 패키지가 무엇인지를 확인할 수 있습니다.

Command line instructions

To add this bundle*, enter: sudo swupd bundle-add kernel-install

To search for bundles and their contents, enter: swupd search kernel-install

For additional swupd commands, enter: swupd —-help

앞서 Kernel설치가 자유롭다고 하였는데 이와 관련한 화면입니다. clr-boot-manager 명령어를 통하여 현재 설치된 커널을 확인하거나 부팅 커널등을 지정할 수 있습니다.

저도 생소한 OS라서 무척이나 고생하였스비다. Clear Linux와 관련한 정보를 얻을 수 있는 곳이 Community나 공식적인 Wiki말고는 없는 것도 어려움입니다. 어떤 패키지를 설치하여야 하는데 어떻게 해야할지 확인할 수 없었습니다. 이것저것 찾다가 보니까 swupd search-file 이라는 옵션이 있더군요. 아래가 관련 화면입니다.

3.
이상의 목적을 Chelsio, Solarflare 및 Mellanox의 성능을 각각 Clear Linux와 Centos에서 비교하는 것입니다. 이를 위해 두가지가 필요합니다. RNIC이 Clear Linux에 정상적으로 설치할 수 있는지, 두번째는 설치한 후 가속기능이 정상적으로 동작하는지를 확인하는 것입니다. 시험은 다음과 같이 진행하고 있습니다.

두대의 서버를 준비합니다. 한대는 Clear Linux와 Centos를 설치합니다. 우선 Clear Linux를 설치합니다. 나머지는 Rocky Linux를 설치한 서버입니다.
시험방식은 socperf를 이용하여 Rocky Linux와 Clear Linux/Centos사이의 성능을 측정합니다.

현재 서버에 설치한 RNIC은 Chelsio의 T4-CR입니다. Chelsio Driver를 설치하여야 하는데 소스를 이용하여 컴파일을 하여야 합니다. 소스는 Chelsio Download Center에서 받을 수 있습니다. 현재 제공하는 버전은 Chelsio Unified Wire v3.16.0.0 for Linux 입니다. 받아서 압축을 푼 후 컴파일을 시도합니다.

우선 build만 가능한지 확인하였습니다. 설치하려고 하는 드라이버가 offload기능이 있는 nic driver입니다. 아래는 컴파일한 결과입니다.

smallake@clearlinux~/Downloads/ChelsioUwire-3.16.0.0 $ sudo make toe_ipv4 
Building for kernel 5.15.2-1096.native
cat: /etc/issue: No such file or directory
cp: cannot stat '/boot/initrd-5.15.2-1096.native': No such file or directory

Building for kernel 5.15.2-1096.native
cat: /etc/issue: No such file or directory


##################################################
#   Building Offload Capable Network Drivers     #
##################################################
Makefile:117: *** NO utsrelease.  Stop.
##################################################
#      Building Network Drivers (offload)        #
##################################################
Makefile:117: *** NO utsrelease.  Stop.


***********************
*      Summary        *
***********************
CONFIG = UNIFIED_WIRE
Protocol                      Modules\Libraries\Tools       Action         Status    
------------------------------------------------------------------------------------------
Network(NIC_IPV4)             cxgb4                         Build          Failed

smallake@clearlinux~/Downloads/ChelsioUwire-3.16.0.0 $ sudo make toe_ipv4

Building for kernel 5.15.2-1096.native

cat: /etc/issue: No such file or directory

cp: cannot stat '/boot/initrd-5.15.2-1096.native': No such file or directory

Building for kernel 5.15.2-1096.native

cat: /etc/issue: No such file or directory

##################################################

# Building Offload Capable Network Drivers #

##################################################

Makefile:117: *** NO utsrelease. Stop.

##################################################

# Building Network Drivers (offload) #

##################################################

Makefile:117: *** NO utsrelease. Stop.

***********************

* Summary *

***********************

CONFIG = UNIFIED_WIRE

Protocol Modules\Libraries\Tools Action Status

------------------------------------------------------------------------------------------

Network(NIC_IPV4) cxgb4 Build Failed

Kernel 컴파일을 해 본적도 없어서 원인을 찾는데 고생하였습니다. kernel source를 swupd를 이용하여 설치하는 것부터 헤맸습니다.
이런 저런 시도끝의 결과입니다.

smallake@clearlinux~/Downloads/ChelsioUwire-3.16.0.0 $ sudo make toe_ipv4_install SKIP_RPM=1 
Building for kernel 5.15.2-1096.native
cat: /etc/issue: No such file or directory
cp: cannot stat '/boot/initrd-5.15.2-1096.native': No such file or directory

##################################################
#          Cleaning Source/Build/RPMDir          #
##################################################
Building for kernel 5.15.2-1096.native
cat: /etc/issue: No such file or directory


cat: /etc/issue: No such file or directory
Uninstalling all previously installed drivers/libs. This step may take some time.
  File "/home/smallake/Downloads/ChelsioUwire-3.16.0.0/scripts/uninstall.py", line 37
    fd = os.open ('uninstall.log',os.O_CREAT|os.O_WRONLY,0644)
                                                         ^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers
/boot/initrd-5.15.2-1096.native does not exist

Usage: lsinitrd [options] [<initramfs file> [<filename> [<filename> [...] ]]]
Usage: lsinitrd [options] -k <kernel version>

-h, --help                  print a help message and exit.
-s, --size                  sort the contents of the initramfs by size.
-m, --mod                   list modules.
-f, --file <filename>       print the contents of <filename>.
--unpack                    unpack the initramfs, instead of displaying the contents.
                            If optional filenames are given, will only unpack specified files,
                            else the whole image will be unpacked. Won't unpack anything from early cpio part.
--unpackearly               unpack the early microcode part of the initramfs.
                            Same as --unpack, but only unpack files from early cpio part.
-v, --verbose               unpack verbosely.
-k, --kver <kernel version> inspect the initramfs of <kernel version>.

Preparing Diags...
'/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/chelsio_adapter_config_v4/bin/t5seeprom_bins/t5seeprom_x86_64' -> '/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/chelsio_adapter_config_v4/bin/t5seeprom'
'/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/chelsio_adapter_config_v4/bin/t6seeprom_bins/t6seeprom_x86_64' -> '/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/chelsio_adapter_config_v4/bin/t6seeprom'
##################################################
# Building Chelsio-utils(tools)                  #
##################################################
#######################
####  cxgbtool/cop ####
#######################
cc -O2 -Wall -Werror     -c -o cxgbtool.o cxgbtool.c
cc   cxgbtool.o   -o cxgbtool
g++ -O2 -Wall -Werror -g -DUSE_STD_VECTOR    -c -o cop.o cop.cc
g++ -O2 -Wall -Werror -g -DUSE_STD_VECTOR    -c -o classifier.o classifier.cc
g++ -O2 -Wall -Werror -g -DUSE_STD_VECTOR    -c -o ipfilter.o ipfilter.cc
g++  -g -o cop cop.o classifier.o ipfilter.o
cc -O2 -Wall -Werror -g    -c -o opt.o opt.c
cc  -g -o opt opt.o
'/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/cudbg/bin/cudbg_app_x86_64' -> '/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/cudbg/app/cudbg_app'
##################################################
# Installing Chelsio-utils(tools)                #
##################################################
#######################
####  cxgbtool/cop ####
#######################
install -m 755 cxgbtool /sbin
gzip -c cxgbtool.8 > cxgbtool.8.gz
gzip -c cxgbtool-stor.8 > cxgbtool-stor.8.gz
make[2]: Nothing to be done for 'all'.
'/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/cudbg/app//cudbg_app' -> '/sbin/cudbg_app'
Copying VPD binaries to /lib/firmware/cxgb4/config   :  DONE
Copying scripts to /sbin   :  DONE
##################################################
#   Building Offload Capable Network Drivers     #
##################################################
Preparing for build ...
#### Preparing cxgb4 ####
make[3]: Nothing to be done for 'prep'.
#######################
####  cxgb4        ####
#######################
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20211116 releases/gcc-11.2.0-439-gde15d42914
  You are using:           gcc (Clear Linux OS for Intel Architecture) 11.2.1 20220215 releases/gcc-11.2.0-770-g2c9485a496
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_main.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/l2t.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/clip_tbl.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_ethtool.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_cxgbtool.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_filter.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_debugfs.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/t4_hw.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/t4_linux_debugfs.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/sge.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/trace.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/smt.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/srq.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_ptp.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_mps.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_cudbg.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/fastlz_api.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/fastlz.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_lib.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_wtp.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_flash_utils.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_common.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/ocqp.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_thermal.o
  LD [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4.o
  MODPOST /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/Module.symvers
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4.mod.o
  LD [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4.ko
##################################################
#    Installing Offload Capable Network Drivers  #
##################################################
Preparing for build ...
#### Preparing cxgb4 ####
make[3]: Nothing to be done for 'prep'.
#######################
####  cxgb4        ####
#######################
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20211116 releases/gcc-11.2.0-439-gde15d42914
  You are using:           gcc (Clear Linux OS for Intel Architecture) 11.2.1 20220215 releases/gcc-11.2.0-770-g2c9485a496
install: creating directory '/lib/modules/5.15.2-1096.native/updates'
install: creating directory '/lib/modules/5.15.2-1096.native/updates/drivers'
install: creating directory '/lib/modules/5.15.2-1096.native/updates/drivers/net'
install: creating directory '/lib/modules/5.15.2-1096.native/updates/drivers/net/cxgb4'
'cxgb4.ko' -> '/lib/modules/5.15.2-1096.native/updates/drivers/net/cxgb4/cxgb4.ko'
install -D -v -m 644 t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw-1.26.6.0.bin;
't4fw-1.26.6.0.bin' -> '/lib/firmware/cxgb4/t4fw-1.26.6.0.bin'
install -D -v -m 644 t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw-1.26.6.0.bin;
't5fw-1.26.6.0.bin' -> '/lib/firmware/cxgb4/t5fw-1.26.6.0.bin'
install -D -v -m 644 t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw-1.26.6.0.bin;
't6fw-1.26.6.0.bin' -> '/lib/firmware/cxgb4/t6fw-1.26.6.0.bin'
/bin/ln -sf /lib/firmware/cxgb4/t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw.bin
/bin/ln -sf /lib/firmware/cxgb4/t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw.bin
/bin/ln -sf /lib/firmware/cxgb4/t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw.bin
install -D -v -m 644 t4-config.txt /lib/firmware/cxgb4/t4-config.txt;
't4-config.txt' -> '/lib/firmware/cxgb4/t4-config.txt'
install -D -v -m 644 t5-config.txt /lib/firmware/cxgb4/t5-config.txt;
't5-config.txt' -> '/lib/firmware/cxgb4/t5-config.txt'
install -D -v -m 644 t6-config.txt /lib/firmware/cxgb4/t6-config.txt;
't6-config.txt' -> '/lib/firmware/cxgb4/t6-config.txt'
install -D -v -m 644 aq1202_fw.cld /lib/firmware/cxgb4/aq1202_fw.cld;
'aq1202_fw.cld' -> '/lib/firmware/cxgb4/aq1202_fw.cld'
install -D -v -m 644 bcm8483.bin /lib/firmware/cxgb4/bcm8483.bin;
'bcm8483.bin' -> '/lib/firmware/cxgb4/bcm8483.bin'
depmod
install -D  -m 644 t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw-1.26.6.0.bin;
install -D  -m 644 t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw-1.26.6.0.bin;
install -D  -m 644 t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw-1.26.6.0.bin;
/bin/ln -sf /lib/firmware/cxgb4/t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw.bin
/bin/ln -sf /lib/firmware/cxgb4/t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw.bin
/bin/ln -sf /lib/firmware/cxgb4/t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw.bin
install -D  -m 644 t4-config.txt /lib/firmware/cxgb4/t4-config.txt;
install -D  -m 644 t5-config.txt /lib/firmware/cxgb4/t5-config.txt;
install -D  -m 644 t6-config.txt /lib/firmware/cxgb4/t6-config.txt;
install -D  -m 644 aq1202_fw.cld /lib/firmware/cxgb4/aq1202_fw.cld;
install -D  -m 644 bcm8483.bin /lib/firmware/cxgb4/bcm8483.bin;
install -D  -m 644 t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw-1.26.6.0.bin;
install -D  -m 644 t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw-1.26.6.0.bin;
install -D  -m 644 t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw-1.26.6.0.bin;
/bin/ln -sf /lib/firmware/cxgb4/t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw.bin
/bin/ln -sf /lib/firmware/cxgb4/t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw.bin
/bin/ln -sf /lib/firmware/cxgb4/t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw.bin
install -D  -m 644 t4-config.txt /lib/firmware/cxgb4/t4-config.txt;
install -D  -m 644 t5-config.txt /lib/firmware/cxgb4/t5-config.txt;
install -D  -m 644 t6-config.txt /lib/firmware/cxgb4/t6-config.txt;
install -D  -m 644 aq1202_fw.cld /lib/firmware/cxgb4/aq1202_fw.cld;
install -D  -m 644 bcm8483.bin /lib/firmware/cxgb4/bcm8483.bin;
##################################################
#      Building Network Drivers (offload)        #
##################################################
Preparing for build ...
#### Preparing toecore ####
#### Preparing cxgb4 ####
make[3]: Nothing to be done for 'prep'.
#### Preparing t4_tom ####
#######################
####  toecore      ####
#######################
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20211116 releases/gcc-11.2.0-439-gde15d42914
  You are using:           gcc (Clear Linux OS for Intel Architecture) 11.2.1 20220215 releases/gcc-11.2.0-770-g2c9485a496
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/toecore/module_support.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/toecore/offload.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/toecore/toedev.o
  LD [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/toecore/toecore.o
  MODPOST /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/toecore/Module.symvers
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/toecore/toecore.mod.o
  LD [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/toecore/toecore.ko
#######################
####  cxgb4        ####
#######################
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20211116 releases/gcc-11.2.0-439-gde15d42914
  You are using:           gcc (Clear Linux OS for Intel Architecture) 11.2.1 20220215 releases/gcc-11.2.0-770-g2c9485a496
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_main.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/l2t.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/clip_tbl.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_ethtool.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_cxgbtool.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_filter.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_debugfs.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/t4_hw.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/t4_linux_debugfs.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/sge.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/trace.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/smt.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/srq.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_ptp.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_mps.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_cudbg.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/fastlz_api.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/fastlz.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_lib.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_wtp.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_flash_utils.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_common.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/ocqp.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_thermal.o
  LD [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4.o
  MODPOST /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/Module.symvers
  LD [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4.ko
#######################
####  t4_tom       ####
#######################
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20211116 releases/gcc-11.2.0-439-gde15d42914
  You are using:           gcc (Clear Linux OS for Intel Architecture) 11.2.1 20220215 releases/gcc-11.2.0-770-g2c9485a496
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/cpl_io.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/cpl_sock.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/listen.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/module_support.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/tom.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/t4_ddp.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/tom_sysctl.o
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/t4_tls.o
  LD [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/t4_tom.o
  MODPOST /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/Module.symvers
  CC [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/t4_tom.mod.o
  LD [M]  /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/t4_tom/t4_tom.ko
##################################################
#      Installing Network Drivers (offload)      #
##################################################
Preparing for build ...
#### Preparing toecore ####
#### Preparing cxgb4 ####
make[3]: Nothing to be done for 'prep'.
#### Preparing t4_tom ####
#######################
####  toecore      ####
#######################
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20211116 releases/gcc-11.2.0-439-gde15d42914
  You are using:           gcc (Clear Linux OS for Intel Architecture) 11.2.1 20220215 releases/gcc-11.2.0-770-g2c9485a496
#######################
####  cxgb4        ####
#######################
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20211116 releases/gcc-11.2.0-439-gde15d42914
  You are using:           gcc (Clear Linux OS for Intel Architecture) 11.2.1 20220215 releases/gcc-11.2.0-770-g2c9485a496
#######################
####  t4_tom       ####
#######################
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20211116 releases/gcc-11.2.0-439-gde15d42914
  You are using:           gcc (Clear Linux OS for Intel Architecture) 11.2.1 20220215 releases/gcc-11.2.0-770-g2c9485a496
install: creating directory '/lib/modules/5.15.2-1096.native/updates/drivers/net/offload'
'toecore.ko' -> '/lib/modules/5.15.2-1096.native/updates/drivers/net/offload/toecore.ko'
removed '/lib/modules/5.15.2-1096.native/updates/drivers/net/cxgb4/cxgb4.ko'
'cxgb4.ko' -> '/lib/modules/5.15.2-1096.native/updates/drivers/net/cxgb4/cxgb4.ko'
install: creating directory '/lib/modules/5.15.2-1096.native/updates/drivers/net/offload/t4_tom'
't4_tom.ko' -> '/lib/modules/5.15.2-1096.native/updates/drivers/net/offload/t4_tom/t4_tom.ko'
install -D -v -m 644 t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw-1.26.6.0.bin;
removed '/lib/firmware/cxgb4/t4fw-1.26.6.0.bin'
't4fw-1.26.6.0.bin' -> '/lib/firmware/cxgb4/t4fw-1.26.6.0.bin'
install -D -v -m 644 t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw-1.26.6.0.bin;
removed '/lib/firmware/cxgb4/t5fw-1.26.6.0.bin'
't5fw-1.26.6.0.bin' -> '/lib/firmware/cxgb4/t5fw-1.26.6.0.bin'
install -D -v -m 644 t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw-1.26.6.0.bin;
removed '/lib/firmware/cxgb4/t6fw-1.26.6.0.bin'
't6fw-1.26.6.0.bin' -> '/lib/firmware/cxgb4/t6fw-1.26.6.0.bin'
/bin/ln -sf /lib/firmware/cxgb4/t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw.bin
/bin/ln -sf /lib/firmware/cxgb4/t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw.bin
/bin/ln -sf /lib/firmware/cxgb4/t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw.bin
install -D -v -m 644 t4-config.txt /lib/firmware/cxgb4/t4-config.txt;
removed '/lib/firmware/cxgb4/t4-config.txt'
't4-config.txt' -> '/lib/firmware/cxgb4/t4-config.txt'
install -D -v -m 644 t5-config.txt /lib/firmware/cxgb4/t5-config.txt;
removed '/lib/firmware/cxgb4/t5-config.txt'
't5-config.txt' -> '/lib/firmware/cxgb4/t5-config.txt'
install -D -v -m 644 t6-config.txt /lib/firmware/cxgb4/t6-config.txt;
removed '/lib/firmware/cxgb4/t6-config.txt'
't6-config.txt' -> '/lib/firmware/cxgb4/t6-config.txt'
install -D -v -m 644 aq1202_fw.cld /lib/firmware/cxgb4/aq1202_fw.cld;
removed '/lib/firmware/cxgb4/aq1202_fw.cld'
'aq1202_fw.cld' -> '/lib/firmware/cxgb4/aq1202_fw.cld'
install -D -v -m 644 bcm8483.bin /lib/firmware/cxgb4/bcm8483.bin;
removed '/lib/firmware/cxgb4/bcm8483.bin'
'bcm8483.bin' -> '/lib/firmware/cxgb4/bcm8483.bin'
depmod


***********************
*      Summary        *
***********************
CONFIG = UNIFIED_WIRE
Protocol                      Modules\Libraries\Tools       Action         Status    
------------------------------------------------------------------------------------------
Chelsio-utils(tools)          cxgbtool/cop                  Install        Successful
Network(NIC_IPV4)             cxgb4                         Install        Successful
Network-Offload(TOE)          t4_tom                        Install        Successful
UDP_Offload                   t4_tom                        Install        Successful

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

smallake@clearlinux~/Downloads/ChelsioUwire-3.16.0.0 $ sudo make toe_ipv4_install SKIP_RPM=1

Building for kernel 5.15.2-1096.native

cat: /etc/issue: No such file or directory

cp: cannot stat '/boot/initrd-5.15.2-1096.native': No such file or directory

##################################################

# Cleaning Source/Build/RPMDir #

##################################################

Building for kernel 5.15.2-1096.native

cat: /etc/issue: No such file or directory

Uninstalling all previously installed drivers/libs. This step may take some time.

File "/home/smallake/Downloads/ChelsioUwire-3.16.0.0/scripts/uninstall.py", line 37

fd = os.open ('uninstall.log',os.O_CREAT|os.O_WRONLY,0644)

SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers

/boot/initrd-5.15.2-1096.native does not exist

Usage: lsinitrd [options] [<initramfs file> [<filename> [<filename> [...] ]]]

Usage: lsinitrd [options] -k <kernel version>

-h, --help print a help message and exit.

-s, --size sort the contents of the initramfs by size.

-m, --mod list modules.

-f, --file <filename> print the contents of <filename>.

--unpack unpack the initramfs, instead of displaying the contents.

If optional filenames are given, will only unpack specified files,

else the whole image will be unpacked. Won't unpack anything from early cpio part.

--unpackearly unpack the early microcode part of the initramfs.

Same as --unpack, but only unpack files from early cpio part.

-v, --verbose unpack verbosely.

-k, --kver <kernel version> inspect the initramfs of <kernel version>.

Preparing Diags...

'/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/chelsio_adapter_config_v4/bin/t5seeprom_bins/t5seeprom_x86_64' -> '/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/chelsio_adapter_config_v4/bin/t5seeprom'

'/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/chelsio_adapter_config_v4/bin/t6seeprom_bins/t6seeprom_x86_64' -> '/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/chelsio_adapter_config_v4/bin/t6seeprom'

##################################################

# Building Chelsio-utils(tools) #

##################################################

#######################

#### cxgbtool/cop ####

#######################

cc -O2 -Wall -Werror -c -o cxgbtool.o cxgbtool.c

cc cxgbtool.o -o cxgbtool

g++ -O2 -Wall -Werror -g -DUSE_STD_VECTOR -c -o cop.o cop.cc

g++ -O2 -Wall -Werror -g -DUSE_STD_VECTOR -c -o classifier.o classifier.cc

g++ -O2 -Wall -Werror -g -DUSE_STD_VECTOR -c -o ipfilter.o ipfilter.cc

g++ -g -o cop cop.o classifier.o ipfilter.o

cc -O2 -Wall -Werror -g -c -o opt.o opt.c

cc -g -o opt opt.o

'/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/cudbg/bin/cudbg_app_x86_64' -> '/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/cudbg/app/cudbg_app'

##################################################

# Installing Chelsio-utils(tools) #

##################################################

#######################

#### cxgbtool/cop ####

#######################

install -m 755 cxgbtool /sbin

gzip -c cxgbtool.8 > cxgbtool.8.gz

gzip -c cxgbtool-stor.8 > cxgbtool-stor.8.gz

make[2]: Nothing to be done for 'all'.

'/home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/tools/cudbg/app//cudbg_app' -> '/sbin/cudbg_app'

Copying VPD binaries to /lib/firmware/cxgb4/config : DONE

Copying scripts to /sbin : DONE

##################################################

# Building Offload Capable Network Drivers #

##################################################

Preparing for build ...

#### Preparing cxgb4 ####

make[3]: Nothing to be done for 'prep'.

#######################

#### cxgb4 ####

#######################

warning: the compiler differs from the one used to build the kernel

The kernel was built by: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20211116 releases/gcc-11.2.0-439-gde15d42914

You are using: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20220215 releases/gcc-11.2.0-770-g2c9485a496

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_main.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/l2t.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/clip_tbl.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_ethtool.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_cxgbtool.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_filter.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_debugfs.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/t4_hw.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/t4_linux_debugfs.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/sge.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/trace.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/smt.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/srq.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_ptp.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_mps.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_cudbg.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/fastlz_api.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/fastlz.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_lib.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_wtp.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_flash_utils.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cudbg_common.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/ocqp.o

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4_thermal.o

LD [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4.o

MODPOST /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/Module.symvers

CC [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4.mod.o

LD [M] /home/smallake/Downloads/ChelsioUwire-3.16.0.0/build/src/network/cxgb4/cxgb4.ko

##################################################

# Installing Offload Capable Network Drivers #

##################################################

Preparing for build ...

#### Preparing cxgb4 ####

make[3]: Nothing to be done for 'prep'.

#######################

#### cxgb4 ####

#######################

warning: the compiler differs from the one used to build the kernel

The kernel was built by: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20211116 releases/gcc-11.2.0-439-gde15d42914

You are using: gcc (Clear Linux OS for Intel Architecture) 11.2.1 20220215 releases/gcc-11.2.0-770-g2c9485a496

install: creating directory '/lib/modules/5.15.2-1096.native/updates'

install: creating directory '/lib/modules/5.15.2-1096.native/updates/drivers'

install: creating directory '/lib/modules/5.15.2-1096.native/updates/drivers/net'

install: creating directory '/lib/modules/5.15.2-1096.native/updates/drivers/net/cxgb4'

'cxgb4.ko' -> '/lib/modules/5.15.2-1096.native/updates/drivers/net/cxgb4/cxgb4.ko'

install -D -v -m 644 t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw-1.26.6.0.bin;

't4fw-1.26.6.0.bin' -> '/lib/firmware/cxgb4/t4fw-1.26.6.0.bin'

install -D -v -m 644 t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw-1.26.6.0.bin;

't5fw-1.26.6.0.bin' -> '/lib/firmware/cxgb4/t5fw-1.26.6.0.bin'

install -D -v -m 644 t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw-1.26.6.0.bin;

't6fw-1.26.6.0.bin' -> '/lib/firmware/cxgb4/t6fw-1.26.6.0.bin'

/bin/ln -sf /lib/firmware/cxgb4/t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw.bin

/bin/ln -sf /lib/firmware/cxgb4/t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw.bin

/bin/ln -sf /lib/firmware/cxgb4/t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw.bin

install -D -v -m 644 t4-config.txt /lib/firmware/cxgb4/t4-config.txt;

't4-config.txt' -> '/lib/firmware/cxgb4/t4-config.txt'

install -D -v -m 644 t5-config.txt /lib/firmware/cxgb4/t5-config.txt;

't5-config.txt' -> '/lib/firmware/cxgb4/t5-config.txt'

install -D -v -m 644 t6-config.txt /lib/firmware/cxgb4/t6-config.txt;

't6-config.txt' -> '/lib/firmware/cxgb4/t6-config.txt'

install -D -v -m 644 aq1202_fw.cld /lib/firmware/cxgb4/aq1202_fw.cld;

'aq1202_fw.cld' -> '/lib/firmware/cxgb4/aq1202_fw.cld'

install -D -v -m 644 bcm8483.bin /lib/firmware/cxgb4/bcm8483.bin;

'bcm8483.bin' -> '/lib/firmware/cxgb4/bcm8483.bin'

depmod

install -D -m 644 t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw-1.26.6.0.bin;

install -D -m 644 t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw-1.26.6.0.bin;

install -D -m 644 t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw-1.26.6.0.bin;

/bin/ln -sf /lib/firmware/cxgb4/t4fw-1.26.6.0.bin /lib/firmware/cxgb4/t4fw.bin

/bin/ln -sf /lib/firmware/cxgb4/t5fw-1.26.6.0.bin /lib/firmware/cxgb4/t5fw.bin

/bin/ln -sf /lib/firmware/cxgb4/t6fw-1.26.6.0.bin /lib/firmware/cxgb4/t6fw.bin

install -D -m 644 t4-config.txt /lib/firmware/cxgb4/t4-config.txt;

install -D -m 644 t5-config.txt /lib/firmware/cxgb4/t5-config.txt;

install -D -m 644 t6-config.txt /lib/firmware/cxgb4/t6-config.txt;

install -D -m 644 aq1202_fw.cld /lib/firmware/cxgb4/aq1202_fw.cld;

install -D -m 644 bcm8483.bin /lib/firmware/cxgb4/bcm8483.bin;