1.
일반적으로 레이턴시를 측정하기 위해서 syslog를 이용합니다. 어플리케이션 로그로 레이턴시를 측정할 때 빠지는 부분이 있습니다. 네트워크 어댑터와 Userspace사이의 시간을 측정할 수 없습니다. Network Latency입니다. Network Latency는 10G 네트워크 카드를 구매할 때와 튜닝할 때 중요합니다. 10G 네트워크 카드로 많이 사용하는 Mellanox, Solarflare, Exanic 및 각 제조업체가 제공하는 TCP Accelerator의 튜닝을 할 때 숫자가 필요하기 때문입니다.
이러한 요구에 부응하는 기술이 Timestamping입니다. 리눅스 커널 문서중 timestamping입니다.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 |
1. Control Interfaces The interfaces for receiving network packages timestamps are: * SO_TIMESTAMP Generates a timestamp for each incoming packet in (not necessarily monotonic) system time. Reports the timestamp via recvmsg() in a control message in usec resolution. SO_TIMESTAMP is defined as SO_TIMESTAMP_NEW or SO_TIMESTAMP_OLD based on the architecture type and time_t representation of libc. Control message format is in struct __kernel_old_timeval for SO_TIMESTAMP_OLD and in struct __kernel_sock_timeval for SO_TIMESTAMP_NEW options respectively. * SO_TIMESTAMPNS Same timestamping mechanism as SO_TIMESTAMP, but reports the timestamp as struct timespec in nsec resolution. SO_TIMESTAMPNS is defined as SO_TIMESTAMPNS_NEW or SO_TIMESTAMPNS_OLD based on the architecture type and time_t representation of libc. Control message format is in struct timespec for SO_TIMESTAMPNS_OLD and in struct __kernel_timespec for SO_TIMESTAMPNS_NEW options respectively. * IP_MULTICAST_LOOP + SO_TIMESTAMP[NS] Only for multicast:approximate transmit timestamp obtained by reading the looped packet receive timestamp. * SO_TIMESTAMPING Generates timestamps on reception, transmission or both. Supports multiple timestamp sources, including hardware. Supports generating timestamps for stream sockets. 1.1 SO_TIMESTAMP (also SO_TIMESTAMP_OLD and SO_TIMESTAMP_NEW): This socket option enables timestamping of datagrams on the reception path. Because the destination socket, if any, is not known early in the network stack, the feature has to be enabled for all packets. The same is true for all early receive timestamp options. For interface details, see `man 7 socket`. Always use SO_TIMESTAMP_NEW timestamp to always get timestamp in struct __kernel_sock_timeval format. SO_TIMESTAMP_OLD returns incorrect timestamps after the year 2038 on 32 bit machines. 1.2 SO_TIMESTAMPNS (also SO_TIMESTAMPNS_OLD and SO_TIMESTAMPNS_NEW): This option is identical to SO_TIMESTAMP except for the returned data type. Its struct timespec allows for higher resolution (ns) timestamps than the timeval of SO_TIMESTAMP (ms). Always use SO_TIMESTAMPNS_NEW timestamp to always get timestamp in struct __kernel_timespec format. SO_TIMESTAMPNS_OLD returns incorrect timestamps after the year 2038 on 32 bit machines. 1.3 SO_TIMESTAMPING (also SO_TIMESTAMPING_OLD and SO_TIMESTAMPING_NEW): Supports multiple types of timestamp requests. As a result, this socket option takes a bitmap of flags, not a boolean. In err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val)); val is an integer with any of the following bits set. Setting other bit returns EINVAL and does not change the current state. The socket option configures timestamp generation for individual sk_buffs (1.3.1), timestamp reporting to the socket's error queue (1.3.2) and options (1.3.3). Timestamp generation can also be enabled for individual sendmsg calls using cmsg (1.3.4). 1.3.1 Timestamp Generation Some bits are requests to the stack to try to generate timestamps. Any combination of them is valid. Changes to these bits apply to newly created packets, not to packets already in the stack. As a result, i500t is possible to selectively request timestamps for a subset of packets (e.g., for sampling) by embedding an send() call within two setsockopt calls, one to enable timestamp generation and one to disable it. Timestamps may also be generated for reasons other than being requested by a particular socket, such as when receive timestamping is enabled system wide, as explained earlier. SOF_TIMESTAMPING_RX_HARDWARE: Request rx timestamps generated by the network adapter. SOF_TIMESTAMPING_RX_SOFTWARE: Request rx timestamps when data enters the kernel. These timestamps are generated just after a device driver hands a packet to the kernel receive stack. SOF_TIMESTAMPING_TX_HARDWARE: Request tx timestamps generated by the network adapter. This flag can be enabled via both socket options and control messages. SOF_TIMESTAMPING_TX_SOFTWARE: Request tx timestamps when data leaves the kernel. These timestamps are generated in the device driver as close as possible, but always prior to, passing the packet to the network interface. Hence, they require driver support and may not be available for all devices. This flag can be enabled via both socket options and control messages. SOF_TIMESTAMPING_TX_SCHED: Request tx timestamps prior to entering the packet scheduler. Kernel transmit latency is, if long, often dominated by queuing delay. The difference between this timestamp and one taken at SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent of protocol processing. The latency incurred in protocol processing, if any, can be computed by subtracting a userspace timestamp taken immediately before send() from this timestamp. On machines with virtual devices where a transmitted packet travels through multiple devices and, hence, multiple packet schedulers, a timestamp is generated at each layer. This allows for fine grained measurement of queuing delay. This flag can be enabled via both socket options and control messages. SOF_TIMESTAMPING_TX_ACK: Request tx timestamps when all data in the send buffer has been acknowledged. This only makes sense for reliable protocols. It is currently only implemented for TCP. For that protocol, it may over-report measurement, because the timestamp is generated when all data up to and including the buffer at send() was acknowled <pre class="height-set:true width-set:true width:100 width-unit:1 lang:default decode:true ">// SPDX-License-Identifier: GPL-2.0-only /* * This program demonstrates how the various time stamping features in * the Linux kernel work. It emulates the behavior of a PTP * implementation in stand-alone master mode by sending PTPv1 Sync * multicasts once every second. It looks for similar packets, but * beyond that doesn't actually implement PTP. * * Outgoing packets are time stamped with SO_TIMESTAMPING with or * without hardware support. * * Incoming packets are time stamped with SO_TIMESTAMPING with or * without hardware support, SIOCGSTAMP[NS] (per-socket time stamp) and * SO_TIMESTAMP[NS]. * * Copyright (C) 2009 Intel Corporatio <pre class="height-set:true width-set:true width:100 width-unit:1 lang:default decode:true ">// SPDX-License-Identifier: GPL-2.0-only /* * This program demonstrates how the various time stamping features in * the Linux kernel work. It emulates the behavior of a PTP * implementation in stand-alone master mode by sending PTPv1 Sync * multicasts once every second. It looks for similar packets, but * beyond that doesn't actually implement PTP. * * Outgoing packets are time stamped with SO_TIMESTAMPING with or * without hardware support. * * Incoming packets are time stamped with SO_TIMESTAMPING with or * without hardware support, SIOCGSTAMP[NS] (per-socket time stamp) and * SO_TIMESTAMP[NS]. * * Copyright (C) 2009 Intel Corporation. * Author: Patrick Ohly <patrick.ohly@intel.com> */ #include <stdio.h> #include <stdlib.h> #include <errno.h> #include <string.h> #include <sys/time.h> #include <sys/socket.h> #include <sys/select.h> #include <sys/ioctl.h> #include <arpa/inet.h> #include <net/if.h> #include <asm/types.h> #include <linux/net_tstamp.h> #include <linux/errqueue.h> #include <linux/sockios.h> #ifndef SO_TIMESTAMPING # define SO_TIMESTAMPING 37 # define SCM_TIMESTAMPING SO_TIMESTAMPING #endif #ifndef SO_TIMESTAMPNS # define SO_TIMESTAMPNS 35 #endif static void usage(const char *error) { if (error) printf("invalid option: %s\n", error); printf("timestamping <interface> [bind_phc_index] [option]*\n\n" "Options:\n" " IP_MULTICAST_LOOP - looping outgoing multicasts\n" " SO_TIMESTAMP - normal software time stamping, ms resolution\n" " SO_TIMESTAMPNS - more accurate software time stamping\n" " SOF_TIMESTAMPING_TX_HARDWARE - hardware time stamping of outgoing packets\n" " SOF_TIMESTAMPING_TX_SOFTWARE - software fallback for outgoing packets\n" " SOF_TIMESTAMPING_RX_HARDWARE - hardware time stamping of incoming packets\n" " SOF_TIMESTAMPING_RX_SOFTWARE - software fallback for incoming packets\n" " SOF_TIMESTAMPING_SOFTWARE - request reporting of software time stamps\n" " SOF_TIMESTAMPING_RAW_HARDWARE - request reporting of raw HW time stamps\n" " SOF_TIMESTAMPING_BIND_PHC - request to bind a PHC of PTP vclock\n" " SIOCGSTAMP - check last socket time stamp\n" " SIOCGSTAMPNS - more accurate socket time stamp\n" " PTPV2 - use PTPv2 messages\n"); exit(1); } static void bail(const char *error) { printf("%s: %s\n", error, strerror(errno)); exit(1); } static const unsigned char sync[] = { 0x00, 0x01, 0x00, 0x01, 0x5f, 0x44, 0x46, 0x4c, 0x54, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01, /* fake uuid */ 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, n. * Author: Patrick Ohly <patrick.ohly@intel.com> */ #include <stdio.h> #include <stdlib.h> #include <errno.h> #include <string.h> #include <sys/time.h> #include <sys/socket.h> #include <sys/select.h> #include <sys/ioctl.h> #include <arpa/inet.h> #include <net/if.h> #include <asm/types.h> #include <linux/net_tstamp.h> #include <linux/errqueue.h> #include <linux/sockios.h> #ifndef SO_TIMESTAMPING # define SO_TIMESTAMPING 37 # define SCM_TIMESTAMPING SO_TIMESTAMPING #endif #ifndef SO_TIMESTAMPNS # define SO_TIMESTAMPNS 35 #endif static void usage(const char *error) { if (error) printf("invalid option: %s\n", error); printf("timestamping <interface> [bind_phc_index] [option]*\n\n" "Options:\n" " IP_MULTICAST_LOOP - looping outgoing multicasts\n" " SO_TIMESTAMP - normal software time stamping, ms resolution\n" " SO_TIMESTAMPNS - more accurate software time stamping\n" " SOF_TIMESTAMPING_TX_HARDWARE - hardware time stamping of outgoing packets\n" " SOF_TIMESTAMPING_TX_SOFTWARE - software fallback for outgoing packets\n" " SOF_TIMESTAMPING_RX_HARDWARE - hardware time stamping of incoming packets\n" " SOF_TIMESTAMPING_RX_SOFTWARE - software fallback for incoming packets\n" " SOF_TIMESTAMPING_SOFTWARE - request reporting of software time stamps\n" " SOF_TIMESTAMPING_RAW_HARDWARE - request reporting of raw HW time stamps\n" " SOF_TIMESTAMPING_BIND_PHC - request to bind a PHC of PTP vclock\n" " SIOCGSTAMP - check last socket time stamp\n" " SIOCGSTAMPNS - more accurate socket time stamp\n" " PTPV2 - use PTPv2 messages\n"); exit(1); } static void bail(const char *error) { printf("%s: %s\n", error, strerror(errno)); exit(1); } static const unsigned char sync[] = { 0x00, 0x01, 0x00, 0x01, 0x5f, 0x44, 0x46, 0x4c, 0x54, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01, /* fake uuid */ 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, printf("Starting to send packets with size %u bytes at intervals of %u ms\n", cfg.cfg_pkt_size, cfg.cfg_interval_ms); if (cfg.cfg_max_packets > 0) { printf("Will send %u packets\n", cfg.cfg_max_packets); } else { printf("Will send packets indefinitely (Ctrl+C to stop)\n"); } /* Record start time */ time(&start_time); /* Send packets */ while ((pkt_num++ < cfg.cfg_max_packets || (cfg.cfg_max_packets == 0))) { /* Fill the packet with a pattern */ memset(buffer, 0, cfg.cfg_pkt_size); snprintf(buffer, cfg.cfg_pkt_size, "Packet %u", pkt_num); for (size_t i = strlen(buffer) + 1; i < cfg.cfg_pkt_size; i++) { buffer[i] = (char)(i % 256); } /* Send the packet */ ssize_t sent; if (cfg.cfg_protocol == IPPROTO_UDP) { sent = sendto(sock, buffer, cfg.cfg_pkt_size, 0, (struct sockaddr *)&dest_addr, sizeof(dest_addr)); } else { sent = send(sock, buffer, cfg.cfg_pkt_size, 0); } if (sent < 0) { perror("send failed"); break; } total_sent++; if (!cfg.cfg_quiet) { printf("Sent packet %u, %zd bytes\n", pkt_num, sent); } /* Show stats periodically */ time(¤t_time); if (current_time - start_time >= 10) { /* Every 10 seconds */ double rate = (double)total_sent / (current_time - start_time); printf("Status: Sent %lu packets in %ld seconds (%.2f packets/sec)\n", total_sent, (long)(current_time - start_time), rate); start_time = current_time; total_sent = 0; } /* Wait for specified interval */ usleep(cfg.cfg_interval_ms * 1000); } printf("Sent %u packets. Exiting.\n", pkt_num - 1); /* Clean up */ free(buffer); free(cfg.cfg_dest_ip); close(sock); return 0; } 0x00, 0x01, 0x00, 0x37, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x49, 0x05, 0xcd, 0x01, 0x29, 0xb1, 0x8d, 0xb0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, /* fake uuid */ 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x00, 0x00, 0x00, 0x37, 0x00, 0x00, 0x00, 0x04, 0x44, 0x46, 0x4c, 0x54, 0x00, 0x00, 0xf0, 0x60, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0xf0, 0x60, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x44, 0x46, 0x4c, 0x54, 0x00, 0x01, /* fake uuid */ 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }; static const unsigned char sync_v2[] = { 0x00, 0x02, 0x00, 0x2C, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFF, 0xFE, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, }; static void sendpacket(int sock, struct sockaddr *addr, socklen_t addr_len, int ptpv2) { size_t sync_len = ptpv2 ? sizeof(sync_v2) : sizeof(sync); const void *sync_p = ptpv2 ? sync_v2 : sync; struct timeval now; int res; res = sendto(sock, sync_p, sync_len, 0, addr, addr_len); gettimeofday(&now, 0); if (res < 0) printf("%s: %s\n", "send", strerror(errno)); else printf("%ld.%06ld: sent %d bytes\n", (long)now.tv_sec, (long)now.tv_usec, res); } static void printpacket(struct msghdr *msg, int res, char *data, int sock, int recvmsg_flags, int siocgstamp, int siocgstampns, int ptpv2) { struct sockaddr_in *from_addr = (struct sockaddr_in *)msg->msg_name; size_t sync_len = ptpv2 ? sizeof(sync_v2) : sizeof(sync); const void *sync_p = ptpv2 ? sync_v2 : sync; struct cmsghdr *cmsg; struct timeval tv; struct timespec ts; struct timeval now; gettimeofday(&now, 0); printf("%ld.%06ld: received %s data, %d bytes from %s, %zu bytes control messages\n", (long)now.tv_sec, (long)now.tv_usec, (recvmsg_flags & MSG_ERRQUEUE) ? "error" : "regular", res, inet_ntoa(from_addr->sin_addr), msg->msg_controllen); for (cmsg = CMSG_FIRSTHDR(msg); cmsg; cmsg = CMSG_NXTHDR(msg, cmsg)) { printf(" cmsg len %zu: ", cmsg->cmsg_len); switch (cmsg->cmsg_level) { case SOL_SOCKET: printf("SOL_SOCKET "); switch (cmsg->cmsg_type) { case SO_TIMESTAMP: { struct timeval *stamp = (struct timeval *)CMSG_DATA(cmsg); printf("SO_TIMESTAMP %ld.%06ld", (long)stamp->tv_sec, (long)stamp->tv_usec); break; } case SO_TIMESTAMPNS: { struct timespec *stamp = (struct timespec *)CMSG_DATA(cmsg); printf("SO_TIMESTAMPNS %ld.%09ld", (long)stamp->tv_sec, (long)stamp->tv_nsec); break; } case SO_TIMESTAMPING: { struct timespec *stamp = (struct timespec *)CMSG_DATA(cmsg); printf("SO_TIMESTAMPING "); printf("SW %ld.%09ld ", (long)stamp->tv_sec, (long)stamp->tv_nsec); stamp++; /* skip deprecated HW transformed */ stamp++; printf("HW raw %ld.%09ld", (long)stamp->tv_sec, (long)stamp->tv_nsec); break; } default: printf("type %d", cmsg->cmsg_type); break; } break; case IPPROTO_IP: printf("IPPROTO_IP "); switch (cmsg->cmsg_type) { case IP_RECVERR: { struct sock_extended_err *err = (struct sock_extended_err *)CMSG_DATA(cmsg); printf("IP_RECVERR ee_errno '%s' ee_origin %d => %s", strerror(err->ee_errno), err->ee_origin, #ifdef SO_EE_ORIGIN_TIMESTAMPING err->ee_origin == SO_EE_ORIGIN_TIMESTAMPING ? "bounced packet" : "unexpected origin" #else "probably SO_EE_ORIGIN_TIMESTAMPING" #endif ); if (res < sync_len) printf(" => truncated data?!"); else if (!memcmp(sync_p, data + res - sync_len, sync_len)) printf(" => GOT OUR DATA BACK (HURRAY!)"); break; } case IP_PKTINFO: { struct in_pktinfo *pktinfo = (struct in_pktinfo *)CMSG_DATA(cmsg); printf("IP_PKTINFO interface index %u", pktinfo->ipi_ifindex); break; } default: printf("type %d", cmsg->cmsg_type); break; } break; default: printf("level %d type %d", cmsg->cmsg_level, cmsg->cmsg_type); break; } printf("\n"); } if (siocgstamp) { if (ioctl(sock, SIOCGSTAMP, &tv)) printf(" %s: %s\n", "SIOCGSTAMP", strerror(errno)); else printf("SIOCGSTAMP %ld.%06ld\n", (long)tv.tv_sec, (long)tv.tv_usec); } if (siocgstampns) { if (ioctl(sock, SIOCGSTAMPNS, &ts)) printf(" %s: %s\n", "SIOCGSTAMPNS", strerror(errno)); else printf("SIOCGSTAMPNS %ld.%09ld\n", (long)ts.tv_sec, (long)ts.tv_nsec); } } static void recvpacket(int sock, int recvmsg_flags, int siocgstamp, int siocgstampns, int ptpv2) { char data[256]; struct msghdr msg; struct iovec entry; struct sockaddr_in from_addr; struct { struct cmsghdr cm; char control[512]; } control; int res; memset(&msg, 0, sizeof(msg)); msg.msg_iov = &entry; msg.msg_iovlen = 1; entry.iov_base = data; entry.iov_len = sizeof(data); msg.msg_name = (caddr_t)&from_addr; msg.msg_namelen = sizeof(from_addr); msg.msg_control = &control; msg.msg_controllen = sizeof(control); res = recvmsg(sock, &msg, recvmsg_flags|MSG_DONTWAIT); if (res < 0) { printf("%s %s: %s\n", "recvmsg", (recvmsg_flags & MSG_ERRQUEUE) ? "error" : "regular", strerror(errno)); } else { printpacket(&msg, res, data, sock, recvmsg_flags, siocgstamp, siocgstampns, ptpv2); } } int main(int argc, char **argv) { int so_timestamp = 0; int so_timestampns = 0; int siocgstamp = 0; int siocgstampns = 0; int ip_multicast_loop = 0; int ptpv2 = 0; char *interface; int i; int enabled = 1; int sock; struct ifreq device; struct ifreq hwtstamp; struct hwtstamp_config hwconfig, hwconfig_requested; struct so_timestamping so_timestamping_get = { 0, 0 }; struct so_timestamping so_timestamping = { 0, 0 }; struct sockaddr_in addr; struct ip_mreq imr; struct in_addr iaddr; int val; socklen_t len; struct timeval next; size_t if_len; if (argc < 2) usage(0); interface = argv[1]; if_len = strlen(interface); if (if_len >= IFNAMSIZ) { printf("interface name exceeds IFNAMSIZ\n"); exit(1); } if (argc >= 3 && sscanf(argv[2], "%d", &so_timestamping.bind_phc) == 1) val = 3; else val = 2; for (i = val; i < argc; i++) { if (!strcasecmp(argv[i], "SO_TIMESTAMP")) so_timestamp = 1; else if (!strcasecmp(argv[i], "SO_TIMESTAMPNS")) so_timestampns = 1; else if (!strcasecmp(argv[i], "SIOCGSTAMP")) siocgstamp = 1; else if (!strcasecmp(argv[i], "SIOCGSTAMPNS")) siocgstampns = 1; else if (!strcasecmp(argv[i], "IP_MULTICAST_LOOP")) ip_multicast_loop = 1; else if (!strcasecmp(argv[i], "PTPV2")) ptpv2 = 1; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_TX_HARDWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_TX_HARDWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_TX_SOFTWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_TX_SOFTWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_RX_HARDWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_RX_HARDWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_RX_SOFTWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_RX_SOFTWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_SOFTWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_SOFTWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_RAW_HARDWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_RAW_HARDWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_BIND_PHC")) so_timestamping.flags |= SOF_TIMESTAMPING_BIND_PHC; else usage(argv[i]); } sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP); if (sock < 0) bail("socket"); memset(&device, 0, sizeof(device)); memcpy(device.ifr_name, interface, if_len + 1); if (ioctl(sock, SIOCGIFADDR, &device) < 0) bail("getting interface IP address"); memset(&hwtstamp, 0, sizeof(hwtstamp)); memcpy(hwtstamp.ifr_name, interface, if_len + 1); hwtstamp.ifr_data = (void *)&hwconfig; memset(&hwconfig, 0, sizeof(hwconfig)); hwconfig.tx_type = (so_timestamping.flags & SOF_TIMESTAMPING_TX_HARDWARE) ? HWTSTAMP_TX_ON : HWTSTAMP_TX_OFF; hwconfig.rx_filter = (so_timestamping.flags & SOF_TIMESTAMPING_RX_HARDWARE) ? ptpv2 ? HWTSTAMP_FILTER_PTP_V2_L4_SYNC : HWTSTAMP_FILTER_PTP_V1_L4_SYNC : HWTSTAMP_FILTER_NONE; hwconfig_requested = hwconfig; if (ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) < 0) { if ((errno == EINVAL || errno == ENOTSUP) && hwconfig_requested.tx_type == HWTSTAMP_TX_OFF && hwconfig_requested.rx_filter == HWTSTAMP_FILTER_NONE) printf("SIOCSHWTSTAMP: disabling hardware time stamping not possible\n"); else bail("SIOCSHWTSTAMP"); } printf("SIOCSHWTSTAMP: tx_type %d requested, got %d; rx_filter %d requested, got %d\n", hwconfig_requested.tx_type, hwconfig.tx_type, hwconfig_requested.rx_filter, hwconfig.rx_filter); /* bind to PTP port */ addr.sin_family = AF_INET; addr.sin_addr.s_addr = htonl(INADDR_ANY); addr.sin_port = htons(319 /* PTP event port */); if (bind(sock, (struct sockaddr *)&addr, sizeof(struct sockaddr_in)) < 0) bail("bind"); if (setsockopt(sock, SOL_SOCKET, SO_BINDTODEVICE, interface, if_len)) bail("bind device"); /* set multicast group for outgoing packets */ inet_aton("224.0.1.130", &iaddr); /* alternate PTP domain 1 */ addr.sin_addr = iaddr; imr.imr_multiaddr.s_addr = iaddr.s_addr; imr.imr_interface.s_addr = ((struct sockaddr_in *)&device.ifr_addr)->sin_addr.s_addr; if (setsockopt(sock, IPPROTO_IP, IP_MULTICAST_IF, &imr.imr_interface.s_addr, sizeof(struct in_addr)) < 0) bail("set multicast"); /* join multicast group, loop our own packet */ if (setsockopt(sock, IPPROTO_IP, IP_ADD_MEMBERSHIP, &imr, sizeof(struct ip_mreq)) < 0) bail("join multicast group"); if (setsockopt(sock, IPPROTO_IP, IP_MULTICAST_LOOP, &ip_multicast_loop, sizeof(enabled)) < 0) { bail("loop multicast"); } /* set socket options for time stamping */ if (so_timestamp && setsockopt(sock, SOL_SOCKET, SO_TIMESTAMP, &enabled, sizeof(enabled)) < 0) bail("setsockopt SO_TIMESTAMP"); if (so_timestampns && setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPNS, &enabled, sizeof(enabled)) < 0) bail("setsockopt SO_TIMESTAMPNS"); if (so_timestamping.flags && setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &so_timestamping, sizeof(so_timestamping)) < 0) bail("setsockopt SO_TIMESTAMPING"); /* request IP_PKTINFO for debugging purposes */ if (setsockopt(sock, SOL_IP, IP_PKTINFO, &enabled, sizeof(enabled)) < 0) printf("%s: %s\n", "setsockopt IP_PKTINFO", strerror(errno)); /* verify socket options */ len = sizeof(val); if (getsockopt(sock, SOL_SOCKET, SO_TIMESTAMP, &val, &len) < 0) printf("%s: %s\n", "getsockopt SO_TIMESTAMP", strerror(errno)); else printf("SO_TIMESTAMP %d\n", val); if (getsockopt(sock, SOL_SOCKET, SO_TIMESTAMPNS, &val, &len) < 0) printf("%s: %s\n", "getsockopt SO_TIMESTAMPNS", strerror(errno)); else printf("SO_TIMESTAMPNS %d\n", val); len = sizeof(so_timestamping_get); if (getsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &so_timestamping_get, &len) < 0) { printf("%s: %s\n", "getsockopt SO_TIMESTAMPING", strerror(errno)); } else { printf("SO_TIMESTAMPING flags %d, bind phc %d\n", so_timestamping_get.flags, so_timestamping_get.bind_phc); if (so_timestamping_get.flags != so_timestamping.flags || so_timestamping_get.bind_phc != so_timestamping.bind_phc) printf(" not expected, flags %d, bind phc %d\n", so_timestamping.flags, so_timestamping.bind_phc); } /* send packets forever every five seconds */ gettimeofday(&next, 0); next.tv_sec = (next.tv_sec + 1) / 5 * 5; next.tv_usec = 0; while (1) { struct timeval now; struct timeval delta; long delta_us; int res; fd_set readfs, errorfs; gettimeofday(&now, 0); delta_us = (long)(next.tv_sec - now.tv_sec) * 1000000 + (long)(next.tv_usec - now.tv_usec); if (delta_us > 0) { /* continue waiting for timeout or data */ delta.tv_sec = delta_us / 1000000; delta.tv_usec = delta_us % 1000000; FD_ZERO(&readfs); FD_ZERO(&errorfs); FD_SET(sock, &readfs); FD_SET(sock, &errorfs); printf("%ld.%06ld: select %ldus\n", (long)now.tv_sec, (long)now.tv_usec, delta_us); res = select(sock + 1, &readfs, 0, &errorfs, &delta); gettimeofday(&now, 0); printf("%ld.%06ld: select returned: %d, %s\n", (long)now.tv_sec, (long)now.tv_usec, res, res < 0 ? strerror(errno) : "success"); if (res > 0) { if (FD_ISSET(sock, &readfs)) printf("ready for reading\n"); if (FD_ISSET(sock, &errorfs)) printf("has error\n"); recvpacket(sock, 0, siocgstamp, siocgstampns, ptpv2); recvpacket(sock, MSG_ERRQUEUE, siocgstamp, siocgstampns, ptpv2); } } else { /* write one packet */ sendpacket(sock, (struct sockaddr *)&addr, sizeof(addr), ptpv2); next.tv_sec += 5; continue; } } return 0; } ged: the cumulative acknowledgment. The mechanism ignores SACK and FACK. This flag can be enabled via both socket options and control messages. 1.3.2 Timestamp Reporting The other three bits control which timestamps will be reported in a generated control message. Changes to the bits take immediate effect at the timestamp reporting locations in the stack. Timestamps are only reported for packets that also have the relevant timestamp generation request set. SOF_TIMESTAMPING_SOFTWARE: Report any software timestamps when available. SOF_TIMESTAMPING_SYS_HARDWARE: This option is deprecated and ignored. SOF_TIMESTAMPING_RAW_HARDWARE: Report hardware timestamps as generated by SOF_TIMESTAMPING_TX_HARDWARE when available. 1.3.3 Timestamp Options The interface supports the options SOF_TIMESTAMPING_OPT_ID: Generate a unique identifier along with each packet. A process can have multiple concurrent timestamping requests outstanding. Packets can be reordered in the transmit path, for instance in the packet scheduler. In that case timestamps will be queued onto the error queue out of order from the original send() calls. It is not always possible to uniquely match timestamps to the original send() calls based on timestamp order or payload inspection alone, then. This option associates each packet at send() with a unique identifier and returns that along with the timestamp. The identifier is derived from a per-socket u32 counter (that wraps). For datagram sockets, the counter increments with each sent packet. For stream500 sockets, it increments with every byte. The counter starts at zero. It is initialized the first time that the socket option is enabled. It is reset each time the option is enabled after having been disabled. Resetting the counter does not change the identifiers of existing packets in the system. This option is implemented only for transmit timestamps. There, the timestamp is always looped along with a struct sock_extended_err. The option modifies field ee_data to pass an id that is unique among all possibly concurrently outstanding timestamp requests for that socket. SOF_TIMESTAMPING_OPT_CMSG: Support recv() cmsg for all timestamped packets. Control messages are already supported unconditionally on all packets with receive timestamps and on IPv6 packets with transmit timestamp. This option extends them to IPv4 packets with transmit timestamp. One use case is to correlate packets with their egress device, by enabling socket option IP_PKTINFO simultaneously. SOF_TIMESTAMPING_OPT_TSONLY: Applies to transmit timestamps only. Makes the kernel return the timestamp as a cmsg alongside an empty packet, as opposed to alongside the original packet. This reduces the amount of memory charged to the socket's receive budget (SO_RCVBUF) and delivers the timestamp even if sysctl net.core.tstamp_allow_data is 0. This option disables SOF_TIMESTAMPING_OPT_CMSG. SOF_TIMESTAMPING_OPT_STATS: Optional stats that are obtained along with the transmit timestamps. It must be used together with SOF_TIMESTAMPING_OPT_TSONLY. When the transmit timestamp is available, the stats are available in a separate control message of type SCM_TIMESTAMPING_OPT_STATS, as a list of TLVs (struct nlattr) of types. These stats allow the application to associate various transport layer stats with the transmit timestamps, such as how long a certain block of data was limited by peer's receiver window. SOF_TIMESTAMPING_OPT_PKTINFO: Enable the SCM_TIMESTAMPING_PKTINFO control message for incoming500 packets with hardware timestamps. The message contains struct scm_ts_pktinfo, which supplies the index of the real interface which received the packet and its length at layer 2. A valid (non-zero) interface index will be returned only if CONFIG_NET_RX_BUSY_POLL is enabled and the driver is using NAPI. The struct contains also two other fields, but they are reserved and undefined. SOF_TIMESTAMPING_OPT_TX_SWHW: Request both hardware and software timestamps for outgoing packets when SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE are enabled at the same time. If both timestamps are generated, two separate messages will be looped to the socket's error queue, each containing just one timestamp. New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate regardless of the setting of sysctl net.core.tstamp_allow_data. An exception is when a process needs additional cmsg data, for instance SOL_IP/IP_PKTINFO to detect the egress network interface. Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on having access to the contents of the original packet, so cannot be combined with SOF_TIMESTAMPING_OPT_TSONLY. 1.3.4. Enabling timestamps via control messages In addition to socket options, timestamp generation can be requested per write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1). Using this feature, applications can sample timestamps per sendmsg() without paying the overhead of enabling and disabling timestamps via setsockopt: struct msghdr *msg; ... cmsg = CMSG_FIRSTHDR(msg); cmsg->cmsg_level = SOL_SOCKET; cmsg->cmsg_type = SO_TIMESTAMPING; cmsg->cmsg_len = CMSG_LEN(sizeof(__u32)); *((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED | SOF_TIMESTAMPING_TX_SOFTWARE | SOF_TIMESTAMPING_TX_ACK; err = sendmsg(fd, msg, 0); The SOF_TIMESTAMPING_TX_* flags set via cmsg will override the SOF_TIMESTAMPING_TX_* flags set via setsockopt. Moreover, applications must still entimestampingable timestamp reporting via setsockopt to receive timestamps: __u32 val = SOF_TIMESTAMPING_SOFTWARE | SOF_TIMESTAMPING_OPT_ID /* or any other flag */; err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val)); 1.4 Bytestream Timestamps The SO_TIMESTAMPING interface supports timestamping of bytes in a bytestream. Each request is interpreted as a request for when the entire contents of the buffer has passed a timestamping point. That is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record when all bytes have reached the device driver, regardless of how many packets the data has been converted into. In general, bytestreams have no natural delimiters and therefore correlating a timestamp with data is non-trivial. A range of bytes may be split across segments, any segments may be merged (possibly coalescing sections of previously segmented buffers associated with independent send() calls). Segments can be reordered and the same byte range can coexist in multiple segments for protocols that implement retransmissions. It is essential that all timestamps implement the same semantics, regardless of these possible transformations, as otherwise they are incomparable. Handling "rare" corner cases differently from the simple case (a 1:1 mapping from buffer to skb) is insufficient because performance debugging often needs to focus on such outliers. In practice, timestamps can be correlated with segments of a bytestream consistently, if both semantics of the timestamp and the timing of measurement are chosen correctly. This challenge is no different from deciding on a strategy for IP fragmentation. There, the definition is that only the first fragment is timestamped. For bytestreams, we chose that a timestamp is generated only when all bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to implement and reason about. An implementation that has to take into account SACK would be more complex due to possible transmission holes and out of order arrival. On the host, TCP can also break the simple 1:1 mapping from buffer to skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The implementation ensures correctness in all cases by tracking the individual last byte passed to send(), even if it is no longer the last byte after an skbuff extend or merge operation. It stores the relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff has only one such field, only one timestamp can be generated. In rare cases, a timestamp request can be missed if two requests are collapsed onto the same skb. A process can detect this situation by enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at send time with the value returned for each timestamp. It can prevent the situation by always flushing the TCP stack in between requests, for instance by enabling TCP_NODELAY and disabling TCP_CORK and autocork.SOF_TIMESTAMPING_RX_HARDWARE | SOF_TIMESTAMPING_RAW_HARDWARE | SOF_TIMESTAMPING_SYS_HARDWARE | SOF_TIMESTAMPING_SOFTWARE; These precautions ensure that the timestamp is generated only when all bytes have passed a timestamp point, assuming that the network stack itself does not reorder the segments. The stack indeed tries to avoid reordering. The one exception is under administrator control: it is possible to construct a packet scheduler configuration that delays segments from the same stream differently. Such a setup would be unusual. 2 Data Interfaces Timestamps are read using the ancillary data feature of recvmsg(). See `man 3 cmsg` for details of this interface. The socket manual page (`man 7 socket`) describes how timestamps generated with SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved. 2.1 SCM_TIMESTAMPING records These timestamps are returned in a control message with cmsg_level SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type For SO_TIMESTAMPING_OLD: struct scm_timestamping { struct timespec ts[3]; }; For SO_TIMESTAMPING_NEW: struct scm_timestamping64 { struct __kernel_timespec ts[3]; Always use SO_TIMESTAMPING_NEW timestamp to always get timestamp in struct scm_timestamping64 format. SO_TIMESTAMPING_OLD returns incorrect timestamps after the year 2038 on 32 bit machines. The structure can return up to three timestamps. This is a legacy feature. At least one field is non-zero at any time. Most timestamps are passed in ts[0]. Hardware timestamps are passed in ts[2]. ts[1] used to hold hardware timestamps converted to system time. Instead, expose the hardware clock device on the NIC directly as a HW PTP clock source, to allow time conversion in userspace and optionally synchronize system time with a userspace PTP stack such as linuxptp. For the PTP clock API, see Documentation/driver-api/ptp.rst. Note that if the SO_TIMESTAMP or SO_TIMESTAMPNS option is enabled together with SO_TIMESTAMPING using SOF_TIMESTAMPING_SOFTWARE, a false software timestamp will be generated in the recvmsg() call and passed in ts[0] when a real software timestamp is missing. This happens also on hardware transmit timestamps. 2.1.1 Transmit timestamps with MSG_ERRQUEUE For transmit timestamps the outgoing packet is looped back to the socket's error queue with the send timestamp(s) attached. A process receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE set and with a msg_control buffer sufficiently large to receive the relevant metadata structures. The recvmsg call returns the original outgoing data packet with two ancillary messages attached. A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR embeds a struct sock_extended_err. This defines the error type. For timestamps, the ee_errno field is ENOMSG. The other ancillary message will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This embeds the struct scm_timestamping. 2.1.1.2 Timestamp types The semantics of the three struct timespec are defined by field ee_info in the extended error structure. It contains a value of type SCM_TSTAMP_* to define the actual timestamp passed in scm_timestamping. The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_* control fields discussed previously, with one exception. For legacy reasons, SCM_TSTAMP_SND is equal to zero and can be set for both SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It is the first if ts[2] is non-zero, the second otherwise, in which case the timestamp is stored in ts[0]. 2.1.1.3 Fragmentation Fragmentation of outgoing datagrams is rare, but is possible, e.g., by explicitly disabling PMTU discovery. If an outgoing packet is fragmented, then only the first fragment is timestamped and returned to the sending socket. 2.1.1.4 Packet Payload The calling application is often not interested in receiving the whole packet payload that it passed to the stack originally: the socket error queue mechanism is just a method to piggyback the timestamp on. In this case, the application can choose to read datagrams with a smaller buffer, possibly even of length 0. The payload is truncated accordingly. Until the process calls recvmsg() on the error queue, however, the full packet is queued, taking up budget from SO_RCVBUF. 2.1.1.5 Blocking Read Reading from the error queue is always a non-blocking operation. To block waiting on a timestamp, use poll or select. poll() will return POLLERR in pollfd.revents if any data is ready on the error queue. There is no need to pass this flag in pollfd.events. This flag is ignored on request. See also `man 2 poll`. 2.1.2 Receive timestamps On reception, there is no reason to read from the socket error queue. The SCM_TIMESTAMPING ancillary data is sent along with the packet data on a normal recvmsg(). Since this is not a socket error, it is not accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case, the meaning of the three fields in struct scm_timestamping is implicitly defined. ts[0] holds a software timestamp if set, ts[1] is again deprecated and ts[2] holds a hardware timestamp if set. 3. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP Hardware time stamping must also be initialized for each device driver that is expected to do hardware time stamping. The parameter is defined in include/uapi/linux/net_tstamp.h as: struct hwtstamp_config { int flags; /* no flags defined right now, must be zero */ int tx_type; /* HWTSTAMP_TX_* */ int rx_filter; /* HWTSTAMP_FILTER_* */ }; Desired behavior is passed into the kernel and to a specific device by calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose ifr_data points to a struct hwtstamp_config. The tx_type and rx_filter are hints to the driver what it is expected to do. If the requested fine-grained filtering for incoming packets is not supported, the driver may time stamp more than just the requested types of packets. Drivers are free to use a more permissive configuration than the requested configuration. It is expected that drivers should only implement directly the most generic mode that can be supported. For example if the hardware can support HWTSTAMP_FILTER_V2_EVENT, then it should generally always upscale HWTSTAMP_FILTER_V2_L2_SYNC_MESSAGE, and so forth, as HWTSTAMP_FILTER_V2_EVENT is more generic (and more useful to applications). A driver which supports hardware time stamping shall update the struct with the actual, possibly more permissive configuration. If the requested packets cannot be time stamped, then nothing should be changed and ERANGE shall be returned (in contrast to EINVAL, which indicates that SIOCSHWTSTAMP is not supported at all). Only a processes with admin rights may change the configuration. User space is responsible to ensure that multiple processes don't interfere with each other and that the settings are reset. Any process can read the actual configuration by passing this structure to ioctl(SIOCGHWTSTAMP) in the same way. However, this has not been implemented in all drivers. /* possible values for hwtstamp_config->tx_type */ enum { /* * no outgoing packet will need hardware time stamping; * should a packet arrive which asks for it, no hardware * time stamping will be done */ HWTSTAMP_TX_OFF, /* * enables hardware time stamping for outgoing packets; * the sender of the packet decides which are to be * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE * before sending the packet */ HWTSTAMP_TX_ON, }; /* possible values for hwtstamp_config->rx_filter */ enum { /* time stamp no incoming packet at all */ HWTSTAMP_FILTER_NONE, /* time stamp any incoming packet */ HWTSTAMP_FILTER_ALL, /* return value: time stamp all packets requested plus some others */ HWTSTAMP_FILTER_SOME, /* PTP v1, UDP, any kind of event packet */ HWTSTAMP_FILTER_PTP_V1_L4_EVENT, /* for the complete list of values, please check * the include file include/uapi/linux/net_tstamp.h */ }; 3.1 Hardware Timestamping Implementation: Device Drivers A driver which supports hardware time stamping must support the SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with the actual values as described in the section on SIOCSHWTSTAMP. It should also support SIOCGHWTSTAMP. Time stamps for received packets must be stored in the skb. To get a pointer to the shared time stamp structure of the skb call skb_hwtstamps(). Then set the time stamps in the structure: struct skb_shared_hwtstamps { /* hardware time stamp transformed into duration * since arbitrary point in time */ ktime_t hwtstamp; }; Time stamps for outgoing packets are to be generated as follows: - In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) is set no-zero. If yes, then the driver is expected to do hardware time stamping. - If this is possible for the skb and requested, then declare that the driver is doing the time stamping by setting the flag SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; You might want to keep a pointer to the associated skb for the next step and not free the skb. A driver not supporting hardware time stamping doesn't do that. A driver must never touch sk_buff::tstamp! It is used to store software generated time stamps by the network subsystem. - Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware as possible. skb_tx_timestamp() provides a software time stamp if requested and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set). - As soon as the driver has sent the packet and/or obtained a hardware time stamp for it, it passes the time stamp back by calling skb_hwtstamp_tx() with the original skb, the raw hardware time stamp. skb_hwtstamp_tx() clones the original skb and adds the timestamps, therefore the original skb has to be freed now. If obtaining the hardware time stamp somehow fails, then the driver should not fall back to software time stamping. The rationale is that this would occur at a later time in the processing pipeline than other software time stamping and therefore could lead to unexpected deltas between time stamps. |
이상을 기초로 하여 SO_TIMESTAMPING: powering fleetwide RPC monitoring은 실제 사용설명을 합니다.
아래는 2009년 Patrick Ohly가 개발한 Socket timestamping 프로그램입니다.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 |
// SPDX-License-Identifier: GPL-2.0-only /* * This program demonstrates how the various time stamping features in * the Linux kernel work. It emulates the behavior of a PTP * implementation in stand-alone master mode by sending PTPv1 Sync * multicasts once every second. It looks for similar packets, but * beyond that doesn't actually implement PTP. * * Outgoing packets are time stamped with SO_TIMESTAMPING with or * without hardware support. * * Incoming packets are time stamped with SO_TIMESTAMPING with or * without hardware support, SIOCGSTAMP[NS] (per-socket time stamp) and * SO_TIMESTAMP[NS]. * * Copyright (C) 2009 Intel Corporation. * Author: Patrick Ohly <patrick.ohly@intel.com> */ #include <stdio.h> #include <stdlib.h> #include <errno.h> #include <string.h> #include <sys/time.h> #include <sys/socket.h> #include <sys/select.h> #include <sys/ioctl.h> #include <arpa/inet.h> #include <net/if.h> #include <asm/types.h> #include <linux/net_tstamp.h> #include <linux/errqueue.h> #include <linux/sockios.h> #ifndef SO_TIMESTAMPING # define SO_TIMESTAMPING 37 # define SCM_TIMESTAMPING SO_TIMESTAMPING #endif #ifndef SO_TIMESTAMPNS # define SO_TIMESTAMPNS 35 #endif static void usage(const char *error) { if (error) printf("invalid option: %s\n", error); printf("timestamping <interface> [bind_phc_index] [option]*\n\n" "Options:\n" " IP_MULTICAST_LOOP - looping outgoing multicasts\n" " SO_TIMESTAMP - normal software time stamping, ms resolution\n" " SO_TIMESTAMPNS - more accurate software time stamping\n" " SOF_TIMESTAMPING_TX_HARDWARE - hardware time stamping of outgoing packets\n" " SOF_TIMESTAMPING_TX_SOFTWARE - software fallback for outgoing packets\n" " SOF_TIMESTAMPING_RX_HARDWARE - hardware time stamping of incoming packets\n" " SOF_TIMESTAMPING_RX_SOFTWARE - software fallback for incoming packets\n" " SOF_TIMESTAMPING_SOFTWARE - request reporting of software time stamps\n" " SOF_TIMESTAMPING_RAW_HARDWARE - request reporting of raw HW time stamps\n" " SOF_TIMESTAMPING_BIND_PHC - request to bind a PHC of PTP vclock\n" " SIOCGSTAMP - check last socket time stamp\n" " SIOCGSTAMPNS - more accurate socket time stamp\n" " PTPV2 - use PTPv2 messages\n"); exit(1); } static void bail(const char *error) { printf("%s: %s\n", error, strerror(errno)); exit(1); } static const unsigned char sync[] = { 0x00, 0x01, 0x00, 0x01, 0x5f, 0x44, 0x46, 0x4c, 0x54, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01, /* fake uuid */ 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x00, 0x01, 0x00, 0x37, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x49, 0x05, 0xcd, 0x01, 0x29, 0xb1, 0x8d, 0xb0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, /* fake uuid */ 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x00, 0x00, 0x00, 0x37, 0x00, 0x00, 0x00, 0x04, 0x44, 0x46, 0x4c, 0x54, 0x00, 0x00, 0xf0, 0x60, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0xf0, 0x60, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x44, 0x46, 0x4c, 0x54, 0x00, 0x01, /* fake uuid */ 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }; static const unsigned char sync_v2[] = { 0x00, 0x02, 0x00, 0x2C, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFF, 0xFE, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, }; static void sendpacket(int sock, struct sockaddr *addr, socklen_t addr_len, int ptpv2) { size_t sync_len = ptpv2 ? sizeof(sync_v2) : sizeof(sync); const void *sync_p = ptpv2 ? sync_v2 : sync; struct timeval now; int res; res = sendto(sock, sync_p, sync_len, 0, addr, addr_len); gettimeofday(&now, 0); if (res < 0) printf("%s: %s\n", "send", strerror(errno)); else printf("%ld.%06ld: sent %d bytes\n", (long)now.tv_sec, (long)now.tv_usec, res); } static void printpacket(struct msghdr *msg, int res, char *data, int sock, int recvmsg_flags, int siocgstamp, int siocgstampns, int ptpv2) { struct sockaddr_in *from_addr = (struct sockaddr_in *)msg->msg_name; size_t sync_len = ptpv2 ? sizeof(sync_v2) : sizeof(sync); const void *sync_p = ptpv2 ? sync_v2 : sync; struct cmsghdr *cmsg; struct timeval tv; struct timespec ts; struct timeval now; gettimeofday(&now, 0); printf("%ld.%06ld: received %s data, %d bytes from %s, %zu bytes control messages\n", (long)now.tv_sec, (long)now.tv_usec, (recvmsg_flags & MSG_ERRQUEUE) ? "error" : "regular", res, inet_ntoa(from_addr->sin_addr), msg->msg_controllen); for (cmsg = CMSG_FIRSTHDR(msg); cmsg; cmsg = CMSG_NXTHDR(msg, cmsg)) { printf(" cmsg len %zu: ", cmsg->cmsg_len); switch (cmsg->cmsg_level) { case SOL_SOCKET: printf("SOL_SOCKET "); switch (cmsg->cmsg_type) { case SO_TIMESTAMP: { struct timeval *stamp = (struct timeval *)CMSG_DATA(cmsg); printf("SO_TIMESTAMP %ld.%06ld", (long)stamp->tv_sec, (long)stamp->tv_usec); break; } case SO_TIMESTAMPNS: { struct timespec *stamp = (struct timespec *)CMSG_DATA(cmsg); printf("SO_TIMESTAMPNS %ld.%09ld", (long)stamp->tv_sec, (long)stamp->tv_nsec); break; } case SO_TIMESTAMPING: { struct timespec *stamp = (struct timespec *)CMSG_DATA(cmsg); printf("SO_TIMESTAMPING "); printf("SW %ld.%09ld ", (long)stamp->tv_sec, (long)stamp->tv_nsec); stamp++; /* skip deprecated HW transformed */ stamp++; printf("HW raw %ld.%09ld", (long)stamp->tv_sec, (long)stamp->tv_nsec); break; } default: printf("type %d", cmsg->cmsg_type); break; } break; case IPPROTO_IP: printf("IPPROTO_IP "); switch (cmsg->cmsg_type) { case IP_RECVERR: { struct sock_extended_err *err = (struct sock_extended_err *)CMSG_DATA(cmsg); printf("IP_RECVERR ee_errno '%s' ee_origin %d => %s", strerror(err->ee_errno), err->ee_origin, #ifdef SO_EE_ORIGIN_TIMESTAMPING err->ee_origin == SO_EE_ORIGIN_TIMESTAMPING ? "bounced packet" : "unexpected origin" #else "probably SO_EE_ORIGIN_TIMESTAMPING" #endif ); if (res < sync_len) printf(" => truncated data?!"); else if (!memcmp(sync_p, data + res - sync_len, sync_len)) printf(" => GOT OUR DATA BACK (HURRAY!)"); break; } case IP_PKTINFO: { struct in_pktinfo *pktinfo = (struct in_pktinfo *)CMSG_DATA(cmsg); printf("IP_PKTINFO interface index %u", pktinfo->ipi_ifindex); break; } default: printf("type %d", cmsg->cmsg_type); break; } break; default: printf("level %d type %d", cmsg->cmsg_level, cmsg->cmsg_type); break; } printf("\n"); } if (siocgstamp) { if (ioctl(sock, SIOCGSTAMP, &tv)) printf(" %s: %s\n", "SIOCGSTAMP", strerror(errno)); else printf("SIOCGSTAMP %ld.%06ld\n", (long)tv.tv_sec, (long)tv.tv_usec); } if (siocgstampns) { if (ioctl(sock, SIOCGSTAMPNS, &ts)) printf(" %s: %s\n", "SIOCGSTAMPNS", strerror(errno)); else printf("SIOCGSTAMPNS %ld.%09ld\n", (long)ts.tv_sec, (long)ts.tv_nsec); } } static void recvpacket(int sock, int recvmsg_flags, int siocgstamp, int siocgstampns, int ptpv2) { char data[256]; struct msghdr msg; struct iovec entry; struct sockaddr_in from_addr; struct { struct cmsghdr cm; char control[512]; } control; int res; memset(&msg, 0, sizeof(msg)); msg.msg_iov = &entry; msg.msg_iovlen = 1; entry.iov_base = data; entry.iov_len = sizeof(data); msg.msg_name = (caddr_t)&from_addr; msg.msg_namelen = sizeof(from_addr); msg.msg_control = &control; msg.msg_controllen = sizeof(control); res = recvmsg(sock, &msg, recvmsg_flags|MSG_DONTWAIT); if (res < 0) { printf("%s %s: %s\n", "recvmsg", (recvmsg_flags & MSG_ERRQUEUE) ? "error" : "regular", strerror(errno)); } else { printpacket(&msg, res, data, sock, recvmsg_flags, siocgstamp, siocgstampns, ptpv2); } } int main(int argc, char **argv) { int so_timestamp = 0; int so_timestampns = 0; int siocgstamp = 0; int siocgstampns = 0; int ip_multicast_loop = 0; int ptpv2 = 0; char *interface; int i; int enabled = 1; int sock; struct ifreq device; struct ifreq hwtstamp; struct hwtstamp_config hwconfig, hwconfig_requested; struct so_timestamping so_timestamping_get = { 0, 0 }; struct so_timestamping so_timestamping = { 0, 0 }; struct sockaddr_in addr; struct ip_mreq imr; struct in_addr iaddr; int val; socklen_t len; struct timeval next; size_t if_len; if (argc < 2) usage(0); interface = argv[1]; if_len = strlen(interface); if (if_len >= IFNAMSIZ) { printf("interface name exceeds IFNAMSIZ\n"); exit(1); } if (argc >= 3 && sscanf(argv[2], "%d", &so_timestamping.bind_phc) == 1) val = 3; else val = 2; for (i = val; i < argc; i++) { if (!strcasecmp(argv[i], "SO_TIMESTAMP")) so_timestamp = 1; else if (!strcasecmp(argv[i], "SO_TIMESTAMPNS")) so_timestampns = 1; else if (!strcasecmp(argv[i], "SIOCGSTAMP")) siocgstamp = 1; else if (!strcasecmp(argv[i], "SIOCGSTAMPNS")) siocgstampns = 1; else if (!strcasecmp(argv[i], "IP_MULTICAST_LOOP")) ip_multicast_loop = 1; else if (!strcasecmp(argv[i], "PTPV2")) ptpv2 = 1; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_TX_HARDWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_TX_HARDWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_TX_SOFTWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_TX_SOFTWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_RX_HARDWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_RX_HARDWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_RX_SOFTWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_RX_SOFTWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_SOFTWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_SOFTWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_RAW_HARDWARE")) so_timestamping.flags |= SOF_TIMESTAMPING_RAW_HARDWARE; else if (!strcasecmp(argv[i], "SOF_TIMESTAMPING_BIND_PHC")) so_timestamping.flags |= SOF_TIMESTAMPING_BIND_PHC; else usage(argv[i]); } sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP); if (sock < 0) bail("socket"); memset(&device, 0, sizeof(device)); memcpy(device.ifr_name, interface, if_len + 1); if (ioctl(sock, SIOCGIFADDR, &device) < 0) bail("getting interface IP address"); memset(&hwtstamp, 0, sizeof(hwtstamp)); memcpy(hwtstamp.ifr_name, interface, if_len + 1); hwtstamp.ifr_data = (void *)&hwconfig; memset(&hwconfig, 0, sizeof(hwconfig)); hwconfig.tx_type = (so_timestamping.flags & SOF_TIMESTAMPING_TX_HARDWARE) ? HWTSTAMP_TX_ON : HWTSTAMP_TX_OFF; hwconfig.rx_filter = (so_timestamping.flags & SOF_TIMESTAMPING_RX_HARDWARE) ? ptpv2 ? HWTSTAMP_FILTER_PTP_V2_L4_SYNC : HWTSTAMP_FILTER_PTP_V1_L4_SYNC : HWTSTAMP_FILTER_NONE; hwconfig_requested = hwconfig; if (ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) < 0) { if ((errno == EINVAL || errno == ENOTSUP) && hwconfig_requested.tx_type == HWTSTAMP_TX_OFF && hwconfig_requested.rx_filter == HWTSTAMP_FILTER_NONE) printf("SIOCSHWTSTAMP: disabling hardware time stamping not possible\n"); else bail("SIOCSHWTSTAMP"); } printf("SIOCSHWTSTAMP: tx_type %d requested, got %d; rx_filter %d requested, got %d\n", hwconfig_requested.tx_type, hwconfig.tx_type, hwconfig_requested.rx_filter, hwconfig.rx_filter); /* bind to PTP port */ addr.sin_family = AF_INET; addr.sin_addr.s_addr = htonl(INADDR_ANY); addr.sin_port = htons(319 /* PTP event port */); if (bind(sock, (struct sockaddr *)&addr, sizeof(struct sockaddr_in)) < 0) bail("bind"); if (setsockopt(sock, SOL_SOCKET, SO_BINDTODEVICE, interface, if_len)) bail("bind device"); /* set multicast group for outgoing packets */ inet_aton("224.0.1.130", &iaddr); /* alternate PTP domain 1 */ addr.sin_addr = iaddr; imr.imr_multiaddr.s_addr = iaddr.s_addr; imr.imr_interface.s_addr = ((struct sockaddr_in *)&device.ifr_addr)->sin_addr.s_addr; if (setsockopt(sock, IPPROTO_IP, IP_MULTICAST_IF, &imr.imr_interface.s_addr, sizeof(struct in_addr)) < 0) bail("set multicast"); /* join multicast group, loop our own packet */ if (setsockopt(sock, IPPROTO_IP, IP_ADD_MEMBERSHIP, &imr, sizeof(struct ip_mreq)) < 0) bail("join multicast group"); if (setsockopt(sock, IPPROTO_IP, IP_MULTICAST_LOOP, &ip_multicast_loop, sizeof(enabled)) < 0) { bail("loop multicast"); } /* set socket options for time stamping */ if (so_timestamp && setsockopt(sock, SOL_SOCKET, SO_TIMESTAMP, &enabled, sizeof(enabled)) < 0) bail("setsockopt SO_TIMESTAMP"); if (so_timestampns && setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPNS, &enabled, sizeof(enabled)) < 0) bail("setsockopt SO_TIMESTAMPNS"); if (so_timestamping.flags && setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &so_timestamping, sizeof(so_timestamping)) < 0) bail("setsockopt SO_TIMESTAMPING"); /* request IP_PKTINFO for debugging purposes */ if (setsockopt(sock, SOL_IP, IP_PKTINFO, &enabled, sizeof(enabled)) < 0) printf("%s: %s\n", "setsockopt IP_PKTINFO", strerror(errno)); /* verify socket options */ len = sizeof(val); if (getsockopt(sock, SOL_SOCKET, SO_TIMESTAMP, &val, &len) < 0) printf("%s: %s\n", "getsockopt SO_TIMESTAMP", strerror(errno)); else printf("SO_TIMESTAMP %d\n", val); if (getsockopt(sock, SOL_SOCKET, SO_TIMESTAMPNS, &val, &len) < 0) printf("%s: %s\n", "getsockopt SO_TIMESTAMPNS", strerror(errno)); else printf("SO_TIMESTAMPNS %d\n", val); len = sizeof(so_timestamping_get); if (getsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &so_timestamping_get, &len) < 0) { printf("%s: %s\n", "getsockopt SO_TIMESTAMPING", strerror(errno)); } else { printf("SO_TIMESTAMPING flags %d, bind phc %d\n", so_timestamping_get.flags, so_timestamping_get.bind_phc); if (so_timestamping_get.flags != so_timestamping.flags || so_timestamping_get.bind_phc != so_timestamping.bind_phc) printf(" not expected, flags %d, bind phc %d\n", so_timestamping.flags, so_timestamping.bind_phc); } /* send packets forever every five seconds */ gettimeofday(&next, 0); next.tv_sec = (next.tv_sec + 1) / 5 * 5; next.tv_usec = 0; while (1) { struct timeval now; struct timeval delta; long delta_us; int res; fd_set readfs, errorfs; gettimeofday(&now, 0); delta_us = (long)(next.tv_sec - now.tv_sec) * 1000000 + (long)(next.tv_usec - now.tv_usec); if (delta_us > 0) { /* continue waiting for timeout or data */ delta.tv_sec = delta_us / 1000000; delta.tv_usec = delta_us % 1000000; FD_ZERO(&readfs); FD_ZERO(&errorfs); FD_SET(sock, &readfs); FD_SET(sock, &errorfs); printf("%ld.%06ld: select %ldus\n", (long)now.tv_sec, (long)now.tv_usec, delta_us); res = select(sock + 1, &readfs, 0, &errorfs, &delta); gettimeofday(&now, 0); printf("%ld.%06ld: select returned: %d, %s\n", (long)now.tv_sec, (long)now.tv_usec, res, res < 0 ? strerror(errno) : "success"); if (res > 0) { if (FD_ISSET(sock, &readfs)) printf("ready for reading\n"); if (FD_ISSET(sock, &errorfs)) printf("has error\n"); recvpacket(sock, 0, siocgstamp, siocgstampns, ptpv2); recvpacket(sock, MSG_ERRQUEUE, siocgstamp, siocgstampns, ptpv2); } } else { /* write one packet */ sendpacket(sock, (struct sockaddr *)&addr, sizeof(addr), ptpv2); next.tv_sec += 5; continue; } } return 0; } |
DMA환경에서 많이 사용하는 Solarflare, Mellanox, Exanic은 고유한 TCP Accelerator를 제공합니다. 관련한 프로젝트를 보면 timestamping과 관련한 툴을 각각 제공합니다.
먼저 Solarflare. AMD(옛날 Solarflare) OpenOnload에서 제공하는 hwtimestamping을 이용합니다.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 |
/* Example application to demonstrate use of the timestamping API * * This application will receive packets, and display their * hardware timestamps. * * Invoke with "--help" to see the options it supports. * * Example: * (host1)$ rx_timestamping * UDP socket created, listening on port 9000 * Selecting software timestamping mode. * (host2)$ echo payload | nc -u host1 9000 * Packet 0 - 8 bytes timestamp 1395768726.443243000 */ #include <unistd.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <inttypes.h> #include <errno.h> #include <time.h> #include <getopt.h> #include <netdb.h> #include <sys/types.h> #include <sys/socket.h> #include <sys/ioctl.h> #include <net/if.h> #ifdef ONLOADEXT_AVAILABLE #include "onload/extensions_timestamping.h" #endif /* Use the kernel definitions if possible - * But if not, use our own local definitions, and Onload will allow it. * - Though you still need a reasonably recent kernel to get hardware * timestamping. */ #ifdef NO_KERNEL_TS_INCLUDE #include <time.h> struct hwtstamp_config { int flags; /* no flags defined right now, must be zero */ int tx_type; /* HWTSTAMP_TX_* */ int rx_filter; /* HWTSTAMP_FILTER_* */ }; enum { SOF_TIMESTAMPING_TX_HARDWARE = (1<<0), SOF_TIMESTAMPING_TX_SOFTWARE = (1<<1), SOF_TIMESTAMPING_RX_HARDWARE = (1<<2), SOF_TIMESTAMPING_RX_SOFTWARE = (1<<3), SOF_TIMESTAMPING_SOFTWARE = (1<<4), SOF_TIMESTAMPING_SYS_HARDWARE = (1<<5), SOF_TIMESTAMPING_RAW_HARDWARE = (1<<6), SOF_TIMESTAMPING_MASK = (SOF_TIMESTAMPING_RAW_HARDWARE - 1) | SOF_TIMESTAMPING_RAW_HARDWARE }; #else #include <linux/net_tstamp.h> #include <linux/sockios.h> #endif /* These are defined in socket.h, but older versions might not have all 3 */ #ifndef SO_TIMESTAMP #define SO_TIMESTAMP 29 #endif #ifndef SO_TIMESTAMPNS #define SO_TIMESTAMPNS 35 #endif #ifndef SO_TIMESTAMPING #define SO_TIMESTAMPING 37 #endif /* Seconds.nanoseconds format */ #define TIME_FMT "%" PRIu64 ".%.9" PRIu64 " " #define OTIME_FMT "%" PRIu64 ".%.9" PRIu32 " " /* Assert-like macros */ #define TEST(x) \ do { \ if( ! (x) ) { \ fprintf(stderr, "ERROR: '%s' failed\n", #x); \ fprintf(stderr, "ERROR: at %s:%d\n", __FILE__, __LINE__); \ exit(1); \ } \ } while( 0 ) #define TRY(x) \ do { \ int __rc = (x); \ if( __rc < 0 ) { \ fprintf(stderr, "ERROR: TRY(%s) failed\n", #x); \ fprintf(stderr, "ERROR: at %s:%d\n", __FILE__, __LINE__); \ fprintf(stderr, "ERROR: rc=%d errno=%d (%s)\n", \ __rc, errno, strerror(errno)); \ exit(1); \ } \ } while( 0 ) struct configuration { char const* cfg_ioctl; /* e.g. eth6 - calls the ts enable ioctl */ unsigned short cfg_port; /* listen port */ int cfg_protocol; /* udp or tcp? */ unsigned int cfg_max_packets; /* Stop after this many (0=forever) */ int cfg_ext; /* Use extension API? */ }; /* Commandline options, configuration etc. */ void print_help(void) { printf("Usage:\n" "\t--ioctl\t<ethX>\tDevice to send timestamping enable ioctl. " "Default: None\n" "\t--port\t<num>\tPort to listen on. " "Default: 9000\n" "\t--proto\t[TCP|UDP]. " "Default: UDP\n" "\t--max\t<num>\tStop after n packets. " "Default: Run forever\n" #ifdef ONLOADEXT_AVAILABLE "\t--ext\t\tUse extensions API rather than SO_TIMESTAMPING.\n" #endif ); exit(-1); } static void get_protcol(struct configuration* cfg, const char* protocol) { if( 0 == strcasecmp(protocol, "UDP") ) { cfg->cfg_protocol = IPPROTO_UDP; } else if( 0 == strcasecmp(protocol, "TCP") ) { cfg->cfg_protocol = IPPROTO_TCP; } else { printf("ERROR: '%s' is not a recognised protocol (TCP or UCP).\n", protocol); exit(-EINVAL); } } static void parse_options( int argc, char** argv, struct configuration* cfg ) { int option_index = 0; int opt; static struct option long_options[] = { { "ioctl", required_argument, 0, 'i' }, { "port", required_argument, 0, 'p' }, { "proto", required_argument, 0, 'P' }, { "max", required_argument, 0, 'n' }, { "ext", no_argument, 0, 'e' }, { 0, no_argument, 0, 0 } }; const char* optstring = "i:p:P:n:"; /* Defaults */ bzero(cfg, sizeof(struct configuration)); cfg->cfg_port = 9000; cfg->cfg_protocol = IPPROTO_UDP; opt = getopt_long(argc, argv, optstring, long_options, &option_index); while( opt != -1 ) { switch( opt ) { case 'i': cfg->cfg_ioctl = optarg; break; case 'p': cfg->cfg_port = atoi(optarg); break; case 'P': get_protcol(cfg, optarg); break; case 'n': cfg->cfg_max_packets = atoi(optarg); break; #ifdef ONLOADEXT_AVAILABLE case 'e': cfg->cfg_ext = 1; break; #endif default: print_help(); break; } opt = getopt_long(argc, argv, optstring, long_options, &option_index); } } /* Connection */ static void make_address(unsigned short port, struct sockaddr_in* host_address) { bzero(host_address, sizeof(struct sockaddr_in)); host_address->sin_family = AF_INET; host_address->sin_port = htons(port); host_address->sin_addr.s_addr = INADDR_ANY; } /* This requires a bit of explanation. * Typically, you have to enable hardware timestamping on an interface. * Any application can do it, and then it's available to everyone. * The easiest way to do this, is just to run sfptpd. * * But in case you need to do it manually; here is the code, but * that's only supported on reasonably recent versions * * Option: --ioctl ethX * * NOTE: * Usage of the ioctl call is discouraged. A better method, if using * hardware timestamping, would be to use sfptpd as it will effectively * make the ioctl call for you. * */ static void do_ioctl(struct configuration* cfg, int sock) { #ifdef SIOCSHWTSTAMP struct ifreq ifr; struct hwtstamp_config hwc; #endif if( cfg->cfg_ioctl == NULL ) return; #ifdef SIOCSHWTSTAMP bzero(&ifr, sizeof(ifr)); snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", cfg->cfg_ioctl); /* Standard kernel ioctl options */ hwc.flags = 0; hwc.tx_type = 0; hwc.rx_filter = HWTSTAMP_FILTER_ALL; ifr.ifr_data = (char*)&hwc; TRY( ioctl(sock, SIOCSHWTSTAMP, &ifr) ); return; #else (void) sock; printf("SIOCHWTSTAMP ioctl not supported on this kernel.\n"); exit(-ENOTSUP); return; #endif } /* This routine selects the correct socket option to enable timestamping. */ static void do_ts_sockopt(struct configuration* cfg, int sock) { printf("Selecting hardware timestamping mode.\n"); #ifdef ONLOADEXT_AVAILABLE if( cfg->cfg_ext ) TRY(onload_timestamping_request(sock, ONLOAD_TIMESTAMPING_FLAG_RX_NIC | ONLOAD_TIMESTAMPING_FLAG_RX_CPACKET)); else #endif { int enable = SOF_TIMESTAMPING_RX_HARDWARE | SOF_TIMESTAMPING_RAW_HARDWARE | SOF_TIMESTAMPING_SYS_HARDWARE | SOF_TIMESTAMPING_SOFTWARE; TRY(setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &enable, sizeof(int))); } } static int add_socket(struct configuration* cfg) { int s; struct sockaddr_in host_address; int domain = SOCK_DGRAM; if ( cfg->cfg_protocol == IPPROTO_TCP ) domain = SOCK_STREAM; make_address(cfg->cfg_port, &host_address); s = socket(PF_INET, domain, cfg->cfg_protocol); TEST(s >= 0); TRY(bind(s, (struct sockaddr*)&host_address, sizeof(host_address)) ); printf("Socket created, listening on port %d\n", cfg->cfg_port); return s; } static int accept_child(int parent) { int child; socklen_t clilen; struct sockaddr_in cli_addr; clilen = sizeof(cli_addr); TRY(listen(parent, 1)); child = accept(parent, (struct sockaddr* ) &cli_addr, &clilen); TEST(child >= 0); printf("Socket accepted\n"); return child; } /* Processing */ static void print_time(struct timespec* ts) { if( ts != NULL ) { /* Hardware timestamping provides three timestamps - * system (software) * transformed (hw converted to sw) * raw (hardware) * in that order - though depending on socket option, you may have 0 in * some of them. */ printf("timestamps " TIME_FMT TIME_FMT TIME_FMT "\n", (uint64_t)ts[0].tv_sec, (uint64_t)ts[0].tv_nsec, (uint64_t)ts[1].tv_sec, (uint64_t)ts[1].tv_nsec, (uint64_t)ts[2].tv_sec, (uint64_t)ts[2].tv_nsec ); } else { printf( "no timestamp\n" ); } } /* Given a packet, extract the timestamp(s) */ static void handle_time(struct msghdr* msg, struct configuration* cfg) { struct timespec* ts = NULL; struct cmsghdr* cmsg; for( cmsg = CMSG_FIRSTHDR(msg); cmsg; cmsg = CMSG_NXTHDR(msg,cmsg) ) { if( cmsg->cmsg_level != SOL_SOCKET ) continue; switch( cmsg->cmsg_type ) { case SO_TIMESTAMPNS: ts = (struct timespec*) CMSG_DATA(cmsg); break; case SO_TIMESTAMPING: #ifdef ONLOADEXT_AVAILABLE if( cfg->cfg_ext ) { struct onload_timestamp* ts = (struct onload_timestamp*) CMSG_DATA(cmsg); printf("timestamps " OTIME_FMT OTIME_FMT "\n", ts[0].sec, ts[0].nsec, ts[1].sec, ts[1].nsec); return; } #endif ts = (struct timespec*) CMSG_DATA(cmsg); break; default: /* Ignore other cmsg options */ break; } } print_time(ts); } /* Receive a packet, and print out the timestamps from it */ static int do_recv(int sock, unsigned int pkt_num, struct configuration* cfg) { struct msghdr msg; struct iovec iov; struct sockaddr_in host_address; char buffer[2048]; char control[1024]; int got; /* recvmsg header structure */ make_address(0, &host_address); iov.iov_base = buffer; iov.iov_len = 2048; msg.msg_iov = &iov; msg.msg_iovlen = 1; msg.msg_name = &host_address; msg.msg_namelen = sizeof(struct sockaddr_in); msg.msg_control = control; msg.msg_controllen = 1024; /* block for message */ got = recvmsg(sock, &msg, 0); if( !got && errno == EAGAIN ) return 0; printf("Packet %d - %d bytes\t", pkt_num, got); handle_time(&msg, cfg); return got; }; int main(int argc, char** argv) { struct configuration cfg; int parent, sock, got; unsigned int pkt_num = 0; parse_options(argc, argv, &cfg); /* Initialise */ parent = add_socket(&cfg); do_ioctl(&cfg, parent); sock = parent; if( cfg.cfg_protocol == IPPROTO_TCP ) sock = accept_child(parent); do_ts_sockopt(&cfg, sock); /* Run forever */ while((pkt_num++ < cfg.cfg_max_packets || (cfg.cfg_max_packets == 0) ) ) { got = do_recv(sock, pkt_num, &cfg); /* TCP can detect an exit; for UDP, zero payload packets are valid */ if ( got == 0 && cfg.cfg_protocol == IPPROTO_TCP ) { printf( "recvmsg returned 0 - end of stream\n" ); break; } } close(sock); if( cfg.cfg_protocol == IPPROTO_TCP ) close(parent); return 0; } |
libvma의 경우 별도의 도구를 제공하지 않고 리눅스커널에 포함되어 있는 timestamping.c를 이용하는 방법을 제시합니다.
2.
이상의 개념과 도구를 이용하여 실 환경에 적용할 수 있을까요? 이런 목적으로 이루어진 프로젝트가 있습니다. 일본 iij engineering이 소개한 How to measure network latency using hardware timestamps은 앞서의 so_timestamps를 이용하여 네트워크 레이턴시를 측정하는 방법을 소개합니다. 앞서 프로그램과 다른 점은 측정한 값을 파일로 저장하고 그래프로 출력할 수 있도록 코드를 수정한 점입니다.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 |
/**************************************************************************\ ** Example for RX timestamping sockets API ** 2014/04/03 ** (c) Level 5 Networks Limited. \**************************************************************************/ /* Example application to demonstrate use of the timestamping API * * This application will receive packets, and display their * hardware timestamps. * * Invoke with "--help" to see the options it supports. */ #include <errno.h> #include <getopt.h> #include <inttypes.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <time.h> #include <unistd.h> #include <arpa/inet.h> #include <net/if.h> #include <netdb.h> #include <sys/ioctl.h> #include <sys/socket.h> #include <sys/time.h> #include <sys/types.h> static uint64_t total_received = 0; static uint64_t *nic_kernel_latency_numbers = NULL; static uint64_t *nic_user_latency_numbers = NULL; static uint64_t *kernel_user_latency_numbers = NULL; /* Use the kernel definitions if possible - * But if not, use our own local definitions, and Onload will allow it. * - Though you still need a reasonably recent kernel to get hardware * timestamping. */ #ifdef NO_KERNEL_TS_INCLUDE #include <time.h> struct hwtstamp_config { int flags; /* no flags defined right now, must be zero */ int tx_type; /* HWTSTAMP_TX_* */ int rx_filter; /* HWTSTAMP_FILTER_* */ }; enum { SOF_TIMESTAMPING_TX_HARDWARE = (1 << 0), SOF_TIMESTAMPING_TX_SOFTWARE = (1 << 1), SOF_TIMESTAMPING_RX_HARDWARE = (1 << 2), SOF_TIMESTAMPING_RX_SOFTWARE = (1 << 3), SOF_TIMESTAMPING_SOFTWARE = (1 << 4), SOF_TIMESTAMPING_SYS_HARDWARE = (1 << 5), SOF_TIMESTAMPING_RAW_HARDWARE = (1 << 6), SOF_TIMESTAMPING_MASK = (SOF_TIMESTAMPING_RAW_HARDWARE - 1) | SOF_TIMESTAMPING_RAW_HARDWARE }; #else #include <linux/net_tstamp.h> #include <linux/sockios.h> #endif /* These are defined in socket.h, but older versions might not have all 3 */ #ifndef SO_TIMESTAMP #define SO_TIMESTAMP 29 #endif #ifndef SO_TIMESTAMPNS #define SO_TIMESTAMPNS 35 #endif #ifndef SO_TIMESTAMPING #define SO_TIMESTAMPING 37 #endif /* Seconds.nanoseconds format */ #define TIME_FMT "%" PRIu64 ".%.9" PRIu64 " " #define OTIME_FMT "%" PRIu64 ".%.9" PRIu32 " " /* Assert-like macros */ #define TEST(x) \ do { \ if (!(x)) { \ fprintf(stderr, "ERROR: '%s' failed\n", #x); \ fprintf(stderr, "ERROR: at %s:%d\n", __FILE__, __LINE__); \ exit(1); \ } \ } while (0) #define TRY(x) \ do { \ int __rc = (x); \ if (__rc < 0) { \ fprintf(stderr, "ERROR: TRY(%s) failed\n", #x); \ fprintf(stderr, "ERROR: at %s:%d\n", __FILE__, __LINE__); \ fprintf(stderr, "ERROR: rc=%d errno=%d (%s)\n", __rc, errno, \ strerror(errno)); \ exit(1); \ } \ } while (0) struct configuration { char const *cfg_ioctl; /* e.g. eth6 - calls the ts enable ioctl */ unsigned short cfg_port; /* listen port */ int cfg_protocol; /* udp or tcp? */ unsigned int cfg_max_packets; /* Stop after this many (0=forever) */ int cfg_ext; /* Use extension API? */ }; /* Commandline options, configuration etc. */ void print_help(void) { printf("Usage:\n" "\t--ioctl\t<ethX>\tDevice to send timestamping enable ioctl. " "Default: None\n" "\t--port\t<num>\tPort to listen on. " "Default: 9000\n" "\t--proto\t[TCP|UDP]. " "Default: UDP\n" "\t--max\t<num>\tStop after n packets. " "Default: Run forever\n"); exit(-1); } static void get_protcol(struct configuration *cfg, const char *protocol) { if (0 == strcasecmp(protocol, "UDP")) { cfg->cfg_protocol = IPPROTO_UDP; } else if (0 == strcasecmp(protocol, "TCP")) { cfg->cfg_protocol = IPPROTO_TCP; } else { printf("ERROR: '%s' is not a recognised protocol (TCP or UCP).\n", protocol); exit(-EINVAL); } } static void parse_options(int argc, char **argv, struct configuration *cfg) { int option_index = 0; int opt; static struct option long_options[] = { {"ioctl", required_argument, 0, 'i'}, {"port", required_argument, 0, 'p'}, {"proto", required_argument, 0, 'P'}, {"max", required_argument, 0, 'n'}, {"ext", no_argument, 0, 'e'}, {0, no_argument, 0, 0}}; const char *optstring = "i:p:P:n:"; /* Defaults */ bzero(cfg, sizeof(struct configuration)); cfg->cfg_port = 9000; cfg->cfg_protocol = IPPROTO_UDP; opt = getopt_long(argc, argv, optstring, long_options, &option_index); while (opt != -1) { switch (opt) { case 'i': cfg->cfg_ioctl = optarg; break; case 'p': cfg->cfg_port = atoi(optarg); break; case 'P': get_protcol(cfg, optarg); break; case 'n': cfg->cfg_max_packets = atoi(optarg); break; default: print_help(); break; } opt = getopt_long(argc, argv, optstring, long_options, &option_index); } } /* Connection */ static void make_address(unsigned short port, struct sockaddr_in *host_address) { bzero(host_address, sizeof(struct sockaddr_in)); host_address->sin_family = AF_INET; host_address->sin_port = htons(port); host_address->sin_addr.s_addr = INADDR_ANY; } /* This requires a bit of explanation. * Typically, you have to enable hardware timestamping on an interface. * Any application can do it, and then it's available to everyone. * The easiest way to do this, is just to run sfptpd. * * But in case you need to do it manually; here is the code, but * that's only supported on reasonably recent versions * * Option: --ioctl ethX * * NOTE: * Usage of the ioctl call is discouraged. A better method, if using * hardware timestamping, would be to use sfptpd as it will effectively * make the ioctl call for you. * */ static void do_ioctl(struct configuration *cfg, int sock) { #ifdef SIOCSHWTSTAMP struct ifreq ifr; struct hwtstamp_config hwc; #endif if (cfg->cfg_ioctl == NULL) return; #ifdef SIOCSHWTSTAMP bzero(&ifr, sizeof(ifr)); snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", cfg->cfg_ioctl); /* Standard kernel ioctl options */ hwc.flags = 0; hwc.tx_type = 0; hwc.rx_filter = HWTSTAMP_FILTER_ALL; ifr.ifr_data = (char *)&hwc; TRY(ioctl(sock, SIOCSHWTSTAMP, &ifr)); return; #else (void)sock; printf("SIOCHWTSTAMP ioctl not supported on this kernel.\n"); exit(-ENOTSUP); return; #endif } /* This routine selects the correct socket option to enable timestamping. */ static void do_ts_sockopt(struct configuration *cfg, int sock) { printf("Selecting hardware timestamping mode.\n"); { int enable = SOF_TIMESTAMPING_RX_HARDWARE | SOF_TIMESTAMPING_RAW_HARDWARE | SOF_TIMESTAMPING_SYS_HARDWARE | SOF_TIMESTAMPING_SOFTWARE; TRY(setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &enable, sizeof(int))); printf("enabled timestamping sockopt\n"); } } static int add_socket(struct configuration *cfg) { int s; struct sockaddr_in host_address; int domain = SOCK_DGRAM; if (cfg->cfg_protocol == IPPROTO_TCP) domain = SOCK_STREAM; make_address(cfg->cfg_port, &host_address); s = socket(PF_INET, domain, cfg->cfg_protocol); TEST(s >= 0); TRY(bind(s, (struct sockaddr *)&host_address, sizeof(host_address))); printf("Socket created, listening on port %d\n", cfg->cfg_port); return s; } static int accept_child(int parent) { int child; socklen_t clilen; struct sockaddr_in cli_addr; clilen = sizeof(cli_addr); TRY(listen(parent, 1)); child = accept(parent, (struct sockaddr *)&cli_addr, &clilen); TEST(child >= 0); printf("Socket accepted\n"); return child; } /* Processing */ static void print_time(struct timespec *ts) { if (ts == NULL) { printf("no timestamp\n"); return; } /* Hardware timestamping provides three timestamps - * system (software) * transformed (hw converted to sw) * raw (hardware) * in that order - though depending on socket option, you may have 0 in * some of them. */ // printf("timestamps " TIME_FMT TIME_FMT TIME_FMT "\n", // (uint64_t)ts[0].tv_sec, (uint64_t)ts[0].tv_nsec, // (uint64_t)ts[1].tv_sec, (uint64_t)ts[1].tv_nsec, // (uint64_t)ts[2].tv_sec, (uint64_t)ts[2].tv_nsec ); struct timeval time_user; gettimeofday(&time_user, NULL); // printf("time_user : %d.%06d\n", (int) time_user.tv_sec, // (int) time_user.tv_usec); static uint64_t diff_nic_kernel = 0; static uint64_t diff_nic_user = 0; static uint64_t diff_kernel_user = 0; static int64_t nic_kernel_total_diff = 0; uint64_t old_diff_nic_kernel = diff_nic_kernel; uint64_t nanoseconds_nic = ts[2].tv_sec * 1000000000 + ts[2].tv_nsec; uint64_t nanoseconds_kernel = ts[0].tv_sec * 1000000000 + ts[0].tv_nsec; uint64_t nanoseconds_user = time_user.tv_sec * 1000000000 + time_user.tv_usec * 1000; //printf("nic: %ld, kernel: %ld, user: %ld\n", nanoseconds_nic, // nanoseconds_kernel, nanoseconds_user); diff_nic_kernel = (ts[0].tv_sec - ts[2].tv_sec) * 1000000000 + (ts[0].tv_nsec - ts[2].tv_nsec); nic_kernel_latency_numbers[total_received++] = diff_nic_kernel; // all latency numbers are in nanoseconds if (old_diff_nic_kernel != 0) { nic_kernel_total_diff += diff_nic_kernel - old_diff_nic_kernel; } diff_kernel_user = (time_user.tv_sec - ts[0].tv_sec) * 1000000000 + (time_user.tv_usec * 1000 - ts[0].tv_nsec); diff_nic_user = (time_user.tv_sec - ts[2].tv_sec) * 1000000000 + (time_user.tv_usec * 1000 - ts[2].tv_nsec); nic_user_latency_numbers[total_received] = diff_nic_user; // all latency numbers are in nanoseconds kernel_user_latency_numbers[total_received] = diff_kernel_user; // all latency numbers are in nanoseconds // printf("Kernel timestamp %lds %ldns\n", ts[0].tv_sec, ts[0].tv_nsec); // printf("Kernel timestamp %lds %ldns\n", ts[2].tv_sec, ts[2].tv_nsec); // printf("Difference NIC->Kernel: %ld, change of diff_nic_kernel: %ld, at // %ld\n", diff_nic_kernel, diff_nic_kernel-old_diff_nic_kernel, // nic_kernel_total_diff); // printf("Difference NIC->User: %ld\n", diff_nic_user); // printf("Difference Kernel->User: %ld\n", diff_kernel_user); } /* Given a packet, extract the timestamp(s) */ static void handle_time(struct msghdr *msg, struct configuration *cfg) { struct timespec *ts = NULL; struct cmsghdr *cmsg; for (cmsg = CMSG_FIRSTHDR(msg); cmsg; cmsg = CMSG_NXTHDR(msg, cmsg)) { if (cmsg->cmsg_level != SOL_SOCKET) continue; switch (cmsg->cmsg_type) { case SO_TIMESTAMPNS: ts = (struct timespec *)CMSG_DATA(cmsg); break; case SO_TIMESTAMPING: ts = (struct timespec *)CMSG_DATA(cmsg); break; default: /* Ignore other cmsg options */ break; } } print_time(ts); } void broadcast(const char *buffer, int buffer_len) { static int sockfd = -1; static struct sockaddr_in server_addr; if (sockfd == -1) { // Create a UDP socket sockfd = socket(AF_INET, SOCK_DGRAM, 0); if (sockfd < 0) { perror("socket"); exit(1); } // Enable broadcast option int broadcastEnable = 1; if (setsockopt(sockfd, SOL_SOCKET, SO_BROADCAST, &broadcastEnable, sizeof(broadcastEnable)) < 0) { perror("setsockopt"); exit(1); } // Set up the server address structure memset(&server_addr, 0, sizeof(server_addr)); server_addr.sin_family = AF_INET; server_addr.sin_port = htons(4200); server_addr.sin_addr.s_addr = inet_addr("255.255.255.255"); } // Send the UDP packet if (sendto(sockfd, buffer, buffer_len, 0, (struct sockaddr *)&server_addr, sizeof(server_addr)) < 0) { perror("sendto"); exit(1); } } /* Receive a packet, and print out the timestamps from it */ static int do_recv(int sock, unsigned int pkt_num, struct configuration *cfg) { struct msghdr msg; struct iovec iov; struct sockaddr_in host_address; char buffer[2048]; char control[1024]; int got; /* recvmsg header structure */ make_address(0, &host_address); iov.iov_base = buffer; iov.iov_len = 2048; msg.msg_iov = &iov; msg.msg_iovlen = 1; msg.msg_name = &host_address; msg.msg_namelen = sizeof(struct sockaddr_in); msg.msg_control = control; msg.msg_controllen = 1024; /* block for message */ got = recvmsg(sock, &msg, 0); if (!got && errno == EAGAIN) return 0; // printf("Packet %d - %d bytes\n", pkt_num, got); handle_time(&msg, cfg); broadcast(buffer, got); return got; }; int main(int argc, char **argv) { struct configuration cfg; int parent, sock, got; unsigned int pkt_num = 0; parse_options(argc, argv, &cfg); /* Initialise */ parent = add_socket(&cfg); do_ioctl(&cfg, parent); sock = parent; if (cfg.cfg_protocol == IPPROTO_TCP) sock = accept_child(parent); do_ts_sockopt(&cfg, sock); nic_user_latency_numbers = malloc(cfg.cfg_max_packets * sizeof(uint64_t)); nic_kernel_latency_numbers = malloc(cfg.cfg_max_packets * sizeof(uint64_t)); kernel_user_latency_numbers = malloc(cfg.cfg_max_packets * sizeof(uint64_t)); /* Run forever */ while ((pkt_num++ < cfg.cfg_max_packets || (cfg.cfg_max_packets == 0))) { got = do_recv(sock, pkt_num, &cfg); /* TCP can detect an exit; for UDP, zero payload packets are valid */ if (got == 0 && cfg.cfg_protocol == IPPROTO_TCP) { printf("recvmsg returned 0 - end of stream\n"); break; } } FILE *f = fopen("latency.txt", "w"); for (int i = 0; i < total_received; ++i) { fprintf(f, "%ld,%ld,%ld\n", nic_user_latency_numbers[i], nic_kernel_latency_numbers[i], kernel_user_latency_numbers[i]); } fclose(f); close(sock); if (cfg.cfg_protocol == IPPROTO_TCP) close(parent); return 0; } |
원래 프로젝트 레포지토리에는 Rust로 개발된 packet sender가 있지만 아래는 C로 된 packet sender입니다.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 |
/* Packet sender for RX timestamp testing * This program sends packets to test the rx_timestamp application */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> #include <time.h> #include <getopt.h> #include <arpa/inet.h> #include <sys/socket.h> #include <sys/types.h> struct configuration { char *cfg_dest_ip; /* destination IP */ unsigned short cfg_port; /* destination port */ int cfg_protocol; /* udp or tcp? */ unsigned int cfg_max_packets; /* Stop after this many (0=forever) */ unsigned int cfg_pkt_size; /* Packet payload size */ unsigned int cfg_interval_ms; /* Interval between packets in ms */ int cfg_quiet; /* Suppress output per packet */ }; void print_help(void) { printf("Usage:\n" "\t--ip\t<addr>\tDestination IP address. " "Default: 127.0.0.1\n" "\t--port\t<num>\tDestination port. " "Default: 9000\n" "\t--proto\t[TCP|UDP]. " "Default: UDP\n" "\t--size\t<num>\tPacket payload size. " "Default: 64\n" "\t--interval\t<ms>\tInterval between packets in milliseconds. " "Default: 1000\n" "\t--max\t<num>\tStop after n packets. " "Default: Run forever\n" "\t--quiet\tSuppress per-packet output\n"); exit(-1); } static void get_protocol(struct configuration *cfg, const char *protocol) { if (0 == strcasecmp(protocol, "UDP")) { cfg->cfg_protocol = IPPROTO_UDP; } else if (0 == strcasecmp(protocol, "TCP")) { cfg->cfg_protocol = IPPROTO_TCP; } else { printf("ERROR: '%s' is not a recognised protocol (TCP or UDP).\n", protocol); exit(-1); } } static void parse_options(int argc, char **argv, struct configuration *cfg) { int option_index = 0; int opt; static struct option long_options[] = { {"ip", required_argument, 0, 'a'}, {"port", required_argument, 0, 'p'}, {"proto", required_argument, 0, 'P'}, {"size", required_argument, 0, 's'}, {"interval", required_argument, 0, 't'}, {"max", required_argument, 0, 'n'}, {"quiet", no_argument, 0, 'q'}, {"help", no_argument, 0, 'h'}, {0, no_argument, 0, 0} }; const char *optstring = "a:p:P:s:t:n:qh"; /* Defaults */ memset(cfg, 0, sizeof(struct configuration)); cfg->cfg_dest_ip = strdup("127.0.0.1"); cfg->cfg_port = 9000; cfg->cfg_protocol = IPPROTO_UDP; cfg->cfg_pkt_size = 64; cfg->cfg_interval_ms = 1000; cfg->cfg_quiet = 0; opt = getopt_long(argc, argv, optstring, long_options, &option_index); while (opt != -1) { switch (opt) { case 'a': free(cfg->cfg_dest_ip); cfg->cfg_dest_ip = strdup(optarg); break; case 'p': cfg->cfg_port = atoi(optarg); break; case 'P': get_protocol(cfg, optarg); break; case 's': cfg->cfg_pkt_size = atoi(optarg); break; case 't': cfg->cfg_interval_ms = atoi(optarg); break; case 'n': cfg->cfg_max_packets = atoi(optarg); break; case 'q': cfg->cfg_quiet = 1; break; case 'h': default: print_help(); break; } opt = getopt_long(argc, argv, optstring, long_options, &option_index); } } int main(int argc, char **argv) { struct configuration cfg; struct sockaddr_in dest_addr; int sock; unsigned int pkt_num = 0; char *buffer; time_t start_time, current_time; uint64_t total_sent = 0; /* Parse command line options */ parse_options(argc, argv, &cfg); /* Create socket */ if (cfg.cfg_protocol == IPPROTO_UDP) { sock = socket(AF_INET, SOCK_DGRAM, 0); } else { sock = socket(AF_INET, SOCK_STREAM, 0); } if (sock < 0) { perror("socket creation failed"); exit(EXIT_FAILURE); } /* Prepare destination address */ memset(&dest_addr, 0, sizeof(dest_addr)); dest_addr.sin_family = AF_INET; dest_addr.sin_port = htons(cfg.cfg_port); if (inet_pton(AF_INET, cfg.cfg_dest_ip, &dest_addr.sin_addr) <= 0) { perror("Invalid address"); exit(EXIT_FAILURE); } /* For TCP, we need to connect */ if (cfg.cfg_protocol == IPPROTO_TCP) { if (connect(sock, (struct sockaddr *)&dest_addr, sizeof(dest_addr)) < 0) { perror("TCP connection failed"); exit(EXIT_FAILURE); } printf("Connected to %s:%d via TCP\n", cfg.cfg_dest_ip, cfg.cfg_port); } else { printf("Ready to send UDP packets to %s:%d\n", cfg.cfg_dest_ip, cfg.cfg_port); } /* Allocate buffer for packet data */ buffer = malloc(cfg.cfg_pkt_size); if (!buffer) { perror("Memory allocation failed"); exit(EXIT_FAILURE); } printf("Starting to send packets with size %u bytes at intervals of %u ms\n", cfg.cfg_pkt_size, cfg.cfg_interval_ms); if (cfg.cfg_max_packets > 0) { printf("Will send %u packets\n", cfg.cfg_max_packets); } else { printf("Will send packets indefinitely (Ctrl+C to stop)\n"); } /* Record start time */ time(&start_time); /* Send packets */ while ((pkt_num++ < cfg.cfg_max_packets || (cfg.cfg_max_packets == 0))) { /* Fill the packet with a pattern */ memset(buffer, 0, cfg.cfg_pkt_size); snprintf(buffer, cfg.cfg_pkt_size, "Packet %u", pkt_num); for (size_t i = strlen(buffer) + 1; i < cfg.cfg_pkt_size; i++) { buffer[i] = (char)(i % 256); } /* Send the packet */ ssize_t sent; if (cfg.cfg_protocol == IPPROTO_UDP) { sent = sendto(sock, buffer, cfg.cfg_pkt_size, 0, (struct sockaddr *)&dest_addr, sizeof(dest_addr)); } else { sent = send(sock, buffer, cfg.cfg_pkt_size, 0); } if (sent < 0) { perror("send failed"); break; } total_sent++; if (!cfg.cfg_quiet) { printf("Sent packet %u, %zd bytes\n", pkt_num, sent); } /* Show stats periodically */ time(¤t_time); if (current_time - start_time >= 10) { /* Every 10 seconds */ double rate = (double)total_sent / (current_time - start_time); printf("Status: Sent %lu packets in %ld seconds (%.2f packets/sec)\n", total_sent, (long)(current_time - start_time), rate); start_time = current_time; total_sent = 0; } /* Wait for specified interval */ usleep(cfg.cfg_interval_ms * 1000); } printf("Sent %u packets. Exiting.\n", pkt_num - 1); /* Clean up */ free(buffer); free(cfg.cfg_dest_ip); close(sock); return 0; } |
이상을 실제 업무에 적용한 것이 Timestamping を使ってネットワークレイテンシを分析することで、ゲスト VM の Disk 性能低下問題を解決した입니다. 가상환경에서 네트워크 성능향상을 위해 위 툴을 적용하였습니다.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 |
/**************************************************************************\ ** Example for TX timestamping sockets API ** 2025/05/05 ** Based on the RX timestamping example \**************************************************************************/ /* Example application to demonstrate use of the timestamping API * * This application will send packets, and display their * hardware timestamps. * * Invoke with "--help" to see the options it supports. */ #include <errno.h> #include <getopt.h> #include <inttypes.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <time.h> #include <unistd.h> #include <arpa/inet.h> #include <net/if.h> #include <netdb.h> #include <sys/ioctl.h> #include <sys/socket.h> #include <sys/time.h> #include <sys/types.h> static uint64_t total_sent = 0; static uint64_t *user_nic_latency_numbers = NULL; static uint64_t *user_kernel_latency_numbers = NULL; static uint64_t *kernel_nic_latency_numbers = NULL; /* Use the kernel definitions if possible - * But if not, use our own local definitions, and Onload will allow it. * - Though you still need a reasonably recent kernel to get hardware * timestamping. */ #ifdef NO_KERNEL_TS_INCLUDE #include <time.h> struct hwtstamp_config { int flags; /* no flags defined right now, must be zero */ int tx_type; /* HWTSTAMP_TX_* */ int rx_filter; /* HWTSTAMP_FILTER_* */ }; enum { SOF_TIMESTAMPING_TX_HARDWARE = (1 << 0), SOF_TIMESTAMPING_TX_SOFTWARE = (1 << 1), SOF_TIMESTAMPING_RX_HARDWARE = (1 << 2), SOF_TIMESTAMPING_RX_SOFTWARE = (1 << 3), SOF_TIMESTAMPING_SOFTWARE = (1 << 4), SOF_TIMESTAMPING_SYS_HARDWARE = (1 << 5), SOF_TIMESTAMPING_RAW_HARDWARE = (1 << 6), SOF_TIMESTAMPING_MASK = (SOF_TIMESTAMPING_RAW_HARDWARE - 1) | SOF_TIMESTAMPING_RAW_HARDWARE }; #else #include <linux/net_tstamp.h> #include <linux/sockios.h> #endif /* These are defined in socket.h, but older versions might not have all 3 */ #ifndef SO_TIMESTAMP #define SO_TIMESTAMP 29 #endif #ifndef SO_TIMESTAMPNS #define SO_TIMESTAMPNS 35 #endif #ifndef SO_TIMESTAMPING #define SO_TIMESTAMPING 37 #endif /* Seconds.nanoseconds format */ #define TIME_FMT "%" PRIu64 ".%.9" PRIu64 " " #define OTIME_FMT "%" PRIu64 ".%.9" PRIu32 " " /* Assert-like macros */ #define TEST(x) \ do { \ if (!(x)) { \ fprintf(stderr, "ERROR: '%s' failed\n", #x); \ fprintf(stderr, "ERROR: at %s:%d\n", __FILE__, __LINE__); \ exit(1); \ } \ } while (0) #define TRY(x) \ do { \ int __rc = (x); \ if (__rc < 0) { \ fprintf(stderr, "ERROR: TRY(%s) failed\n", #x); \ fprintf(stderr, "ERROR: at %s:%d\n", __FILE__, __LINE__); \ fprintf(stderr, "ERROR: rc=%d errno=%d (%s)\n", __rc, errno, \ strerror(errno)); \ exit(1); \ } \ } while (0) struct configuration { char const *cfg_ioctl; /* e.g. eth6 - calls the ts enable ioctl */ char const *cfg_dest_ip; /* destination IP */ unsigned short cfg_port; /* destination port */ int cfg_protocol; /* udp or tcp? */ unsigned int cfg_max_packets; /* Stop after this many (0=forever) */ unsigned int cfg_pkt_size; /* Packet payload size */ unsigned int cfg_interval_ms; /* Interval between packets in ms */ }; /* Commandline options, configuration etc. */ void print_help(void) { printf("Usage:\n" "\t--ioctl\t<ethX>\tDevice to send timestamping enable ioctl. " "Default: None\n" "\t--ip\t<addr>\tDestination IP address. " "Default: 127.0.0.1\n" "\t--port\t<num>\tDestination port. " "Default: 9000\n" "\t--proto\t[TCP|UDP]. " "Default: UDP\n" "\t--size\t<num>\tPacket payload size. " "Default: 64\n" "\t--interval\t<ms>\tInterval between packets in milliseconds. " "Default: 1000\n" "\t--max\t<num>\tStop after n packets. " "Default: Run forever\n"); exit(-1); } static void get_protocol(struct configuration *cfg, const char *protocol) { if (0 == strcasecmp(protocol, "UDP")) { cfg->cfg_protocol = IPPROTO_UDP; } else if (0 == strcasecmp(protocol, "TCP")) { cfg->cfg_protocol = IPPROTO_TCP; } else { printf("ERROR: '%s' is not a recognised protocol (TCP or UDP).\n", protocol); exit(-EINVAL); } } static void parse_options(int argc, char **argv, struct configuration *cfg) { int option_index = 0; int opt; static struct option long_options[] = { {"ioctl", required_argument, 0, 'i'}, {"ip", required_argument, 0, 'a'}, {"port", required_argument, 0, 'p'}, {"proto", required_argument, 0, 'P'}, {"size", required_argument, 0, 's'}, {"interval", required_argument, 0, 't'}, {"max", required_argument, 0, 'n'}, {0, no_argument, 0, 0} }; const char *optstring = "i:a:p:P:s:t:n:"; /* Defaults */ bzero(cfg, sizeof(struct configuration)); cfg->cfg_dest_ip = "127.0.0.1"; cfg->cfg_port = 9000; cfg->cfg_protocol = IPPROTO_UDP; cfg->cfg_pkt_size = 64; cfg->cfg_interval_ms = 1000; opt = getopt_long(argc, argv, optstring, long_options, &option_index); while (opt != -1) { switch (opt) { case 'i': cfg->cfg_ioctl = optarg; break; case 'a': cfg->cfg_dest_ip = optarg; break; case 'p': cfg->cfg_port = atoi(optarg); break; case 'P': get_protocol(cfg, optarg); break; case 's': cfg->cfg_pkt_size = atoi(optarg); break; case 't': cfg->cfg_interval_ms = atoi(optarg); break; case 'n': cfg->cfg_max_packets = atoi(optarg); break; default: print_help(); break; } opt = getopt_long(argc, argv, optstring, long_options, &option_index); } } /* Connection */ static void make_address(const char *ip, unsigned short port, struct sockaddr_in *host_address) { bzero(host_address, sizeof(struct sockaddr_in)); host_address->sin_family = AF_INET; host_address->sin_port = htons(port); host_address->sin_addr.s_addr = inet_addr(ip); } /* This requires a bit of explanation. * Typically, you have to enable hardware timestamping on an interface. * Any application can do it, and then it's available to everyone. * The easiest way to do this, is just to run sfptpd. * * But in case you need to do it manually; here is the code, but * that's only supported on reasonably recent versions * * Option: --ioctl ethX * * NOTE: * Usage of the ioctl call is discouraged. A better method, if using * hardware timestamping, would be to use sfptpd as it will effectively * make the ioctl call for you. * */ static void do_ioctl(struct configuration *cfg, int sock) { #ifdef SIOCSHWTSTAMP struct ifreq ifr; struct hwtstamp_config hwc; #endif if (cfg->cfg_ioctl == NULL) return; #ifdef SIOCSHWTSTAMP bzero(&ifr, sizeof(ifr)); snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", cfg->cfg_ioctl); /* Standard kernel ioctl options */ hwc.flags = 0; hwc.tx_type = HWTSTAMP_TX_ON; /* Enable TX timestamping */ hwc.rx_filter = HWTSTAMP_FILTER_NONE; /* Not using RX timestamps */ ifr.ifr_data = (char *)&hwc; TRY(ioctl(sock, SIOCSHWTSTAMP, &ifr)); return; #else (void)sock; printf("SIOCHWTSTAMP ioctl not supported on this kernel.\n"); exit(-ENOTSUP); return; #endif } /* This routine selects the correct socket option to enable timestamping. */ static void do_ts_sockopt(struct configuration *cfg, int sock) { printf("Selecting hardware timestamping mode.\n"); { int enable = SOF_TIMESTAMPING_TX_HARDWARE | SOF_TIMESTAMPING_RAW_HARDWARE | SOF_TIMESTAMPING_SYS_HARDWARE | SOF_TIMESTAMPING_SOFTWARE; TRY(setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &enable, sizeof(int))); printf("enabled timestamping sockopt\n"); } } static int create_socket(struct configuration *cfg) { int s; int domain = SOCK_DGRAM; if (cfg->cfg_protocol == IPPROTO_TCP) domain = SOCK_STREAM; s = socket(PF_INET, domain, cfg->cfg_protocol); TEST(s >= 0); printf("Socket created\n"); return s; } /* Processing */ static void print_time(struct timespec *ts, struct timeval *time_user) { if (ts == NULL) { printf("no timestamp\n"); return; } /* Hardware timestamping provides three timestamps - * system (software) * transformed (hw converted to sw) * raw (hardware) * in that order - though depending on socket option, you may have 0 in * some of them. */ static uint64_t diff_user_kernel = 0; static uint64_t diff_user_nic = 0; static uint64_t diff_kernel_nic = 0; static int64_t user_kernel_total_diff = 0; uint64_t old_diff_user_kernel = diff_user_kernel; uint64_t nanoseconds_nic = ts[2].tv_sec * 1000000000 + ts[2].tv_nsec; uint64_t nanoseconds_kernel = ts[0].tv_sec * 1000000000 + ts[0].tv_nsec; uint64_t nanoseconds_user = time_user->tv_sec * 1000000000 + time_user->tv_usec * 1000; diff_user_kernel = (ts[0].tv_sec - time_user->tv_sec) * 1000000000 + (ts[0].tv_nsec - time_user->tv_usec * 1000); user_kernel_latency_numbers[total_sent] = diff_user_kernel; // all latency numbers are in nanoseconds if (old_diff_user_kernel != 0) { user_kernel_total_diff += diff_user_kernel - old_diff_user_kernel; } diff_kernel_nic = (ts[2].tv_sec - ts[0].tv_sec) * 1000000000 + (ts[2].tv_nsec - ts[0].tv_nsec); diff_user_nic = (ts[2].tv_sec - time_user->tv_sec) * 1000000000 + (ts[2].tv_nsec - time_user->tv_usec * 1000); user_nic_latency_numbers[total_sent] = diff_user_nic; // all latency numbers are in nanoseconds kernel_nic_latency_numbers[total_sent] = diff_kernel_nic; // all latency numbers are in nanoseconds printf("Packet %lu: User->NIC: %ld ns, User->Kernel: %ld ns, Kernel->NIC: %ld ns\n", total_sent, diff_user_nic, diff_user_kernel, diff_kernel_nic); } /* Given a packet, extract the timestamp(s) */ static void handle_time(struct msghdr *msg, struct timeval *time_user) { struct timespec *ts = NULL; struct cmsghdr *cmsg; for (cmsg = CMSG_FIRSTHDR(msg); cmsg; cmsg = CMSG_NXTHDR(msg, cmsg)) { if (cmsg->cmsg_level != SOL_SOCKET) continue; switch (cmsg->cmsg_type) { case SO_TIMESTAMPNS: ts = (struct timespec *)CMSG_DATA(cmsg); break; case SO_TIMESTAMPING: ts = (struct timespec *)CMSG_DATA(cmsg); break; default: /* Ignore other cmsg options */ break; } } print_time(ts, time_user); } /* Send a packet, then wait for and process its timestamp */ static int do_send(int sock, struct sockaddr_in *dest_addr, unsigned int pkt_num, struct configuration *cfg) { char buffer[2048]; char control[1024]; struct msghdr msg; struct iovec iov; int sent; struct timeval time_user; fd_set readfds; struct timespec timeout; int result; /* Create packet payload - fill with packet number */ memset(buffer, 0, sizeof(buffer)); snprintf(buffer, sizeof(buffer), "Packet %u", pkt_num); /* Fill the rest with a pattern */ size_t header_len = strlen(buffer); for (size_t i = header_len; i < cfg->cfg_pkt_size && i < sizeof(buffer); i++) { buffer[i] = (char)(i % 256); } /* Get user-space timestamp before sending */ gettimeofday(&time_user, NULL); /* Send the packet */ sent = sendto(sock, buffer, cfg->cfg_pkt_size, 0, (struct sockaddr *)dest_addr, sizeof(struct sockaddr_in)); if (sent < 0) { perror("sendto"); return -1; } /* Now wait for the timestamp control message to come back */ FD_ZERO(&readfds); FD_SET(sock, &readfds); /* Set timeout to 1 second */ timeout.tv_sec = 1; timeout.tv_nsec = 0; result = pselect(sock + 1, &readfds, NULL, NULL, &timeout, NULL); if (result <= 0) { if (result == 0) fprintf(stderr, "Timeout waiting for timestamp\n"); else perror("pselect"); return -1; } /* Prepare to receive control message with timestamp */ iov.iov_base = buffer; iov.iov_len = sizeof(buffer); memset(&msg, 0, sizeof(msg)); msg.msg_iov = &iov; msg.msg_iovlen = 1; msg.msg_control = control; msg.msg_controllen = sizeof(control); /* Receive timestamp control message */ result = recvmsg(sock, &msg, MSG_ERRQUEUE); if (result < 0) { perror("recvmsg"); return -1; } /* Process timestamp */ handle_time(&msg, &time_user); total_sent++; return sent; } int main(int argc, char **argv) { struct configuration cfg; int sock; unsigned int pkt_num = 0; struct sockaddr_in dest_addr; parse_options(argc, argv, &cfg); /* Initialise */ sock = create_socket(&cfg); do_ioctl(&cfg, sock); do_ts_sockopt(&cfg, sock); /* Prepare destination address */ make_address(cfg.cfg_dest_ip, cfg.cfg_port, &dest_addr); /* For TCP we need to connect */ if (cfg.cfg_protocol == IPPROTO_TCP) { TRY(connect(sock, (struct sockaddr *)&dest_addr, sizeof(dest_addr))); printf("Connected to %s:%d\n", cfg.cfg_dest_ip, cfg.cfg_port); } /* Allocate memory for latency measurements */ user_nic_latency_numbers = malloc(cfg.cfg_max_packets * sizeof(uint64_t)); user_kernel_latency_numbers = malloc(cfg.cfg_max_packets * sizeof(uint64_t)); kernel_nic_latency_numbers = malloc(cfg.cfg_max_packets * sizeof(uint64_t)); if (!user_nic_latency_numbers || !user_kernel_latency_numbers || !kernel_nic_latency_numbers) { fprintf(stderr, "Failed to allocate memory for latency measurements\n"); exit(1); } printf("Starting to send packets to %s:%d\n", cfg.cfg_dest_ip, cfg.cfg_port); /* Send packets and measure timestamps */ while ((pkt_num++ < cfg.cfg_max_packets || (cfg.cfg_max_packets == 0))) { int sent = do_send(sock, &dest_addr, pkt_num, &cfg); if (sent < 0) { fprintf(stderr, "Error sending packet %d\n", pkt_num); break; } /* Wait specified interval before sending next packet */ usleep(cfg.cfg_interval_ms * 1000); } printf("Sent %lu packets\n", total_sent); /* Save latency data to file */ FILE *f = fopen("tx_latency.txt", "w"); if (f) { fprintf(f, "user_nic,user_kernel,kernel_nic\n"); for (uint64_t i = 0; i < total_sent; ++i) { fprintf(f, "%ld,%ld,%ld\n", user_nic_latency_numbers[i], user_kernel_latency_numbers[i], kernel_nic_latency_numbers[i]); } fclose(f); printf("Latency data saved to tx_latency.txt\n"); } else { perror("Failed to open output file"); } /* Clean up */ free(user_nic_latency_numbers); free(user_kernel_latency_numbers); free(kernel_nic_latency_numbers); close(sock); return 0; } |
