1.
요즘 Low Latency에 대한 관심이 줄었습니다. 서버 튜닝도 먼 옛날의 이야기입니다. 그래도 중요하다고 생각하고 시간이 날 때마다 살펴보려고 합니다. 오늘 이야기는 2013년 여름때 있었던 일입니다. 그 때 메모를 해두었지만 마무리를 못했던 글입니다.
몇 일전 고객에게 공급할 트레이딩서버를 받았습니다. 이제 튜닝을 하여야 합니다. 튜닝을 하려면 측정방법을 표준화하여야 합니다. 그래야 측정값에 대한 비교가 가능합니다. 측정은 이전에 소개하였던 프로그램을 이용하였습니다.
lmbench
sockperf를 이용한 레이턴시 측정
어플리케이션 로그와 레이턴시
리눅스 서버를 튜닝할 때 성능에 직접적인 영향을 미치는 것을 선택하라고 하면 네트워크카드와 CPU의 C-State입니다. 네트워크카드는 제품입니다. 그래서 예산과 업무에 따라 적절히 선택하면 됩니다. ZeroServer의 경우 Chelsio를 기본으로 제안합니다. 튜닝을 하려고 한 서버도 Chelsio를 설치하였습니다. 다음은 CPU 상태입니다. Linux에서 C-State관리는 아래 글을 참고하세요.
측정을 진행하면서 CPU의 C-State를 보여주는 정보가 부족하여 프로그램을 하나 설치하였습니다.
The turbostat tool is part of the cpupowerutils package. It reports on processor topology, frequency, idle power-state statistics, temperature, and power usage on Intel® 64 processors.
Turbostat is useful for identifying servers that are inefficient in terms of power usage or idle time. It also helps to identify the rate of system management interrupts (SMIs) and latency-sensitive applications that may prompt SMIs unnecessarily. Turbostat can be used in conjunction with the powertop utility to identify services that prevent the processor from entering deep sleep states.
Turbostat로 Intel에 개발배포하는 프로그램입니다. 아래와 같은 파일을 받아서 컴파일합니다. 다른 방법도 많습니다.
|
1 2 |
wget http://mirror.linux.org.au/linux/kernel/people/lenb/acpi/utils/pmtools-latest/turbostat/turbostat.c wget http://mirror.linux.org.au/linux/kernel/people/lenb/acpi/utils/pmtools-latest/turbostat/Makefile |
2.
Supermicro의 BIOS는 Dell이나 HP와 메뉴가 다릅니다. 그리고 매뉴얼도 충분하지 않습니다. 하나씩 선택값을 바꾸어서 시험하는 방법을 취했습니다. Bios 중 Power Technology는 Disabled, Energy Efficient and Custom의 선택이 가능합니다. 먼저 energy efficient로 한 후 측정을 하였습니다. 아래는 측정값들입니다.
|
1 2 3 4 5 6 7 8 9 10 |
[root@kproject1 turbostat]# ./turbostat -v GenuineIntel 13 CPUID levels; family:model:stepping 0x6:3a:9 (6:58:9) 16 * 133 = 2133 MHz max efficiency 37 * 133 = 4933 MHz TSC frequency core CPU %c0 GHz TSC %c1 %c3 %c6 %pc3 %pc6 45.09 1.62 3.70 45.86 0.00 9.04 0.00 0.00 0 0 44.24 1.61 3.70 46.19 0.00 9.57 0.00 0.00 1 1 45.63 1.63 3.70 45.80 0.00 8.57 0.00 0.00 2 2 50.17 1.62 3.70 43.39 0.00 6.44 0.00 0.00 3 3 40.34 1.63 3.70 48.07 0.00 11.59 0.00 0.00 |
다음으로 Bios의 Power Management를 Disabled로 설정하고 측정했습니다.
|
1 2 3 4 5 6 7 8 9 10 |
[root@kproject1 turbostat]# ./turbostat -v GenuineIntel 13 CPUID levels; family:model:stepping 0x6:3a:9 (6:58:9) 16 * 133 = 2133 MHz max efficiency 37 * 133 = 4933 MHz TSC frequency core CPU %c0 GHz TSC %c1 %c3 %c6 %pc3 %pc6 100.00 3.70 3.70 0.00 0.00 0.00 0.00 0.00 0 0 100.00 3.70 3.70 0.00 0.00 0.00 0.00 0.00 1 1 100.00 3.70 3.70 0.00 0.00 0.00 0.00 0.00 2 2 100.00 3.70 3.70 0.00 0.00 0.00 0.00 0.00 3 3 100.00 3.70 3.70 0.00 0.00 0.00 0.00 0.00 |
차이를 아시겠나요? Redhat가 Redhat Summit에서 발표한 Performance Analysis and Tuning 에서는 다음과 같이 소개하고 있습니다.
설치해서 사용해보시길 바랍니다. 아래는 소스입니다. 검색하면 최신 버전을 확인하실 수 있습니다. Windows에서의 CPU 전원관리에서 소개하였던 윈도우 소스와 비교해보시면 어떨까요?
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 |
/* /* * turbostat -- show CPU frequency and C-state residency * on modern Intel turbo-capable processors. * * Copyright (c) 2010, Intel Corporation. * Len Brown <len.brown@intel.com> * * This program is free software; you can redistribute it and/or modify it * under the terms and conditions of the GNU General Public License, * version 2, as published by the Free Software Foundation. * * This program is distributed in the hope it will be useful, but WITHOUT * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for * more details. * * You should have received a copy of the GNU General Public License along with * this program; if not, write to the Free Software Foundation, Inc., * 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #define MSR_TSC 0x10 #define MSR_NEHALEM_PLATFORM_INFO 0xCE #define MSR_NEHALEM_TURBO_RATIO_LIMIT 0x1AD #define MSR_APERF 0xE8 #define MSR_MPERF 0xE7 #define MSR_PKG_C2_RESIDENCY 0x60D /* SNB only */ #define MSR_PKG_C3_RESIDENCY 0x3F8 #define MSR_PKG_C6_RESIDENCY 0x3F9 #define MSR_PKG_C7_RESIDENCY 0x3FA /* SNB only */ #define MSR_CORE_C3_RESIDENCY 0x3FC #define MSR_CORE_C6_RESIDENCY 0x3FD #define MSR_CORE_C7_RESIDENCY 0x3FE /* SNB only */ char *proc_stat = "/proc/stat"; unsigned int interval_sec = 5; /* set with -i interval_sec */ unsigned int verbose; /* set with -v */ unsigned int skip_c0; unsigned int skip_c1; unsigned int do_nhm_cstates; unsigned int do_snb_cstates; unsigned int has_aperf; unsigned int units = 1000000000; /* Ghz etc */ unsigned int genuine_intel; unsigned int has_invariant_tsc; unsigned int do_nehalem_platform_info; unsigned int do_nehalem_turbo_ratio_limit; unsigned int extra_msr_offset; double bclk; unsigned int show_pkg; unsigned int show_core; unsigned int show_cpu; int aperf_mperf_unstable; int backwards_count; char *progname; int need_reinitialize; int num_cpus; struct counters { unsigned long long tsc; /* per thread */ unsigned long long aperf; /* per thread */ unsigned long long mperf; /* per thread */ unsigned long long c1; /* per thread (calculated) */ unsigned long long c3; /* per core */ unsigned long long c6; /* per core */ unsigned long long c7; /* per core */ unsigned long long pc2; /* per package */ unsigned long long pc3; /* per package */ unsigned long long pc6; /* per package */ unsigned long long pc7; /* per package */ unsigned long long extra_msr; /* per thread */ int pkg; int core; int cpu; struct counters *next; }; struct counters *cnt_even; struct counters *cnt_odd; struct counters *cnt_delta; struct counters *cnt_average; struct timeval tv_even; struct timeval tv_odd; struct timeval tv_delta; unsigned long long get_msr(int cpu, off_t offset) { ssize_t retval; unsigned long long msr; char pathname[32]; int fd; sprintf(pathname, "/dev/cpu/%d/msr", cpu); fd = open(pathname, O_RDONLY); if (fd < 0) { perror(pathname); need_reinitialize = 1; return 0; } retval = pread(fd, &msr, sizeof msr, offset); if (retval != sizeof msr) { fprintf(stderr, "cpu%d pread(..., 0x%zx) = %jd\n", cpu, offset, retval); exit(-2); } close(fd); return msr; } void print_header(void) { if (show_pkg) fprintf(stderr, "pkg "); if (show_core) fprintf(stderr, "core"); if (show_cpu) fprintf(stderr, " CPU"); if (do_nhm_cstates) fprintf(stderr, " %%c0 "); if (has_aperf) fprintf(stderr, " GHz"); fprintf(stderr, " TSC"); if (do_nhm_cstates) fprintf(stderr, " %%c1 "); if (do_nhm_cstates) fprintf(stderr, " %%c3 "); if (do_nhm_cstates) fprintf(stderr, " %%c6 "); if (do_snb_cstates) fprintf(stderr, " %%c7 "); if (do_snb_cstates) fprintf(stderr, " %%pc2 "); if (do_nhm_cstates) fprintf(stderr, " %%pc3 "); if (do_nhm_cstates) fprintf(stderr, " %%pc6 "); if (do_snb_cstates) fprintf(stderr, " %%pc7 "); if (extra_msr_offset) fprintf(stderr, " MSR 0x%x ", extra_msr_offset); putc('\n', stderr); } void dump_cnt(struct counters *cnt) { fprintf(stderr, "package: %d ", cnt->pkg); fprintf(stderr, "core:: %d ", cnt->core); fprintf(stderr, "CPU: %d ", cnt->cpu); fprintf(stderr, "TSC: %016llX\n", cnt->tsc); fprintf(stderr, "c3: %016llX\n", cnt->c3); fprintf(stderr, "c6: %016llX\n", cnt->c6); fprintf(stderr, "c7: %016llX\n", cnt->c7); fprintf(stderr, "aperf: %016llX\n", cnt->aperf); fprintf(stderr, "pc2: %016llX\n", cnt->pc2); fprintf(stderr, "pc3: %016llX\n", cnt->pc3); fprintf(stderr, "pc6: %016llX\n", cnt->pc6); fprintf(stderr, "pc7: %016llX\n", cnt->pc7); fprintf(stderr, "msr0x%x: %016llX\n", extra_msr_offset, cnt->extra_msr); } void dump_list(struct counters *cnt) { printf("dump_list 0x%p\n", cnt); for (; cnt; cnt = cnt->next) dump_cnt(cnt); } void print_cnt(struct counters *p) { double interval_float; interval_float = tv_delta.tv_sec + tv_delta.tv_usec/1000000.0; /* topology columns, print blanks on 1st (average) line */ if (p == cnt_average) { if (show_pkg) fprintf(stderr, " "); if (show_core) fprintf(stderr, " "); if (show_cpu) fprintf(stderr, " "); } else { if (show_pkg) fprintf(stderr, "%4d", p->pkg); if (show_core) fprintf(stderr, "%4d", p->core); if (show_cpu) fprintf(stderr, "%4d", p->cpu); } /* %c0 */ if (do_nhm_cstates) { if (!skip_c0) fprintf(stderr, "%7.2f", 100.0 * p->mperf/p->tsc); else fprintf(stderr, " ****"); } /* GHz */ if (has_aperf) { if (!aperf_mperf_unstable) { fprintf(stderr, "%5.2f", 1.0 * p->tsc / units * p->aperf / p->mperf / interval_float); } else { if (p->aperf > p->tsc || p->mperf > p->tsc) { fprintf(stderr, " ****"); } else { fprintf(stderr, "%4.1f*", 1.0 * p->tsc / units * p->aperf / p->mperf / interval_float); } } } /* TSC */ fprintf(stderr, "%5.2f", 1.0 * p->tsc/units/interval_float); if (do_nhm_cstates) { if (!skip_c1) fprintf(stderr, "%7.2f", 100.0 * p->c1/p->tsc); else fprintf(stderr, " ****"); } if (do_nhm_cstates) fprintf(stderr, "%7.2f", 100.0 * p->c3/p->tsc); if (do_nhm_cstates) fprintf(stderr, "%7.2f", 100.0 * p->c6/p->tsc); if (do_snb_cstates) fprintf(stderr, "%7.2f", 100.0 * p->c7/p->tsc); if (do_snb_cstates) fprintf(stderr, "%7.2f", 100.0 * p->pc2/p->tsc); if (do_nhm_cstates) fprintf(stderr, "%7.2f", 100.0 * p->pc3/p->tsc); if (do_nhm_cstates) fprintf(stderr, "%7.2f", 100.0 * p->pc6/p->tsc); if (do_snb_cstates) fprintf(stderr, "%7.2f", 100.0 * p->pc7/p->tsc); if (extra_msr_offset) fprintf(stderr, " 0x%016llx", p->extra_msr); putc('\n', stderr); } void print_counters(struct counters *counters) { struct counters *cnt; print_header(); if (num_cpus > 1) print_cnt(cnt_average); for (cnt = counters; cnt != NULL; cnt = cnt->next) print_cnt(cnt); } #define SUBTRACT_COUNTER(after, before, delta) (delta = (after - before), (before > after)) int compute_delta(struct counters *after, struct counters *before, struct counters *delta) { int errors = 0; int perf_err = 0; skip_c0 = skip_c1 = 0; for ( ; after && before && delta; after = after->next, before = before->next, delta = delta->next) { if (before->cpu != after->cpu) { printf("cpu configuration changed: %d != %d\n", before->cpu, after->cpu); return -1; } if (SUBTRACT_COUNTER(after->tsc, before->tsc, delta->tsc)) { fprintf(stderr, "cpu%d TSC went backwards %llX to %llX\n", before->cpu, before->tsc, after->tsc); errors++; } /* check for TSC < 1 Mcycles over interval */ if (delta->tsc < (1000 * 1000)) { fprintf(stderr, "Insanely slow TSC rate," " TSC stops in idle?\n"); fprintf(stderr, "You can disable all c-states" " by booting with \"idle=poll\"\n"); fprintf(stderr, "or just the deep ones with" " \"processor.max_cstate=1\"\n"); exit(-3); } if (SUBTRACT_COUNTER(after->c3, before->c3, delta->c3)) { fprintf(stderr, "cpu%d c3 counter went backwards %llX to %llX\n", before->cpu, before->c3, after->c3); errors++; } if (SUBTRACT_COUNTER(after->c6, before->c6, delta->c6)) { fprintf(stderr, "cpu%d c6 counter went backwards %llX to %llX\n", before->cpu, before->c6, after->c6); errors++; } if (SUBTRACT_COUNTER(after->c7, before->c7, delta->c7)) { fprintf(stderr, "cpu%d c7 counter went backwards %llX to %llX\n", before->cpu, before->c7, after->c7); errors++; } if (SUBTRACT_COUNTER(after->pc2, before->pc2, delta->pc2)) { fprintf(stderr, "cpu%d pc2 counter went backwards %llX to %llX\n", before->cpu, before->pc2, after->pc2); errors++; } if (SUBTRACT_COUNTER(after->pc3, before->pc3, delta->pc3)) { fprintf(stderr, "cpu%d pc3 counter went backwards %llX to %llX\n", before->cpu, before->pc3, after->pc3); errors++; } if (SUBTRACT_COUNTER(after->pc6, before->pc6, delta->pc6)) { fprintf(stderr, "cpu%d pc6 counter went backwards %llX to %llX\n", before->cpu, before->pc6, after->pc6); errors++; } if (SUBTRACT_COUNTER(after->pc7, before->pc7, delta->pc7)) { fprintf(stderr, "cpu%d pc7 counter went backwards %llX to %llX\n", before->cpu, before->pc7, after->pc7); errors++; } perf_err = SUBTRACT_COUNTER(after->aperf, before->aperf, delta->aperf); if (perf_err) { fprintf(stderr, "cpu%d aperf counter went backwards %llX to %llX\n", before->cpu, before->aperf, after->aperf); } perf_err |= SUBTRACT_COUNTER(after->mperf, before->mperf, delta->mperf); if (perf_err) { fprintf(stderr, "cpu%d mperf counter went backwards %llX to %llX\n", before->cpu, before->mperf, after->mperf); } if (perf_err) { if (!aperf_mperf_unstable) { fprintf(stderr, "%s: APERF or MPERF went backwards *\n", progname); fprintf(stderr, "* Frequency results do not cover entire interval *\n"); fprintf(stderr, "* fix this by running Linux-2.6.30 or later *\n"); aperf_mperf_unstable = 1; } /* * mperf delta is likely a huge "positive" number * can not use it for calculating c0 time */ skip_c0 = 1; skip_c1 = 1; } /* * As mperf and tsc collection are not atomic, * it is possible for mperf's non-halted cycles * to exceed TSC's all cycles: show c1 = 0% in that case. */ if (delta->mperf > delta->tsc) delta->c1 = 0; else /* normal case, derive c1 */ delta->c1 = delta->tsc - delta->mperf - delta->c3 - delta->c6 - delta->c7; if (delta->mperf == 0) delta->mperf = 1; /* divide by 0 protection */ /* * for "extra msr", just copy the latest w/o subtracting */ delta->extra_msr = after->extra_msr; if (errors) { fprintf(stderr, "ERROR cpu%d before:\n", before->cpu); dump_cnt(before); fprintf(stderr, "ERROR cpu%d after:\n", before->cpu); dump_cnt(after); errors = 0; } } return 0; } void compute_average(struct counters *delta, struct counters *avg) { struct counters *sum; sum = calloc(1, sizeof(struct counters)); if (sum == NULL) { perror("calloc sum"); exit(1); } for (; delta; delta = delta->next) { sum->tsc += delta->tsc; sum->c1 += delta->c1; sum->c3 += delta->c3; sum->c6 += delta->c6; sum->c7 += delta->c7; sum->aperf += delta->aperf; sum->mperf += delta->mperf; sum->pc2 += delta->pc2; sum->pc3 += delta->pc3; sum->pc6 += delta->pc6; sum->pc7 += delta->pc7; } avg->tsc = sum->tsc/num_cpus; avg->c1 = sum->c1/num_cpus; avg->c3 = sum->c3/num_cpus; avg->c6 = sum->c6/num_cpus; avg->c7 = sum->c7/num_cpus; avg->aperf = sum->aperf/num_cpus; avg->mperf = sum->mperf/num_cpus; avg->pc2 = sum->pc2/num_cpus; avg->pc3 = sum->pc3/num_cpus; avg->pc6 = sum->pc6/num_cpus; avg->pc7 = sum->pc7/num_cpus; free(sum); } void get_counters(struct counters *cnt) { for ( ; cnt; cnt = cnt->next) { cnt->tsc = get_msr(cnt->cpu, MSR_TSC); if (do_nhm_cstates) cnt->c3 = get_msr(cnt->cpu, MSR_CORE_C3_RESIDENCY); if (do_nhm_cstates) cnt->c6 = get_msr(cnt->cpu, MSR_CORE_C6_RESIDENCY); if (do_snb_cstates) cnt->c7 = get_msr(cnt->cpu, MSR_CORE_C7_RESIDENCY); if (has_aperf) cnt->aperf = get_msr(cnt->cpu, MSR_APERF); if (has_aperf) cnt->mperf = get_msr(cnt->cpu, MSR_MPERF); if (do_snb_cstates) cnt->pc2 = get_msr(cnt->cpu, MSR_PKG_C2_RESIDENCY); if (do_nhm_cstates) cnt->pc3 = get_msr(cnt->cpu, MSR_PKG_C3_RESIDENCY); if (do_nhm_cstates) cnt->pc6 = get_msr(cnt->cpu, MSR_PKG_C6_RESIDENCY); if (do_snb_cstates) cnt->pc7 = get_msr(cnt->cpu, MSR_PKG_C7_RESIDENCY); if (extra_msr_offset) cnt->extra_msr = get_msr(cnt->cpu, extra_msr_offset); } } void print_nehalem_info(void) { unsigned long long msr; unsigned int ratio; if (!do_nehalem_platform_info) return; msr = get_msr(0, MSR_NEHALEM_PLATFORM_INFO); ratio = (msr >> 40) & 0xFF; fprintf(stderr, "%d * %.0f = %.0f MHz max efficiency\n", ratio, bclk, ratio * bclk); ratio = (msr >> 8) & 0xFF; fprintf(stderr, "%d * %.0f = %.0f MHz TSC frequency\n", ratio, bclk, ratio * bclk); if (verbose > 1) fprintf(stderr, "MSR_NEHALEM_PLATFORM_INFO: 0x%llx\n", msr); if (!do_nehalem_turbo_ratio_limit) return; msr = get_msr(0, MSR_NEHALEM_TURBO_RATIO_LIMIT); ratio = (msr >> 24) & 0xFF; if (ratio) fprintf(stderr, "%d * %.0f = %.0f MHz max turbo 4 active cores\n", ratio, bclk, ratio * bclk); ratio = (msr >> 16) & 0xFF; if (ratio) fprintf(stderr, "%d * %.0f = %.0f MHz max turbo 3 active cores\n", ratio, bclk, ratio * bclk); ratio = (msr >> 8) & 0xFF; if (ratio) fprintf(stderr, "%d * %.0f = %.0f MHz max turbo 2 active cores\n", ratio, bclk, ratio * bclk); ratio = (msr >> 0) & 0xFF; if (ratio) fprintf(stderr, "%d * %.0f = %.0f MHz max turbo 1 active cores\n", ratio, bclk, ratio * bclk); } void free_counter_list(struct counters *list) { struct counters *p; for (p = list; p; ) { struct counters *free_me; free_me = p; p = p->next; free(free_me); } } void free_all_counters(void) { free_counter_list(cnt_even); cnt_even = NULL; free_counter_list(cnt_odd); cnt_odd = NULL; free_counter_list(cnt_delta); cnt_delta = NULL; free_counter_list(cnt_average); cnt_average = NULL; } void insert_counters(struct counters **list, struct counters *new) { struct counters *prev; /* * list was empty */ if (*list == NULL) { new->next = *list; *list = new; return; } show_cpu = 1; /* there is more than one CPU */ /* * insert on front of list. * It is sorted by ascending package#, core#, cpu# */ if (((*list)->pkg > new->pkg) || (((*list)->pkg == new->pkg) && ((*list)->core > new->core)) || (((*list)->pkg == new->pkg) && ((*list)->core == new->core) && ((*list)->cpu > new->cpu))) { new->next = *list; *list = new; return; } prev = *list; while (prev->next && (prev->next->pkg < new->pkg)) { prev = prev->next; show_pkg = 1; /* there is more than 1 package */ } while (prev->next && (prev->next->pkg == new->pkg) && (prev->next->core < new->core)) { prev = prev->next; show_core = 1; /* there is more than 1 core */ } while (prev->next && (prev->next->pkg == new->pkg) && (prev->next->core == new->core) && (prev->next->cpu < new->cpu)) { prev = prev->next; } /* * insert after "prev" */ new->next = prev->next; prev->next = new; } void alloc_new_counters(int pkg, int core, int cpu) { struct counters *new; if (verbose > 1) printf("pkg%d core%d, cpu%d\n", pkg, core, cpu); new = (struct counters *)calloc(1, sizeof(struct counters)); if (new == NULL) { perror("calloc"); exit(1); } new->pkg = pkg; new->core = core; new->cpu = cpu; insert_counters(&cnt_odd, new); new = (struct counters *)calloc(1, sizeof(struct counters)); if (new == NULL) { perror("calloc"); exit(1); } new->pkg = pkg; new->core = core; new->cpu = cpu; insert_counters(&cnt_even, new); new = (struct counters *)calloc(1, sizeof(struct counters)); if (new == NULL) { perror("calloc"); exit(1); } new->pkg = pkg; new->core = core; new->cpu = cpu; insert_counters(&cnt_delta, new); new = (struct counters *)calloc(1, sizeof(struct counters)); if (new == NULL) { perror("calloc"); exit(1); } new->pkg = pkg; new->core = core; new->cpu = cpu; cnt_average = new; } int get_physical_package_id(int cpu) { char path[64]; FILE *filep; int pkg; sprintf(path, "/sys/devices/system/cpu/cpu%d/topology/physical_package_id", cpu); filep = fopen(path, "r"); if (filep == NULL) { perror(path); exit(1); } fscanf(filep, "%d", &pkg); fclose(filep); return pkg; } int get_core_id(int cpu) { char path[64]; FILE *filep; int core; sprintf(path, "/sys/devices/system/cpu/cpu%d/topology/core_id", cpu); filep = fopen(path, "r"); if (filep == NULL) { perror(path); exit(1); } fscanf(filep, "%d", &core); fclose(filep); return core; } /* * run func(index, cpu) on every cpu in /proc/stat */ int for_all_cpus(void (func)(int, int, int)) { FILE *fp; int cpu_count; int retval; fp = fopen(proc_stat, "r"); if (fp == NULL) { perror(proc_stat); exit(1); } retval = fscanf(fp, "cpu %*d %*d %*d %*d %*d %*d %*d %*d %*d %*d\n"); if (retval != 0) { perror("/proc/stat format"); exit(1); } for (cpu_count = 0; ; cpu_count++) { int cpu; retval = fscanf(fp, "cpu%u %*d %*d %*d %*d %*d %*d %*d %*d %*d %*d\n", &cpu); if (retval != 1) break; func(get_physical_package_id(cpu), get_core_id(cpu), cpu); } fclose(fp); return cpu_count; } void re_initialize(void) { printf("turbostat: topology changed, re-initializing.\n"); free_all_counters(); num_cpus = for_all_cpus(alloc_new_counters); need_reinitialize = 0; printf("num_cpus is now %d\n", num_cpus); } void dummy(int pkg, int core, int cpu) { return; } /* * check to see if a cpu came on-line */ void verify_num_cpus(void) { int new_num_cpus; new_num_cpus = for_all_cpus(dummy); if (new_num_cpus != num_cpus) { if (verbose) printf("num_cpus was %d, is now %d\n", num_cpus, new_num_cpus); need_reinitialize = 1; } } void turbostat_loop() { restart: get_counters(cnt_even); gettimeofday(&tv_even, (struct timezone *)NULL); while (1) { verify_num_cpus(); if (need_reinitialize) { re_initialize(); goto restart; } sleep(interval_sec); get_counters(cnt_odd); gettimeofday(&tv_odd, (struct timezone *)NULL); compute_delta(cnt_odd, cnt_even, cnt_delta); timersub(&tv_odd, &tv_even, &tv_delta); compute_average(cnt_delta, cnt_average); print_counters(cnt_delta); if (need_reinitialize) { re_initialize(); goto restart; } sleep(interval_sec); get_counters(cnt_even); gettimeofday(&tv_even, (struct timezone *)NULL); compute_delta(cnt_even, cnt_odd, cnt_delta); timersub(&tv_even, &tv_odd, &tv_delta); compute_average(cnt_delta, cnt_average); print_counters(cnt_delta); } } void check_dev_msr() { struct stat sb; if (stat("/dev/cpu/0/msr", &sb)) { fprintf(stderr, "no /dev/cpu/0/msr\n"); fprintf(stderr, "Try \"# modprobe msr\"\n"); exit(-5); } } void check_super_user() { if (getuid() != 0) { fprintf(stderr, "must be root\n"); exit(-6); } } int has_nehalem_turbo_ratio_limit(unsigned int family, unsigned int model) { if (!genuine_intel) return 0; if (family != 6) return 0; switch (model) { case 0x1A: /* Core i7, Xeon 5500 series - Bloomfield, Gainstown NHM-EP */ case 0x1E: /* Core i7 and i5 Processor - Clarksfield, Lynnfield, Jasper Forest */ case 0x1F: /* Core i7 and i5 Processor - Nehalem */ case 0x25: /* Westmere Client - Clarkdale, Arrandale */ case 0x2C: /* Westmere EP - Gulftown */ case 0x2A: /* SNB */ case 0x2D: /* SNB Xeon */ return 1; case 0x2E: /* Nehalem-EX Xeon - Beckton */ case 0x2F: /* Westmere-EX Xeon - Eagleton */ default: return 0; } } int is_snb(unsigned int family, unsigned int model) { if (!genuine_intel) return 0; switch (model) { case 0x2A: case 0x2D: return 1; } return 0; } double discover_bclk(unsigned int family, unsigned int model) { if (is_snb(family, model)) return 100.00; else return 133.33; } void check_cpuid() { unsigned int eax, ebx, ecx, edx, max_level; unsigned int fms, family, model, stepping; eax = ebx = ecx = edx = 0; asm("cpuid" : "=a" (max_level), "=b" (ebx), "=c" (ecx), "=d" (edx) : "a" (0)); if (ebx == 0x756e6547 && edx == 0x49656e69 && ecx == 0x6c65746e) genuine_intel = 1; if (verbose) fprintf(stderr, "%.4s%.4s%.4s ", (char *)&ebx, (char *)&edx, (char *)&ecx); asm("cpuid" : "=a" (fms), "=c" (ecx), "=d" (edx) : "a" (1) : "ebx"); family = (fms >> 8) & 0xf; model = (fms >> 4) & 0xf; stepping = fms & 0xf; if (family == 6 || family == 0xf) model += ((fms >> 16) & 0xf) << 4; if (verbose) fprintf(stderr, "%d CPUID levels; family:model:stepping 0x%x:%x:%x (%d:%d:%d)\n", max_level, family, model, stepping, family, model, stepping); if (!(edx & (1 << 5))) { fprintf(stderr, "CPUID: no MSR\n"); exit(1); } /* * check max extended function levels of CPUID. * This is needed to check for invariant TSC. * This check is valid for both Intel and AMD. */ ebx = ecx = edx = 0; asm("cpuid" : "=a" (max_level), "=b" (ebx), "=c" (ecx), "=d" (edx) : "a" (0x80000000)); if (max_level < 0x80000007) { fprintf(stderr, "CPUID: no invariant TSC (max_level 0x%x)\n", max_level); exit(1); } /* * Non-Stop TSC is advertised by CPUID.EAX=0x80000007: EDX.bit8 * this check is valid for both Intel and AMD */ asm("cpuid" : "=a" (eax), "=b" (ebx), "=c" (ecx), "=d" (edx) : "a" (0x80000007)); has_invariant_tsc = edx & (1 << 8); if (!has_invariant_tsc) { fprintf(stderr, "No invariant TSC\n"); exit(1); } /* * APERF/MPERF is advertised by CPUID.EAX=0x6: ECX.bit0 * this check is valid for both Intel and AMD */ asm("cpuid" : "=a" (eax), "=b" (ebx), "=c" (ecx), "=d" (edx) : "a" (0x6)); has_aperf = ecx & (1 << 0); if (!has_aperf) { fprintf(stderr, "No APERF MSR\n"); exit(1); } do_nehalem_platform_info = genuine_intel && has_invariant_tsc; do_nhm_cstates = genuine_intel; /* all Intel w/ non-stop TSC have NHM counters */ do_snb_cstates = is_snb(family, model); bclk = discover_bclk(family, model); do_nehalem_turbo_ratio_limit = has_nehalem_turbo_ratio_limit(family, model); } void usage() { fprintf(stderr, "%s: [-v] [-M MSR#] [-i interval_sec | command ...]\n", progname); exit(1); } /* * in /dev/cpu/ return success for names that are numbers * ie. filter out ".", "..", "microcode". */ int dir_filter(const struct dirent *dirp) { if (isdigit(dirp->d_name[0])) return 1; else return 0; } int open_dev_cpu_msr(int dummy1) { return 0; } void turbostat_init() { check_cpuid(); check_dev_msr(); check_super_user(); num_cpus = for_all_cpus(alloc_new_counters); if (verbose) print_nehalem_info(); } int fork_it(char **argv) { int retval; pid_t child_pid; get_counters(cnt_even); gettimeofday(&tv_even, (struct timezone *)NULL); child_pid = fork(); if (!child_pid) { /* child */ execvp(argv[0], argv); } else { int status; /* parent */ if (child_pid == -1) { perror("fork"); exit(1); } signal(SIGINT, SIG_IGN); signal(SIGQUIT, SIG_IGN); if (waitpid(child_pid, &status, 0) == -1) { perror("wait"); exit(1); } } get_counters(cnt_odd); gettimeofday(&tv_odd, (struct timezone *)NULL); retval = compute_delta(cnt_odd, cnt_even, cnt_delta); timersub(&tv_odd, &tv_even, &tv_delta); compute_average(cnt_delta, cnt_average); if (!retval) print_counters(cnt_delta); fprintf(stderr, "%.6f sec\n", tv_delta.tv_sec + tv_delta.tv_usec/1000000.0);; return 0; } void cmdline(int argc, char **argv) { int opt; progname = argv[0]; while ((opt = getopt(argc, argv, "+vi:M:")) != -1) { switch (opt) { case 'v': verbose++; break; case 'i': interval_sec = atoi(optarg); break; case 'M': sscanf(optarg, "%x", &extra_msr_offset); if (verbose > 1) fprintf(stderr, "MSR 0x%X\n", extra_msr_offset); break; default: usage(); } } } int main(int argc, char **argv) { cmdline(argc, argv); if (verbose > 1) fprintf(stderr, "turbostat Dec 6, 2010" " - Len Brown <lenb@kernel.org>\n"); if (verbose > 1) fprintf(stderr, "http://userweb.kernel.org/~lenb/acpi/utils/pmtools/turbostat/\n"); turbostat_init(); /* * if any params left, it must be a command to fork */ if (argc - optind) return fork_it(argv + optind); else turbostat_loop(); return 0; } |
3.
아래는 Supermicro가 내놓은 Low Latency와 관련한 자료입니다. ZeroServer를 Supermicro로 했던 이유중 하나가 BIOS에서 제공하는 기능입니다. 이에 대한 설명을 포함합니다.
그리고 앞서 소개했던 Red Hat Summit에서 발표하였던 Performance Analysis and Tuning입니다. 꼭 한번 읽어보세요.
