[X86][SchedModel] Add masked gather/scatter overrides to Znver4Model - #212997
[X86][SchedModel] Add masked gather/scatter overrides to Znver4Model#212997amd-subharad wants to merge 2 commits into
Conversation
The Znver4 scheduling model (also used by Znver5) had no overrides for the AVX-512 masked gather and scatter instructions, so they fell back to the default vector load/store write and were costed as ordinary memory ops. On these parts the instructions are microcoded sequences whose throughput, uops and latency scale strongly with the number of elements and the element width, so the default is far from the real hardware behaviour. Add SchedWriteRes overrides covering every AVX-512 masked gather (VPGATHER* / VGATHER*) and scatter (VPSCATTER* / VSCATTER*) shape. Each entry is keyed by (#elements, element width); the dword- and qword-index encodings of a shape (e.g. VPGATHERDQ and VPGATHERQQ for 8 x i64) share one entry, since the microcoded cost tracks the element count and width rather than the index width. Throughput and uop counts were measured on Znver5 (Ryzen 9 9950X, which reuses Znver4Model) with mask-reloading microbenchmarks under perf; reciprocal throughput is reproduced through load-/store-pipe and AGU occupancy via ReleaseAtCycles. Scatter latency is estimated (throughput and uops are measured). The 2-element (128-bit) shapes are force-scalarised by the vectoriser and were not measured; their values are extrapolated as half the 4-element entry so llvm-mca stays self-consistent. The llvm-mca Znver4 resource tests are updated for the new entries.
|
Hello @amd-subharad 👋 Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description. Frequently asked questionsHow do I add reviewers? This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically. You can also add reviewers manually using the Reviewers section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using What if there are no comments? If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers. Are any special GitHub settings required to contribute to LLVM? We only require contributors to have a public email address associated with their GitHub commits, see this section of LLVM Developer Policy for details. If you have questions, feel free to leave a comment on this PR, or ask on LLVM Discord or LLVM Discourse. Thank you, |
|
@llvm/pr-subscribers-backend-x86 Author: Sumukh J Bharadwaj (amd-subharad) ChangesSummaryThe Znver4 scheduling model (also used by Znver5) had no overrides for the MotivationOn Zen4/Zen5 these are microcoded sequences whose throughput, uops and latency Implementation notes
Tests
regenerated to reflect the new gather/scatter numbers. Patch is 53.43 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/212997.diff 3 Files Affected:
diff --git a/llvm/lib/Target/X86/X86ScheduleZnver4.td b/llvm/lib/Target/X86/X86ScheduleZnver4.td
index ac4d31de8dbfe..1d058397f3455 100644
--- a/llvm/lib/Target/X86/X86ScheduleZnver4.td
+++ b/llvm/lib/Target/X86/X86ScheduleZnver4.td
@@ -509,6 +509,111 @@ defm : Zn4WriteResInt<WriteLoad, [Zn4AGU012, Zn4Load], !add(Znver4Model.LoadLate
// Does not cost anything by itself, only has latency, matching that of the WriteLoad,
defm : Zn4WriteResInt<WriteVecMaskedGatherWriteback, [], !add(Znver4Model.LoadLatency, 1), [], 0>;
+// AVX-512 masked GATHER / SCATTER, per shape.
+//
+// Zen4/Zen5 implement these as microcoded sequences (~4-5 macro-ops per
+// element), so the cost is a function of the number of elements and the
+// element width, not of the index width. Throughput and uops were measured on
+// Znver5 (Ryzen 9 9950X, which reuses Znver4Model) with mask-reloading
+// microbenchmarks under perf:
+// shape gather: tput(cyc) uops lat scatter: tput(cyc) uops
+// v4x32 4.0 20 17 6.0 28
+// v8x32 6.2 33 23 9.0 49
+// v16x32 12.7 65 30 17.0 89
+// v4x64 4.0 20 19 5.0 28
+// v8x64 8.0 41 25 9.0 49
+// Reciprocal throughput is reproduced through load-pipe (Zn4Load, 3 units) /
+// store-pipe (Zn4Store, 2 units) and AGU (Zn4AGU012, 3 units) occupancy:
+// ReleaseAtCycles = round(tput * NumUnits). Scatter latency is estimated
+// (throughput and uops are measured).
+//
+// Each entry is keyed by (#elements, element width); the dword- and
+// qword-index encodings of a shape share it (e.g. VPGATHERDQ and VPGATHERQQ
+// for 8 x i64). The 2-element (128-bit) shapes are force-scalarised by the
+// vectoriser and were not measured; their values are extrapolated as half the
+// 4-element entry and exist only so llvm-mca stays self-consistent.
+def Zn4WriteVPGATHERQDZ128 : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [6, 6]; let Latency = 14; let NumMicroOps = 10;
+}
+def : InstRW<[Zn4WriteVPGATHERQDZ128, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERQDZ128rm, VGATHERQPSZ128rm)>;
+def Zn4WriteVPGATHERDDZ128 : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [12, 12]; let Latency = 17; let NumMicroOps = 20;
+}
+def : InstRW<[Zn4WriteVPGATHERDDZ128, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERDDZ128rm, VGATHERDPSZ128rm,
+ VPGATHERQDZ256rm, VGATHERQPSZ256rm)>;
+def Zn4WriteVPGATHERDDZ256 : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [19, 19]; let Latency = 23; let NumMicroOps = 33;
+}
+def : InstRW<[Zn4WriteVPGATHERDDZ256, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERDDZ256rm, VGATHERDPSZ256rm,
+ VPGATHERQDZrm, VGATHERQPSZrm)>;
+def Zn4WriteVPGATHERDDZ : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [38, 38]; let Latency = 30; let NumMicroOps = 65;
+}
+def : InstRW<[Zn4WriteVPGATHERDDZ, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERDDZrm, VGATHERDPSZrm)>;
+def Zn4WriteVPGATHERQQZ128 : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [6, 6]; let Latency = 15; let NumMicroOps = 10;
+}
+def : InstRW<[Zn4WriteVPGATHERQQZ128, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERQQZ128rm, VGATHERQPDZ128rm,
+ VPGATHERDQZ128rm, VGATHERDPDZ128rm)>;
+def Zn4WriteVPGATHERQQZ256 : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [12, 12]; let Latency = 19; let NumMicroOps = 20;
+}
+def : InstRW<[Zn4WriteVPGATHERQQZ256, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERQQZ256rm, VGATHERQPDZ256rm,
+ VPGATHERDQZ256rm, VGATHERDPDZ256rm)>;
+def Zn4WriteVPGATHERQQZ : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
+ let ReleaseAtCycles = [24, 24]; let Latency = 25; let NumMicroOps = 41;
+}
+def : InstRW<[Zn4WriteVPGATHERQQZ, WriteVecMaskedGatherWriteback],
+ (instrs VPGATHERQQZrm, VGATHERQPDZrm,
+ VPGATHERDQZrm, VGATHERDPDZrm)>;
+
+def Zn4WriteVPSCATTERQDZ128 : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [9, 6]; let Latency = 6; let NumMicroOps = 14;
+}
+def : InstRW<[Zn4WriteVPSCATTERQDZ128],
+ (instrs VPSCATTERQDZ128mr, VSCATTERQPSZ128mr)>;
+def Zn4WriteVPSCATTERDDZ128 : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [18, 12]; let Latency = 10; let NumMicroOps = 28;
+}
+def : InstRW<[Zn4WriteVPSCATTERDDZ128],
+ (instrs VPSCATTERDDZ128mr, VSCATTERDPSZ128mr,
+ VPSCATTERQDZ256mr, VSCATTERQPSZ256mr)>;
+def Zn4WriteVPSCATTERDDZ256 : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [27, 18]; let Latency = 14; let NumMicroOps = 49;
+}
+def : InstRW<[Zn4WriteVPSCATTERDDZ256],
+ (instrs VPSCATTERDDZ256mr, VSCATTERDPSZ256mr,
+ VPSCATTERQDZmr, VSCATTERQPSZmr)>;
+def Zn4WriteVPSCATTERDDZ : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [51, 34]; let Latency = 24; let NumMicroOps = 89;
+}
+def : InstRW<[Zn4WriteVPSCATTERDDZ],
+ (instrs VPSCATTERDDZmr, VSCATTERDPSZmr)>;
+def Zn4WriteVPSCATTERQQZ128 : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [8, 5]; let Latency = 5; let NumMicroOps = 14;
+}
+def : InstRW<[Zn4WriteVPSCATTERQQZ128],
+ (instrs VPSCATTERQQZ128mr, VSCATTERQPDZ128mr,
+ VPSCATTERDQZ128mr, VSCATTERDPDZ128mr)>;
+def Zn4WriteVPSCATTERQQZ256 : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [15, 10]; let Latency = 9; let NumMicroOps = 28;
+}
+def : InstRW<[Zn4WriteVPSCATTERQQZ256],
+ (instrs VPSCATTERQQZ256mr, VSCATTERQPDZ256mr,
+ VPSCATTERDQZ256mr, VSCATTERDPDZ256mr)>;
+def Zn4WriteVPSCATTERQQZ : SchedWriteRes<[Zn4AGU012, Zn4Store]> {
+ let ReleaseAtCycles = [27, 18]; let Latency = 14; let NumMicroOps = 49;
+}
+def : InstRW<[Zn4WriteVPSCATTERQQZ],
+ (instrs VPSCATTERQQZmr, VSCATTERQPDZmr,
+ VPSCATTERDQZmr, VSCATTERDPDZmr)>;
+
def Zn4WriteMOVSlow : SchedWriteRes<[Zn4AGU012, Zn4Load]> {
let Latency = !add(Znver4Model.LoadLatency, 1);
let ReleaseAtCycles = [3, 1];
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s
index 14b8e5f36c666..22983b0161a2b 100644
--- a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s
@@ -1495,10 +1495,10 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 4 1.00 vfmadd231ps %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 11 1.00 * vfmadd231ps (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 11 1.00 * vfmadd231ps (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: 1 5 0.33 * vgatherdpd (%rax,%ymm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vgatherdps (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vgatherqpd (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vgatherqps (%rax,%zmm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 41 25 8.00 * vgatherdpd (%rax,%ymm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 65 30 12.67 * vgatherdps (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 41 25 8.00 * vgatherqpd (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 33 23 6.33 * vgatherqps (%rax,%zmm1,2), %ymm2 {%k1}
# CHECK-NEXT: 1 2 1.00 vmaxpd %zmm16, %zmm17, %zmm19
# CHECK-NEXT: 1 9 1.00 * vmaxpd (%rax), %zmm17, %zmm19
# CHECK-NEXT: 1 9 1.00 * vmaxpd (%rax){1to8}, %zmm17, %zmm19
@@ -1732,10 +1732,10 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 1 0.50 vpcmpequq %zmm0, %zmm1, %k2 {%k3}
# CHECK-NEXT: 1 8 0.50 * vpcmpequq (%rax), %zmm1, %k2 {%k3}
# CHECK-NEXT: 1 8 0.50 * vpcmpequq (%rax){1to8}, %zmm1, %k2 {%k3}
-# CHECK-NEXT: 1 5 0.33 * vpgatherdq (%rax,%ymm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vpgatherdd (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vpgatherqq (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 1 5 0.33 * vpgatherqd (%rax,%zmm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 41 25 8.00 * vpgatherdq (%rax,%ymm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 65 30 12.67 * vpgatherdd (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 41 25 8.00 * vpgatherqq (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 33 23 6.33 * vpgatherqd (%rax,%zmm1,2), %ymm2 {%k1}
# CHECK-NEXT: 1 5 1.00 vpmovdb %zmm19, %xmm16
# CHECK-NEXT: 1 11 1.50 * vpmovdb %zmm19, (%rax)
# CHECK-NEXT: 1 5 1.00 vpmovdb %zmm19, %xmm16 {%k1}
@@ -1970,10 +1970,10 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 2 1 0.50 vpermq %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 2 8 0.50 * vpermq (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 2 8 0.50 * vpermq (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: 1 1 1.00 * vpscatterdd %zmm1, (%rdx,%zmm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vpscatterdq %zmm1, (%rdx,%ymm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vpscatterqd %ymm1, (%rdx,%zmm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vpscatterqq %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 89 24 17.00 * vpscatterdd %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 49 14 9.00 * vpscatterdq %zmm1, (%rdx,%ymm0,4) {%k1}
+# CHECK-NEXT: 49 14 9.00 * vpscatterqd %ymm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 49 14 9.00 * vpscatterqq %zmm1, (%rdx,%zmm0,4) {%k1}
# CHECK-NEXT: 1 1 1.00 vpshufd $0, %zmm16, %zmm19
# CHECK-NEXT: 1 8 1.00 * vpshufd $0, (%rax), %zmm19
# CHECK-NEXT: 1 8 1.00 * vpshufd $0, (%rax){1to16}, %zmm19
@@ -2037,10 +2037,10 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 1 1.00 vpunpcklqdq %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 8 1.00 * vpunpcklqdq (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 8 1.00 * vpunpcklqdq (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: 1 1 1.00 * vscatterdps %zmm1, (%rdx,%zmm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vscatterdpd %zmm1, (%rdx,%ymm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vscatterqps %ymm1, (%rdx,%zmm0,4) {%k1}
-# CHECK-NEXT: 1 1 1.00 * vscatterqpd %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 89 24 17.00 * vscatterdps %zmm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 49 14 9.00 * vscatterdpd %zmm1, (%rdx,%ymm0,4) {%k1}
+# CHECK-NEXT: 49 14 9.00 * vscatterqps %ymm1, (%rdx,%zmm0,4) {%k1}
+# CHECK-NEXT: 49 14 9.00 * vscatterqpd %zmm1, (%rdx,%zmm0,4) {%k1}
# CHECK-NEXT: 1 2 1.00 vshuff32x4 $0, %zmm16, %zmm17, %zmm19
# CHECK-NEXT: 3 9 1.00 * vshuff32x4 $0, (%rax), %zmm17, %zmm19
# CHECK-NEXT: 3 9 1.00 * vshuff32x4 $0, (%rax){1to16}, %zmm17, %zmm19
@@ -2233,7 +2233,7 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK: Resource pressure per iteration:
# CHECK-NEXT: [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12.0] [12.1] [13] [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
-# CHECK-NEXT: 5.33 5.33 5.33 - - - - - 219.50 1119.00 676.50 351.00 312.50 312.50 17.00 215.67 215.67 215.67 204.67 204.67 204.67 16.50 16.50
+# CHECK-NEXT: 158.00 158.00 158.00 - - - - - 219.50 1119.00 676.50 351.00 312.50 312.50 17.00 336.33 336.33 336.33 272.00 272.00 272.00 96.50 96.50
# CHECK: Resource pressure by instruction:
# CHECK-NEXT: [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12.0] [12.1] [13] [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
@@ -2552,10 +2552,10 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - 1.00 1.00 - - - - - - - - - - - - - vfmadd231ps %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - 1.00 1.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vfmadd231ps (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - 1.00 1.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vfmadd231ps (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherdpd (%rax,%ymm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherdps (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherqpd (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vgatherqps (%rax,%zmm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 8.00 8.00 8.00 - - - - - - - - - - - - 8.00 8.00 8.00 8.00 8.00 8.00 - - vgatherdpd (%rax,%ymm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 12.67 12.67 12.67 - - - - - - - - - - - - 12.67 12.67 12.67 12.67 12.67 12.67 - - vgatherdps (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 8.00 8.00 8.00 - - - - - - - - - - - - 8.00 8.00 8.00 8.00 8.00 8.00 - - vgatherqpd (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 6.33 6.33 6.33 - - - - - - - - - - - - 6.33 6.33 6.33 6.33 6.33 6.33 - - vgatherqps (%rax,%zmm1,2), %ymm2 {%k1}
# CHECK-NEXT: - - - - - - - - 1.00 1.00 - - - - - - - - - - - - - vmaxpd %zmm16, %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 1.00 1.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vmaxpd (%rax), %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 1.00 1.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vmaxpd (%rax){1to8}, %zmm17, %zmm19
@@ -2789,10 +2789,10 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpcmpequq %zmm0, %zmm1, %k2 {%k3}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpequq (%rax), %zmm1, %k2 {%k3}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpequq (%rax){1to8}, %zmm1, %k2 {%k3}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherdq (%rax,%ymm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherdd (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherqq (%rax,%zmm1,2), %zmm2 {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpgatherqd (%rax,%zmm1,2), %ymm2 {%k1}
+# CHECK-NEXT: 8.00 8.00 8.00 - - - - - - - - - - - - 8.00 8.00 8.00 8.00 8.00 8.00 - - vpgatherdq (%rax,%ymm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 12.67 12.67 12.67 - - - - - - - - - - - - 12.67 12.67 12.67 12.67 12.67 12.67 - - vpgatherdd (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 8.00 8.00 8.00 - - - - - - - - - - - - 8.00 8.00 8.00 8.00 8.00 8.00 - - vpgatherqq (%rax,%zmm1,2), %zmm2 {%k1}
+# CHECK-NEXT: 6.33 6.33 6.33 - - - - - - - - - - - - 6.33 6.33 6.33 6.33 6.33 6.33 - - vpgatherqd (%rax,%zmm1,2), %ymm2 {%k1}
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - - - - - - - - - - - - vpmovdb %zmm19, %xmm16
# CHECK-NEXT: - - - - - - - - - 1.50 1.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpmovdb %zmm19, (%rax)
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - - - - - - - - - - - - vpmovdb %zmm19, %xmm16 {%k1}
@@ -3027,10 +3027,10 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - - - - - - - - - - - - vpermq %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpermq (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 0.50 0.50 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpermq (%rax){1to8}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterdd %zmm1, (%rdx,%zmm0,4) {%k1}
-# CHECK-NEXT: 0.33 0.33 0.33 - - - - - - - - - - - - 0.67 0.67 0.67 - - - 1.00 1.00 vpscatterdq %zmm1, (%rdx,%ymm0,4) {%k1}
-# CHECK-NE...
[truncated]
|
|
Prerequisite for #199488 (the per-shape gather/scatter TTI cost model), which reads these schedule-model numbers via |
| // v8x32 6.2 33 23 9.0 49 | ||
| // v16x32 12.7 65 30 17.0 89 | ||
| // v4x64 4.0 20 19 5.0 28 | ||
| // v8x64 8.0 41 25 9.0 49 |
There was a problem hiding this comment.
no need to include all this in the comments as its in the data - KISS
| // for 8 x i64). The 2-element (128-bit) shapes are force-scalarised by the | ||
| // vectoriser and were not measured; their values are extrapolated as half the | ||
| // 4-element entry and exist only so llvm-mca stays self-consistent. | ||
| def Zn4WriteVPGATHERQDZ128 : SchedWriteRes<[Zn4AGU012, Zn4Load]> { |
MattPD
left a comment
There was a problem hiding this comment.
I have left a few questions inline.
The measurements below use the merge base plus this PR at b25f7cd3a575, in an X86-only assertions build.
#199488 is stacked on this one and reads these values as its body term, so I reviewed it only with this prerequisite applied.
| // for 8 x i64). The 2-element (128-bit) shapes are force-scalarised by the | ||
| // vectoriser and were not measured; their values are extrapolated as half the | ||
| // 4-element entry and exist only so llvm-mca stays self-consistent. | ||
| def Zn4WriteVPGATHERQDZ128 : SchedWriteRes<[Zn4AGU012, Zn4Load]> { |
There was a problem hiding this comment.
Following up on @RKSimon's "No FPU pipe usage?" question, I think the answer is tied to the AGU occupancy, so the two may need to move together. Comparing the new gather against a plain vector load in this model, with llvm-mca -instruction-tables=full:
16x32 EVEX gather : AGU0/1/2 12.67 Load 12.67 FP45 0.00 FPSt 0.00
vmovaps zmm load : AGU 0.00 Load 0.33 FP45 0.50 FPSt 0.00
An ordinary vector load charges Zn4FP45 and no AGU. The gather does the reverse. Zn4AGU0/1/2 are also members of Zn4Int, and WriteLoad, WriteStore and WriteLEA charge them, so in this model a gather throttles scalar address generation harder than its element count suggests, while leaving the vector pipes free.
One observation that may help: Zn4FPSt has a single unit, and one cycle per element gives 16, 8, 8 and 4 for the 16x32, 8x32, 8x64 and 4x64 scatter shapes, against your measured 17.0, 9.0, 9.0 and 5.0. Would charging physical occupancy on the AGU and load or store pipes, and putting the limiting occupancy on the FP pipes, reproduce the same throughput?
| } | ||
| def : InstRW<[Zn4WriteVPGATHERQQZ, WriteVecMaskedGatherWriteback], | ||
| (instrs VPGATHERQQZrm, VGATHERQPDZrm, | ||
| VPGATHERDQZrm, VGATHERDPDZrm)>; |
There was a problem hiding this comment.
The VEX gathers stay on the old default write, so modeled reciprocal throughput for the same logical operation differs by about 19 times inside one model:
vpgatherdd %ymm0, (%rax,%ymm1,2), %ymm2 VEX 1 uop 0.33 cyc
vpgatherdd (%rax,%ymm1,4), %ymm2 {%k1} EVEX 33 uops 6.33 cyc
The scheduler model is selected by TuneCPU rather than by the ISA, so -mtune=znver4 -march=x86-64-v3 schedules VEX gathers with this model. SkylakeServer carries both encodings, so there is precedent for covering both. Would adding matching VEX entries be in scope here, or is it better as a follow-up with a note in the .td?
| def : InstRW<[Zn4WriteVPSCATTERDDZ], | ||
| (instrs VPSCATTERDDZmr, VSCATTERDPSZmr)>; | ||
| def Zn4WriteVPSCATTERQQZ128 : SchedWriteRes<[Zn4AGU012, Zn4Store]> { | ||
| let ReleaseAtCycles = [8, 5]; let Latency = 5; let NumMicroOps = 14; |
There was a problem hiding this comment.
Applying round(tput * NumUnits) per resource lets a non-bottleneck resource round past the intended limiter. This 2x64 scatter targets 2.5. The store pipe gives round(2.5 * 2) = 5, which is 2.50, and the AGU gives round(2.5 * 3) = 8, which is 2.67. llvm-mca takes the maximum:
vpscatterdq %xmm1, (%rdx,%xmm0,4) {%k1} Block RThroughput: 2.7
No other entry flips today. Would deriving the bottleneck exactly, and setting the other resources to their physical occupancy, avoid this by construction?
| // Each entry is keyed by (#elements, element width); the dword- and | ||
| // qword-index encodings of a shape share it (e.g. VPGATHERDQ and VPGATHERQQ | ||
| // for 8 x i64). The 2-element (128-bit) shapes are force-scalarised by the | ||
| // vectoriser and were not measured; their values are extrapolated as half the |
There was a problem hiding this comment.
The halving holds for ReleaseAtCycles and NumMicroOps, but not for Latency in any of the four entries:
2-element entry source 4-element latency half installed latency
gather 2x32 17 8.5 14
gather 2x64 19 9.5 15
scatter 2x32 10 5 6
scatter 2x64 9 4.5 5
Could the rule actually used for those four be stated, so a later maintainer can regenerate them? Given the request to trim this block, dropping the halving claim may be simpler than documenting a second rule.
| // vectoriser and were not measured; their values are extrapolated as half the | ||
| // 4-element entry and exist only so llvm-mca stays self-consistent. | ||
| def Zn4WriteVPGATHERQDZ128 : SchedWriteRes<[Zn4AGU012, Zn4Load]> { | ||
| let ReleaseAtCycles = [6, 6]; let Latency = 14; let NumMicroOps = 10; |
There was a problem hiding this comment.
On these same entries, "exist only so llvm-mca stays self-consistent" may understate their reach. A SchedWriteRes also feeds the machine scheduler, and llvm/test/CodeGen/X86/avx512-gather-scatter-intrin.ll shows intrinsics lowering to the XMM forms, for example vpgatherqq (%rdi,%xmm1,8), %xmm0 {%k1}. The vectorizer does force scalarize those widths, so the auto-vectorized path is unreachable, but the intrinsic path is not. Would it be worth softening that sentence?
| def : InstRW<[Zn4WriteVPSCATTERQDZ128], | ||
| (instrs VPSCATTERQDZ128mr, VSCATTERQPSZ128mr)>; | ||
| def Zn4WriteVPSCATTERDDZ128 : SchedWriteRes<[Zn4AGU012, Zn4Store]> { | ||
| let ReleaseAtCycles = [18, 12]; let Latency = 10; let NumMicroOps = 28; |
There was a problem hiding this comment.
The measured table gives 4x32 scatter 6.0 cycles at 28 uops and 4x64 scatter 5.0 cycles at 28 uops. The uop counts are equal and the throughputs differ by 1.0 cycle. The 8-element scatter pair is identical, and the 4-element gather pair is identical, so the width dependence appears once.
Halving carries it into a result that looks inverted:
vpscatterdq 2 x 64-bit data, 16 bytes written 2.7
vpscatterqd 2 x 32-bit data, 8 bytes written 3.0
The shape writing twice as much data is the cheaper one. Repetition count and observed variance for that 4-element pair would show whether the difference exceeds run-to-run noise.
| // Zen4/Zen5 implement these as microcoded sequences (~4-5 macro-ops per | ||
| // element), so the cost is a function of the number of elements and the | ||
| // element width, not of the index width. Throughput and uops were measured on | ||
| // Znver5 (Ryzen 9 9950X, which reuses Znver4Model) with mask-reloading |
There was a problem hiding this comment.
Since the measurements come from a Zen 5 part, it seems worth saying so in the user-facing text too, and naming who inherits them. X86.td binds Znver4Model to znver4, znver5 and znver6, so a third microarchitecture picks these up without being mentioned. Zen 5 widened the vector datapath where Zen 4 double-pumps, so for microcoded 512-bit shapes the two may differ.
There is one independent Zen 4 cross-check available. uops.info measures VPSCATTERQQ (VSIB_ZMM, K, ZMM) on Zen 4 at 48 uops, with 18 restricted to the store ports. That matches the 8x64 entry's ReleaseAtCycles of 18 on Zn4Store well, and sits one uop below its 49. Since the benchmarks reloaded the mask, was that overhead subtracted before filling in NumMicroOps?
If you shorten this block for the existing review request, keeping the provenance line seems worth it. It is the only record that these are not Zen 4 measurements.
| (instrs VPGATHERDDZ256rm, VGATHERDPSZ256rm, | ||
| VPGATHERQDZrm, VGATHERQPSZrm)>; | ||
| def Zn4WriteVPGATHERDDZ : SchedWriteRes<[Zn4AGU012, Zn4Load]> { | ||
| let ReleaseAtCycles = [38, 38]; let Latency = 30; let NumMicroOps = 65; |
There was a problem hiding this comment.
This moves gather latency from 5 to between 14 and 30, and NumMicroOps from 1 to between 10 and 89, for the machine scheduler on znver4, znver5 and znver6. Only the llvm-mca tests exercise the new values, and the existing znver-tuned tests under llvm/test/CodeGen/X86/ contain no gather or scatter. A later retune could therefore change znver4 codegen with every test still passing. Would a small llc -mcpu=znver4 test over a gather and scatter loop be worth adding?
RKSimon
left a comment
There was a problem hiding this comment.
Agner / uops.info all indicate that these use a lot of FPU pipes as well - we need to model this
Review pointed out that Agner and uops.info both show these sequences using the vector pipes heavily, which the original entries did not model at all: they charged only the AGU and the load or store pipes. Charge Zn4FPU0123 on every entry, and take its occupancy from the measured bottleneck instead of fitting each resource independently. Every other resource now carries its plain physical occupancy, which by construction stops a non-limiting resource from rounding past the intended bottleneck -- the 2 x i64 scatter previously reported 2.7 cycles against an intended 2.5. Retarget the numbers from Znver5 to Znver4. The original values came from a 9950X on the grounds that it reuses Znver4Model, but the two parts diverge by 25-55%, and on every shape rather than only the 512-bit ones. Since the model is named for Znver4 and znver5/znver6 merely inherit it, it now carries Znver4 data and the newer parts are modeled pessimistically; a per-part split is left as future work. Key the entries by index width where the forms diverge. Sharing one entry between the dword- and qword-index encodings was wrong for six shapes, worst at the 8 x i64 scatter: 13.94 cycles with a qword index against 11.05 with a dword one, at equal uop counts on identical address sets. The four 2-element shapes and both 4-element gathers agree to within 0.03 cycles and keep a shared entry. Bind the AVX2 (VEX) gathers, which had been left on the default write and diverged from it by more than an order of magnitude. All but the 8 x i32 form measure like their EVEX counterparts and reuse those entries; VPGATHERDDY is genuinely cheaper and gets its own. Measure gather latency from a dependency chain that feeds the loaded data back into the index register, with the feedback op's own latency subtracted. Several latencies had been carried over rather than measured -- the 8 x i32 gather was 29 cycles against 20 measured, the 4 x i64 gather 23 against 16. Scatter latency remains estimated, as there is no dependent consumer to time. State the mask-reload rule so it can be regenerated. The instruction destroys its own mask and so cannot be measured alone; substituting an equal-cost kxnorw for the kmovd isolates the reload, and only the 2-element gathers change (5.00 to 3.98 cycles), so only they have it subtracted. Sub-cycle accounting stops there: identical code in two processes agrees to a p90 of 0.24 cycles, but the same instructions in a different schedule move by a p90 of 1.35, so the table is accurate to about a cycle regardless. llvm-mca now tracks hardware within 1.3% on every scatter and within 1.1% on every gather of four elements or more. Add a CodeGen test so the values are exercised by something other than the llvm-mca resource tests.
5164b70 to
64b7a0b
Compare
Summary
The Znver4 scheduling model (also used by Znver5) had no overrides for the
AVX-512 masked gather/scatter instructions, so they fell back to the default
vector load/store write. This adds per-shape
SchedWriteResoverrides for themasked gather (
VPGATHER*/VGATHER*) and scatter (VPSCATTER*/VSCATTER*) instructions covering every EVEX shape.Motivation
On Zen4/Zen5 these are microcoded sequences whose throughput, uops and latency
scale strongly with the number of elements and the element width, so the
default cost is far from the real hardware. Giving the scheduler and llvm-mca
honest numbers is a prerequisite for driving the TTI gather/scatter cost model
from the schedule model (follow-up PR).
Implementation notes
used in
X86ScheduleZnver4.td(e.g.Zn4WriteVPGATHERDDZ128), with the FPvariant sharing the same
InstRWas the integer representative.(#elements, element width)and shared between thedword- and qword-index encodings of a shape, since the microcoded cost
tracks element count and width rather than index width.
reuses
Znver4Model) with mask-reloading microbenchmarks under perf;reciprocal throughput is reproduced through load-/store-pipe and AGU
occupancy via
ReleaseAtCycles. Scatter latency is estimated (throughputand uops are measured).
were not measured; their values are extrapolated as half the 4-element entry,
purely so llvm-mca stays self-consistent.
Tests
llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.sllvm/test/tools/llvm-mca/X86/Znver4/resources-avx512vl.sregenerated to reflect the new gather/scatter numbers.