text:
Disable bookend by default.

https://github.com/Lightning-AI/lightning-thunder/pull/974 hided the RoPE regression that used to block this PR, so the benchmark numbers look neutral.
$ pwd
/opt/pytorch/nvfuser

$ python tools/benchmark_thunder.py --storage ~/workspace main:main wjy/bookend:main
$ pytest-benchmark --storage ~/workspace compare 0017 0018 --group-by name

https://gist.github.com/wujingyue/a87080535d5f482f954b3ec8e835a66a
0017 -- without this PR
0018 -- with this PR
For a couple of benchmarks, 0018 wins by ~20%. But those are noise and disappear the second time I ran them.
cc @tfogal

What are the results that can be inferred from the "before" and "after" outputs in the pull request description? Is the change good or bad?
Could you please also update the PR description with the context and history of this flag?
Here's a slightly easier-to-interpret benchmark output obtained with:
pytest thunder/benchmarks/targets.py --benchmark-autosave -k "thunder]"
git checkout wjy/bookend
pytest thunder/benchmarks/targets.py --benchmark-autosave -k "thunder]"
pytest-benchmark compare 0001 0002 --group-by='name'

Click for the benchmark results
# pytest-benchmark compare 0002 0003 --group-by='name'

----------------------------------------------------------------------------------- benchmark 'test_batch_norm[backward-thunder]': 2 tests ----------------------------------------------------------------------------------
Name (time in us)                                         Min                 Max                Mean            StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_batch_norm[backward-thunder] (0003_af807f9)     105.2827 (1.0)      157.0910 (1.03)     114.4125 (1.0)      5.7068 (1.0)      113.4938 (1.0)      1.6492 (1.47)       87;115        8.7403 (1.0)         938          10
test_batch_norm[backward-thunder] (0002_7d62ae1)     106.7905 (1.01)     152.1134 (1.0)      115.3195 (1.01)     5.8931 (1.03)     114.3012 (1.01)     1.1246 (1.0)        68;119        8.6716 (0.99)        925          10
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_batch_norm[forward-thunder]': 2 tests ----------------------------------------------------------------------------------
Name (time in us)                                       Min                 Max               Mean             StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_batch_norm[forward-thunder] (0002_7d62ae1)     85.2062 (1.0)      146.1624 (1.0)      89.1900 (1.0)       7.3847 (1.0)      87.1541 (1.0)      1.2275 (1.0)        76;135       11.2120 (1.0)        1174          10
test_batch_norm[forward-thunder] (0003_af807f9)     85.7533 (1.01)     358.3039 (2.45)     90.1488 (1.01)     10.9547 (1.48)     87.9145 (1.01)     1.4099 (1.15)       68;130       11.0928 (0.99)       1167          10
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_batch_norm[inference-thunder]': 2 tests ---------------------------------------------------------------------------------
Name (time in us)                                         Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_batch_norm[inference-thunder] (0003_af807f9)     65.3689 (1.0)      123.2382 (1.0)      69.6323 (1.00)     7.3964 (1.0)      67.6332 (1.00)     1.2386 (1.08)      102;172       14.3612 (1.00)       1514          10
test_batch_norm[inference-thunder] (0002_7d62ae1)     65.4180 (1.00)     187.4234 (1.52)     69.4771 (1.0)      7.5675 (1.02)     67.6296 (1.0)      1.1489 (1.0)        94;159       14.3932 (1.0)        1530          10
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_qkv_split_rope[Llama-2-13b-hf-backward-bs1-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                                                       Min                   Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-13b-hf-backward-bs1-thunder] (0002_7d62ae1)     694.0509 (1.0)      1,169.6634 (1.04)     724.5289 (1.0)      67.7069 (1.11)     707.0149 (1.0)      7.8348 (1.26)       92;117        1.3802 (1.0)        1438           1
test_litgpt_qkv_split_rope[Llama-2-13b-hf-backward-bs1-thunder] (0003_af807f9)     764.7262 (1.10)     1,124.4356 (1.0)      791.4923 (1.09)     61.2606 (1.0)      775.9267 (1.10)     6.1975 (1.0)        83;101        1.2634 (0.92)       1305           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-13b-hf-backward-bs2-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                                     Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-13b-hf-backward-bs2-thunder] (0003_af807f9)     1.3085 (1.0)      1.6463 (1.0)      1.3347 (1.0)      0.0615 (1.0)      1.3191 (1.0)      0.0058 (1.0)         48;59  749.2476 (1.0)         764           1
test_litgpt_qkv_split_rope[Llama-2-13b-hf-backward-bs2-thunder] (0002_7d62ae1)     1.3846 (1.06)     1.9526 (1.19)     1.4139 (1.06)     0.0725 (1.18)     1.3951 (1.06)     0.0069 (1.18)        49;56  707.2559 (0.94)        723           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-13b-hf-forward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                                      Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-13b-hf-forward-bs1-thunder] (0003_af807f9)     350.7216 (1.0)      697.8686 (1.0)      367.4164 (1.0)      39.1957 (1.0)      357.5522 (1.0)      4.6874 (1.02)       71;148        2.7217 (1.0)        1429           2
test_litgpt_qkv_split_rope[Llama-2-13b-hf-forward-bs1-thunder] (0002_7d62ae1)     371.7090 (1.06)     738.3525 (1.06)     390.0441 (1.06)     42.2033 (1.08)     379.0190 (1.06)     4.5733 (1.0)        71;149        2.5638 (0.94)       1351           2
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-13b-hf-forward-bs2-thunder]': 2 tests ---------------------------------------------------------------------------------------
Name (time in us)                                                                        Min                   Max                  Mean              StdDev                Median                IQR            Outliers         OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-13b-hf-forward-bs2-thunder] (0002_7d62ae1)       742.3386 (1.0)      1,957.5511 (1.0)        794.2925 (1.0)      113.6537 (1.46)       756.8570 (1.0)      13.2808 (1.17)      133;169  1,258.9821 (1.0)        1353           1
test_litgpt_qkv_split_rope[Llama-2-13b-hf-forward-bs2-thunder] (0003_af807f9)     2,951.6988 (3.98)     3,402.1954 (1.74)     2,986.4246 (3.76)      78.0704 (1.0)      2,964.9280 (3.92)     11.3915 (1.0)         21;37    334.8486 (0.27)        340           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-13b-hf-inference-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                                        Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-13b-hf-inference-bs1-thunder] (0003_af807f9)     345.7631 (1.0)      943.5895 (1.59)     361.2745 (1.0)      39.6749 (1.12)     351.5279 (1.0)      3.5630 (1.0)        71;160        2.7680 (1.0)        1454           2
test_litgpt_qkv_split_rope[Llama-2-13b-hf-inference-bs1-thunder] (0002_7d62ae1)     359.2505 (1.04)     592.0776 (1.0)      373.5317 (1.03)     35.3059 (1.0)      364.3120 (1.04)     3.8987 (1.09)       67;141        2.6771 (0.97)       1395           2
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-13b-hf-inference-bs2-thunder]': 2 tests --------------------------------------------------------------------------------------
Name (time in us)                                                                          Min                   Max                  Mean             StdDev                Median               IQR            Outliers         OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-13b-hf-inference-bs2-thunder] (0002_7d62ae1)       729.4999 (1.0)      1,633.2669 (1.0)        772.4898 (1.0)      98.6939 (1.33)       739.6389 (1.0)      9.6960 (1.0)       137;196  1,294.5154 (1.0)        1384           1
test_litgpt_qkv_split_rope[Llama-2-13b-hf-inference-bs2-thunder] (0003_af807f9)     2,937.5702 (4.03)     3,373.0241 (2.07)     2,968.3295 (3.84)     74.2754 (1.0)      2,948.3577 (3.99)     9.9218 (1.02)        18;37    336.8898 (0.26)        342           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-70b-hf-backward-bs1-thunder]': 2 tests --------------------------------------------------------------------------------------
Name (time in us)                                                                         Min                   Max                  Mean             StdDev                Median               IQR            Outliers         OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-70b-hf-backward-bs1-thunder] (0002_7d62ae1)       797.2918 (1.0)      1,434.2456 (1.0)        824.3178 (1.0)      58.9251 (1.0)        809.1033 (1.0)      7.7584 (1.69)        79;98  1,213.1244 (1.0)        1252           1
test_litgpt_qkv_split_rope[Llama-2-70b-hf-backward-bs1-thunder] (0003_af807f9)     1,267.4648 (1.59)     1,849.1708 (1.29)     1,292.9274 (1.57)     62.9070 (1.07)     1,277.3133 (1.58)     4.5910 (1.0)         50;67    773.4387 (0.64)        790           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-70b-hf-backward-bs2-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                                     Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-70b-hf-backward-bs2-thunder] (0002_7d62ae1)     1.4380 (1.0)      1.6930 (1.0)      1.4622 (1.0)      0.0543 (1.0)      1.4489 (1.0)      0.0055 (1.0)         44;47  683.8781 (1.0)         696           1
test_litgpt_qkv_split_rope[Llama-2-70b-hf-backward-bs2-thunder] (0003_af807f9)     2.7236 (1.89)     3.0583 (1.81)     2.7500 (1.88)     0.0612 (1.13)     2.7340 (1.89)     0.0065 (1.19)        26;30  363.6368 (0.53)        368           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_qkv_split_rope[Llama-2-70b-hf-forward-bs1-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                                                      Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-70b-hf-forward-bs1-thunder] (0003_af807f9)     712.7421 (1.0)      1,184.3797 (1.0)      749.3845 (1.0)      76.4985 (1.0)      727.3431 (1.0)      14.2963 (1.55)       79;184        1.3344 (1.0)        1406           1
test_litgpt_qkv_split_rope[Llama-2-70b-hf-forward-bs1-thunder] (0002_7d62ae1)     804.1477 (1.13)     1,831.7960 (1.55)     840.1958 (1.12)     79.6607 (1.04)     816.3569 (1.12)      9.2032 (1.0)       120;161        1.1902 (0.89)       1247           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-70b-hf-forward-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                                    Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-70b-hf-forward-bs2-thunder] (0002_7d62ae1)     1.5136 (1.0)      2.1330 (1.0)      1.5449 (1.0)      0.0742 (1.01)     1.5252 (1.0)      0.0095 (1.0)         37;72  647.3100 (1.0)         662           1
test_litgpt_qkv_split_rope[Llama-2-70b-hf-forward-bs2-thunder] (0003_af807f9)     4.8474 (3.20)     5.2076 (2.44)     4.8862 (3.16)     0.0734 (1.0)      4.8653 (3.19)     0.0196 (2.06)        11;19  204.6573 (0.32)        207           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_qkv_split_rope[Llama-2-70b-hf-inference-bs1-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                                                        Min                   Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-70b-hf-inference-bs1-thunder] (0003_af807f9)     698.7941 (1.0)      1,549.7650 (1.08)     733.4003 (1.0)      78.3481 (1.03)     712.2126 (1.0)      9.5465 (1.10)       77;163        1.3635 (1.0)        1432           1
test_litgpt_qkv_split_rope[Llama-2-70b-hf-inference-bs1-thunder] (0002_7d62ae1)     813.5061 (1.16)     1,429.8861 (1.0)      847.5727 (1.16)     75.7869 (1.0)      823.8426 (1.16)     8.6692 (1.0)       118;183        1.1798 (0.87)       1234           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-70b-hf-inference-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                                      Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-70b-hf-inference-bs2-thunder] (0002_7d62ae1)     1.5361 (1.0)      2.2752 (1.0)      1.5632 (1.0)      0.0712 (1.02)     1.5460 (1.0)      0.0078 (1.0)         32;69  639.6965 (1.0)         653           1
test_litgpt_qkv_split_rope[Llama-2-70b-hf-inference-bs2-thunder] (0003_af807f9)     4.8346 (3.15)     5.1960 (2.28)     4.8682 (3.11)     0.0697 (1.0)      4.8489 (3.14)     0.0124 (1.59)        11;23  205.4133 (0.32)        207           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_qkv_split_rope[Llama-2-7b-hf-backward-bs1-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                                                      Min                   Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-7b-hf-backward-bs1-thunder] (0002_7d62ae1)     584.3821 (1.0)        908.6430 (1.0)      610.5621 (1.0)      58.4833 (1.0)      595.8304 (1.0)      8.3027 (1.16)      110;115        1.6378 (1.0)        1718           1
test_litgpt_qkv_split_rope[Llama-2-7b-hf-backward-bs1-thunder] (0003_af807f9)     645.5909 (1.10)     1,116.6600 (1.23)     674.4252 (1.10)     64.0627 (1.10)     657.9980 (1.10)     7.1451 (1.0)        99;103        1.4827 (0.91)       1550           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-7b-hf-backward-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                                    Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-7b-hf-backward-bs2-thunder] (0003_af807f9)     1.0702 (1.0)      1.4659 (1.02)     1.1018 (1.0)      0.0644 (1.10)     1.0838 (1.0)      0.0086 (1.27)        64;94  907.6409 (1.0)         935           1
test_litgpt_qkv_split_rope[Llama-2-7b-hf-backward-bs2-thunder] (0002_7d62ae1)     1.1199 (1.05)     1.4419 (1.0)      1.1455 (1.04)     0.0585 (1.0)      1.1311 (1.04)     0.0068 (1.0)         55;57  872.9678 (0.96)        894           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-7b-hf-forward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                                     Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-7b-hf-forward-bs1-thunder] (0003_af807f9)     300.7683 (1.0)      925.9235 (1.23)     317.7686 (1.0)      39.3588 (1.0)      307.9849 (1.0)      4.8517 (1.0)        80;169        3.1469 (1.0)        1657           2
test_litgpt_qkv_split_rope[Llama-2-7b-hf-forward-bs1-thunder] (0002_7d62ae1)     318.4380 (1.06)     752.0844 (1.0)      339.3701 (1.07)     42.0364 (1.07)     327.0565 (1.06)     8.0263 (1.65)      101;191        2.9466 (0.94)       1569           2
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-7b-hf-forward-bs2-thunder]': 2 tests --------------------------------------------------------------------------------------
Name (time in us)                                                                       Min                   Max                  Mean             StdDev                Median                IQR            Outliers         OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-7b-hf-forward-bs2-thunder] (0002_7d62ae1)       630.3820 (1.0)      1,164.4987 (1.0)        676.0690 (1.0)      93.1502 (1.40)       644.9758 (1.0)      11.2671 (1.08)      153;195  1,479.1390 (1.0)        1587           1
test_litgpt_qkv_split_rope[Llama-2-7b-hf-forward-bs2-thunder] (0003_af807f9)     2,396.1221 (3.80)     2,796.2234 (2.40)     2,427.0360 (3.59)     66.4130 (1.0)      2,409.1471 (3.74)     10.4014 (1.0)         22;43    412.0252 (0.28)        419           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-7b-hf-inference-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                                       Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-7b-hf-inference-bs1-thunder] (0003_af807f9)     296.5410 (1.0)      646.1765 (1.0)      313.4811 (1.0)      37.2798 (1.0)      302.6894 (1.0)      5.5807 (1.27)      100;225        3.1900 (1.0)        1685           2
test_litgpt_qkv_split_rope[Llama-2-7b-hf-inference-bs1-thunder] (0002_7d62ae1)     305.7760 (1.03)     845.9550 (1.31)     322.6342 (1.03)     38.4419 (1.03)     312.8296 (1.03)     4.3854 (1.0)        81;178        3.0995 (0.97)       1637           2
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-2-7b-hf-inference-bs2-thunder]': 2 tests --------------------------------------------------------------------------------------
Name (time in us)                                                                         Min                   Max                  Mean             StdDev                Median                IQR            Outliers         OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-2-7b-hf-inference-bs2-thunder] (0002_7d62ae1)       616.3642 (1.0)      1,340.0312 (1.0)        659.3661 (1.0)      90.8849 (1.24)       628.3810 (1.0)       9.2218 (1.0)       158;239  1,516.6082 (1.0)        1635           1
test_litgpt_qkv_split_rope[Llama-2-7b-hf-inference-bs2-thunder] (0003_af807f9)     2,383.3956 (3.87)     3,035.8839 (2.27)     2,414.1565 (3.66)     73.2880 (1.0)      2,393.8799 (3.81)     10.7132 (1.16)        24;50    414.2234 (0.27)        421           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-3-70B-backward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                                  Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-70B-backward-bs1-thunder] (0002_7d62ae1)     1.4378 (1.0)      1.7232 (1.0)      1.4622 (1.0)      0.0540 (1.0)      1.4487 (1.0)      0.0069 (1.10)        43;49  683.9142 (1.0)         695           1
test_litgpt_qkv_split_rope[Llama-3-70B-backward-bs1-thunder] (0003_af807f9)     2.3773 (1.65)     2.6925 (1.56)     2.4095 (1.65)     0.0647 (1.20)     2.3917 (1.65)     0.0062 (1.0)         32;38  415.0192 (0.61)        422           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-3-70B-backward-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                                  Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-70B-backward-bs2-thunder] (0002_7d62ae1)     2.7201 (1.0)      3.6348 (1.0)      2.7515 (1.0)      0.0752 (1.07)     2.7322 (1.0)      0.0074 (1.0)         25;48  363.4352 (1.0)         368           1
test_litgpt_qkv_split_rope[Llama-3-70B-backward-bs2-thunder] (0003_af807f9)     5.2817 (1.94)     5.8238 (1.60)     5.3119 (1.93)     0.0705 (1.0)      5.2941 (1.94)     0.0124 (1.67)        11;15  188.2558 (0.52)        190           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-3-70B-forward-bs1-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                                 Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-70B-forward-bs1-thunder] (0003_af807f9)     1.2318 (1.0)      2.3138 (1.26)     1.2672 (1.0)      0.0864 (1.24)     1.2450 (1.0)      0.0100 (1.18)        45;84  789.1414 (1.0)         812           1
test_litgpt_qkv_split_rope[Llama-3-70B-forward-bs1-thunder] (0002_7d62ae1)     1.4364 (1.17)     1.8374 (1.0)      1.4654 (1.16)     0.0695 (1.0)      1.4473 (1.16)     0.0084 (1.0)         38;69  682.4206 (0.86)        698           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-3-70B-forward-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                                 Min                Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-70B-forward-bs2-thunder] (0002_7d62ae1)     2.8456 (1.0)       3.1610 (1.0)      2.8739 (1.0)      0.0658 (1.0)      2.8566 (1.0)      0.0096 (1.01)        19;37  347.9541 (1.0)         352           1
test_litgpt_qkv_split_rope[Llama-3-70B-forward-bs2-thunder] (0003_af807f9)     9.4542 (3.32)     10.1548 (3.21)     9.4917 (3.30)     0.0979 (1.49)     9.4654 (3.31)     0.0095 (1.0)          6;13  105.3553 (0.30)        106           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-3-70B-inference-bs1-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                                   Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-70B-inference-bs1-thunder] (0003_af807f9)     1.2227 (1.0)      1.6614 (1.0)      1.2524 (1.0)      0.0718 (1.18)     1.2330 (1.0)      0.0094 (1.34)        42;94  798.4752 (1.0)         823           1
test_litgpt_qkv_split_rope[Llama-3-70B-inference-bs1-thunder] (0002_7d62ae1)     1.4575 (1.19)     1.8029 (1.09)     1.4811 (1.18)     0.0610 (1.0)      1.4665 (1.19)     0.0070 (1.0)         33;59  675.1863 (0.85)        687           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-3-70B-inference-bs2-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                                   Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-70B-inference-bs2-thunder] (0002_7d62ae1)     2.8999 (1.0)      3.2380 (1.0)      2.9247 (1.0)      0.0624 (1.02)     2.9091 (1.0)      0.0068 (1.0)         17;38  341.9184 (1.0)         346           1
test_litgpt_qkv_split_rope[Llama-3-70B-inference-bs2-thunder] (0003_af807f9)     9.4774 (3.27)     9.7856 (3.02)     9.5055 (3.25)     0.0610 (1.0)      9.4886 (3.26)     0.0114 (1.69)         6;13  105.2022 (0.31)        106           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-3-8B-backward-bs1-thunder]': 2 tests --------------------------------------------------------------------------------------
Name (time in us)                                                                     Min                   Max                  Mean             StdDev                Median               IQR            Outliers         OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-8B-backward-bs1-thunder] (0002_7d62ae1)       846.4498 (1.0)      1,683.6631 (1.02)       877.5587 (1.0)      69.3678 (1.11)       858.7760 (1.0)      9.9610 (1.15)       79;136  1,139.5249 (1.0)        1182           1
test_litgpt_qkv_split_rope[Llama-3-8B-backward-bs1-thunder] (0003_af807f9)     1,289.4548 (1.52)     1,657.3532 (1.0)      1,319.7025 (1.50)     62.2520 (1.0)      1,302.0590 (1.52)     8.6315 (1.0)         58;86    757.7465 (0.66)        776           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-3-8B-backward-bs2-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                                 Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-8B-backward-bs2-thunder] (0002_7d62ae1)     1.5472 (1.0)      2.2090 (1.0)      1.5772 (1.0)      0.0658 (1.0)      1.5609 (1.0)      0.0074 (1.0)         41;47  634.0259 (1.0)         648           1
test_litgpt_qkv_split_rope[Llama-3-8B-backward-bs2-thunder] (0003_af807f9)     2.7692 (1.79)     3.1402 (1.42)     2.7981 (1.77)     0.0670 (1.02)     2.7789 (1.78)     0.0080 (1.09)        28;39  357.3829 (0.56)        362           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_qkv_split_rope[Llama-3-8B-forward-bs1-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                                                  Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-8B-forward-bs1-thunder] (0003_af807f9)     728.9578 (1.0)      1,562.5572 (1.0)      765.3614 (1.0)      82.5845 (1.0)      743.7759 (1.0)      12.4075 (1.34)       72;141        1.3066 (1.0)        1374           1
test_litgpt_qkv_split_rope[Llama-3-8B-forward-bs1-thunder] (0002_7d62ae1)     806.3838 (1.11)     2,276.1622 (1.46)     844.4545 (1.10)     92.5043 (1.12)     818.0570 (1.10)      9.2398 (1.0)       121;174        1.1842 (0.91)       1243           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-3-8B-forward-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                                Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-8B-forward-bs2-thunder] (0002_7d62ae1)     1.5113 (1.0)      1.9503 (1.0)      1.5395 (1.0)      0.0719 (1.0)      1.5207 (1.0)      0.0080 (1.0)         36;67  649.5438 (1.0)         663           1
test_litgpt_qkv_split_rope[Llama-3-8B-forward-bs2-thunder] (0003_af807f9)     4.8575 (3.21)     5.3206 (2.73)     4.8971 (3.18)     0.0810 (1.13)     4.8750 (3.21)     0.0163 (2.03)        11;24  204.2012 (0.31)        207           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_qkv_split_rope[Llama-3-8B-inference-bs1-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                                                    Min                   Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-8B-inference-bs1-thunder] (0003_af807f9)     721.5971 (1.0)      1,388.0963 (1.12)     755.7033 (1.0)      81.8033 (1.18)     734.0536 (1.0)      8.9910 (1.02)       72;156        1.3233 (1.0)        1390           1
test_litgpt_qkv_split_rope[Llama-3-8B-inference-bs1-thunder] (0002_7d62ae1)     810.9948 (1.12)     1,238.3582 (1.0)      842.1366 (1.11)     69.3210 (1.0)      821.0498 (1.12)     8.7816 (1.0)        79;188        1.1875 (0.90)       1241           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Llama-3-8B-inference-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                                  Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Llama-3-8B-inference-bs2-thunder] (0002_7d62ae1)     1.5341 (1.0)      2.3818 (1.0)      1.5610 (1.0)      0.0744 (1.0)      1.5431 (1.0)      0.0066 (1.0)         33;71  640.6284 (1.0)         654           1
test_litgpt_qkv_split_rope[Llama-3-8B-inference-bs2-thunder] (0003_af807f9)     4.8434 (3.16)     5.2719 (2.21)     4.8776 (3.12)     0.0757 (1.02)     4.8571 (3.15)     0.0123 (1.85)        11;23  205.0172 (0.32)        207           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_qkv_split_rope[Mistral-7B-v0.1-backward-bs1-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                                                        Min                   Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-backward-bs1-thunder] (0002_7d62ae1)     434.7549 (1.0)        749.5859 (1.0)      447.2606 (1.0)      30.9177 (1.0)      440.1420 (1.0)      4.5002 (1.0)         56;71        2.2358 (1.0)        1152           2
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-backward-bs1-thunder] (0003_af807f9)     726.7059 (1.67)     1,173.3929 (1.57)     757.9373 (1.69)     61.5078 (1.99)     740.6930 (1.68)     9.8660 (2.19)      105;121        1.3194 (0.59)       1378           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Mistral-7B-v0.1-backward-bs2-thunder]': 2 tests ---------------------------------------------------------------------------------------
Name (time in us)                                                                          Min                   Max                  Mean              StdDev                Median                IQR            Outliers         OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-backward-bs2-thunder] (0002_7d62ae1)       857.2140 (1.0)      2,932.6929 (1.67)       915.5342 (1.0)      110.8944 (1.88)       892.1660 (1.0)      10.5926 (1.53)       78;151  1,092.2585 (1.0)        1132           1
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-backward-bs2-thunder] (0003_af807f9)     1,464.5117 (1.71)     1,752.9912 (1.0)      1,492.1034 (1.63)      59.1187 (1.0)      1,475.6303 (1.65)      6.9442 (1.0)         47;60    670.1948 (0.61)        684           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Mistral-7B-v0.1-forward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                                       Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-forward-bs1-thunder] (0003_af807f9)     378.2709 (1.0)      626.3824 (1.0)      393.9628 (1.0)      37.8802 (1.0)      384.2914 (1.0)      4.5793 (1.15)       67;138        2.5383 (1.0)        1325           2
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-forward-bs1-thunder] (0002_7d62ae1)     415.6900 (1.10)     775.7447 (1.24)     430.7088 (1.09)     38.9930 (1.03)     421.1981 (1.10)     3.9654 (1.0)        60;128        2.3218 (0.91)       1205           2
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Mistral-7B-v0.1-forward-bs2-thunder]': 2 tests --------------------------------------------------------------------------------------
Name (time in us)                                                                         Min                   Max                  Mean             StdDev                Median                IQR            Outliers         OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-forward-bs2-thunder] (0002_7d62ae1)       837.1561 (1.0)      1,192.0980 (1.0)        869.7725 (1.0)      73.5546 (1.0)        847.4486 (1.0)       9.9819 (1.0)       114;149  1,149.7259 (1.0)        1198           1
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-forward-bs2-thunder] (0003_af807f9)     2,531.1769 (3.02)     2,950.9207 (2.48)     2,564.6287 (2.95)     74.8823 (1.02)     2,544.0152 (3.00)     13.0227 (1.30)        22;39    389.9200 (0.34)        396           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Mistral-7B-v0.1-inference-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                                         Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-inference-bs1-thunder] (0003_af807f9)     373.5200 (1.0)      606.8365 (1.0)      389.2410 (1.0)      34.9560 (1.0)      379.8949 (1.0)      4.3000 (1.27)       71;161        2.5691 (1.0)        1337           2
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-inference-bs1-thunder] (0002_7d62ae1)     419.0879 (1.12)     707.4785 (1.17)     433.1831 (1.11)     38.1303 (1.09)     423.4959 (1.11)     3.3810 (1.0)        59;154        2.3085 (0.90)       1197           2
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[Mistral-7B-v0.1-inference-bs2-thunder]': 2 tests --------------------------------------------------------------------------------------
Name (time in us)                                                                           Min                   Max                  Mean             StdDev                Median                IQR            Outliers         OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-inference-bs2-thunder] (0002_7d62ae1)       843.5724 (1.0)      1,395.1687 (1.0)        872.6096 (1.0)      69.5579 (1.01)       852.1848 (1.0)       7.8045 (1.0)        71;159  1,145.9879 (1.0)        1191           1
test_litgpt_qkv_split_rope[Mistral-7B-v0.1-inference-bs2-thunder] (0003_af807f9)     2,520.8388 (2.99)     2,975.1719 (2.13)     2,551.4443 (2.92)     68.9350 (1.0)      2,532.2558 (2.97)     10.1347 (1.30)        22;47    391.9349 (0.34)        398           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[phi-2-backward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                              Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[phi-2-backward-bs1-thunder] (0003_af807f9)     250.3768 (1.0)      550.0579 (1.0)      269.2825 (1.0)      23.1929 (1.0)      263.7329 (1.0)      7.5824 (1.43)       97;129        3.7136 (1.0)        1340           3
test_litgpt_qkv_split_rope[phi-2-backward-bs1-thunder] (0002_7d62ae1)     320.6285 (1.28)     636.3902 (1.16)     332.8162 (1.24)     26.3851 (1.14)     326.8181 (1.24)     5.2936 (1.0)         77;81        3.0047 (0.81)       1559           2
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_qkv_split_rope[phi-2-backward-bs2-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                                              Min                   Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[phi-2-backward-bs2-thunder] (0003_af807f9)     455.4261 (1.0)        667.0589 (1.0)      469.7736 (1.0)      31.1865 (1.0)      462.3112 (1.0)      4.5116 (1.0)         62;79        2.1287 (1.0)        1098           2
test_litgpt_qkv_split_rope[phi-2-backward-bs2-thunder] (0002_7d62ae1)     655.9831 (1.44)     1,405.0310 (2.11)     686.7280 (1.46)     67.8123 (2.17)     669.5744 (1.45)     6.9996 (1.55)      100;124        1.4562 (0.68)       1520           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[phi-2-forward-bs1-thunder]': 2 tests --------------------------------------------------------------------------------------
Name (time in us)                                                             Min                     Max                Mean                StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[phi-2-forward-bs1-thunder] (0003_af807f9)     181.4912 (1.0)      197,413.4845 (270.02)   550.7718 (1.69)     8,424.9395 (182.42)   186.8945 (1.0)      5.5142 (1.0)          1;69        1.8156 (0.59)        548          10
test_litgpt_qkv_split_rope[phi-2-forward-bs1-thunder] (0002_7d62ae1)     299.0745 (1.65)         731.1094 (1.0)      325.2010 (1.0)         46.1840 (1.0)      313.1008 (1.68)     8.8327 (1.60)       91;177        3.0750 (1.0)        1662           2
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[phi-2-forward-bs2-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                                             Min                   Max                Mean              StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[phi-2-forward-bs2-thunder] (0003_af807f9)     302.6519 (1.0)        592.6825 (1.0)      322.4569 (1.0)       44.3886 (1.0)      310.2297 (1.0)       5.1839 (1.0)        89;206        3.1012 (1.0)        1652           2
test_litgpt_qkv_split_rope[phi-2-forward-bs2-thunder] (0002_7d62ae1)     581.3180 (1.92)     1,491.1001 (2.52)     634.7485 (1.97)     102.3185 (2.31)     599.5221 (1.93)     14.4583 (2.79)      175;252        1.5754 (0.51)       1745           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[phi-2-inference-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                               Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[phi-2-inference-bs1-thunder] (0003_af807f9)     164.5835 (1.0)      240.3515 (1.0)      170.4853 (1.0)       9.5016 (1.0)      167.7159 (1.0)      2.6341 (1.0)         38;63        5.8656 (1.0)         606          10
test_litgpt_qkv_split_rope[phi-2-inference-bs1-thunder] (0002_7d62ae1)     227.4519 (1.38)     595.7210 (2.48)     244.2094 (1.43)     35.2736 (3.71)     234.6397 (1.40)     5.2988 (2.01)       95;161        4.0948 (0.70)       1461           3
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_qkv_split_rope[phi-2-inference-bs2-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                               Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_qkv_split_rope[phi-2-inference-bs2-thunder] (0003_af807f9)     295.1627 (1.0)      705.4375 (1.0)      310.9849 (1.0)      37.0849 (1.0)      301.0614 (1.0)      4.4014 (1.0)        89;190        3.2156 (1.0)        1700           2
test_litgpt_qkv_split_rope[phi-2-inference-bs2-thunder] (0002_7d62ae1)     354.7384 (1.20)     901.7615 (1.28)     371.4070 (1.19)     41.1443 (1.11)     360.8322 (1.20)     4.7694 (1.08)       73;158        2.6925 (0.84)       1412           2
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-13b-hf-backward-bs1-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                           Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-13b-hf-backward-bs1-thunder] (0003_af807f9)     1.3099 (1.0)      1.7339 (1.08)     1.3406 (1.0)      0.0588 (1.04)     1.3266 (1.0)      0.0116 (1.0)         49;50  745.9298 (1.0)         773           1
test_litgpt_sdpa[Llama-2-13b-hf-backward-bs1-thunder] (0002_7d62ae1)     1.3128 (1.00)     1.6020 (1.0)      1.3430 (1.00)     0.0564 (1.0)      1.3280 (1.00)     0.0126 (1.09)        49;53  744.6175 (1.00)        769           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-13b-hf-backward-bs2-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                           Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-13b-hf-backward-bs2-thunder] (0003_af807f9)     2.2296 (1.0)      2.6329 (1.02)     2.2786 (1.00)     0.0671 (1.0)      2.2574 (1.00)     0.0325 (1.92)        31;35  438.8728 (1.00)        453           1
test_litgpt_sdpa[Llama-2-13b-hf-backward-bs2-thunder] (0002_7d62ae1)     2.2323 (1.00)     2.5866 (1.0)      2.2779 (1.0)      0.0672 (1.00)     2.2573 (1.0)      0.0169 (1.0)         38;58  438.9917 (1.0)         452           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_sdpa[Llama-2-13b-hf-forward-bs1-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                                            Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-13b-hf-forward-bs1-thunder] (0002_7d62ae1)     579.2058 (1.0)      1,041.7593 (1.0)      620.0286 (1.0)      74.1246 (1.0)      600.3540 (1.00)     15.0963 (1.0)        96;176        1.6128 (1.0)        1736           1
test_litgpt_sdpa[Llama-2-13b-hf-forward-bs1-thunder] (0003_af807f9)     579.2943 (1.00)     2,263.8310 (2.17)     620.0800 (1.00)     83.0757 (1.12)     600.0979 (1.0)      15.4283 (1.02)       90;179        1.6127 (1.00)       1728           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_sdpa[Llama-2-13b-hf-forward-bs2-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                                            Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-13b-hf-forward-bs2-thunder] (0003_af807f9)     852.7138 (1.0)      1,298.3428 (1.0)      892.5657 (1.0)      74.9483 (1.0)      872.3312 (1.0)      14.9092 (1.0)        67;120        1.1204 (1.0)        1177           1
test_litgpt_sdpa[Llama-2-13b-hf-forward-bs2-thunder] (0002_7d62ae1)     859.1330 (1.01)     1,512.7151 (1.17)     904.0847 (1.01)     82.4785 (1.10)     879.7138 (1.01)     17.6303 (1.18)       82;151        1.1061 (0.99)       1179           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_sdpa[Llama-2-13b-hf-inference-bs1-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                                              Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-13b-hf-inference-bs1-thunder] (0003_af807f9)     561.0739 (1.0)      1,052.2390 (1.05)     598.2303 (1.0)      73.1401 (1.03)     578.3923 (1.0)      14.5058 (1.09)       99;191        1.6716 (1.0)        1795           1
test_litgpt_sdpa[Llama-2-13b-hf-inference-bs1-thunder] (0002_7d62ae1)     563.1847 (1.00)     1,000.1352 (1.0)      598.6921 (1.00)     71.0962 (1.0)      579.9218 (1.00)     13.3282 (1.0)        94;177        1.6703 (1.00)       1802           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_sdpa[Llama-2-13b-hf-inference-bs2-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                                              Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-13b-hf-inference-bs2-thunder] (0003_af807f9)     838.3119 (1.0)      1,269.0648 (1.0)      878.5282 (1.0)      71.7226 (1.0)      858.9663 (1.0)      14.9016 (1.0)        69;143        1.1383 (1.0)        1198           1
test_litgpt_sdpa[Llama-2-13b-hf-inference-bs2-thunder] (0002_7d62ae1)     841.5119 (1.00)     1,310.6540 (1.03)     881.0570 (1.00)     72.6201 (1.01)     859.7393 (1.00)     17.0770 (1.15)       65;160        1.1350 (1.00)       1197           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-70b-hf-backward-bs1-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                           Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-70b-hf-backward-bs1-thunder] (0003_af807f9)     1.8584 (1.0)      2.2121 (1.01)     1.8938 (1.0)      0.0575 (1.01)     1.8801 (1.0)      0.0091 (1.10)        36;48  528.0275 (1.0)         542           1
test_litgpt_sdpa[Llama-2-70b-hf-backward-bs1-thunder] (0002_7d62ae1)     1.8654 (1.00)     2.1812 (1.0)      1.8987 (1.00)     0.0568 (1.0)      1.8852 (1.00)     0.0083 (1.0)         34;47  526.6766 (1.00)        542           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-70b-hf-backward-bs2-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                           Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-70b-hf-backward-bs2-thunder] (0003_af807f9)     3.3492 (1.0)      3.7620 (1.0)      3.4425 (1.0)      0.0707 (1.0)      3.4244 (1.00)     0.0746 (1.0)         44;18  290.4881 (1.0)         303           1
test_litgpt_sdpa[Llama-2-70b-hf-backward-bs2-thunder] (0002_7d62ae1)     3.3598 (1.00)     3.7977 (1.01)     3.4429 (1.00)     0.0748 (1.06)     3.4210 (1.0)      0.0840 (1.13)        43;17  290.4566 (1.00)        301           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_sdpa[Llama-2-70b-hf-forward-bs1-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                                            Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-70b-hf-forward-bs1-thunder] (0002_7d62ae1)     744.1589 (1.0)      1,181.3319 (1.0)      783.0882 (1.0)      73.4369 (1.0)      763.5048 (1.0)      15.3653 (1.0)        75;133        1.2770 (1.0)        1359           1
test_litgpt_sdpa[Llama-2-70b-hf-forward-bs1-thunder] (0003_af807f9)     745.4790 (1.00)     1,254.4044 (1.06)     788.1032 (1.01)     76.9314 (1.05)     767.7749 (1.01)     16.0800 (1.05)       75;137        1.2689 (0.99)       1356           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-70b-hf-forward-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                          Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-70b-hf-forward-bs2-thunder] (0002_7d62ae1)     1.1986 (1.0)      1.6545 (1.00)     1.2418 (1.0)      0.0714 (1.0)      1.2239 (1.0)      0.0190 (1.00)        44;76  805.2533 (1.0)         840           1
test_litgpt_sdpa[Llama-2-70b-hf-forward-bs2-thunder] (0003_af807f9)     1.2012 (1.00)     1.6497 (1.0)      1.2465 (1.00)     0.0762 (1.07)     1.2255 (1.00)     0.0190 (1.0)         48;88  802.2391 (1.00)        836           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_sdpa[Llama-2-70b-hf-inference-bs1-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                                              Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-70b-hf-inference-bs1-thunder] (0002_7d62ae1)     726.2100 (1.0)      1,932.8678 (1.64)     767.1202 (1.0)      79.5447 (1.07)     745.5619 (1.0)      14.2128 (1.0)        82;162        1.3036 (1.0)        1381           1
test_litgpt_sdpa[Llama-2-70b-hf-inference-bs1-thunder] (0003_af807f9)     728.1760 (1.00)     1,177.0283 (1.0)      767.3991 (1.00)     74.3279 (1.0)      747.4781 (1.00)     14.5973 (1.03)       76;140        1.3031 (1.00)       1379           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-70b-hf-inference-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                            Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-70b-hf-inference-bs2-thunder] (0002_7d62ae1)     1.1790 (1.0)      1.6227 (1.0)      1.2254 (1.0)      0.0700 (1.0)      1.2087 (1.0)      0.0198 (1.0)         46;68  816.0383 (1.0)         862           1
test_litgpt_sdpa[Llama-2-70b-hf-inference-bs2-thunder] (0003_af807f9)     1.1822 (1.00)     1.6426 (1.01)     1.2335 (1.01)     0.0742 (1.06)     1.2135 (1.00)     0.0243 (1.23)        47;75  810.6705 (0.99)        853           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-7b-hf-backward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                          Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-7b-hf-backward-bs1-thunder] (0003_af807f9)     1.1244 (1.0)      1.4010 (1.0)      1.1577 (1.0)      0.0580 (1.0)      1.1432 (1.0)      0.0128 (1.0)         56;57  863.8102 (1.0)         898           1
test_litgpt_sdpa[Llama-2-7b-hf-backward-bs1-thunder] (0002_7d62ae1)     1.1308 (1.01)     1.4761 (1.05)     1.1672 (1.01)     0.0606 (1.04)     1.1511 (1.01)     0.0148 (1.16)        61;74  856.7708 (0.99)        891           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-7b-hf-backward-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                          Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-7b-hf-backward-bs2-thunder] (0003_af807f9)     1.8644 (1.0)      2.1984 (1.0)      1.9003 (1.0)      0.0605 (1.0)      1.8838 (1.0)      0.0091 (1.0)         37;56  526.2462 (1.0)         541           1
test_litgpt_sdpa[Llama-2-7b-hf-backward-bs2-thunder] (0002_7d62ae1)     1.8673 (1.00)     2.2590 (1.03)     1.9145 (1.01)     0.0671 (1.11)     1.8928 (1.00)     0.0146 (1.60)        48;78  522.3292 (0.99)        540           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-7b-hf-forward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                           Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-7b-hf-forward-bs1-thunder] (0002_7d62ae1)     393.3010 (1.0)      692.6279 (1.07)     414.1771 (1.0)      36.7627 (1.0)      403.3570 (1.0)      8.6322 (1.26)       68;199        2.4144 (1.0)        1270           2
test_litgpt_sdpa[Llama-2-7b-hf-forward-bs1-thunder] (0003_af807f9)     395.3180 (1.01)     649.3055 (1.0)      415.2267 (1.00)     37.4481 (1.02)     405.7527 (1.01)     6.8429 (1.0)        62;128        2.4083 (1.00)       1266           2
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-7b-hf-forward-bs2-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                                           Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-7b-hf-forward-bs2-thunder] (0003_af807f9)     752.0411 (1.0)      1,228.8257 (1.0)      795.0294 (1.0)      80.3188 (1.0)      773.5630 (1.0)      16.5491 (1.03)       75;134        1.2578 (1.0)        1345           1
test_litgpt_sdpa[Llama-2-7b-hf-forward-bs2-thunder] (0002_7d62ae1)     756.0453 (1.01)     2,652.6223 (2.16)     796.6722 (1.00)     91.6864 (1.14)     775.3037 (1.00)     16.0692 (1.0)        71;137        1.2552 (1.00)       1341           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-7b-hf-inference-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                             Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-7b-hf-inference-bs1-thunder] (0002_7d62ae1)     386.2884 (1.0)      616.4701 (1.0)      403.7395 (1.0)      35.7568 (1.0)      394.4829 (1.0)      6.4295 (1.0)        67;132        2.4768 (1.0)        1297           2
test_litgpt_sdpa[Llama-2-7b-hf-inference-bs1-thunder] (0003_af807f9)     386.6360 (1.00)     959.7170 (1.56)     404.7183 (1.00)     38.0444 (1.06)     395.1873 (1.00)     6.7521 (1.05)       65;131        2.4709 (1.00)       1292           2
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-2-7b-hf-inference-bs2-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                                             Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-2-7b-hf-inference-bs2-thunder] (0002_7d62ae1)     726.1587 (1.0)      1,153.9040 (1.0)      760.8553 (1.0)      69.5719 (1.0)      742.7672 (1.0)      12.8904 (1.0)        71;139        1.3143 (1.0)        1386           1
test_litgpt_sdpa[Llama-2-7b-hf-inference-bs2-thunder] (0003_af807f9)     726.3003 (1.00)     1,160.8726 (1.01)     766.4930 (1.01)     71.2820 (1.02)     746.4320 (1.00)     15.3859 (1.19)       78;174        1.3046 (0.99)       1386           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-3-70B-backward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                        Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-70B-backward-bs1-thunder] (0002_7d62ae1)     5.8379 (1.0)      6.7056 (1.00)     6.1370 (1.00)     0.2096 (1.09)     6.1084 (1.0)      0.3621 (1.0)          60;0  162.9453 (1.00)        172           1
test_litgpt_sdpa[Llama-3-70B-backward-bs1-thunder] (0003_af807f9)     5.9037 (1.01)     6.6988 (1.0)      6.1268 (1.0)      0.1922 (1.0)      6.1280 (1.00)     0.3732 (1.03)         82;0  163.2175 (1.0)         172           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_litgpt_sdpa[Llama-3-70B-backward-bs2-thunder]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                                         Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-70B-backward-bs2-thunder] (0003_af807f9)     11.5088 (1.0)      13.2036 (1.0)      12.1713 (1.0)      0.4187 (1.0)      12.0606 (1.0)      0.6521 (1.03)         30;0  82.1608 (1.0)          88           1
test_litgpt_sdpa[Llama-3-70B-backward-bs2-thunder] (0002_7d62ae1)     11.5375 (1.00)     13.3934 (1.01)     12.1953 (1.00)     0.4456 (1.06)     12.1097 (1.00)     0.6339 (1.0)          27;0  81.9989 (1.00)         87           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-3-70B-forward-bs1-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                       Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-70B-forward-bs1-thunder] (0002_7d62ae1)     2.0951 (1.0)      2.6860 (1.0)      2.1835 (1.0)      0.0895 (1.0)      2.1583 (1.00)     0.0778 (1.12)        32;25  457.9856 (1.0)         492           1
test_litgpt_sdpa[Llama-3-70B-forward-bs1-thunder] (0003_af807f9)     2.0963 (1.00)     2.7096 (1.01)     2.1840 (1.00)     0.0900 (1.01)     2.1515 (1.0)      0.0696 (1.0)         29;27  457.8770 (1.00)        492           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-3-70B-forward-bs2-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                       Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-70B-forward-bs2-thunder] (0002_7d62ae1)     3.9442 (1.0)      4.5874 (1.0)      4.1473 (1.0)      0.1255 (1.01)     4.1246 (1.0)      0.1658 (1.01)         77;6  241.1190 (1.0)         267           1
test_litgpt_sdpa[Llama-3-70B-forward-bs2-thunder] (0003_af807f9)     3.9446 (1.00)     4.6420 (1.01)     4.1528 (1.00)     0.1242 (1.0)      4.1333 (1.00)     0.1642 (1.0)          77;7  240.8024 (1.00)        266           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-3-70B-inference-bs1-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                         Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-70B-inference-bs1-thunder] (0002_7d62ae1)     2.0874 (1.0)      2.6314 (1.0)      2.1788 (1.00)     0.0885 (1.02)     2.1545 (1.00)     0.0805 (1.19)        36;25  458.9663 (1.00)        496           1
test_litgpt_sdpa[Llama-3-70B-inference-bs1-thunder] (0003_af807f9)     2.0927 (1.00)     2.6575 (1.01)     2.1729 (1.0)      0.0867 (1.0)      2.1447 (1.0)      0.0675 (1.0)         30;26  460.2050 (1.0)         495           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-3-70B-inference-bs2-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                         Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-70B-inference-bs2-thunder] (0003_af807f9)     3.8403 (1.0)      4.6658 (1.02)     4.1167 (1.0)      0.1433 (1.44)     4.0879 (1.0)      0.2091 (1.88)         99;3  242.9114 (1.0)         269           1
test_litgpt_sdpa[Llama-3-70B-inference-bs2-thunder] (0002_7d62ae1)     3.9973 (1.04)     4.5966 (1.0)      4.1361 (1.00)     0.0998 (1.0)      4.1354 (1.01)     0.1114 (1.0)         59;13  241.7712 (1.00)        267           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-3-8B-backward-bs1-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                       Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-8B-backward-bs1-thunder] (0003_af807f9)     3.1718 (1.0)      3.5221 (1.0)      3.2188 (1.0)      0.0638 (1.0)      3.1964 (1.0)      0.0322 (1.15)        25;40  310.6762 (1.0)         321           1
test_litgpt_sdpa[Llama-3-8B-backward-bs1-thunder] (0002_7d62ae1)     3.1868 (1.00)     3.5845 (1.02)     3.2314 (1.00)     0.0702 (1.10)     3.2083 (1.00)     0.0280 (1.0)         21;28  309.4594 (1.00)        318           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-3-8B-backward-bs2-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                                       Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-8B-backward-bs2-thunder] (0003_af807f9)     5.8634 (1.0)      6.5238 (1.0)      6.1390 (1.0)      0.1565 (1.0)      6.1329 (1.00)     0.2692 (1.03)         66;0  162.8923 (1.0)         172           1
test_litgpt_sdpa[Llama-3-8B-backward-bs2-thunder] (0002_7d62ae1)     5.8687 (1.00)     6.8505 (1.05)     6.1495 (1.00)     0.1874 (1.20)     6.1241 (1.0)      0.2616 (1.0)          64;1  162.6159 (1.00)        172           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-3-8B-forward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                      Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-8B-forward-bs1-thunder] (0003_af807f9)     1.1782 (1.0)      1.6252 (1.0)      1.2189 (1.0)      0.0742 (1.0)      1.1999 (1.0)      0.0143 (1.0)         45;84  820.3978 (1.0)         853           1
test_litgpt_sdpa[Llama-3-8B-forward-bs1-thunder] (0002_7d62ae1)     1.1797 (1.00)     1.7074 (1.05)     1.2265 (1.01)     0.0810 (1.09)     1.2007 (1.00)     0.0216 (1.51)       51;131  815.3237 (0.99)        851           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-3-8B-forward-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                      Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-8B-forward-bs2-thunder] (0003_af807f9)     2.0850 (1.0)      2.5439 (1.0)      2.1822 (1.0)      0.0804 (1.0)      2.1620 (1.0)      0.0666 (1.0)         41;28  458.2538 (1.0)         492           1
test_litgpt_sdpa[Llama-3-8B-forward-bs2-thunder] (0002_7d62ae1)     2.0908 (1.00)     2.6358 (1.04)     2.1937 (1.01)     0.0860 (1.07)     2.1914 (1.01)     0.0822 (1.23)        42;25  455.8464 (0.99)        491           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-3-8B-inference-bs1-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                        Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-8B-inference-bs1-thunder] (0003_af807f9)     1.1615 (1.0)      1.6609 (1.01)     1.2038 (1.0)      0.0767 (1.0)      1.1837 (1.0)      0.0141 (1.10)       45;104  830.6990 (1.0)         861           1
test_litgpt_sdpa[Llama-3-8B-inference-bs1-thunder] (0002_7d62ae1)     1.1646 (1.00)     1.6452 (1.0)      1.2043 (1.00)     0.0785 (1.02)     1.1840 (1.00)     0.0128 (1.0)         44;85  830.3618 (1.00)        865           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Llama-3-8B-inference-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                        Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Llama-3-8B-inference-bs2-thunder] (0002_7d62ae1)     2.0784 (1.0)      2.6221 (1.00)     2.1736 (1.00)     0.0857 (1.07)     2.1515 (1.00)     0.0704 (1.12)        50;25  460.0721 (1.00)        496           1
test_litgpt_sdpa[Llama-3-8B-inference-bs2-thunder] (0003_af807f9)     2.0797 (1.00)     2.6139 (1.0)      2.1716 (1.0)      0.0798 (1.0)      2.1478 (1.0)      0.0626 (1.0)         42;25  460.4893 (1.0)         497           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Mistral-7B-v0.1-backward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                            Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Mistral-7B-v0.1-backward-bs1-thunder] (0003_af807f9)     1.1187 (1.0)      1.5379 (1.0)      1.1550 (1.0)      0.0615 (1.0)      1.1379 (1.0)      0.0145 (1.14)        64;75  865.8172 (1.0)         893           1
test_litgpt_sdpa[Mistral-7B-v0.1-backward-bs1-thunder] (0002_7d62ae1)     1.1293 (1.01)     1.7336 (1.13)     1.1619 (1.01)     0.0621 (1.01)     1.1465 (1.01)     0.0127 (1.0)         57;61  860.6931 (0.99)        886           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Mistral-7B-v0.1-backward-bs2-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                                            Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Mistral-7B-v0.1-backward-bs2-thunder] (0002_7d62ae1)     1.8653 (1.0)      2.1913 (1.0)      1.8980 (1.0)      0.0605 (1.0)      1.8834 (1.0)      0.0124 (1.0)         35;36  526.8625 (1.0)         539           1
test_litgpt_sdpa[Mistral-7B-v0.1-backward-bs2-thunder] (0003_af807f9)     1.8659 (1.00)     2.3277 (1.06)     1.9221 (1.01)     0.0718 (1.19)     1.8962 (1.01)     0.0397 (3.20)        51;52  520.2634 (0.99)        542           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Mistral-7B-v0.1-forward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                             Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Mistral-7B-v0.1-forward-bs1-thunder] (0002_7d62ae1)     391.1126 (1.0)      627.7347 (1.0)      410.3859 (1.0)      35.6988 (1.0)      401.0565 (1.0)      7.0345 (1.0)        63;127        2.4367 (1.0)        1278           2
test_litgpt_sdpa[Mistral-7B-v0.1-forward-bs1-thunder] (0003_af807f9)     394.0261 (1.01)     636.1902 (1.01)     413.3117 (1.01)     36.2730 (1.02)     404.1928 (1.01)     7.4009 (1.05)       63;123        2.4195 (0.99)       1269           2
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Mistral-7B-v0.1-forward-bs2-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                                             Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Mistral-7B-v0.1-forward-bs2-thunder] (0002_7d62ae1)     742.3330 (1.0)      1,176.4467 (1.0)      781.4618 (1.0)      71.1537 (1.0)      762.4086 (1.0)      15.3910 (1.0)        73;143        1.2797 (1.0)        1356           1
test_litgpt_sdpa[Mistral-7B-v0.1-forward-bs2-thunder] (0003_af807f9)     744.8113 (1.00)     1,245.7203 (1.06)     788.8545 (1.01)     73.7364 (1.04)     769.3600 (1.01)     15.4523 (1.00)       72;143        1.2677 (0.99)       1357           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Mistral-7B-v0.1-inference-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                               Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Mistral-7B-v0.1-inference-bs1-thunder] (0003_af807f9)     387.6337 (1.0)      619.1484 (1.0)      405.8014 (1.00)     34.4214 (1.0)      396.7521 (1.00)     6.8070 (1.01)       62;128        2.4643 (1.00)       1290           2
test_litgpt_sdpa[Mistral-7B-v0.1-inference-bs1-thunder] (0002_7d62ae1)     388.3103 (1.00)     624.7417 (1.01)     405.6102 (1.0)      35.1960 (1.02)     396.4835 (1.0)      6.7082 (1.0)        66;128        2.4654 (1.0)        1288           2
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[Mistral-7B-v0.1-inference-bs2-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                                               Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[Mistral-7B-v0.1-inference-bs2-thunder] (0002_7d62ae1)     728.6654 (1.0)      1,154.5601 (1.0)      765.3275 (1.00)     69.4871 (1.0)      747.3780 (1.00)     12.9414 (1.0)        70;137        1.3066 (1.00)       1385           1
test_litgpt_sdpa[Mistral-7B-v0.1-inference-bs2-thunder] (0003_af807f9)     729.2670 (1.00)     1,163.0999 (1.01)     764.8012 (1.0)      70.2739 (1.01)     746.1195 (1.0)      13.8627 (1.07)       70;137        1.3075 (1.0)        1374           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[phi-2-backward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                    Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[phi-2-backward-bs1-thunder] (0003_af807f9)     403.7034 (1.0)      688.2851 (1.14)     422.8839 (1.0)      32.6433 (1.14)     414.6987 (1.0)      8.5860 (1.0)         75;90        2.3647 (1.0)        1240           2
test_litgpt_sdpa[phi-2-backward-bs1-thunder] (0002_7d62ae1)     404.6785 (1.00)     604.3180 (1.0)      424.2038 (1.00)     28.6210 (1.0)      417.2465 (1.01)     9.3944 (1.09)        61;68        2.3574 (1.00)       1233           2
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_sdpa[phi-2-backward-bs2-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                                    Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[phi-2-backward-bs2-thunder] (0003_af807f9)     737.2461 (1.0)      1,267.1743 (1.0)      776.5200 (1.0)      68.2146 (1.0)      758.2656 (1.0)      14.0037 (1.0)        93;117        1.2878 (1.0)        1359           1
test_litgpt_sdpa[phi-2-backward-bs2-thunder] (0002_7d62ae1)     743.0026 (1.01)     1,309.2570 (1.03)     789.2596 (1.02)     71.6335 (1.05)     767.3190 (1.01)     17.4753 (1.25)       91;200        1.2670 (0.98)       1347           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_sdpa[phi-2-forward-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                   Min                 Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[phi-2-forward-bs1-thunder] (0002_7d62ae1)     247.2997 (1.0)      449.1131 (1.0)      268.8610 (1.0)      32.5021 (1.0)      259.4520 (1.0)      11.1421 (1.05)       84;147        3.7194 (1.0)        1369           3
test_litgpt_sdpa[phi-2-forward-bs1-thunder] (0003_af807f9)     273.5443 (1.11)     581.2070 (1.29)     300.2961 (1.12)     42.0052 (1.29)     288.8890 (1.11)     10.6218 (1.0)       105;192        3.3300 (0.90)       1823           2
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_sdpa[phi-2-forward-bs2-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                   Min                 Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[phi-2-forward-bs2-thunder] (0003_af807f9)     303.4091 (1.0)      613.2890 (1.01)     330.8793 (1.0)      43.2174 (1.04)     319.6201 (1.0)      10.7515 (1.0)        86;163        3.0222 (1.0)        1651           2
test_litgpt_sdpa[phi-2-forward-bs2-thunder] (0002_7d62ae1)     305.9146 (1.01)     608.2444 (1.0)      334.4957 (1.01)     41.6714 (1.0)      323.7431 (1.01)     10.7656 (1.00)       86;162        2.9896 (0.99)       1621           2
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_litgpt_sdpa[phi-2-inference-bs1-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                     Min                 Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[phi-2-inference-bs1-thunder] (0003_af807f9)     219.7370 (1.0)      434.2711 (1.09)     242.1392 (1.01)     29.0945 (1.07)     232.5710 (1.00)     10.9000 (1.52)      113;186        4.1299 (0.99)       1500           3
test_litgpt_sdpa[phi-2-inference-bs1-thunder] (0002_7d62ae1)     222.4781 (1.01)     399.8874 (1.0)      239.7175 (1.0)      27.1006 (1.0)      231.9521 (1.0)       7.1518 (1.0)        87;151        4.1716 (1.0)        1493           3
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_litgpt_sdpa[phi-2-inference-bs2-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                     Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_litgpt_sdpa[phi-2-inference-bs2-thunder] (0002_7d62ae1)     276.3611 (1.0)      522.6703 (1.0)      299.2186 (1.0)      37.5992 (1.0)      289.2773 (1.0)      8.7139 (1.0)       101;180        3.3420 (1.0)        1808           2
test_litgpt_sdpa[phi-2-inference-bs2-thunder] (0003_af807f9)     277.8745 (1.01)     615.3460 (1.18)     300.6327 (1.00)     41.1332 (1.09)     289.7740 (1.00)     8.7245 (1.00)       92;185        3.3263 (1.00)       1813           2
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_llama2_7b_rmsnorm_grad[backward-thunder]': 2 tests ----------------------------------------------------------------------------------
Name (time in us)                                                     Min                 Max                Mean            StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_7b_rmsnorm_grad[backward-thunder] (0003_af807f9)     104.1109 (1.0)      156.9320 (1.0)      111.8684 (1.0)      6.6203 (1.0)      110.3793 (1.0)      1.4776 (1.0)        68;115        8.9391 (1.0)         963          10
test_llama2_7b_rmsnorm_grad[backward-thunder] (0002_7d62ae1)     105.7466 (1.02)     217.1471 (1.38)     113.5998 (1.02)     7.0634 (1.07)     112.7732 (1.02)     1.5025 (1.02)       56;183        8.8028 (0.98)        944          10
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_llama2_7b_rmsnorm_grad[forward-thunder]': 2 tests ----------------------------------------------------------------------------------
Name (time in us)                                                   Min                 Max               Mean             StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_7b_rmsnorm_grad[forward-thunder] (0002_7d62ae1)     85.6484 (1.0)      249.5101 (1.0)      91.8812 (1.0)      11.3500 (1.0)      88.7657 (1.0)      1.5098 (1.0)        93;147       10.8836 (1.0)        1162          10
test_llama2_7b_rmsnorm_grad[forward-thunder] (0003_af807f9)     86.8296 (1.01)     255.1879 (1.02)     93.3011 (1.02)     12.1471 (1.07)     89.6067 (1.01)     1.7303 (1.15)      100;178       10.7180 (0.98)       1139          10
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_llama2_7b_rmsnorm_grad[inference-thunder]': 2 tests ---------------------------------------------------------------------------------
Name (time in us)                                                     Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_7b_rmsnorm_grad[inference-thunder] (0002_7d62ae1)     70.5724 (1.0)      167.6464 (1.0)      75.4802 (1.0)      9.7523 (1.0)      72.7686 (1.0)      1.4101 (1.0)       103;168       13.2485 (1.0)        1423          10
test_llama2_7b_rmsnorm_grad[inference-thunder] (0003_af807f9)     72.0670 (1.02)     219.7263 (1.31)     76.9016 (1.02)     9.9957 (1.02)     74.1118 (1.02)     1.4535 (1.03)      115;168       13.0036 (0.98)       1401          10
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_llama2_7b_sdpa[backward-thunder]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                            Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_7b_sdpa[backward-thunder] (0003_af807f9)     12.3372 (1.0)      14.0127 (1.00)     13.1034 (1.0)      0.4010 (1.0)      13.0874 (1.01)     0.5787 (1.0)          27;0  76.3163 (1.0)          82           1
test_llama2_7b_sdpa[backward-thunder] (0002_7d62ae1)     12.6262 (1.02)     14.0118 (1.0)      13.1696 (1.01)     0.4154 (1.04)     13.0097 (1.0)      0.6538 (1.13)         35;0  75.9325 (0.99)         82           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_llama2_7b_sdpa[forward-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                          Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_7b_sdpa[forward-thunder] (0002_7d62ae1)     4.0955 (1.0)      4.9473 (1.0)      4.3440 (1.0)      0.1744 (1.0)      4.3114 (1.00)     0.2480 (1.0)          96;2  230.2048 (1.0)         257           1
test_llama2_7b_sdpa[forward-thunder] (0003_af807f9)     4.1297 (1.01)     5.7219 (1.16)     4.3617 (1.00)     0.1952 (1.12)     4.2915 (1.0)      0.2551 (1.03)         60;4  229.2681 (1.00)        257           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_llama2_7b_sdpa[inference-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                            Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_7b_sdpa[inference-thunder] (0002_7d62ae1)     4.0843 (1.0)      4.9366 (1.01)     4.3250 (1.0)      0.1712 (1.15)     4.2849 (1.0)      0.2258 (1.0)          79;3  231.2150 (1.0)         258           1
test_llama2_7b_sdpa[inference-thunder] (0003_af807f9)     4.1113 (1.01)     4.8773 (1.0)      4.3274 (1.00)     0.1488 (1.0)      4.3037 (1.00)     0.2502 (1.11)         89;1  231.0856 (1.00)        259           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------- benchmark 'test_llama2_causal_self_attention_7b[backward-thunder]': 1 tests -------------------------------------------
Name (time in ms)                                                             Min      Max     Mean  StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_causal_self_attention_7b[backward-thunder] (0002_7d62ae1)     44.1746  46.4387  44.9844  0.6344  44.8860  0.7882       8;0  22.2299      24           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------- benchmark 'test_llama2_causal_self_attention_7b[forward-thunder]': 1 tests -------------------------------------------
Name (time in ms)                                                            Min      Max     Mean  StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_causal_self_attention_7b[forward-thunder] (0002_7d62ae1)     20.1249  23.5894  21.3439  0.7482  21.2663  0.9183      15;2  46.8518      52           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------- benchmark 'test_llama2_causal_self_attention_7b[inference-thunder]': 1 tests -------------------------------------------
Name (time in ms)                                                              Min      Max     Mean  StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_causal_self_attention_7b[inference-thunder] (0002_7d62ae1)     20.3437  23.1714  21.6558  0.7925  21.5324  1.2122      19;0  46.1771      51           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------- benchmark 'test_llama2_mlp_7b[backward-thunder]': 1 tests -------------------------------------------
Name (time in ms)                                           Min      Max     Mean  StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_mlp_7b[backward-thunder] (0002_7d62ae1)     53.4709  60.6208  55.8100  2.4141  54.7918  3.3090       4;0  17.9179      20           1
-------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------- benchmark 'test_llama2_mlp_7b[forward-thunder]': 1 tests -------------------------------------------
Name (time in ms)                                          Min      Max     Mean  StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_mlp_7b[forward-thunder] (0002_7d62ae1)     26.6723  30.3003  27.9740  1.1069  27.4585  1.5764       9;0  35.7475      41           1
------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------- benchmark 'test_llama2_mlp_7b[inference-thunder]': 1 tests -------------------------------------------
Name (time in ms)                                            Min      Max     Mean  StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------
test_llama2_mlp_7b[inference-thunder] (0002_7d62ae1)     26.1804  30.8140  27.9133  1.4435  27.5377  2.4451      16;0  35.8252      41           1
--------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------- benchmark 'test_llama_2_7b_hf[forward-thunder]': 1 tests --------------------------------------------
Name (time in ms)                                           Min       Max      Mean  StdDev    Median     IQR  Outliers     OPS  Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------
test_llama_2_7b_hf[forward-thunder] (0003_af807f9)     246.1541  252.7756  249.2240  2.4668  249.6076  3.0707       2;0  4.0125       5           1
---------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------- benchmark 'test_nanogpt_block[backward-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                          Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_block[backward-thunder] (0002_7d62ae1)     1.1298 (1.0)      1.8638 (1.0)      1.2150 (1.0)      0.1039 (1.04)     1.1785 (1.0)      0.0327 (1.49)      101;147  823.0602 (1.0)         887           1
test_nanogpt_block[backward-thunder] (0003_af807f9)     1.1776 (1.04)     1.8980 (1.02)     1.2455 (1.03)     0.1003 (1.0)      1.2175 (1.03)     0.0219 (1.0)         76;79  802.8757 (0.98)        847           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_nanogpt_block[forward-thunder]': 2 tests ----------------------------------------------------------------------------
Name (time in ms)                                         Min               Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_block[forward-thunder] (0002_7d62ae1)     1.1305 (1.0)      1.6516 (1.0)      1.1843 (1.0)      0.0801 (1.0)      1.1627 (1.0)      0.0203 (1.0)         57;79  844.4044 (1.0)         884           1
test_nanogpt_block[forward-thunder] (0003_af807f9)     1.1523 (1.02)     1.6958 (1.03)     1.2150 (1.03)     0.0931 (1.16)     1.1892 (1.02)     0.0203 (1.00)        66;79  823.0244 (0.97)        873           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------- benchmark 'test_nanogpt_block[inference-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                             Min                   Max                  Mean              StdDev              Median                IQR            Outliers         OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_block[inference-thunder] (0002_7d62ae1)     884.0971 (1.0)      1,727.0413 (1.23)       946.3297 (1.0)      102.8528 (1.0)      913.9495 (1.0)      20.8449 (1.03)      108;115  1,056.7142 (1.0)        1134           1
test_nanogpt_block[inference-thunder] (0003_af807f9)     940.8100 (1.06)     1,398.8279 (1.0)      1,010.8412 (1.07)     106.2288 (1.03)     976.5713 (1.07)     20.2400 (1.0)       102;106    989.2751 (0.94)       1062           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_nanogpt_cross_entropy[backward-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                                    Min                   Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_cross_entropy[backward-thunder] (0003_af807f9)     720.9498 (1.0)      1,011.0531 (1.04)     742.5114 (1.0)      53.6764 (1.02)     728.0395 (1.0)      4.9120 (1.0)         87;90        1.3468 (1.0)        1384           1
test_nanogpt_cross_entropy[backward-thunder] (0002_7d62ae1)     725.8193 (1.01)       968.6360 (1.0)      748.2257 (1.01)     52.8014 (1.0)      733.7234 (1.01)     5.5737 (1.13)       86;107        1.3365 (0.99)       1380           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_nanogpt_cross_entropy[forward-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                   Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_cross_entropy[forward-thunder] (0002_7d62ae1)     316.6450 (1.0)      547.5925 (1.00)     330.4315 (1.0)      33.8816 (1.0)      321.8970 (1.0)      3.2916 (1.0)        77;164        3.0263 (1.0)        1587           2
test_nanogpt_cross_entropy[forward-thunder] (0003_af807f9)     318.2003 (1.00)     544.8866 (1.0)      332.2962 (1.01)     34.1028 (1.01)     323.7046 (1.01)     3.8479 (1.17)       77;164        3.0094 (0.99)       1576           2
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_nanogpt_cross_entropy[inference-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                     Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_cross_entropy[inference-thunder] (0003_af807f9)     252.1134 (1.0)      388.9116 (1.0)      261.5832 (1.0)      19.1361 (1.0)      257.0463 (1.0)      2.4037 (1.01)       70;121        3.8229 (1.0)        1323           3
test_nanogpt_cross_entropy[inference-thunder] (0002_7d62ae1)     254.3883 (1.01)     392.9823 (1.01)     263.3677 (1.01)     19.9708 (1.04)     258.6385 (1.01)     2.3803 (1.0)        70;120        3.7970 (0.99)       1314           3
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ benchmark 'test_nanogpt_csa[backward-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                          Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_csa[backward-thunder] (0002_7d62ae1)     785.2069 (1.0)      1,190.6656 (1.0)      836.3728 (1.0)      58.4677 (1.0)      822.5092 (1.0)      19.6176 (1.06)        80;85        1.1956 (1.0)        1272           1
test_nanogpt_csa[backward-thunder] (0003_af807f9)     833.3679 (1.06)     1,192.2210 (1.00)     880.7743 (1.05)     60.0314 (1.03)     866.6501 (1.05)     18.4913 (1.0)         75;79        1.1354 (0.95)       1199           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------- benchmark 'test_nanogpt_csa[forward-thunder]': 2 tests -------------------------------------------------------------------------------------
Name (time in us)                                         Min                   Max                Mean              StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_csa[forward-thunder] (0002_7d62ae1)     809.0981 (1.0)      1,559.7697 (1.04)     882.9235 (1.0)      107.2857 (1.06)     841.4930 (1.0)      23.4980 (1.18)      151;233        1.1326 (1.0)        1232           1
test_nanogpt_csa[forward-thunder] (0003_af807f9)     862.1374 (1.07)     1,493.9131 (1.0)      928.5902 (1.05)     100.8505 (1.0)      893.9039 (1.06)     19.8485 (1.0)       126;162        1.0769 (0.95)       1161           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------- benchmark 'test_nanogpt_csa[inference-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                           Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_csa[inference-thunder] (0002_7d62ae1)     653.0392 (1.0)      1,415.2522 (1.23)     720.6831 (1.0)      99.9357 (1.01)     689.5480 (1.0)      18.1963 (1.08)      148;155        1.3876 (1.0)        1531           1
test_nanogpt_csa[inference-thunder] (0003_af807f9)     712.1842 (1.09)     1,150.4064 (1.0)      773.1471 (1.07)     99.0819 (1.0)      742.2704 (1.08)     16.9072 (1.0)       133;139        1.2934 (0.93)       1402           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_nanogpt_gelu[backward-thunder]': 2 tests ---------------------------------------------------------------------------------
Name (time in us)                                          Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_gelu[backward-thunder] (0003_af807f9)     88.0430 (1.0)      279.8670 (1.49)     95.6577 (1.0)      7.8540 (1.27)     93.9611 (1.0)      2.6505 (2.29)        60;63       10.4539 (1.0)        1168          10
test_nanogpt_gelu[backward-thunder] (0002_7d62ae1)     89.3592 (1.01)     187.2511 (1.0)      96.6674 (1.01)     6.1921 (1.0)      95.3325 (1.01)     1.1560 (1.0)         70;94       10.3448 (0.99)       1109          10
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_nanogpt_gelu[forward-thunder]': 2 tests ---------------------------------------------------------------------------------
Name (time in us)                                         Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_gelu[forward-thunder] (0003_af807f9)     63.5417 (1.0)      169.3531 (1.26)     67.4907 (1.0)      7.8576 (1.0)      65.4330 (1.0)      1.1527 (1.0)       102;185       14.8169 (1.0)        1570          10
test_nanogpt_gelu[forward-thunder] (0002_7d62ae1)     63.5945 (1.00)     134.3798 (1.0)      68.3616 (1.01)     8.0329 (1.02)     65.5381 (1.00)     1.4754 (1.28)      144;283       14.6281 (0.99)       1575          10
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_nanogpt_gelu[inference-thunder]': 2 tests ---------------------------------------------------------------------------------
Name (time in us)                                           Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_gelu[inference-thunder] (0002_7d62ae1)     51.5348 (1.0)      212.0233 (1.0)      55.6829 (1.00)     8.0728 (1.01)     53.7632 (1.0)      1.1588 (1.03)      120;170       17.9588 (1.00)       1769          11
test_nanogpt_gelu[inference-thunder] (0003_af807f9)     51.5385 (1.00)     234.1646 (1.10)     55.6079 (1.0)      7.9671 (1.0)      53.8257 (1.00)     1.1250 (1.0)       112;162       17.9830 (1.0)        1748          11
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_nanogpt_gpt2[backward-thunder]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                          Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_gpt2[backward-thunder] (0002_7d62ae1)     11.2561 (1.0)      12.5892 (1.01)     11.6748 (1.0)      0.2504 (1.08)     11.6798 (1.0)      0.2171 (1.0)          28;9  85.6549 (1.0)          90           1
test_nanogpt_gpt2[backward-thunder] (0003_af807f9)     11.5282 (1.02)     12.5232 (1.0)      11.9239 (1.02)     0.2328 (1.0)      11.8392 (1.01)     0.3591 (1.65)         22;0  83.8655 (0.98)         86           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_nanogpt_gpt2[forward-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                        Min                Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_gpt2[forward-thunder] (0002_7d62ae1)     9.4473 (1.0)      11.8571 (1.14)     9.7484 (1.0)      0.4554 (3.52)     9.5758 (1.0)      0.1060 (1.49)        10;22  102.5810 (1.0)         106           1
test_nanogpt_gpt2[forward-thunder] (0003_af807f9)     9.7710 (1.03)     10.3589 (1.0)      9.9151 (1.02)     0.1295 (1.0)      9.8789 (1.03)     0.0711 (1.0)         12;10  100.8566 (0.98)        103           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------- benchmark 'test_nanogpt_gpt2[inference-thunder]': 2 tests -----------------------------------------------------------------------------
Name (time in ms)                                          Min                Max              Mean            StdDev            Median               IQR            Outliers       OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_gpt2[inference-thunder] (0002_7d62ae1)     7.4251 (1.0)       8.1598 (1.0)      7.5928 (1.0)      0.1106 (1.0)      7.5670 (1.0)      0.0793 (1.0)         24;16  131.7035 (1.0)         135           1
test_nanogpt_gpt2[inference-thunder] (0003_af807f9)     8.2603 (1.11)     10.3238 (1.27)     8.4339 (1.11)     0.2729 (2.47)     8.3719 (1.11)     0.0911 (1.15)         9;16  118.5697 (0.90)        122           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_nanogpt_gpt2xl[backward-thunder]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                            Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_gpt2xl[backward-thunder] (0002_7d62ae1)     43.7388 (1.0)      44.8607 (1.0)      44.2504 (1.0)      0.2769 (1.0)      44.2067 (1.0)      0.3822 (1.56)          5;0  22.5987 (1.0)          23           1
test_nanogpt_gpt2xl[backward-thunder] (0003_af807f9)     45.7487 (1.05)     46.9857 (1.05)     46.1097 (1.04)     0.2843 (1.03)     46.0333 (1.04)     0.2448 (1.0)           5;1  21.6874 (0.96)         22           1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_nanogpt_gpt2xl[forward-thunder]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                           Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_gpt2xl[forward-thunder] (0002_7d62ae1)     35.2449 (1.0)      35.8504 (1.0)      35.5559 (1.0)      0.1542 (1.0)      35.5383 (1.0)      0.2412 (2.35)         10;0  28.1247 (1.0)          29           1
test_nanogpt_gpt2xl[forward-thunder] (0003_af807f9)     36.4080 (1.03)     38.9975 (1.09)     36.7238 (1.03)     0.6083 (3.94)     36.5466 (1.03)     0.1024 (1.0)           3;3  27.2303 (0.97)         28           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'test_nanogpt_gpt2xl[inference-thunder]': 2 tests ------------------------------------------------------------------------------
Name (time in ms)                                             Min                Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_gpt2xl[inference-thunder] (0002_7d62ae1)     27.7313 (1.0)      28.7375 (1.0)      28.1935 (1.0)      0.1931 (1.28)     28.1923 (1.0)      0.2355 (1.57)          9;1  35.4692 (1.0)          36           1
test_nanogpt_gpt2xl[inference-thunder] (0003_af807f9)     30.4985 (1.10)     31.2199 (1.09)     30.7468 (1.09)     0.1512 (1.0)      30.7381 (1.09)     0.1496 (1.0)           8;2  32.5238 (0.92)         33           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_nanogpt_layer_norm[backward-thunder]': 2 tests ----------------------------------------------------------------------------------
Name (time in us)                                                 Min                 Max                Mean            StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_layer_norm[backward-thunder] (0003_af807f9)     126.0244 (1.0)      170.7214 (1.0)      134.6516 (1.0)      5.9180 (1.0)      134.3998 (1.0)      1.7478 (1.0)       112;180        7.4266 (1.0)         802          10
test_nanogpt_layer_norm[backward-thunder] (0002_7d62ae1)     129.0852 (1.02)     201.3610 (1.18)     138.4870 (1.03)     7.1475 (1.21)     138.1459 (1.03)     3.4953 (2.00)       111;65        7.2209 (0.97)        773          10
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------- benchmark 'test_nanogpt_layer_norm[forward-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                                Min                 Max                Mean            StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_layer_norm[forward-thunder] (0003_af807f9)      94.5124 (1.0)      241.2692 (1.39)      99.2600 (1.0)      8.9889 (1.03)      97.0111 (1.0)      1.6148 (1.0)        63;123       10.0745 (1.0)        1060          10
test_nanogpt_layer_norm[forward-thunder] (0002_7d62ae1)     105.7446 (1.12)     173.4158 (1.0)      111.3754 (1.12)     8.7182 (1.0)      108.8443 (1.12)     1.9339 (1.20)       60;111        8.9786 (0.89)        945          10
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'test_nanogpt_layer_norm[inference-thunder]': 2 tests ---------------------------------------------------------------------------------
Name (time in us)                                                 Min                 Max               Mean            StdDev             Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_layer_norm[inference-thunder] (0002_7d62ae1)     70.0389 (1.0)      159.5998 (1.04)     75.1544 (1.0)      8.6986 (1.03)     72.1831 (1.0)      1.3727 (1.03)      155;233       13.3059 (1.0)        1426          10
test_nanogpt_layer_norm[inference-thunder] (0003_af807f9)     70.8635 (1.01)     152.8586 (1.0)      75.7162 (1.01)     8.4510 (1.0)      73.1799 (1.01)     1.3313 (1.0)       115;195       13.2072 (0.99)       1409          10
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_nanogpt_mlp[backward-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                          Min                 Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_mlp[backward-thunder] (0002_7d62ae1)     546.5490 (1.0)      887.2091 (1.02)     577.1896 (1.0)      59.1704 (1.08)     561.5931 (1.0)      10.5480 (1.25)      118;143        1.7325 (1.0)        1837           1
test_nanogpt_mlp[backward-thunder] (0003_af807f9)     551.0352 (1.01)     871.0981 (1.0)      579.6110 (1.00)     54.7911 (1.0)      565.4532 (1.01)      8.4143 (1.0)       115;133        1.7253 (1.00)       1832           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------- benchmark 'test_nanogpt_mlp[forward-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                         Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_mlp[forward-thunder] (0002_7d62ae1)     323.6760 (1.0)      1,042.7639 (1.70)     359.8038 (1.0)      44.7757 (1.08)     348.8951 (1.0)      14.5098 (1.00)       77;140        2.7793 (1.0)        1513           2
test_nanogpt_mlp[forward-thunder] (0003_af807f9)     336.2682 (1.04)       611.8745 (1.0)      364.8052 (1.01)     41.3788 (1.0)      354.2870 (1.02)     14.4740 (1.0)        75;129        2.7412 (0.99)       1489           2
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_nanogpt_mlp[inference-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                           Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_mlp[inference-thunder] (0002_7d62ae1)     206.4761 (1.0)      601.1034 (1.43)     219.4239 (1.00)     30.6747 (1.06)     210.9169 (1.00)     3.2547 (1.0)        90;196        4.5574 (1.00)       1607           3
test_nanogpt_mlp[inference-thunder] (0003_af807f9)     206.7247 (1.00)     420.3147 (1.0)      219.0873 (1.0)      28.8350 (1.0)      210.7560 (1.0)      3.8634 (1.19)       94;184        4.5644 (1.0)        1613           3
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------- benchmark 'test_nanogpt_sdpa[backward-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                           Min                   Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_sdpa[backward-thunder] (0003_af807f9)     300.7832 (1.0)        551.0279 (1.0)      325.7900 (1.0)      25.9729 (1.0)      320.1980 (1.0)      12.8177 (1.0)         79;79        3.0695 (1.0)        1647           2
test_nanogpt_sdpa[backward-thunder] (0002_7d62ae1)     313.1281 (1.04)     1,833.9385 (3.33)     339.6545 (1.04)     46.7295 (1.80)     333.0905 (1.04)     16.0111 (1.25)        81;83        2.9442 (0.96)       1590           2
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_nanogpt_sdpa[forward-thunder]': 2 tests ------------------------------------------------------------------------------------
Name (time in us)                                          Min                 Max                Mean             StdDev              Median                IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_sdpa[forward-thunder] (0003_af807f9)     293.2772 (1.0)      557.1146 (1.0)      322.5282 (1.0)      39.7666 (1.0)      312.4697 (1.0)      13.0673 (1.10)       94;156        3.1005 (1.0)        1679           2
test_nanogpt_sdpa[forward-thunder] (0002_7d62ae1)     302.0046 (1.03)     592.2045 (1.06)     331.0693 (1.03)     43.0010 (1.08)     320.1501 (1.02)     11.9164 (1.0)        97;163        3.0205 (0.97)       1667           2
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'test_nanogpt_sdpa[inference-thunder]': 2 tests -----------------------------------------------------------------------------------
Name (time in us)                                            Min                 Max                Mean             StdDev              Median               IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_nanogpt_sdpa[inference-thunder] (0002_7d62ae1)     242.9307 (1.0)      419.2563 (1.0)      262.2773 (1.0)      28.6718 (1.0)      254.3693 (1.0)      9.6212 (1.0)        86;122        3.8128 (1.0)        1380           3
test_nanogpt_sdpa[inference-thunder] (0003_af807f9)     242.9903 (1.00)     782.9460 (1.87)     263.2770 (1.00)     31.9099 (1.11)     255.0247 (1.00)     9.6334 (1.00)       84;125        3.7983 (1.00)       1372           3
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------- benchmark 'test_open_llama_7b[inference-thunder]': 2 tests -------------------------------------------------------------------------------
Name (time in ms)                                             Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_open_llama_7b[inference-thunder] (0002_7d62ae1)     387.6853 (1.0)      406.6569 (1.0)      397.5314 (1.0)      7.1338 (5.15)     398.7915 (1.0)      9.6737 (4.29)          2;0  2.5155 (1.0)           5           1
test_open_llama_7b[inference-thunder] (0003_af807f9)     507.5673 (1.31)     510.9431 (1.26)     509.0542 (1.28)     1.3865 (1.0)      509.1694 (1.28)     2.2527 (1.0)           2;0  1.9644 (0.78)          5           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

I think close attention to test_litgpt_qkv_split_rope is needed. Is it overhead that kills the performance here? How does pure kernel time change?

I think close attention to test_litgpt_qkv_split_rope is needed. Is it overhead that kills the performance here? How does pure kernel time change?

ok, we know that nvfuser kernel performance is bad with rope. Forgot that rope is left as-is in the benchmark...

I'd be curious to see how the rope benchmark regression looks like with this https://github.com/NVIDIA/Fuser/pull/2490. I'll take a look at that.

There are quite some looseversion errors in CI. Does anyone know why?

There are quite some looseversion errors in CI. Does anyone know why?

Yeah, unfortunately, something seems up with the connection to PyPI sometimes... :/
I'll rerun the tests. If you see this, please don't hesitate to ping me on slack or here.

I reran the test with the spurious (looseversion) failure but now:
FAILED thunder/tests/test_jit_general.py::test_litgpt_variants_kvcache[cuda-codellama2-like] - RuntimeError: !detect_exception_in_thread_pool.load() INTERNAL ASSERT FAILED at "/workspace/Fuser/csrc/kernel_cache.cpp":1234, please report a bug with repro script to NVFuser at https://github.com/NVIDIA/Fuser/issues. Detected exception while compiling fusion segments in parallel. Error messages from all threads are printed below.

Error from segmentation group 9: producer->getMemoryType() == MemoryType::Global INTERNAL ASSERT FAILED at "/workspace/Fuser/csrc/device_lower/analysis/sync_information.cpp":699, please report a bug with repro script to NVFuser at https://github.com/NVIDIA/Fuser/issues. Inconsistent parallelization found between TV41 (T41_l[ iblockIdx.x443{( ceilDiv(( ceilDiv(( ceilDiv(( 1 * 16 ), 128) ), 1) ), 1) )}, iUS444{1}, iS442{1}, ithreadIdx.x440{128} ]) and TV17(T17_l[ iblockIdx.x391{( ceilDiv(( ceilDiv(( ceilDiv(( 1 * ( 1 * ( 1 * 16 ) ) ), 128) ), 1) ), 1) )}, iUS392{1}, iS390{1}, ithreadIdx.x388{128} ] ca_pos( 4 )). Producer is required to be in Global Memory based on parallelization strategy. RAW flags: (blockIdx.x threadIdx.x)
Exception raised from SyncMap at /workspace/Fuser/csrc/device_lower/analysis/sync_information.cpp:699 (most recent call first):
frame #0: nvfuser::nvfCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x9a (0x7efd5819d42a in /usr/local/lib/python3.10/dist-packages/nvfuser/_C.cpython-310-x86_64-linux-gnu.so)
frame #1: nvfuser::nvfErrorFail(char const*, char const*, unsigned int, char const*, std::string const&) + 0x3e (0x7efd584e43ce in /usr/local/lib/python3.10/dist-packages/nvfuser/_C.cpython-310-x86_64-linux-gnu.so)
frame #2: nvfuser::SyncMap::SyncMap(nvfuser::Fusion*) + 0x26ee (0x7efd583e279e in /usr/local/lib/python3.10/dist-packages/nvfuser/_C.cpython-310-x86_64-linux-gnu.so)
frame #3: <unknown function> + 0x39bb02 (0x7efd5840bb02 in /usr/local/lib/python3.10/dist-packages/nvfuser/_C.cpython-310-x86_64-linux-gnu.so)
frame #4: nvfuser::GpuLower::GpuLower(nvfuser::Fusion*, nvfuser::CompileParams const&) + 0xe45 (0x7efd5840da75 in /usr/local/lib/python3.10/dist-packages/nvfuser/_C.cpython-310-x86_64-linux-gnu.so)
frame #5: nvfuser::FusionExecutor::compileFusion(nvfuser::Fusion*, nvfuser::KernelArgumentHolder const&, nvfuser::LaunchParams const&, nvfuser::CompileParams, nvfuser::ScheduleHeuristic, long, long, long, long) + 0xc92 (0x7efd584fcd92 in /usr/local/lib/python3.10/dist-packages/nvfuser/_C.cpython-310-x86_64-linux-gnu.so)
frame #6: <unknown function> + 0x68cadd (0x7efd586fcadd in /usr/local/lib/python3.10/dist-packages/nvfuser/_C.cpython-310-x86_64-linux-gnu.so)
frame #7: <unknown function> + 0x68cd0b (0x7efd586fcd0b in /usr/local/lib/python3.10/dist-packages/nvfuser/_C.cpython-310-x86_64-linux-gnu.so)
frame #8: c10::ThreadPool::main_loop(unsigned long) + 0x2b3 (0x7efebeb2c8a3 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0xdc253 (0x7efebdeb0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #10: <unknown function> + 0x94ac3 (0x7efec1065ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #11: clone + 0x44 (0x7efec10f6bf4 in /usr/lib/x86_64-linux-gnu/libc.so.6)

so the benchmark isn't running without the PR?

Those ran out of memory without the PR, but I don't know whether it's an actual improvement or luck.

From the new result, I'm seeing Mistral-7B-v0.1-inference perf regression (11%) for bs1 but perf gain (17%) for bs2

Yes, I'm pretty sure those are noise as I said in the description. I'll call out their names more clearly.

@jjsjann123, @riccardofelluga Did you want to take another look?

@jjsjann123 did you get a chance to check the Einsum benchmarks?

authors: IvanYashchuk jjsjann123 mruberry t-vi wujingyue

repo: Lightning-AI/lightning-thunder