AOTInductor Compile Cache

Introduction

Building a neural network inference engine using compiler can be slow and non-deterministic due to auto-tuning, even if it is on the same computer. The correctness of an inference engine depends on the kernels it used and the input distributions it encounters during inference. If the kernels used for a previously verified inference engine changed, even if the neural network remains the same, there is no guarantee that the inference results will be correct. Consequently, to serve an inference engine for production, the best practice is, either the same engine used for verification is used for production, or a reliable mechanism is in place to cache and reuse the previously tuned kernels when an inference engine is rebuilt from the same neural network. Of course, the former approach is simpler and more straightforward. But in practice, the latter approach is also often used.

In this blog post, I would like to quickly discuss how to leverage the AOTInductor compile cache to ensure consistent and efficient reuse of previously tuned kernels for neural network inference engines.

AOTInductor Compile Cache

TorchInductor will generate compile cache in the cache subdirectory of the TorchInductor cache directory. The content of the cache file might look like this, where each entry corresponds to a specific kernel configuration and its associated tuning results.

cache
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
{
"system": {
"version": {
"triton": "3.8.058b25352786ff1d04468ce113f1986063da0b78640428046a28c8ede4c9126ac-e7b75e513fb868674fd2adc660c6527e4ad9e74f01120d8e2b1ad243329a0c92-28ff26b7e752df455c410418d72fd0afc5b3bfa952ca4bca95b1cfd5a31a0e2f-23d635e690d670bf61798e1259674b78c0ed5ba222ab6a455f329f27a758fc2d-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-a39587ad88b66ea78116454125470474c2a5cd783f9d0ab81d607820596004d2-24b908043301e5dfcdbc4cdb17b9b7313e650cdfd4098b2ebfb998af8994be4e-c5d3403b88b5bb728f8ac78c55163198e418430f126c842892c68cb4b6baaa78-3aaa775f592533d06c13c99c23bd117430b1e7c732c2ae64c571d84d01ad2c39-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-6c4fa53b40b2748935f42b47f987adee8ec272b1fc54d2f27c6eb1cc029ec5ff-966a8105ff35077006b103266c3d8d292a56ad096a2d0dd6833027efb4bd1d50-fcdf0895d9ea3df9b91d6c05aba5aba579f385b8debd474b8525aa137e9b2bb2-b4966729f21f0db6f20fa4cb9b2b5402c5735e1424ffba05dbd8f480dedee790-5d15c5bebef8d7aa51b21fd187e5faa95eba4a213254355bc69e0648013599f7-30106ed84518c6ca7aca08e2c0ee188755f512cc0cb2d7da8914cc48c1ad6dcc-adb54e71d0ffc3bdac437fbc97929769fbbe4ebd03e6361c6357f2a24f7c5954-27b2a5d1e8db008bacefe6019f63922bbd65926de90bb1b527ee597477d2f365-a610dc5c215589aab7a784e1c07acef3e16d53ef00f08de793899964956f4e2a-ebad24d351bb79b3068a60627c74372a07cb8cc9532ce106773402ad2515a2b4-0731cda9ecbdd902acab13604efda0f9dbeaf411bc42bdddff8454b69a62f54b-2c66ab6523cc7dcd8bdc3e68b8f8844657d7bd4fd0492774c9819b046b8f7674-f60c2bb2d8eebe1c191f4b8b819844414dd1bce243645635a094f9f92665a58e-08abee21ce6230a873ed0831f70f9570b7ce39969dbf9b2f28ae1a1992ee1cc7-1a7ae2a0c3de32df60ffc18f21a20f8dcf8bf3bc8cb018bb269b540533868829-0d99c1f30bdb86bd718c98bf026ced2a84a878f79630fa9b3fe099ce46e86ebd-113296c74fb8b8dcf67a1b82f71926842eb66d0c77fe2c4f8d0f315715a38fd4-18ba36f0d8827d4807c631a2b92f9615eb56c8aa7f914964a0bb6f2ba69f344d",
"cuda": "13.4"
},
"device": {
"name": "NVIDIA GeForce RTX 5080"
},
"hash": "49cc03d54ac5bc8584957a4db2c7460d346c6f70b7002fea25dc7a2ef0c7c3a1"
},
"cache": {
"mm": {
"[('cuda', 'torch.bfloat16', 64, 1024, 1024, 1, 0), ('cuda', 'torch.bfloat16', 1024, 2048, 1, 1024, 0)]": {
"high": {
"mm-cqrjyn5jqdhj5q3lohxxeq3fxnpvy6qsigzdm2ykqas7fpkl7zrz": 0.009983999654650688,
"triton_mm-cd5p4wrvq5iidlqri5x2fjrvrjh2ef7hg5tkv5ldomwi2voftd67": 0.03683200106024742,
"triton_mm-csrroaexmdv2c7ph7w4cwochqoha7xws6ek4banbqx5oo6indtjz": 0.013632000423967838,
"triton_mm-ctyeqjd6jv2an637kbcvfkk6mtbeanggv6jkyjarkfexwiz2sewl": 0.018400000408291817,
"triton_mm-cohiwiqm7kfnrthaag35heyouv4sg5mbhafyv3e6gqj7t2dd5oqx": 0.013856000266969204,
"triton_mm-ck4bnfm3vh67gec2gq7xvqw66lyiefipgr55hlnmit5ognovox65": 0.012256000190973282,
"triton_mm-chjdn3rk7iey5a644aao3z35tsdsjzvsmrdzkyhxz5fn3oadkijh": 0.04095999896526337,
"triton_mm-ce7t5tivu2e7ylsdixrbzdbjknui74yyi3d6vxgbyazfj6teb6jx": 0.027936000376939774,
"triton_mm-c6usc4weo57xa55nklmwiuuwrwcyq7z4kuh3fonwqfjy6wtcrxun": 0.013856000266969204,
"triton_mm-crrz34b5rgtt5uogf5ug4tsio6rlrmpbdnsa3kbjpxchx4qftnku": 0.01990400068461895,
"triton_mm-cgvv3rsvsk4nswjlx6iqm2gkojl6bzweyfqo7wsilgq66hmedgzw": 0.024000000208616257,
"triton_mm-co5sj5jtzldgl4dnqqtsrjs3kcd3hewdq5ipv35qfypiqjqjxyea": 0.01881599985063076,
"triton_mm-ctbpmwbdrkvvv5kngs7mt52usdnmxfxpw33djsvxz3baqmdldu45": 0.018144000321626663,
"triton_mm-cszkh2jx4ryai5aktdtdp7bpzyqyth5nkf7ulmhlaelnr4p2ecck": 0.029023999348282814,
"triton_mm-cngbjofxwx7eukrjgzaja7hagkllyk777vkjqb2ebqoubl4voqrg": 0.01583999954164028,
"triton_mm-cw5vnawoymyzkba4cy4t3gqtrpq73bzc34m2zctnezqkudhskbwt": 0.016767999157309532,
"triton_mm-c2timucdtdplhy4lpaftkazykxhmgnb652y4k6qut4jnqobxt6bz": 0.030239999294281006,
"triton_mm-ctjjyomv6k3325epc5mvkmgnb37zwxdo5dinj4dhxnp3huqbugfe": 0.018432000651955605,
"triton_mm-cfsur26ayhe7c2kapmokuidv7k3ztzfgsfjttu2435vbcdlrmbwu": 0.026079999282956123,
"triton_mm-cp6ov26kqg2jyi25l3q2ygwqhgxakelu7sm2zz3pggfmqq5h7ekb": 0.04095999896526337
}
},
"[('cuda', 'torch.bfloat16', 64, 2048, 2048, 1, 0), ('cuda', 'torch.bfloat16', 2048, 1024, 1, 2048, 0)]": {
"high": {
"mm-cqrjyn5jqdhj5q3lohxxeq3fxnpvy6qsigzdm2ykqas7fpkl7zrz": 0.01228800043463707,
"triton_mm-cmwgxwbladuod2lshztiz6ktwrslrbqsz3vx3rzvnxx6x3pbdtzu": 0.061055999249219894,
"triton_mm-cjcfttw3fuotzx6gsolao3pfddpekwqkagdtbzf3iufp2t2ck34f": 0.017920000478625298,
"triton_mm-c6ewckuebz7l66li74hnvdshvk6mjs5h74que6jrbwkli3w7l7yc": 0.02208000048995018,
"triton_mm-cjcp7xil2c3snoxysf3r36z4uoyuywe3roaqe3aoxglnlkreei6w": 0.016063999384641647,
"triton_mm-cnxvhj2eemjdhd7h3r4vdk52vmvk4flq32ohd4uoviigeznurnxd": 0.01228800043463707,
"triton_mm-c3vsb2m6kx5sde3wnniolmitcqj37t75ev47l4iykt3zunuankz7": 0.06761600077152252,
"triton_mm-cppdejonesxil6mjfu55iwnctrazkevff3ewmhjljqd4hiri6pfp": 0.04323200136423111,
"triton_mm-c7snjkmvhyldusr2xifxcn4vadadytmppfcozunfzhasda4i4chn": 0.020864000543951988,
"triton_mm-cpeclnbmjef4zcf34qooorozuzahofx2d2puhvminst4q5b6f7nx": 0.0323840007185936,
"triton_mm-cv3pt6bjxbkkmys7gka5bq5ns2ayb2owazohwy66jt3eckw2kca3": 0.034143999218940735,
"triton_mm-c4jlyq7rb77ebf4rvn4ikhtfnbhd6pqtrbxquteeuzquwime3nrk": 0.03014400042593479,
"triton_mm-crcbalw7w5z67y3jpfc4zfqhhx7irjjgaywopxdonysfd34iinrz": 0.029055999591946602,
"triton_mm-codfiw5qgjj5hqgd6t6zsv2szo4zjoahyfdsvfh56hnbgq7vnv3j": 0.05273599922657013,
"triton_mm-cvydnimncazdmxph7sxasjhvtsyllgmbuof75bklhpu57m7yrxx6": 0.030208000913262367,
"triton_mm-cvztnzlt5tlyricy2bshow75wusnx6cdc2hh6jc6itq643vvmygo": 0.02502400055527687,
"triton_mm-chxhx5e76spsou4xzqh2ifnxdwx4fuytkirzkav5bbhnw4dg2cba": 0.054816000163555145,
"triton_mm-cm26eqy5sxjmedj2xbsngtunj5snlfj6q2i7k4jcrqk6sjbzvljq": 0.03014400042593479,
"triton_mm-cvtl254oljcn7lkpyxehxyff2wpt77qy5ilagkecbxvomyhjgugy": 0.04265600070357323,
"triton_mm-cqes7rpd6r2low2rn5znlbz5u67uxhbfpekix3pakyasdkdtqmhn": 0.0761599987745285
}
}
}
}
}

If this cache will is present or carried to another computer that has the same hardware and software environment, TorchInductor can reuse the cached compilation results, avoiding the need to re-tune the kernels.

In the following example, we demonstrate how to compile a simple MLP model, execute it to trigger compilation, save the compilation cache from development, and reuse the compilation cache for production re-compilation.

compile_aoti.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
import json
import os
import platform
import tarfile
import time
import torch
from torch._inductor.runtime.cache_dir_utils import cache_dir

CACHE_BLOB_PATH = "aoti_cache.tar.gz"
CACHE_META_PATH = "aoti_cache_meta.json"
PACKAGE_PATH = "model.pt2"


class SimpleMLP(torch.nn.Module):

def __init__(self):
super().__init__()
self.fc1 = torch.nn.Linear(1024, 2048, bias=False)
self.fc2 = torch.nn.Linear(2048, 1024, bias=False)

def forward(self, x):
return self.fc2(torch.nn.functional.gelu(self.fc1(x)))


def get_platform_fingerprint() -> dict:
props = torch.cuda.get_device_properties(0)
try:
import triton
triton_version = triton.__version__
except ImportError:
triton_version = "unavailable"

return {
"gpu_name": props.name,
"compute_capability": f"{props.major}.{props.minor}",
"multi_processor_count": props.multi_processor_count,
"total_memory_mb": props.total_memory // (1024 * 1024),
"pytorch_version": torch.__version__,
"cuda_runtime_version": torch.version.cuda,
"triton_version": triton_version,
"python_version": platform.python_version(),
"os_arch": f"{platform.system()}-{platform.machine()}",
}


def check_platform_compatibility(saved_meta: dict, current_meta: dict) -> bool:
all_matched = True
all_keys = sorted(set(saved_meta.keys()) | set(current_meta.keys()))

print(
"[Platform Check] Comparing build environment against cached metadata:"
)
for key in all_keys:
expected = saved_meta.get(key, "<missing>")
actual = current_meta.get(key, "<missing>")
if expected == actual:
print(f" [MATCH] {key}: '{actual}'")
else:
print(f" [MISMATCH] {key}: expected '{expected}', got '{actual}'")
all_matched = False

if all_matched:
print(
"[Platform Check] Summary: ALL criteria matched (exact cache reuse guaranteed)."
)
else:
print(
"[Platform Check] Summary: WARNING — Mismatch detected! Cache keys may miss and re-autotune."
)

return all_matched


def save_aoti_autotune_cache(archive_path: str) -> int:
"""Packs the Inductor 'cache/' subdirectory (autotune decisions) into a portable archive."""
inductor_cache_subdir = os.path.join(cache_dir(), "cache")
if not os.path.isdir(inductor_cache_subdir):
return 0

file_count = 0
with tarfile.open(archive_path, "w:gz") as tar:
for root, _, files in os.walk(inductor_cache_subdir):
for fname in files:
full_path = os.path.join(root, fname)
rel_path = os.path.relpath(full_path, cache_dir())
tar.add(full_path, arcname=rel_path)
file_count += 1
return file_count


def load_aoti_autotune_cache(archive_path: str) -> int:
"""Extracts the portable autotune archive into the active TORCHINDUCTOR_CACHE_DIR."""
target_dir = cache_dir()
os.makedirs(target_dir, exist_ok=True)
with tarfile.open(archive_path, "r:gz") as tar:
members = tar.getmembers()
tar.extractall(path=target_dir, filter="data")
return len(members)


def main():
torch.manual_seed(0)
model = SimpleMLP().cuda().to(torch.bfloat16).eval()
example_inputs = (torch.randn(64,
1024,
device="cuda",
dtype=torch.bfloat16), )
batch_dim = torch.export.Dim("batch", min=1, max=512)

current_platform = get_platform_fingerprint()
has_portable_cache = os.path.exists(CACHE_BLOB_PATH)

# 1. Verify platform compatibility and restore the Inductor 'cache/' folder if present
if has_portable_cache:
if os.path.exists(CACHE_META_PATH):
with open(CACHE_META_PATH, "r") as f:
saved_platform = json.load(f)
check_platform_compatibility(saved_platform, current_platform)
else:
print(
f"[Platform Check] WARNING: '{CACHE_META_PATH}' not found; skipping platform comparison."
)

restored_files = load_aoti_autotune_cache(CACHE_BLOB_PATH)
print(
f"Restored {restored_files} autotune cache files into '{cache_dir()}' from '{CACHE_BLOB_PATH}'."
)
else:
print(
f"No '{CACHE_BLOB_PATH}' found; will autotune if disk cache is cold."
)

with torch.no_grad():
ep = torch.export.export(
model,
example_inputs,
dynamic_shapes={"x": {
0: batch_dim
}},
strict=True,
)

# 2. Compile AOTInductor package (measure strictly compile time)
torch.cuda.synchronize()
t0 = time.perf_counter()
torch._inductor.aoti_compile_and_package(
ep,
package_path=PACKAGE_PATH,
inductor_configs={
"max_autotune": True,
},
)
torch.cuda.synchronize()
compile_sec = time.perf_counter() - t0

print(f"AOTInductor compile time: {compile_sec:.2f}s")

# 3. On cold builds, verify correctness & archive the minimal 'cache/' directory
if not has_portable_cache:
compiled_fn = torch._inductor.aoti_load_package(PACKAGE_PATH)
with torch.no_grad():
ref_out = model(*example_inputs)
aoti_out = compiled_fn(*example_inputs)
torch.testing.assert_close(aoti_out, ref_out, rtol=1e-2, atol=1e-2)
print("Local correctness check passed.")

saved_files = save_aoti_autotune_cache(CACHE_BLOB_PATH)
with open(CACHE_META_PATH, "w") as f:
json.dump(current_platform, f, indent=2)
print(
f"Saved {saved_files} autotune cache files into '{CACHE_BLOB_PATH}' and '{CACHE_META_PATH}'."
)


if __name__ == "__main__":
main()

To run the example, PyTorch Docker image from NVIDIA NGC nvcr.io/nvidia/pytorch:26.10-py3 was used.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Clean old files
$ rm -rf ./dev_cache ./prod_cache aoti_cache.bin aoti_cache.tar.gz aoti_cache_meta.json model.pt2

# Run 1 (Local Dev): Autotunes, verifies correctness, and packs ./dev_cache/cache into aoti_cache.tar.gz
$ TORCHINDUCTOR_CACHE_DIR=./dev_cache \
TRITON_CACHE_DIR=./dev_cache/triton \
TORCH_LOGS="autotuning" \
python compile_aoti.py

# Run 2 (Simulated Prod): Starts with empty ./prod_cache, unpacks aoti_cache.tar.gz, and skips autotuning
$ TORCHINDUCTOR_CACHE_DIR=./prod_cache \
TRITON_CACHE_DIR=./prod_cache/triton \
TORCH_LOGS="autotuning" \
python compile_aoti.py

After local development, the compilation cache is saved in aoti_cache.tar.gz and can be reused for production re-compilation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
$ TORCHINDUCTOR_CACHE_DIR=./dev_cache \
TRITON_CACHE_DIR=./dev_cache/triton \
TORCH_LOGS="autotuning" \
python compile_aoti.py
No 'aoti_cache.tar.gz' found; will autotune if disk cache is cold.
V1116 07:22:40.661000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess ExternKernelBenchmarkRequest(extern_kernels.mm): load 0.000005, create tensor 0.000000, bench 0.142278
V1116 07:22:40.664000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cd5p4wrvq5iidlqri5x2fjrvrjh2ef7hg5tkv5ldomwi2voftd67, path: /mnt/dev_cache/d5/cd5p4wrvq5iidlqri5x2fjrvrjh2ef7hg5tkv5ldomwi2voftd67.py
V1116 07:22:40.689000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/d5/cd5p4wrvq5iidlqri5x2fjrvrjh2ef7hg5tkv5ldomwi2voftd67.py', self.module_cache_key='cd5p4wrvq5iidlqri5x2fjrvrjh2ef7hg5tkv5ldomwi2voftd67': load 0.002170, create tensor 0.000000, bench 0.024590
V1116 07:22:40.691000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: csrroaexmdv2c7ph7w4cwochqoha7xws6ek4banbqx5oo6indtjz, path: /mnt/dev_cache/sr/csrroaexmdv2c7ph7w4cwochqoha7xws6ek4banbqx5oo6indtjz.py
V1116 07:22:40.713000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/sr/csrroaexmdv2c7ph7w4cwochqoha7xws6ek4banbqx5oo6indtjz.py', self.module_cache_key='csrroaexmdv2c7ph7w4cwochqoha7xws6ek4banbqx5oo6indtjz': load 0.002307, create tensor 0.000000, bench 0.021628
V1116 07:22:40.715000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: ctyeqjd6jv2an637kbcvfkk6mtbeanggv6jkyjarkfexwiz2sewl, path: /mnt/dev_cache/ty/ctyeqjd6jv2an637kbcvfkk6mtbeanggv6jkyjarkfexwiz2sewl.py
V1116 07:22:40.737000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/ty/ctyeqjd6jv2an637kbcvfkk6mtbeanggv6jkyjarkfexwiz2sewl.py', self.module_cache_key='ctyeqjd6jv2an637kbcvfkk6mtbeanggv6jkyjarkfexwiz2sewl': load 0.002234, create tensor 0.000000, bench 0.021785
V1116 07:22:40.740000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cohiwiqm7kfnrthaag35heyouv4sg5mbhafyv3e6gqj7t2dd5oqx, path: /mnt/dev_cache/oh/cohiwiqm7kfnrthaag35heyouv4sg5mbhafyv3e6gqj7t2dd5oqx.py
V1116 07:22:40.762000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/oh/cohiwiqm7kfnrthaag35heyouv4sg5mbhafyv3e6gqj7t2dd5oqx.py', self.module_cache_key='cohiwiqm7kfnrthaag35heyouv4sg5mbhafyv3e6gqj7t2dd5oqx': load 0.002160, create tensor 0.000000, bench 0.022156
V1116 07:22:40.764000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: ck4bnfm3vh67gec2gq7xvqw66lyiefipgr55hlnmit5ognovox65, path: /mnt/dev_cache/k4/ck4bnfm3vh67gec2gq7xvqw66lyiefipgr55hlnmit5ognovox65.py
V1116 07:22:40.787000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/k4/ck4bnfm3vh67gec2gq7xvqw66lyiefipgr55hlnmit5ognovox65.py', self.module_cache_key='ck4bnfm3vh67gec2gq7xvqw66lyiefipgr55hlnmit5ognovox65': load 0.002336, create tensor 0.000000, bench 0.022160
V1116 07:22:40.789000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: chjdn3rk7iey5a644aao3z35tsdsjzvsmrdzkyhxz5fn3oadkijh, path: /mnt/dev_cache/hj/chjdn3rk7iey5a644aao3z35tsdsjzvsmrdzkyhxz5fn3oadkijh.py
V1116 07:22:40.814000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/hj/chjdn3rk7iey5a644aao3z35tsdsjzvsmrdzkyhxz5fn3oadkijh.py', self.module_cache_key='chjdn3rk7iey5a644aao3z35tsdsjzvsmrdzkyhxz5fn3oadkijh': load 0.002139, create tensor 0.000000, bench 0.024451
V1116 07:22:40.816000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: ce7t5tivu2e7ylsdixrbzdbjknui74yyi3d6vxgbyazfj6teb6jx, path: /mnt/dev_cache/e7/ce7t5tivu2e7ylsdixrbzdbjknui74yyi3d6vxgbyazfj6teb6jx.py
V1116 07:22:40.840000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/e7/ce7t5tivu2e7ylsdixrbzdbjknui74yyi3d6vxgbyazfj6teb6jx.py', self.module_cache_key='ce7t5tivu2e7ylsdixrbzdbjknui74yyi3d6vxgbyazfj6teb6jx': load 0.002221, create tensor 0.000000, bench 0.023701
V1116 07:22:40.842000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c6usc4weo57xa55nklmwiuuwrwcyq7z4kuh3fonwqfjy6wtcrxun, path: /mnt/dev_cache/6u/c6usc4weo57xa55nklmwiuuwrwcyq7z4kuh3fonwqfjy6wtcrxun.py
V1116 07:22:40.865000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/6u/c6usc4weo57xa55nklmwiuuwrwcyq7z4kuh3fonwqfjy6wtcrxun.py', self.module_cache_key='c6usc4weo57xa55nklmwiuuwrwcyq7z4kuh3fonwqfjy6wtcrxun': load 0.002308, create tensor 0.000000, bench 0.022549
V1116 07:22:40.867000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: crrz34b5rgtt5uogf5ug4tsio6rlrmpbdnsa3kbjpxchx4qftnku, path: /mnt/dev_cache/rr/crrz34b5rgtt5uogf5ug4tsio6rlrmpbdnsa3kbjpxchx4qftnku.py
V1116 07:22:40.891000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/rr/crrz34b5rgtt5uogf5ug4tsio6rlrmpbdnsa3kbjpxchx4qftnku.py', self.module_cache_key='crrz34b5rgtt5uogf5ug4tsio6rlrmpbdnsa3kbjpxchx4qftnku': load 0.002243, create tensor 0.000000, bench 0.023495
V1116 07:22:40.893000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cgvv3rsvsk4nswjlx6iqm2gkojl6bzweyfqo7wsilgq66hmedgzw, path: /mnt/dev_cache/gv/cgvv3rsvsk4nswjlx6iqm2gkojl6bzweyfqo7wsilgq66hmedgzw.py
V1116 07:22:40.917000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/gv/cgvv3rsvsk4nswjlx6iqm2gkojl6bzweyfqo7wsilgq66hmedgzw.py', self.module_cache_key='cgvv3rsvsk4nswjlx6iqm2gkojl6bzweyfqo7wsilgq66hmedgzw': load 0.002159, create tensor 0.000000, bench 0.023397
V1116 07:22:40.919000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: co5sj5jtzldgl4dnqqtsrjs3kcd3hewdq5ipv35qfypiqjqjxyea, path: /mnt/dev_cache/o5/co5sj5jtzldgl4dnqqtsrjs3kcd3hewdq5ipv35qfypiqjqjxyea.py
V1116 07:22:40.942000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/o5/co5sj5jtzldgl4dnqqtsrjs3kcd3hewdq5ipv35qfypiqjqjxyea.py', self.module_cache_key='co5sj5jtzldgl4dnqqtsrjs3kcd3hewdq5ipv35qfypiqjqjxyea': load 0.002196, create tensor 0.000000, bench 0.022916
V1116 07:22:40.945000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: ctbpmwbdrkvvv5kngs7mt52usdnmxfxpw33djsvxz3baqmdldu45, path: /mnt/dev_cache/tb/ctbpmwbdrkvvv5kngs7mt52usdnmxfxpw33djsvxz3baqmdldu45.py
V1116 07:22:40.969000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/tb/ctbpmwbdrkvvv5kngs7mt52usdnmxfxpw33djsvxz3baqmdldu45.py', self.module_cache_key='ctbpmwbdrkvvv5kngs7mt52usdnmxfxpw33djsvxz3baqmdldu45': load 0.002147, create tensor 0.000000, bench 0.023867
V1116 07:22:40.971000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cszkh2jx4ryai5aktdtdp7bpzyqyth5nkf7ulmhlaelnr4p2ecck, path: /mnt/dev_cache/sz/cszkh2jx4ryai5aktdtdp7bpzyqyth5nkf7ulmhlaelnr4p2ecck.py
V1116 07:22:40.996000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/sz/cszkh2jx4ryai5aktdtdp7bpzyqyth5nkf7ulmhlaelnr4p2ecck.py', self.module_cache_key='cszkh2jx4ryai5aktdtdp7bpzyqyth5nkf7ulmhlaelnr4p2ecck': load 0.002202, create tensor 0.000000, bench 0.024556
V1116 07:22:40.998000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cngbjofxwx7eukrjgzaja7hagkllyk777vkjqb2ebqoubl4voqrg, path: /mnt/dev_cache/ng/cngbjofxwx7eukrjgzaja7hagkllyk777vkjqb2ebqoubl4voqrg.py
V1116 07:22:41.020000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/ng/cngbjofxwx7eukrjgzaja7hagkllyk777vkjqb2ebqoubl4voqrg.py', self.module_cache_key='cngbjofxwx7eukrjgzaja7hagkllyk777vkjqb2ebqoubl4voqrg': load 0.002171, create tensor 0.000000, bench 0.022327
V1116 07:22:41.023000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cw5vnawoymyzkba4cy4t3gqtrpq73bzc34m2zctnezqkudhskbwt, path: /mnt/dev_cache/w5/cw5vnawoymyzkba4cy4t3gqtrpq73bzc34m2zctnezqkudhskbwt.py
V1116 07:22:41.045000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/w5/cw5vnawoymyzkba4cy4t3gqtrpq73bzc34m2zctnezqkudhskbwt.py', self.module_cache_key='cw5vnawoymyzkba4cy4t3gqtrpq73bzc34m2zctnezqkudhskbwt': load 0.002168, create tensor 0.000000, bench 0.022238
V1116 07:22:41.048000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c2timucdtdplhy4lpaftkazykxhmgnb652y4k6qut4jnqobxt6bz, path: /mnt/dev_cache/2t/c2timucdtdplhy4lpaftkazykxhmgnb652y4k6qut4jnqobxt6bz.py
V1116 07:22:41.072000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/2t/c2timucdtdplhy4lpaftkazykxhmgnb652y4k6qut4jnqobxt6bz.py', self.module_cache_key='c2timucdtdplhy4lpaftkazykxhmgnb652y4k6qut4jnqobxt6bz': load 0.002227, create tensor 0.000000, bench 0.024506
V1116 07:22:41.074000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: ctjjyomv6k3325epc5mvkmgnb37zwxdo5dinj4dhxnp3huqbugfe, path: /mnt/dev_cache/tj/ctjjyomv6k3325epc5mvkmgnb37zwxdo5dinj4dhxnp3huqbugfe.py
V1116 07:22:41.097000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/tj/ctjjyomv6k3325epc5mvkmgnb37zwxdo5dinj4dhxnp3huqbugfe.py', self.module_cache_key='ctjjyomv6k3325epc5mvkmgnb37zwxdo5dinj4dhxnp3huqbugfe': load 0.002155, create tensor 0.000000, bench 0.022694
V1116 07:22:41.100000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cfsur26ayhe7c2kapmokuidv7k3ztzfgsfjttu2435vbcdlrmbwu, path: /mnt/dev_cache/fs/cfsur26ayhe7c2kapmokuidv7k3ztzfgsfjttu2435vbcdlrmbwu.py
V1116 07:22:41.124000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/fs/cfsur26ayhe7c2kapmokuidv7k3ztzfgsfjttu2435vbcdlrmbwu.py', self.module_cache_key='cfsur26ayhe7c2kapmokuidv7k3ztzfgsfjttu2435vbcdlrmbwu': load 0.002608, create tensor 0.000000, bench 0.024043
V1116 07:22:41.126000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cp6ov26kqg2jyi25l3q2ygwqhgxakelu7sm2zz3pggfmqq5h7ekb, path: /mnt/dev_cache/p6/cp6ov26kqg2jyi25l3q2ygwqhgxakelu7sm2zz3pggfmqq5h7ekb.py
V1116 07:22:41.153000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/p6/cp6ov26kqg2jyi25l3q2ygwqhgxakelu7sm2zz3pggfmqq5h7ekb.py', self.module_cache_key='cp6ov26kqg2jyi25l3q2ygwqhgxakelu7sm2zz3pggfmqq5h7ekb': load 0.002146, create tensor 0.000000, bench 0.026611
Autotune Choices Stats:
{"num_choices": 20, "num_triton_choices": 19, "best_kernel": "mm", "best_time": 0.009983999654650688, "best_triton_pos": 1, "best_triton_time": 0.012256000190973282, "best_triton_kernel": "triton_mm_4", "best_triton_kernel_desc": "ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4"}
AUTOTUNE mm(64x1024, 1024x2048)
strides: [1024, 1], [1, 1024]
dtypes: torch.bfloat16, torch.bfloat16
mm 0.0100 ms 100.0%
triton_mm_4 0.0123 ms 81.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4
triton_mm_1 0.0136 ms 73.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
triton_mm_3 0.0139 ms 72.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8
triton_mm_7 0.0139 ms 72.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8
triton_mm_13 0.0158 ms 63.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
triton_mm_14 0.0168 ms 59.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
triton_mm_11 0.0181 ms 55.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
triton_mm_2 0.0184 ms 54.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8
triton_mm_16 0.0184 ms 54.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8
SingleProcess AUTOTUNE benchmarking takes 0.6359 seconds and 1.4817 seconds precompiling for 20 choices
V1116 07:22:41.433000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess ExternKernelBenchmarkRequest(extern_kernels.mm): load 0.000005, create tensor 0.000000, bench 0.030149
V1116 07:22:41.436000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cmwgxwbladuod2lshztiz6ktwrslrbqsz3vx3rzvnxx6x3pbdtzu, path: /mnt/dev_cache/mw/cmwgxwbladuod2lshztiz6ktwrslrbqsz3vx3rzvnxx6x3pbdtzu.py
V1116 07:22:41.462000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/mw/cmwgxwbladuod2lshztiz6ktwrslrbqsz3vx3rzvnxx6x3pbdtzu.py', self.module_cache_key='cmwgxwbladuod2lshztiz6ktwrslrbqsz3vx3rzvnxx6x3pbdtzu': load 0.002256, create tensor 0.000000, bench 0.026758
V1116 07:22:41.465000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cjcfttw3fuotzx6gsolao3pfddpekwqkagdtbzf3iufp2t2ck34f, path: /mnt/dev_cache/jc/cjcfttw3fuotzx6gsolao3pfddpekwqkagdtbzf3iufp2t2ck34f.py
V1116 07:22:41.487000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/jc/cjcfttw3fuotzx6gsolao3pfddpekwqkagdtbzf3iufp2t2ck34f.py', self.module_cache_key='cjcfttw3fuotzx6gsolao3pfddpekwqkagdtbzf3iufp2t2ck34f': load 0.002183, create tensor 0.000000, bench 0.022386
V1116 07:22:41.490000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c6ewckuebz7l66li74hnvdshvk6mjs5h74que6jrbwkli3w7l7yc, path: /mnt/dev_cache/6e/c6ewckuebz7l66li74hnvdshvk6mjs5h74que6jrbwkli3w7l7yc.py
V1116 07:22:41.511000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/6e/c6ewckuebz7l66li74hnvdshvk6mjs5h74que6jrbwkli3w7l7yc.py', self.module_cache_key='c6ewckuebz7l66li74hnvdshvk6mjs5h74que6jrbwkli3w7l7yc': load 0.002131, create tensor 0.000000, bench 0.021764
V1116 07:22:41.514000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cjcp7xil2c3snoxysf3r36z4uoyuywe3roaqe3aoxglnlkreei6w, path: /mnt/dev_cache/jc/cjcp7xil2c3snoxysf3r36z4uoyuywe3roaqe3aoxglnlkreei6w.py
V1116 07:22:41.536000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/jc/cjcp7xil2c3snoxysf3r36z4uoyuywe3roaqe3aoxglnlkreei6w.py', self.module_cache_key='cjcp7xil2c3snoxysf3r36z4uoyuywe3roaqe3aoxglnlkreei6w': load 0.002167, create tensor 0.000000, bench 0.022027
V1116 07:22:41.539000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cnxvhj2eemjdhd7h3r4vdk52vmvk4flq32ohd4uoviigeznurnxd, path: /mnt/dev_cache/nx/cnxvhj2eemjdhd7h3r4vdk52vmvk4flq32ohd4uoviigeznurnxd.py
V1116 07:22:41.561000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/nx/cnxvhj2eemjdhd7h3r4vdk52vmvk4flq32ohd4uoviigeznurnxd.py', self.module_cache_key='cnxvhj2eemjdhd7h3r4vdk52vmvk4flq32ohd4uoviigeznurnxd': load 0.003419, create tensor 0.000000, bench 0.020997
V1116 07:22:41.563000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c3vsb2m6kx5sde3wnniolmitcqj37t75ev47l4iykt3zunuankz7, path: /mnt/dev_cache/3v/c3vsb2m6kx5sde3wnniolmitcqj37t75ev47l4iykt3zunuankz7.py
V1116 07:22:41.590000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/3v/c3vsb2m6kx5sde3wnniolmitcqj37t75ev47l4iykt3zunuankz7.py', self.module_cache_key='c3vsb2m6kx5sde3wnniolmitcqj37t75ev47l4iykt3zunuankz7': load 0.002177, create tensor 0.000000, bench 0.027241
V1116 07:22:41.593000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cppdejonesxil6mjfu55iwnctrazkevff3ewmhjljqd4hiri6pfp, path: /mnt/dev_cache/pp/cppdejonesxil6mjfu55iwnctrazkevff3ewmhjljqd4hiri6pfp.py
V1116 07:22:41.617000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/pp/cppdejonesxil6mjfu55iwnctrazkevff3ewmhjljqd4hiri6pfp.py', self.module_cache_key='cppdejonesxil6mjfu55iwnctrazkevff3ewmhjljqd4hiri6pfp': load 0.002128, create tensor 0.000000, bench 0.024184
V1116 07:22:41.619000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c7snjkmvhyldusr2xifxcn4vadadytmppfcozunfzhasda4i4chn, path: /mnt/dev_cache/7s/c7snjkmvhyldusr2xifxcn4vadadytmppfcozunfzhasda4i4chn.py
V1116 07:22:41.640000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/7s/c7snjkmvhyldusr2xifxcn4vadadytmppfcozunfzhasda4i4chn.py', self.module_cache_key='c7snjkmvhyldusr2xifxcn4vadadytmppfcozunfzhasda4i4chn': load 0.002138, create tensor 0.000000, bench 0.021063
V1116 07:22:41.643000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cpeclnbmjef4zcf34qooorozuzahofx2d2puhvminst4q5b6f7nx, path: /mnt/dev_cache/pe/cpeclnbmjef4zcf34qooorozuzahofx2d2puhvminst4q5b6f7nx.py
V1116 07:22:41.667000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/pe/cpeclnbmjef4zcf34qooorozuzahofx2d2puhvminst4q5b6f7nx.py', self.module_cache_key='cpeclnbmjef4zcf34qooorozuzahofx2d2puhvminst4q5b6f7nx': load 0.002104, create tensor 0.000000, bench 0.023982
V1116 07:22:41.669000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cv3pt6bjxbkkmys7gka5bq5ns2ayb2owazohwy66jt3eckw2kca3, path: /mnt/dev_cache/v3/cv3pt6bjxbkkmys7gka5bq5ns2ayb2owazohwy66jt3eckw2kca3.py
V1116 07:22:41.692000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/v3/cv3pt6bjxbkkmys7gka5bq5ns2ayb2owazohwy66jt3eckw2kca3.py', self.module_cache_key='cv3pt6bjxbkkmys7gka5bq5ns2ayb2owazohwy66jt3eckw2kca3': load 0.002136, create tensor 0.000000, bench 0.023215
V1116 07:22:41.695000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: c4jlyq7rb77ebf4rvn4ikhtfnbhd6pqtrbxquteeuzquwime3nrk, path: /mnt/dev_cache/4j/c4jlyq7rb77ebf4rvn4ikhtfnbhd6pqtrbxquteeuzquwime3nrk.py
V1116 07:22:41.717000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/4j/c4jlyq7rb77ebf4rvn4ikhtfnbhd6pqtrbxquteeuzquwime3nrk.py', self.module_cache_key='c4jlyq7rb77ebf4rvn4ikhtfnbhd6pqtrbxquteeuzquwime3nrk': load 0.002173, create tensor 0.000000, bench 0.022147
V1116 07:22:41.719000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: crcbalw7w5z67y3jpfc4zfqhhx7irjjgaywopxdonysfd34iinrz, path: /mnt/dev_cache/rc/crcbalw7w5z67y3jpfc4zfqhhx7irjjgaywopxdonysfd34iinrz.py
V1116 07:22:41.742000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/rc/crcbalw7w5z67y3jpfc4zfqhhx7irjjgaywopxdonysfd34iinrz.py', self.module_cache_key='crcbalw7w5z67y3jpfc4zfqhhx7irjjgaywopxdonysfd34iinrz': load 0.002204, create tensor 0.000000, bench 0.022645
V1116 07:22:41.744000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: codfiw5qgjj5hqgd6t6zsv2szo4zjoahyfdsvfh56hnbgq7vnv3j, path: /mnt/dev_cache/od/codfiw5qgjj5hqgd6t6zsv2szo4zjoahyfdsvfh56hnbgq7vnv3j.py
V1116 07:22:41.772000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/od/codfiw5qgjj5hqgd6t6zsv2szo4zjoahyfdsvfh56hnbgq7vnv3j.py', self.module_cache_key='codfiw5qgjj5hqgd6t6zsv2szo4zjoahyfdsvfh56hnbgq7vnv3j': load 0.002155, create tensor 0.000000, bench 0.027098
V1116 07:22:41.774000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cvydnimncazdmxph7sxasjhvtsyllgmbuof75bklhpu57m7yrxx6, path: /mnt/dev_cache/vy/cvydnimncazdmxph7sxasjhvtsyllgmbuof75bklhpu57m7yrxx6.py
V1116 07:22:41.797000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/vy/cvydnimncazdmxph7sxasjhvtsyllgmbuof75bklhpu57m7yrxx6.py', self.module_cache_key='cvydnimncazdmxph7sxasjhvtsyllgmbuof75bklhpu57m7yrxx6': load 0.002473, create tensor 0.000000, bench 0.022209
V1116 07:22:41.799000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cvztnzlt5tlyricy2bshow75wusnx6cdc2hh6jc6itq643vvmygo, path: /mnt/dev_cache/vz/cvztnzlt5tlyricy2bshow75wusnx6cdc2hh6jc6itq643vvmygo.py
V1116 07:22:41.821000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/vz/cvztnzlt5tlyricy2bshow75wusnx6cdc2hh6jc6itq643vvmygo.py', self.module_cache_key='cvztnzlt5tlyricy2bshow75wusnx6cdc2hh6jc6itq643vvmygo': load 0.002101, create tensor 0.000000, bench 0.021599
V1116 07:22:41.823000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: chxhx5e76spsou4xzqh2ifnxdwx4fuytkirzkav5bbhnw4dg2cba, path: /mnt/dev_cache/hx/chxhx5e76spsou4xzqh2ifnxdwx4fuytkirzkav5bbhnw4dg2cba.py
V1116 07:22:41.849000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/hx/chxhx5e76spsou4xzqh2ifnxdwx4fuytkirzkav5bbhnw4dg2cba.py', self.module_cache_key='chxhx5e76spsou4xzqh2ifnxdwx4fuytkirzkav5bbhnw4dg2cba': load 0.002681, create tensor 0.000000, bench 0.025591
V1116 07:22:41.852000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cm26eqy5sxjmedj2xbsngtunj5snlfj6q2i7k4jcrqk6sjbzvljq, path: /mnt/dev_cache/m2/cm26eqy5sxjmedj2xbsngtunj5snlfj6q2i7k4jcrqk6sjbzvljq.py
V1116 07:22:41.874000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/m2/cm26eqy5sxjmedj2xbsngtunj5snlfj6q2i7k4jcrqk6sjbzvljq.py', self.module_cache_key='cm26eqy5sxjmedj2xbsngtunj5snlfj6q2i7k4jcrqk6sjbzvljq': load 0.002357, create tensor 0.000000, bench 0.022555
V1116 07:22:41.877000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cvtl254oljcn7lkpyxehxyff2wpt77qy5ilagkecbxvomyhjgugy, path: /mnt/dev_cache/vt/cvtl254oljcn7lkpyxehxyff2wpt77qy5ilagkecbxvomyhjgugy.py
V1116 07:22:41.902000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/vt/cvtl254oljcn7lkpyxehxyff2wpt77qy5ilagkecbxvomyhjgugy.py', self.module_cache_key='cvtl254oljcn7lkpyxehxyff2wpt77qy5ilagkecbxvomyhjgugy': load 0.002104, create tensor 0.000000, bench 0.025068
V1116 07:22:41.904000 2128 torch/_inductor/autotune_process.py:797] [0/0] [__autotuning] benchmark module key: cqes7rpd6r2low2rn5znlbz5u67uxhbfpekix3pakyasdkdtqmhn, path: /mnt/dev_cache/qe/cqes7rpd6r2low2rn5znlbz5u67uxhbfpekix3pakyasdkdtqmhn.py
V1116 07:22:41.933000 2128 torch/_inductor/autotune_process.py:610] [0/0] [__autotuning] InChildProcess self.kernel_name='triton_mm', self.module_path='/mnt/dev_cache/qe/cqes7rpd6r2low2rn5znlbz5u67uxhbfpekix3pakyasdkdtqmhn.py', self.module_cache_key='cqes7rpd6r2low2rn5znlbz5u67uxhbfpekix3pakyasdkdtqmhn': load 0.002103, create tensor 0.000000, bench 0.028835
Autotune Choices Stats:
{"num_choices": 20, "num_triton_choices": 19, "best_kernel": "mm", "best_time": 0.01228800043463707, "best_triton_pos": 1, "best_triton_time": 0.01228800043463707, "best_triton_kernel": "triton_mm_23", "best_triton_kernel_desc": "ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4"}
AUTOTUNE mm(64x2048, 2048x1024)
strides: [2048, 1], [1, 2048]
dtypes: torch.bfloat16, torch.bfloat16
mm 0.0123 ms 100.0%
triton_mm_23 0.0123 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4
triton_mm_22 0.0161 ms 76.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8
triton_mm_20 0.0179 ms 68.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=32, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
triton_mm_26 0.0209 ms 58.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8
triton_mm_21 0.0221 ms 55.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8
triton_mm_33 0.0250 ms 49.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
triton_mm_30 0.0291 ms 42.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
triton_mm_29 0.0301 ms 40.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
triton_mm_35 0.0301 ms 40.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8
SingleProcess AUTOTUNE benchmarking takes 0.5319 seconds and 0.0020 seconds precompiling for 20 choices
AOTInductor compile time: 14.32s
Local correctness check passed.
Saved 1 autotune cache files into 'aoti_cache.tar.gz' and 'aoti_cache_meta.json'.

In production, the compilation cache can be reused and the autotuning process is skipped, ensuring the same kernels that have been verified are used, and the compilation time is significantly reduced. Note that the software and hardware environment, such as CUDA version, GPU architecture, PyTorch version, should match the environment in which the cache was generated.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ TORCHINDUCTOR_CACHE_DIR=./prod_cache \
TRITON_CACHE_DIR=./prod_cache/triton \
TORCH_LOGS="autotuning" \
python compile_aoti.py
[Platform Check] Comparing build environment against cached metadata:
[MATCH] compute_capability: '12.0'
[MATCH] cuda_runtime_version: '13.4'
[MATCH] gpu_name: 'NVIDIA GeForce RTX 5080'
[MATCH] multi_processor_count: '84'
[MATCH] os_arch: 'Linux-x86_64'
[MATCH] python_version: '3.12.3'
[MATCH] pytorch_version: '2.15.0a0+875d815502.nv26.10'
[MATCH] total_memory_mb: '15838'
[MATCH] triton_version: '3.8.0'
[Platform Check] Summary: ALL criteria matched (exact cache reuse guaranteed).
Restored 1 autotune cache files into '/mnt/prod_cache' from 'aoti_cache.tar.gz'.
AOTInductor compile time: 10.72s

Conclusions

Although it is less ideal than reusing the verified inference engine directly for production, if we ever have to recompile the model in a production environment, leveraging the saved compilation cache allows us to skip the autotuning process, ensuring correctness while significantly reducing compilation time.

Author

Lei Mao

Posted on

11-16-2026

Updated on

11-16-2026

Licensed under


Comments