cuda-cutlass-fmha-incremental-rebuild

Incremental rebuilds silently use STALE CUTLASS fused-MHA kernels

The general false-green principles (stale binary, wrong-artifact mtime) are summarised in the ort-test skill's "False-green taxonomy". This skill is the CUDA/CUTLASS-specific detail.

The gotcha (verification-integrity bug)

nvcc-generated depfiles do not track the CUTLASS fused-MHA headers under onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/ (e.g. kernel_forward.h, fmha_launch_template.h). These headers are #included by the fmha_sm*.cu translation units, but the build system does not record that dependency.

Consequence: after you edit one of those headers, an incremental build.sh:

• does not recompile fmha_sm*.cu,
• reports [100%] Built target ... and exits 0,
• leaves the recompiled artifacts — the fmha_sm*.cu.o objects and the libonnxruntime_providers_cuda.so they link into — unchanged (same mtime as the pre-edit build).

(Do not use the gtest test-exe mtime as the stale symptom: in the shared-provider build the exe dlopens the .so and is not relinked, so its mtime stays old even after a correct rebuild — see "How to confirm" below. The reliable diagnostic signal is the fmha_sm*.cu.o / .so mtime.)

So your "successful" rebuild is running the old kernel. Tests that should now pass (or fail) reflect the previous code, not your edit. This silently invalidates any FAIL→PASS / PASS→FAIL verification.

The fix — force recompile the .cu units

Before rebuilding after editing any cutlass_fmha/*.h header:

touch onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/*.cu

Then run the normal build command. This forces the fmha_sm*.cu translation units (and downstream binaries) to recompile against your header change.

How to confirm the rebuild was real (don't trust "[100%] Built")

Confirm that the artifact which actually links the recompiled fmha_sm*.cu.o is newer than your header edit.

⚠️ Do NOT just check the test EXE mtime — it can falsely flag a good build as stale. In the shared-provider build configuration (the default here), the CUDA execution provider is a shared module: the recompiled fmha_sm*.cu.o link into libonnxruntime_providers_cuda.so, and the onnxruntime_provider_test executable dlopens that .so — it is not relinked. So after a correct rebuild the test exe mtime stays old while the .so advances. Checking the exe alone would wrongly conclude the build was stale.

Check the right artifact for your link mode:

• Shared-provider build (default): the .so that links the recompiled .o — build/<dir>/<cfg>/libonnxruntime_providers_cuda.so
• Statically-linked provider: the test exe itself (onnxruntime_provider_test)

Safest check — stat both the recompiled object and the .so, and confirm BOTH are newer than the header edit:

stat -c '%y %n' onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/kernel_forward.h
# in your build dir, e.g. build/Debug_quickbuild/Debug/:
stat -c '%y %n' libonnxruntime_providers_cuda.so
# and the actual recompiled object (path varies by build dir):
find . -name 'fmha_sm80.cu.o' -exec stat -c '%y %n' {} +

If the .so (and the fmha_sm*.cu.o) timestamps are older than (or equal to) the header edit, the build was stale — touch the .cu files and rebuild. The most reliable signal of all is behavioral: a test that was failing now passes (a stale binary cannot flip its result).

Related: pick the right test binary

This is the CUDA/CUTLASS instance of false-green mode 1 (zero-match / wrong binary) — see the ort-test skill's "False-green taxonomy" for the general principle. In short: attention/MEA/Flash boundary gtests (e.g. FlashStructuralEmptyRows*, Attention_Causal_NonPadKVSeqLen_MEA_*) live in onnxruntime_provider_test, which CI runs; onnxruntime_test_all does not contain them and gives a false green. Verify the MEA/Flash boundary fix against onnxruntime_provider_test.

Related: disk frugality on shared GPU dev boxes

Full ORT CUDA builds are large (test binaries ~1 GB each; a build dir can reach tens of GB). On a shared box, /home filling to 100% makes builds fail in non-obvious places — e.g. git submodule sync reporting No space left on device or a config.lock error, not an obvious "disk full" at the compile step.

Before a big rebuild, check free space and clean only clearly-stale, regenerable build directories (old dated experiment dirs). Never delete another agent's active build dir or anything ambiguous:

df -h /home
du -sh build/* | sort -h