Add GENESIS GPU MLP estimation bring-up

Connect GENESIS GPU runs to the PerfTools MLP_NN/v1.5 estimator path for early CI validation. GENESIS now emits a gpu_kernel_region estimation section when GPU MLP profiling is enabled and a padata archive is available, and the gpu_kernel_mlp_v15 section package can consume BenchKit padata archives containing Nsight Compute raw CSV data.

Add an NCU-to-PerfTools input bridge that extracts profile_raw.csv from padata, normalizes the Nsight Compute columns observed on MiyabiG, fills the current v1.5 static GPU spec gaps for known GPUs, and produces a prepared CSV before invoking predict_v15.py.

Add local validation support via scripts/test_estimate_submit.sh, mirroring test_submit.sh style for scheduler submission and adding an --estimate-only mode that can run inside Apptainer with PERFTOOLS/SIF or the corresponding BK_* variables.

Record GENESIS results with the GENESIS-specific Exp p8 and declare the same baseline Exp for estimation. This avoids falling through to the common CASE0 default, which is QWS-specific and is not a valid GENESIS experiment label.

Keep GENESIS padata estimation artifacts as repo-root-relative paths such as results/padata0.tgz, even when run.sh has executed from inside the benchmark input tree. This lets downstream estimate jobs resolve the artifact downloaded from GitLab CI artifacts.

Treat padata upload HTTP 413 as a non-fatal condition in send_results.sh. The result JSON is already ingested, and large profiler archives remain available to downstream estimation jobs through GitLab artifacts even when the Result Server refuses the padata upload size.

Upload generated results/estimation_inputs from send_estimate.sh using the source result UUID from the Estimate JSON, so portal re-estimation and /api/query/estimation-inputs can retrieve GPU MLP prepared inputs produced in the estimate job.

Allow local matrix generation without CI_PIPELINE_SOURCE by recording PARENT_PIPELINE_SOURCE=local, and align the PerfTools smoke-mode documentation with the repository-wide Python 3.12+ runtime expectation.

Temporary bring-up wiring is intentionally explicit in .gitlab-ci.yml: BK_QWS_GPU_MLP_SMOKE, BK_ESTIMATE_RUNNER_TAG=fncx-estimate-python, BK_GPU_MLP_PERFTOOLS_REPO/REF, BK_GENESIS_GPU_MLP_PROFILE, BK_GPU_MLP_NCU_LAUNCH_COUNT, BK_GPU_MLP_SOURCE_GPU, and BK_GPU_MLP_KERNEL_COUNT are provisional switches. Remove or replace them once the real estimator runner/package flow is settled.

Validation: WSL bash -n and shellcheck -S error for changed shell scripts; python py_compile for prepare_gpu_mlp_ncu_input.py; env -u CI_PIPELINE_SOURCE bash scripts/matrix_generate.sh code=qws system=Fugaku; test_send_estimate_inputs.sh, test_genesis_gpu_mlp_estimation.sh, test_bk_profiler.sh, test_estimation_gpu_kernel_mlp_v15.sh, test_qws_gpu_mlp_smoke_estimation.sh, test_result_profile_data.sh, and test_send_results_profile_data.sh.

Signed-off-by: Yoshifumi Nakamura <nakamura@riken.jp>

Author: yoshifuminakamura

.gitlab-ci.yml

@@ -37,6 +37,10 @@ variables:
   BK_ESTIMATE_RUNNER_TAG: "fncx-estimate-python"
   BK_GPU_MLP_PERFTOOLS_REPO: "https://github.com/masaaki-kondo/PerfTools.git"
   BK_GPU_MLP_PERFTOOLS_REF: "main"
+  BK_GENESIS_GPU_MLP_PROFILE: "true"
+  BK_GPU_MLP_NCU_LAUNCH_COUNT: "20"
+  BK_GPU_MLP_SOURCE_GPU: "H100"
+  BK_GPU_MLP_KERNEL_COUNT: "20"
 
 # Extract system and code filters from API variables or commit message
 .filters: &filters

docs/guides/add-estimation-package.md

@@ -114,7 +114,7 @@ export BK_GPU_MLP_PERFTOOLS_REF=main
 `BK_QWS_GPU_MLP_SMOKE` は qws を使った配管確認用、`BK_QWS_GPU_MLP_SMOKE_MODE` は prediction fixture 取り込みと PerfTools 実行の切り替え用、`BK_ESTIMATE_RUNNER_TAG` は推定用 runner/container を手動で逃がすためのものです。
 実際の GPU profiling input と推定 runner の運用が固まったら、専用の package/runner 設定へ置き換え、これらの暫定変数は削除対象として見直してください。
 
-`perftools` smoke mode は GitHub から PerfTools を取得するため、推定 runner/container には `git` と外部接続、Python 3.11、numpy/pandas/torch が必要です。
+`perftools` smoke mode は GitHub から PerfTools を取得するため、推定 runner/container には `git` と外部接続、Python 3.12 以上、numpy/pandas/torch が必要です。
 実運用では smoke mode ではなく、推定 runner/container に PerfTools checkout を用意し、section artifact として実アプリ由来の prepared input CSV を渡してください。
 
 ## 5. metadata に持たせるもの

programs/genesis/estimate.sh

@@ -0,0 +1,66 @@
+#!/bin/bash
+# estimate.sh — GENESIS estimation entrypoint and run-time section metadata.
+
+genesis_declare_estimation_layout() {
+  bk_clear_estimation_defaults
+  bk_clear_estimation_declarations
+  bk_define_current_estimation_package weakscaling
+  bk_define_future_estimation_package instrumented_app_sections_dummy
+  bk_define_baseline_system "${BK_ESTIMATION_BASELINE_SYSTEM:-MiyabiG}"
+  bk_define_baseline_exp "${BK_ESTIMATION_BASELINE_EXP:-${BK_GENESIS_EXP:-p8}}"
+  bk_define_future_system "${BK_ESTIMATION_FUTURE_SYSTEM:-GPU_MLP_TARGET}"
+  bk_define_current_target_nodes "${BK_ESTIMATION_CURRENT_TARGET_NODES:-1}"
+  bk_define_future_target_nodes "${BK_ESTIMATION_FUTURE_TARGET_NODES:-1}"
+  bk_declare_section --side future gpu_kernel_region gpu_kernel_mlp_v15
+}
+
+genesis_emit_estimation_data_from_fom() {
+  local fom="$1"
+  local artifact_path="results/padata0.tgz"
+  local padata_path="$artifact_path"
+
+  case "${BK_GENESIS_GPU_MLP_PROFILE:-false}" in
+    1|true|TRUE|yes|YES|on|ON) ;;
+    *) return 0 ;;
+  esac
+
+  if [[ -n "${GENESIS_BENCHKIT_ROOT:-}" ]]; then
+    padata_path="${GENESIS_BENCHKIT_ROOT}/${artifact_path}"
+  fi
+  if [[ ! -f "$padata_path" ]]; then
+    echo "Genesis GPU MLP estimation requested but profiler archive was not found: ${padata_path}" >&2
+    return 0
+  fi
+
+  bk_emit_declared_section --side future gpu_kernel_region "$fom" "$artifact_path"
+}
+
+source scripts/bk_functions.sh
+source scripts/estimation/common.sh
+
+BK_ESTIMATION_SECTION_DEFAULT_FACTOR="${BK_ESTIMATION_SECTION_DEFAULT_FACTOR:-1.0}"
+BK_GPU_MLP_ARTIFACT_MODE="${BK_GPU_MLP_ARTIFACT_MODE:-ncu}"
+BK_GPU_MLP_SOURCE_GPU="${BK_GPU_MLP_SOURCE_GPU:-H100}"
+BK_GPU_MLP_KERNEL_COUNT="${BK_GPU_MLP_KERNEL_COUNT:-20}"
+export BK_GPU_MLP_ARTIFACT_MODE
+export BK_GPU_MLP_SOURCE_GPU
+export BK_GPU_MLP_KERNEL_COUNT
+
+genesis_declare_estimation_layout
+bk_estimation_apply_declared_defaults
+BK_ESTIMATION_PACKAGE="${BK_ESTIMATION_PACKAGE:-$BK_ESTIMATION_FUTURE_PACKAGE}"
+
+if [[ "${BASH_SOURCE[0]}" != "$0" ]]; then
+  return 0 2>/dev/null || exit 0
+fi
+
+BK_ESTIMATION_INPUT_JSON="$1"
+
+bk_estimation_run_declared_future_package "$BK_ESTIMATION_INPUT_JSON"
+bk_estimation_run_recorded_current_with_weakscaling \
+  "${BK_ESTIMATION_BASELINE_SYSTEM:-MiyabiG}" \
+  "${BK_ESTIMATION_BASELINE_EXP:-}" \
+  "${BK_ESTIMATION_CURRENT_TARGET_NODES:-1}" \
+  "${BK_ESTIMATION_CURRENT_PACKAGE:-weakscaling}"
+
+bk_estimation_write_output "results/estimate_${est_code}_0.json"

programs/genesis/run.sh

@@ -8,13 +8,16 @@ nthreads="$4"
 numproc=$(( numproc_node * nodes ))
 
 source "${PWD}/scripts/bk_functions.sh"
+source "${PWD}/programs/genesis/estimate.sh"
 
 SCRIPT_DIR="${PWD}"
+export GENESIS_BENCHKIT_ROOT="$SCRIPT_DIR"
 REPO_DIR="genesis_benchmark_input"
 REPO_URL="https://github.com/genesis-release-r-ccs/${REPO_DIR}.git"
 BRANCH="main"
 dir_path="npt/genesis2.0beta_3.5fs/apoa1"
 header=p8
+exp="${BK_GENESIS_EXP:-$header}"
 input=${header}.inp
 resultsdir=${SCRIPT_DIR}/results
 artifactsdir=${SCRIPT_DIR}/artifacts
@@ -152,7 +155,15 @@ run_genesis_gh200_gpu() {
     fi
 
     genesis_profiler_tool=$(bk_get_profiler_tool "$genesis_profiler_requested") || return 1
-    genesis_profiler_level="${!profiler_level_var:-${GENESIS_PROFILER_LEVEL:-single}}"
+    local genesis_default_profiler_level="single"
+    case "${BK_GENESIS_GPU_MLP_PROFILE:-false}" in
+      1|true|TRUE|yes|YES|on|ON)
+        genesis_default_profiler_level="detailed"
+        export BK_PROFILER_NCU_RAW_CSV="${BK_PROFILER_NCU_RAW_CSV:-true}"
+        export BK_PROFILER_ARGS="${BK_PROFILER_ARGS:---launch-count ${BK_GPU_MLP_NCU_LAUNCH_COUNT:-20}}"
+        ;;
+    esac
+    genesis_profiler_level="${!profiler_level_var:-${GENESIS_PROFILER_LEVEL:-${genesis_default_profiler_level}}}"
     if [ -n "$genesis_profiler_tool" ]; then
         if [ "$genesis_profiler_tool" = "ncu" ] && ! command -v ncu >/dev/null 2>&1; then
             if [ "$genesis_profiler_explicit" -eq 1 ]; then
@@ -223,14 +234,17 @@ fom_val=$(awk -F'=' '/^[[:space:]]*dynamics[[:space:]]*=/ {
 			print $2;
 			exit
 			}' ${output})
-cd - > /dev/null
+cd "$SCRIPT_DIR" > /dev/null
 
 if [[ -z "$fom_val" ]]; then
     echo "Warning: FOM value not found in ${output}" >&2
     fom_val="nan"   # or 0.0
 fi
 
-bk_emit_result --fom "$fom_val" --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads" >> ${resultsdir}/result
+{
+    bk_emit_result --fom "$fom_val" --exp "$exp" --nodes "$nodes" --numproc-node "$numproc_node" --nthreads "$nthreads"
+    genesis_emit_estimation_data_from_fom "$fom_val"
+} >> ${resultsdir}/result
 # if information is requierd
 #printf "%-10s nodes=%2d numproc=%3d  FOM: %.3f\n" \
 #    "$system" "$nodes" "$numproc" "$fom_val" >> ../results/result

scripts/bk_functions.sh

@@ -908,6 +908,8 @@ bk_profiler_write_meta() {
           else
             _bk_meta_ncu_report_path=""
           fi
+          _bk_meta_ncu_raw_csv_path="raw/${_bk_meta_name}/profile_raw.csv"
+          _bk_meta_ncu_raw_csv_abs="${_bk_meta_stage_dir}/${_bk_meta_ncu_raw_csv_path}"
           ;;
         *)
           _bk_meta_text_path=""
@@ -965,6 +967,13 @@ bk_profiler_write_meta() {
         printf '        {"kind": "ncu_report", "path": "%s"}' "$_bk_meta_ncu_report_path"
         _bk_meta_has_report=1
       fi
+      if [ "${_bk_meta_tool}" = "ncu" ] && [ -f "${_bk_meta_ncu_raw_csv_abs:-}" ]; then
+        if [ "$_bk_meta_has_report" -eq 1 ]; then
+          printf ',\n'
+        fi
+        printf '        {"kind": "ncu_raw_csv", "path": "%s"}' "$_bk_meta_ncu_raw_csv_path"
+        _bk_meta_has_report=1
+      fi
       if [ "$_bk_meta_has_report" -eq 1 ]; then
         printf '\n'
       fi
@@ -1141,6 +1150,19 @@ bk_profiler() {
       else
         echo "bk_profiler[ncu]: failed ${_bk_ncu_rep_name} level=${_bk_profiler_level} status=${_bk_profiler_status}" >&2
       fi
+      case "${BK_PROFILER_NCU_RAW_CSV:-false}" in
+        1|true|TRUE|yes|YES|on|ON)
+          _bk_ncu_report_file=$(bk_profiler_find_ncu_report "$_bk_ncu_rep_dir" || true)
+          if [ -n "$_bk_ncu_report_file" ]; then
+            ncu --import "$_bk_ncu_report_file" \
+              --page raw \
+              --csv \
+              --print-units base \
+              --print-fp \
+              > "${_bk_ncu_rep_dir}/profile_raw.csv" 2> "${_bk_ncu_rep_dir}/profile_raw.csv.log" || true
+          fi
+          ;;
+      esac
       cp -R "$_bk_ncu_rep_dir" "$_bk_stage_dir/raw/${_bk_ncu_rep_name}"
       _bk_profiler_run_names="${_bk_ncu_rep_name}"
       _bk_profiler_run_events="${_bk_profiler_level}"