+
+## 5. よくある問題と対処法
+
+### 5.1 接続テストが失敗する
+
+リモートホスト設定ダイアログの **TEST** ボタンで接続確認が行えます。
+失敗する場合は以下を確認してください。
+
+1. **ホスト名・ポート番号・ユーザ名** が正しいか確認する
+2. リモートホスト側で **公開鍵が登録されているか** 確認する（`~/.ssh/authorized_keys`）
+3. ターミナルで直接 `ssh` コマンドで接続を試みる
+
+ ```bash
+ ssh -i /path/to/key user@hostname
+ ```
+
+4. `known_hosts` にホスト鍵が登録されているか確認する
+
+ ```bash
+ ssh-keygen -F hostname
+ ```
+
+### 5.2 パスフレーズ入力ダイアログが毎回表示される
+
+ssh-agent を使用することでパスフレーズ入力を省略できます。
+[2.3 ssh-agent を使用した認証](#23-ssh-agent-を使用した認証パスフレーズ省略) を参照してください。
+
+また、`~/.ssh/config` に `AddKeysToAgent yes` を設定することで、
+初回接続後は自動的にエージェントが鍵を保持します。
+
+### 5.3 多段SSH接続でタイムアウトする
+
+踏み台サーバの設定で `ServerAliveInterval` を設定して接続を維持してください。
+
+```
+Host bastion
+ ServerAliveInterval 60
+ ServerAliveCountMax 3
+```
+
+### 5.4 SSH接続のデバッグログを出力する
+
+`server.json` の `verboseSsh` を `true` に設定（または環境変数 `WHEEL_VERBOSE_SSH=true`）すると、
+SSH接続時に詳細ログ（`-vvv` オプション相当）が出力されます。
+
+```json
+{
+ "verboseSsh": true
+}
+```
+
+ログはWHEELのログファイル（`wheel.log`）に記録されます。
+問題解決後は `false` に戻すことを推奨します（ログ量が大幅に増加するため）。
+
+--------
+[リモートホスト設定ダイアログに戻る]({{ site.baseurl }}/reference/2_remotehost_screen/)
+
+[リファレンスマニュアルのトップページに戻る]({{ site.baseurl }}/reference/)
diff --git a/documentMD/user_guide/_reference/3_workflow_screen/1_graphview.md b/documentMD/user_guide/_reference/3_workflow_screen/1_graphview.md
index b92bb1954..7bdfe3fdc 100644
--- a/documentMD/user_guide/_reference/3_workflow_screen/1_graphview.md
+++ b/documentMD/user_guide/_reference/3_workflow_screen/1_graphview.md
@@ -67,6 +67,39 @@ WHEELから投入されたジョブの実行状態確認時に、規定の回数
ルートワークフローから順に下位コンポーネントへと移動して、unknown状態となっているタスクコンポーネントを探し、当該ジョブの実行状態がワークフロー全体の実行の成否に影響が無いかを確認してください。
{: .notice--info}
+### プロジェクト操作ボタンエリア（`stopped`/`failed`/`unknown`状態からの再実行について）
+
+プロジェクトが `stopped`、`failed`、または `unknown` 状態のとき、run project ボタンを押すと中断した時点の状態から実行を再開（リスタート）することができます。
+
+#### 各コンポーネントの挙動
+
+リスタート時には、各コンポーネントの状態に応じて以下のように処理されます。
+
+| コンポーネントの状態 | リスタート時の挙動 |
+|:---------------------|:---------------------------------|
+| `finished` | スキップ（再実行されません） |
+| `not-started` | 通常通り実行されます |
+| `running` | 実行中に中断されたため再実行されます |
+| `failed` | 再実行されます |
+
+#### ループコンポーネント（For / Foreach / While）のリスタート挙動
+
+ループコンポーネントが実行途中で中断された場合、リスタートすると完了済みのイテレーション（`finished` 状態のインスタンス）はスキップされ、中断された時点のインデックスから実行を再開します。
+
+#### PSコンポーネントのリスタート挙動
+
+PSコンポーネントが実行途中で中断された場合、リスタートすると完了済み（`finished`）のインスタンスはスキップされ、未完了のインスタンスのみが再実行されます。
+
+#### 保存（save）してからリスタートする場合との違い
+
+`stop project` 後に `save project` を行ってからリスタートすると、全てのコンポーネントの状態が `not-started` にリセットされ、ワークフローが最初から実行されます。
+保存せずに直接 `run project` を押した場合は、中断した時点の状態から再開します（ループコンポーネントは完了済みのイテレーションをスキップして続きから実行）。
+
+#### cleanup project との違い
+
+`cleanup project` ボタンはプロジェクトの実行中に生成されたファイルを全て削除し、コンポーネントの状態を `not-started` にリセットして最初から実行できる状態に戻します。
+リスタートはファイルを削除せず中断時の状態を引き継ぎますが、`cleanup project` はゼロから実行し直す場合に使用します。
+
### プロジェクト操作ボタンエリア
このエリアには、プロジェクトの実行に関わるボタンが表示されます。
@@ -74,8 +107,8 @@ WHEELから投入されたジョブの実行状態確認時に、規定の回数
||構成要素|説明|
|----------|----------|---------------------------------|
-|1|run project ボタン |プロジェクトを実行開始します|
-|2|stop project ボタン |プロジェクトの実行を停止し実行前の状態に戻します|
+|1|run project ボタン |プロジェクトを実行開始します。`stopped`、`failed`、`unknown` 状態のプロジェクトは、中断した時点の状態から実行を再開（リスタート）します。詳細は[上記のリスタートについての説明](#プロジェクト操作ボタンエリアstoppedfailedunknown状態からの再実行について)を参照してください|
+|2|stop project ボタン |プロジェクトの実行を停止し、`stopped` 状態に移行します。各コンポーネントの実行状態はそのまま保持されます|
|3|cleanup project ボタン |プロジェクトの実行中に生成されたファイルなどを削除し、実行開始前の状態に戻します|
### 保存ボタンエリア
diff --git a/documentMD/user_guide/_reference/4_component/00_common.en.md b/documentMD/user_guide/_reference/4_component/00_common.en.md
index e0d465926..997ecfe2b 100644
--- a/documentMD/user_guide/_reference/4_component/00_common.en.md
+++ b/documentMD/user_guide/_reference/4_component/00_common.en.md
@@ -7,6 +7,7 @@ This section describes the specifications common to all components.
## Viewing Properties
When you single-click a component displayed in the workflow creation area, an area for editing the settings (properties) of that component appears.
+Settings are saved when the area is closed, except for some items.
The contents of this area differ for each type of component.
@@ -14,8 +15,8 @@ The contents of this area differ for each type of component.
|| Component | Description |
|----------|----------|---------------------------------|
-|1|close button | Closes the property display |
-|2|clean button | Rewind the state of the component (and any subcomponents) to the most recent saved state |
+|1|close button | Discards changes and closes the property display |
+|2|disable switch | Disables the component (and subcomponents if any) |
|3| Details button | Shows or hides property settings for each group |
### Component Right-Click Menu
@@ -47,7 +48,7 @@ To export a component:
- The archive contains the component and all its descendant components
__What is Exported__
-- Component configuration (component.json files)
+- Component configuration (component.wheel.json files)
- All files and directories within the component
- All descendant (child) components recursively
- Component states are reset to "not-started"
@@ -196,6 +197,7 @@ At the top of the file operation area are buttons for file operations.
|5|upload file button | Uploads a file to the displayed hierarchy |
|6| Download button | Downloads the selected file or directory |
|7|share file button | Displays the path of the currently selected file or directory |
+|8|edit file button | Opens the selected file in a text editor |
__About buttons for working with files and directories during selection__
If the selected file or directory is not supported, the button is disabled.
diff --git a/documentMD/user_guide/_reference/4_component/03_For.en.md b/documentMD/user_guide/_reference/4_component/03_For.en.md
index 25bb24ac1..736bf52f1 100644
--- a/documentMD/user_guide/_reference/4_component/03_For.en.md
+++ b/documentMD/user_guide/_reference/4_component/03_For.en.md
@@ -14,7 +14,7 @@ You can set the following properties for the For component:
Sets the starting value of the index.
### end
-Sets the closing price of the index.
+Sets the ending value of the index.
### step
Sets the update width for index updates.
@@ -29,24 +29,34 @@ If unspecified, all directories are saved.
For details, see [For Component Run-time Behavior](#for-component-run-time-behavior) below.
+### skip copy
+
+Sets the list of files, directories, or glob patterns to exclude from the copy operation between loop iterations.
+
+Files and directories matching the specified patterns are not copied when WHEEL creates a new iteration directory from the previous one.
+This is useful for excluding large output files or temporary files generated during previous iterations.
+
+Enter the desired pattern in the input field and click the + button to add it.
+Glob patterns (e.g., `*.log`, `output_*`, `results/`) are supported.
+
### For Component Run-time Behavior
When the For component runs for the first time, the component directory is copied with the index value appended.
When all the subcomponents in the copied directory have finished executing, a new index value is calculated and further directories are copied based on that value.
-This process is repeated sequentially until the index value exceeds the closing price.
-When the closing price is exceeded, the directory is copied to the original directory, and processing of the For component ends.
-Note that even if you set a negative value for step, if the opening price is > closing price, the operation will be successful.
-In this case, execution ends when the index falls below the closing price.
+This process is repeated sequentially until the index value exceeds the end value.
+When the end value is exceeded, the directory is copied to the original directory, and processing of the For component ends.
+Note that even if you set a negative value for step, if the start value > end value, the operation will be successful.
+In this case, execution ends when the index falls below the end value.
For example, a `for` component with start=1, end=3, step=2 is processed as follows:
1. Copy `for` directory as `for_1` directory
2. Sequentially execute components in the `for_1` directory
-3. index Calculation 1 +2 = 3 => equal to the closing price of 3, run the next loop
+3. index Calculation 1 +2 = 3 => equal to the end value of 3, run the next loop
4. Copy `for_1` directory as `for_3` directory
5. Sequentially execute components in the `for_3` directory
-6. index Calculation 3 +2 = 5 => Since the closing price has exceeded 3, the closing process is performed.
+6. index Calculation 3 +2 = 5 => Since the end value of 3 has been exceeded, the ending process is performed.
7. Copy `for_3` directory as `for` directory
If the number of instance to keep value is set to nonzero, delete the old directories (such as `for_1` and `for_3`) that exceed the number set after the 4, 7 operation.
diff --git a/documentMD/user_guide/_reference/4_component/03_For.md b/documentMD/user_guide/_reference/4_component/03_For.md
index 1c83e47b3..be4fb11d3 100644
--- a/documentMD/user_guide/_reference/4_component/03_For.md
+++ b/documentMD/user_guide/_reference/4_component/03_For.md
@@ -29,6 +29,16 @@ __インデックス値の参照方法について__
詳しくは後述の[Forコンポーネント実行時の挙動](#forコンポーネント実行時の挙動)で説明します。
+### skip copy
+
+各イテレーション間のコピー操作から除外するファイル、ディレクトリ、または glob パターンのリストを設定します。
+
+指定したパターンにマッチするファイル・ディレクトリは、WHEEL が前のイテレーションのディレクトリから新しいイテレーションディレクトリを作成する際にコピーされません。
+前のイテレーションで生成された大容量の出力ファイルや一時ファイルを除外するのに便利です。
+
+入力欄に任意のパターンを入力し、＋ボタンをクリックで追加します。
+glob パターン（例：`*.log`、`output_*`、`results/`）も使用できます。
+
### Forコンポーネント実行時の挙動
Forコンポーネントが初めて実行されるとき、コンポーネントのディレクトリはインデックスの値を末尾につけた名前でコピーされます。
コピーされたディレクトリ内の下位コンポーネントの実行が全て終了すると、新しいインデックス値が計算されその値に基いてさらにディレクトリがコピーされます。
@@ -51,5 +61,11 @@ Forコンポーネントが初めて実行されるとき、コンポーネン
number of instance to keepの値が0以外に設定されていた場合は、4,7の処理を実行後に設定された数を超える古いディレクトリ(`for_1`や`for_3`など)を削除します。
+### 中断時のリスタート挙動
+
+プロジェクトが `stopped`、`failed`、または `unknown` 状態で中断された後にリスタートすると、Forコンポーネントは完了済みのイテレーション（`finished` 状態のインスタンスディレクトリ）をスキップし、中断された時点のインデックスから実行を再開します。
+
+リスタート前に `save project` を行なった場合は、全コンポーネントの状態が `not-started` にリセットされ、最初のインデックスから実行し直します。
+
--------
[コンポーネントの詳細に戻る]({{ site.baseurl }}/reference/4_component/)
diff --git a/documentMD/user_guide/_reference/4_component/04_while.en.md b/documentMD/user_guide/_reference/4_component/04_while.en.md
index 679d77f73..6e8a164f5 100644
--- a/documentMD/user_guide/_reference/4_component/04_while.en.md
+++ b/documentMD/user_guide/_reference/4_component/04_while.en.md
@@ -30,6 +30,16 @@ If unspecified, all directories are saved.
For details, see [While Component Run-time Behavior](#while-component-run-time-behavior) below.
+### skip copy
+
+Sets the list of files, directories, or glob patterns to exclude from the copy operation between loop iterations.
+
+Files and directories matching the specified patterns are not copied when WHEEL creates a new iteration directory from the previous one.
+This is useful for excluding large output files or temporary files generated during previous iterations.
+
+Enter the desired pattern in the input field and click the + button to add it.
+Glob patterns (e.g., `*.log`, `output_*`, `results/`) are supported.
+
### While Component Run-time Behavior
The While component behaves similarly to the For component, but uses a zero-based number at the end of the directory name instead of an index value.
diff --git a/documentMD/user_guide/_reference/4_component/04_while.md b/documentMD/user_guide/_reference/4_component/04_while.md
index 5d1d5fa26..401ba1dc0 100644
--- a/documentMD/user_guide/_reference/4_component/04_while.md
+++ b/documentMD/user_guide/_reference/4_component/04_while.md
@@ -30,11 +30,27 @@ __インデックス値の参照方法について__
詳しくは後述の[Whileコンポーネント実行時の挙動](#whileコンポーネント実行時の挙動)で説明します。
+### skip copy
+
+各イテレーション間のコピー操作から除外するファイル、ディレクトリ、または glob パターンのリストを設定します。
+
+指定したパターンにマッチするファイル・ディレクトリは、WHEEL が前のイテレーションのディレクトリから新しいイテレーションディレクトリを作成する際にコピーされません。
+前のイテレーションで生成された大容量の出力ファイルや一時ファイルを除外するのに便利です。
+
+入力欄に任意のパターンを入力し、＋ボタンをクリックで追加します。
+glob パターン（例：`*.log`、`output_*`、`results/`）も使用できます。
+
### Whileコンポーネント実行時の挙動
Whileコンポーネントも、Forコンポーネントと同様の挙動をしますがディレクトリ名の末尾にはインデックス値の代わりに、0から始まる数字を1刻みで使用します。
また終了判定もインデックス値の計算ではなく、設定されたシェルスクリプトの戻り値か、javascript式の評価結果を用います。
+### 中断時のリスタート挙動
+
+プロジェクトが `stopped`、`failed`、または `unknown` 状態で中断された後にリスタートすると、Whileコンポーネントは完了済みのイテレーション（`finished` 状態のインスタンスディレクトリ）をスキップし、中断された時点のインデックスから実行を再開します。
+
+リスタート前に `save project` を行なった場合は、全コンポーネントの状態が `not-started` にリセットされ、最初のインデックス（0）から実行し直します。
+
--------
[コンポーネントの詳細に戻る]({{ site.baseurl }}/reference/4_component/)
diff --git a/documentMD/user_guide/_reference/4_component/05_Foreach.en.md b/documentMD/user_guide/_reference/4_component/05_Foreach.en.md
index 9a95214c1..752cd8a8a 100644
--- a/documentMD/user_guide/_reference/4_component/05_Foreach.en.md
+++ b/documentMD/user_guide/_reference/4_component/05_Foreach.en.md
@@ -27,6 +27,16 @@ If unspecified, all directories are saved.
For details, see [Foreach Component Run-time Behavior](#foreach-component-run-time-behavior) below.
+### skip copy
+
+Sets the list of files, directories, or glob patterns to exclude from the copy operation between loop iterations.
+
+Files and directories matching the specified patterns are not copied when WHEEL creates a new iteration directory from the previous one.
+This is useful for excluding large output files or temporary files generated during previous iterations.
+
+Enter the desired pattern in the input field and click the + button to add it.
+Glob patterns (e.g., `*.log`, `output_*`, `results/`) are supported.
+
### Foreach Component Run-time Behavior
The Foreach component behaves the same way as the For component, but the index value is not calculated; instead, the values set in indexList are used starting from the beginning of the list.
Terminates execution of the entire component when it reaches the end of the list.
diff --git a/documentMD/user_guide/_reference/4_component/05_Foreach.md b/documentMD/user_guide/_reference/4_component/05_Foreach.md
index 013a7d422..a18ff05cd 100644
--- a/documentMD/user_guide/_reference/4_component/05_Foreach.md
+++ b/documentMD/user_guide/_reference/4_component/05_Foreach.md
@@ -27,10 +27,26 @@ __インデックス値の参照方法について__
詳しくは後述の[Foreachコンポーネント実行時の挙動](#foreachコンポーネント実行時の挙動)で説明します。
+### skip copy
+
+各イテレーション間のコピー操作から除外するファイル、ディレクトリ、または glob パターンのリストを設定します。
+
+指定したパターンにマッチするファイル・ディレクトリは、WHEEL が前のイテレーションのディレクトリから新しいイテレーションディレクトリを作成する際にコピーされません。
+前のイテレーションで生成された大容量の出力ファイルや一時ファイルを除外するのに便利です。
+
+入力欄に任意のパターンを入力し、＋ボタンをクリックで追加します。
+glob パターン（例：`*.log`、`output_*`、`results/`）も使用できます。
+
### Foreachコンポーネント実行時の挙動
ForeachコンポーネントもForコンポーネントと同様の挙動をしますがインデックス値は計算によって求められるのではなく、indexListに設定された値がリストの先頭から順に使われます。
リストの終端まで実行されるとコンポーネント全体の実行を終了します。
+### 中断時のリスタート挙動
+
+プロジェクトが `stopped`、`failed`、または `unknown` 状態で中断された後にリスタートすると、Foreachコンポーネントは完了済みのイテレーション（`finished` 状態のインスタンスディレクトリ）をスキップし、中断された時点のインデックスから実行を再開します。
+
+リスタート前に `save project` を行なった場合は、全コンポーネントの状態が `not-started` にリセットされ、indexListの先頭から実行し直します。
+
--------
[コンポーネントの詳細に戻る]({{ site.baseurl }}/reference/4_component/)
diff --git a/documentMD/user_guide/_reference/4_component/06_PS.md b/documentMD/user_guide/_reference/4_component/06_PS.md
index e8e4c8188..68378958a 100644
--- a/documentMD/user_guide/_reference/4_component/06_PS.md
+++ b/documentMD/user_guide/_reference/4_component/06_PS.md
@@ -23,6 +23,12 @@ PSコンポーネントは実行開始時に[パラメータ設定ファイル](
このときもパラメータの値を用いてファイル名を変更することができます。
アプリケーションの実行結果ファイルなど、同じ名前のファイルが作成されている場合はこの機能を用いてリネームし、集めてください。
+### 中断時のリスタート挙動
+
+プロジェクトが `stopped`、`failed`、または `unknown` 状態で中断された後にリスタートすると、PSコンポーネントは完了済み（`finished`）のインスタンスディレクトリをスキップし、未完了のインスタンスのみを再実行します。
+
+リスタート前に `save project` を行なった場合は、全コンポーネントの状態が `not-started` にリセットされ、全パラメータの組み合わせについて最初から実行し直します。
+
### parameterFile
diff --git a/documentMD/user_guide/_reference/4_component/13_Continue.en.md b/documentMD/user_guide/_reference/4_component/13_Continue.en.md
new file mode 100644
index 000000000..c1b45cf5b
--- /dev/null
+++ b/documentMD/user_guide/_reference/4_component/13_Continue.en.md
@@ -0,0 +1,39 @@
+---
+title: Continue
+lang: en
+permalink: /reference/4_component/13_Continue.html
+---
+
+
+
+The Continue component can only be created directly under a for, while, or foreach component, and can have a condition expression set, similar to the If component.
+
+When the condition set on this component is met, it advances the parent component's loop to the next index and continues executing the project.
+
+When the result of the condition check is false, this component does nothing.
+
+Among the components at the same level as the Continue component, the execution order (before or after the condition check) of components that have no dependency on the Continue component cannot be specified.
+If you want to control whether a component runs when Continue is triggered, set a dependency with the Continue component.
+Even if there is no direct dependency, it is fine if they are connected through other components.
+
+The properties you can set for the Continue component are as follows.
+
+### condition setting
+Configure the settings for condition evaluation.
+
+#### use javascript expression for condition check
+Similar to the retry decision of the Task component, specifies whether to use a JavaScript expression or a shell script as the condition expression for evaluating true / false.
+
+ - When disabled
+ 
+When disabled, a dropdown list for selecting a shell script is displayed.
+The specified shell script is used as the condition expression to determine true / false.
+
+ - When enabled
+
+When enabled, a JavaScript expression can be entered.
+The entered expression is used as the condition expression to determine true / false.
+
+
+--------
+[Return to Component Details]({{site.baseurl}}/reference/4_component/)
diff --git a/documentMD/user_guide/_reference/4_component/14_Break.en.md b/documentMD/user_guide/_reference/4_component/14_Break.en.md
new file mode 100644
index 000000000..6ddf078aa
--- /dev/null
+++ b/documentMD/user_guide/_reference/4_component/14_Break.en.md
@@ -0,0 +1,39 @@
+---
+title: Break
+lang: en
+permalink: /reference/4_component/14_Break.html
+---
+
+
+
+The Break component can only be created directly under a for, while, or foreach component, and can have a condition expression set, similar to the If component.
+
+When the condition set on this component is met, it interrupts the parent component's loop and continues executing the project as if all loops had finished.
+
+When the result of the condition check is false, this component does nothing.
+
+Among the components at the same level as the Break component, the execution order (before or after the condition check) of components that have no dependency on the Break component cannot be specified.
+If you want to control whether a component runs when Break is triggered, set a dependency with the Break component.
+Even if there is no direct dependency, it is fine if they are connected through other components.
+
+The properties you can set for the Break component are as follows.
+
+### condition setting
+Configure the settings for condition evaluation.
+
+#### use javascript expression for condition check
+Similar to the retry decision of the Task component, specifies whether to use a JavaScript expression or a shell script as the condition expression for evaluating true / false.
+
+ - When disabled
+ 
+When disabled, a dropdown list for selecting a shell script is displayed.
+The specified shell script is used as the condition expression to determine true / false.
+
+ - When enabled
+
+When enabled, a JavaScript expression can be entered.
+The entered expression is used as the condition expression to determine true / false.
+
+
+--------
+[Return to Component Details]({{site.baseurl}}/reference/4_component/)
diff --git a/documentMD/user_guide/_reference/4_component/15_HPCISS.en.md b/documentMD/user_guide/_reference/4_component/15_HPCISS.en.md
new file mode 100644
index 000000000..4a10da377
--- /dev/null
+++ b/documentMD/user_guide/_reference/4_component/15_HPCISS.en.md
@@ -0,0 +1,33 @@
+---
+title: HPCI-SS
+lang: en
+permalink: /reference/4_component/15_HPCISS.html
+---
+
+
+
+The HPCI-SS component is a variant of the Storage component that uses HPCI shared storage as the file storage location.
+
+The properties you can set for the HPCI-SS component are as follows.
+
+### host
+You can set the host where files are actually stored.
+However, only hosts that have the `use gfarm` option checked in the remotehost settings can be set as the host.
+
+Additionally, only hosts that can transfer files to HPCI shared storage using gfarm commands (gfcp, gfpcopy, etc.) are available.
+
+### directory path
+
+
+Similar to the Storage component, this is the path where files are actually stored.
+However, you must specify the path on HPCI shared storage, not a path on the host.
+
+### Constraints
+HPCI shared storage does not support overwriting copies to directories that already exist.
+For this reason, when the HPCI-SS component receives a `foo` directory from a preceding component:
+- On the first execution, a `foo` directory is created directly under the path specified in directory path, and the contents of the received `foo` are copied under the `foo` directory.
+- On subsequent executions, a directory named `WHEEL_TMP_XXXXXX` (where XXXXXX is a random string) is created directly under directory path, and the `foo` directory is copied under it.
+
+
+--------
+[Return to Component Details]({{site.baseurl}}/reference/4_component/)
diff --git a/documentMD/user_guide/_reference/4_component/16_HPCISStar.en.md b/documentMD/user_guide/_reference/4_component/16_HPCISStar.en.md
new file mode 100644
index 000000000..b930c7ec1
--- /dev/null
+++ b/documentMD/user_guide/_reference/4_component/16_HPCISStar.en.md
@@ -0,0 +1,37 @@
+---
+title: HPCI-SS-tar
+lang: en
+permalink: /reference/4_component/16_HPCISStar.html
+---
+
+
+
+The HPCI-SS-tar component, like the HPCI-SS component, is a variant of the Storage component for storing files in HPCI shared storage.
+
+Unlike the HPCI-SS component, the HPCI-SS-tar component uses the gfptar command to save files in tar format (gzip compressed) when storing them.
+
+For this reason, unlike the HPCI-SS component, you cannot delete or rename files/directories in the destination HPCI shared storage.
+
+The properties you can set for the HPCI-SS-tar component are as follows.
+
+### host
+You can set the host where files are actually stored.
+However, only hosts that have the `use gfarm` option checked in the remotehost settings can be set as the host.
+
+Additionally, only hosts that can transfer files to HPCI shared storage using the gfptar command are available.
+
+### directory path
+
+
+Similar to the Storage component, this is the path where files are actually stored.
+However, you must specify the path on HPCI shared storage, not a path on the host.
+
+### Constraints
+The HPCI-SS-tar component creates a tar archive with the path name specified in `directory path`.
+Therefore, if a file or directory already exists at directory path, an error will occur.
+When running a project containing this component multiple times, either rewrite directory path or click the `remove storage directory` button on the component property screen to delete the archive directory before running.
+
+
+
+--------
+[Return to Component Details]({{site.baseurl}}/reference/4_component/)
diff --git a/documentMD/user_guide/_reference/4_component/index.en.md b/documentMD/user_guide/_reference/4_component/index.en.md
index af1465750..e6163cc70 100644
--- a/documentMD/user_guide/_reference/4_component/index.en.md
+++ b/documentMD/user_guide/_reference/4_component/index.en.md
@@ -13,9 +13,13 @@ Learn more about each component.
* [For](03_For.html)
* [While](04_while.html)
* [Foreach](05_Foreach.html)
+* [Continue](13_Continue.html)
+* [Break](14_Break.html)
* [PS](06_PS.html)
* [Workflow](07_Workflow.html)
* [Storage](08_Storage.html)
+* [HPCI-SS](15_HPCISS.html)
+* [HPCI-SS-tar](16_HPCISStar.html)
* [Source](09_Source.html)
* [Viewer](10_Viewer.html)
* [Stepjob](11_Stepjob.html)
diff --git a/documentMD/user_guide/_reference/index.md b/documentMD/user_guide/_reference/index.md
index 9f7e725e5..c7d5c70a8 100644
--- a/documentMD/user_guide/_reference/index.md
+++ b/documentMD/user_guide/_reference/index.md
@@ -10,6 +10,8 @@ permalink: /reference/
[2. リモートホスト設定画面](2_remotehost_screen/)
+ * [SSH認証の設定](2_remotehost_screen/ssh_auth/)
+
__3. ワークフロー画面__
* [グラフビュー](3_workflow_screen/1_graphview.html)
diff --git a/package-lock.json b/package-lock.json
index 1e3c7d3c9..2ed89d9e7 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -7054,6 +7054,62 @@
"node": ">= 0.8"
}
},
+ "node_modules/c12": {
+ "version": "3.3.4",
+ "resolved": "https://registry.npmjs.org/c12/-/c12-3.3.4.tgz",
+ "integrity": "sha512-cM0ApFQSBXuourJejzwv/AuPRvAxordTyParRVcHjjtXirtkzM0uK2L9TTn9s0cXZbG7E55jCivRQzoxYmRAlA==",
+ "license": "MIT",
+ "dependencies": {
+ "chokidar": "^5.0.0",
+ "confbox": "^0.2.4",
+ "defu": "^6.1.6",
+ "dotenv": "^17.3.1",
+ "exsolve": "^1.0.8",
+ "giget": "^3.2.0",
+ "jiti": "^2.6.1",
+ "ohash": "^2.0.11",
+ "pathe": "^2.0.3",
+ "perfect-debounce": "^2.1.0",
+ "pkg-types": "^2.3.0",
+ "rc9": "^3.0.1"
+ },
+ "peerDependencies": {
+ "magicast": "*"
+ },
+ "peerDependenciesMeta": {
+ "magicast": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/c12/node_modules/chokidar": {
+ "version": "5.0.0",
+ "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-5.0.0.tgz",
+ "integrity": "sha512-TQMmc3w+5AxjpL8iIiwebF73dRDF4fBIieAqGn9RGCWaEVwQ6Fb2cGe31Yns0RRIzii5goJ1Y7xbMwo1TxMplw==",
+ "license": "MIT",
+ "dependencies": {
+ "readdirp": "^5.0.0"
+ },
+ "engines": {
+ "node": ">= 20.19.0"
+ },
+ "funding": {
+ "url": "https://paulmillr.com/funding/"
+ }
+ },
+ "node_modules/c12/node_modules/readdirp": {
+ "version": "5.0.0",
+ "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-5.0.0.tgz",
+ "integrity": "sha512-9u/XQ1pvrQtYyMpZe7DXKv2p5CNvyVwzUB6uhLAnQwHMSgKMBR62lc7AHljaeteeHXn11XTAaLLUVZYVZyuRBQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 20.19.0"
+ },
+ "funding": {
+ "type": "individual",
+ "url": "https://paulmillr.com/funding/"
+ }
+ },
"node_modules/cacache": {
"version": "15.3.0",
"resolved": "https://registry.npmjs.org/cacache/-/cacache-15.3.0.tgz",
@@ -7964,6 +8020,12 @@
"integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==",
"license": "MIT"
},
+ "node_modules/confbox": {
+ "version": "0.2.4",
+ "resolved": "https://registry.npmjs.org/confbox/-/confbox-0.2.4.tgz",
+ "integrity": "sha512-ysOGlgTFbN2/Y6Cg3Iye8YKulHw+R2fNXHrgSmXISQdMnomY6eNDprVdW9R5xBguEqI954+S6709UyiO7B+6OQ==",
+ "license": "MIT"
+ },
"node_modules/configstore": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/configstore/-/configstore-5.0.1.tgz",
@@ -9011,6 +9073,12 @@
"url": "https://github.com/sponsors/ljharb"
}
},
+ "node_modules/defu": {
+ "version": "6.1.7",
+ "resolved": "https://registry.npmjs.org/defu/-/defu-6.1.7.tgz",
+ "integrity": "sha512-7z22QmUWiQ/2d0KkdYmANbRUVABpZ9SNYyH5vx6PZ+nE5bcC0l7uFvEfHlyld/HcGBFTL536ClDt3DEcSlEJAQ==",
+ "license": "MIT"
+ },
"node_modules/degenerator": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/degenerator/-/degenerator-5.0.1.tgz",
@@ -9050,6 +9118,12 @@
"node": ">= 0.8"
}
},
+ "node_modules/destr": {
+ "version": "2.0.5",
+ "resolved": "https://registry.npmjs.org/destr/-/destr-2.0.5.tgz",
+ "integrity": "sha512-ugFTXCtDZunbzasqBxrK93Ik/DRYsO6S/fedkWEMKqt04xZ4csmnmwGDBAb07QWNaGMAmnTIemsYZCksjATwsA==",
+ "license": "MIT"
+ },
"node_modules/destroy": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/destroy/-/destroy-1.2.0.tgz",
@@ -9107,10 +9181,9 @@
}
},
"node_modules/dotenv": {
- "version": "17.2.3",
- "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.2.3.tgz",
- "integrity": "sha512-JVUnt+DUIzu87TABbhPmNfVdBDt18BLOWjMUFJMSi/Qqg7NTYtabbvSNJGOJ7afbRuv9D/lngizHtP7QyLQ+9w==",
- "dev": true,
+ "version": "17.4.2",
+ "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.4.2.tgz",
+ "integrity": "sha512-nI4U3TottKAcAD9LLud4Cb7b2QztQMUEfHbvhTH09bqXTxnSie8WnjPALV/WMCrJZ6UV/qHJ6L03OqO3LcdYZw==",
"license": "BSD-2-Clause",
"engines": {
"node": ">=12"
@@ -10547,6 +10620,12 @@
"node": ">=6.6.0"
}
},
+ "node_modules/exsolve": {
+ "version": "1.0.8",
+ "resolved": "https://registry.npmjs.org/exsolve/-/exsolve-1.0.8.tgz",
+ "integrity": "sha512-LmDxfWXwcTArk8fUEnOfSZpHOJ6zOMUJKOtFLFqJLoKJetuQG874Uc7/Kki7zFLzYybmZhp1M7+98pfMqeX8yA==",
+ "license": "MIT"
+ },
"node_modules/extend": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz",
@@ -11444,6 +11523,15 @@
"assert-plus": "^1.0.0"
}
},
+ "node_modules/giget": {
+ "version": "3.2.0",
+ "resolved": "https://registry.npmjs.org/giget/-/giget-3.2.0.tgz",
+ "integrity": "sha512-GvHTWcykIR/fP8cj8dMpuMMkvaeJfPvYnhq0oW+chSeIr+ldX21ifU2Ms6KBoyKZQZmVaUAAhQ2EZ68KJF8a7A==",
+ "license": "MIT",
+ "bin": {
+ "giget": "dist/cli.mjs"
+ }
+ },
"node_modules/github-from-package": {
"version": "0.0.0",
"resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz",
@@ -13449,6 +13537,15 @@
"url": "https://github.com/chalk/supports-color?sponsor=1"
}
},
+ "node_modules/jiti": {
+ "version": "2.7.0",
+ "resolved": "https://registry.npmjs.org/jiti/-/jiti-2.7.0.tgz",
+ "integrity": "sha512-AC/7JofJvZGrrneWNaEnJeOLUx+JlGt7tNa0wZiRPT4MY1wmfKjt2+6O2p2uz2+skll8OZZmJMNqeke7kKbNgQ==",
+ "license": "MIT",
+ "bin": {
+ "jiti": "lib/jiti-cli.mjs"
+ }
+ },
"node_modules/joi": {
"version": "18.1.2",
"resolved": "https://registry.npmjs.org/joi/-/joi-18.1.2.tgz",
@@ -16642,7 +16739,6 @@
"version": "2.0.11",
"resolved": "https://registry.npmjs.org/ohash/-/ohash-2.0.11.tgz",
"integrity": "sha512-RdR9FQrFwNBNXAr4GixM8YaRZRJ5PUWbKYbE5eOsrwAjJW0q2REGcf79oYPsLyskQCZG1PLN+S/K1V00joZAoQ==",
- "dev": true,
"license": "MIT"
},
"node_modules/on-finished": {
@@ -17380,7 +17476,6 @@
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz",
"integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==",
- "dev": true,
"license": "MIT"
},
"node_modules/pause": {
@@ -17525,6 +17620,17 @@
"node": ">=8"
}
},
+ "node_modules/pkg-types": {
+ "version": "2.3.1",
+ "resolved": "https://registry.npmjs.org/pkg-types/-/pkg-types-2.3.1.tgz",
+ "integrity": "sha512-y+ichcgc2LrADuhLNAx8DFjVfgz91pRxfZdI3UDhxHvcVEZsenLO+7XaU5vOp0u/7V/wZ+plyuQxtrDlZJ+yeg==",
+ "license": "MIT",
+ "dependencies": {
+ "confbox": "^0.2.4",
+ "exsolve": "^1.0.8",
+ "pathe": "^2.0.3"
+ }
+ },
"node_modules/plist": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/plist/-/plist-1.2.0.tgz",
@@ -17983,6 +18089,16 @@
"node": ">=0.10.0"
}
},
+ "node_modules/rc9": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/rc9/-/rc9-3.0.1.tgz",
+ "integrity": "sha512-gMDyleLWVE+i6Sgtc0QbbY6pEKqYs97NGi6isHQPqYlLemPoO8dxQ3uGi0f4NiP98c+jMW6cG1Kx9dDwfvqARQ==",
+ "license": "MIT",
+ "dependencies": {
+ "defu": "^6.1.6",
+ "destr": "^2.0.5"
+ }
+ },
"node_modules/read-pkg": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/read-pkg/-/read-pkg-3.0.0.tgz",
@@ -22719,6 +22835,7 @@
"ajv-formats": "^3.0.1",
"axios": "^1.13.1",
"body-parser": "^2.2.0",
+ "c12": "^3.3.4",
"connect-ensure-login": "^0.1.1",
"connect-sqlite3": "^0.9.15",
"cookie-parser": "^1.4.7",
diff --git a/server/app/core/componentJsonIO.js b/server/app/core/componentJsonIO.js
index 7b231283b..064ec82bf 100644
--- a/server/app/core/componentJsonIO.js
+++ b/server/app/core/componentJsonIO.js
@@ -26,7 +26,7 @@ const _internal = {
* @returns {string | undefined} -
*/
function componentJsonReplacer(key, value) {
- if (["handler", "doCleanup", "sbsID", "childLoopRunning"].includes(key)) {
+ if (["handler", "doCleanup", "sbsID", "childLoopRunning", "restartChecked"].includes(key)) {
return undefined;
}
return value;
diff --git a/server/app/core/deliverFile.js b/server/app/core/deliverFile.js
index 67aa02727..35076d29e 100644
--- a/server/app/core/deliverFile.js
+++ b/server/app/core/deliverFile.js
@@ -188,12 +188,17 @@ async function deliverFilesBetweenRemotes(recipe) {
logger.debug("direct remote to remote copy from", recipe.srcRemotehostID, "to", recipe.dstRemotehostID);
- await srcSsh.remoteToRemoteCopy(
+ const rt = await srcSsh.remoteToRemoteCopy(
[`${recipe.srcRoot}/${recipe.srcName}`],
dstHostinfo,
`${recipe.dstRoot}/${recipe.dstName}`,
["-vv", ...rsyncExcludeOptionOfWheelSystemFiles]
);
+ if (rt !== 0) {
+ const err = new Error(`direct remote to remote copy failed with exit code ${rt}`);
+ err.rt = rt;
+ return Promise.reject(err);
+ }
return { type: "direct-remote-copy", src: `${recipe.srcRoot}/${recipe.srcName}`, dst: `${recipe.dstRoot}/${recipe.dstName}` };
}
diff --git a/server/app/core/dispatcher.js b/server/app/core/dispatcher.js
index 8fd1751c7..9fc4859fd 100644
--- a/server/app/core/dispatcher.js
+++ b/server/app/core/dispatcher.js
@@ -31,7 +31,8 @@ import {
logDebug,
logInfo,
logWarn,
- logError
+ logError,
+ _internal
} from "../logSettings.js";
import { cancelDispatchedTasks } from "./taskUtil.js";
import { eventEmitters } from "./global.js";
@@ -78,22 +79,26 @@ const taskDB = new Map();
* @param {string} target - absolute path to check
* @param {string} baseDir - base directory path
* @param {Array} skipCopyPatterns - array of glob patterns
- * @returns {boolean} - true if target matches any pattern
+ * @returns {Promise} - true if target matches any pattern
*/
-function shouldSkipCopy(target, baseDir, skipCopyPatterns) {
+async function shouldSkipCopy(target, baseDir, skipCopyPatterns) {
if (!skipCopyPatterns || skipCopyPatterns.length === 0) {
return false;
}
const relativePath = path.relative(baseDir, target);
- return skipCopyPatterns.some((pattern)=>{
+ for (const pattern of skipCopyPatterns) {
if (hasMagic(pattern)) {
- const matched = glob.sync(pattern, { cwd: baseDir });
- return matched.some((matchedPath)=>{
+ const matched = await glob(pattern, { cwd: baseDir });
+ if (matched.some((matchedPath)=>{
return relativePath === matchedPath || relativePath.startsWith(`${matchedPath}${path.sep}`);
- });
+ })) {
+ return true;
+ }
+ } else if (relativePath === pattern || relativePath.startsWith(`${pattern}${path.sep}`)) {
+ return true;
}
- return relativePath === pattern || relativePath.startsWith(`${pattern}${path.sep}`);
- });
+ }
+ return false;
}
/**
@@ -327,6 +332,7 @@ class Dispatcher extends EventEmitter {
continue;
}
await this._getInputFiles(target);
+ await this._warnMissingInputFiles(target);
if (!await this._checkMandatoryInputFilesExist(target)) {
await this._setComponentState(target, "failed");
this.hasFailedComponent = true;
@@ -810,7 +816,7 @@ class Dispatcher extends EventEmitter {
if (path.basename(target) === statusFilename) {
return false;
}
- if (shouldSkipCopy(target, srcDir, skipCopyPatterns)) {
+ if (await shouldSkipCopy(target, srcDir, skipCopyPatterns)) {
logTrace(this.projectRootDir, `${this.cwfDir}/${component.name}`, "skipping copy due to skipCopy pattern:", target);
return false;
}
@@ -1390,6 +1396,49 @@ class Dispatcher extends EventEmitter {
ee.emit("componentStateChanged", component);
}
+ /**
+ * Check non-mandatory inputFiles and warn if any are missing after file staging.
+ * Unlike mandatory inputFiles, missing non-mandatory files do not fail the component —
+ * a warning is logged and a toast message is sent to the client.
+ * @param {object} component - component to check
+ * @returns {Promise}
+ */
+ async _warnMissingInputFiles(component) {
+ if (!component.inputFiles) {
+ return;
+ }
+ const componentDir = this._getComponentDir(component.ID);
+ const remotehostID = isLocal(component) ? null : remoteHost.getID("name", component.host);
+ const ssh = remotehostID ? getSsh(this.projectRootDir, remotehostID) : null;
+ const remoteWorkingDir = remotehostID
+ ? getRemoteWorkingDir(this.projectRootDir, this.projectStartTime, path.resolve(this.cwfDir, component.name), component)
+ : null;
+
+ for (const inputFile of component.inputFiles) {
+ if (inputFile.mandatory === true) {
+ continue;
+ }
+ const renderedName = nunjucks.renderString(inputFile.name, this.env);
+ let missing = false;
+ try {
+ if (isLocal(component)) {
+ await fs.stat(path.join(componentDir, renderedName));
+ } else {
+ const rt = await ssh.exec(`test -e ${path.join(remoteWorkingDir, renderedName)}`, 0, logTrace.bind(null, this.projectRootDir, `${this.cwfDir}/${component.name}`));
+ if (rt !== 0) {
+ missing = true;
+ }
+ }
+ } catch (e) {
+ missing = true;
+ }
+ if (missing) {
+ logWarn(this.projectRootDir, `${this.cwfDir}/${component.name}`, "inputFile not found:", renderedName);
+ await _internal.emitAll(this.projectRootDir, "showMessage", `[${component.name}] inputFile not found: ${renderedName}`);
+ }
+ }
+ }
+
/**
* check if all mandatory inputFiles exist on the host (local or remote) after file staging
* @param {object} component - component to check
@@ -1429,6 +1478,13 @@ class Dispatcher extends EventEmitter {
return true;
}
+ /**
+ * Fetch all inputFiles for a component, delivering them from their sources.
+ * Non-mandatory transfer failures are logged as warnings and ignored.
+ * Mandatory transfer failures cause all remaining transfers to complete before the component is failed.
+ * @param {object} component - the component whose inputFiles should be fetched
+ * @returns {Promise} - array of delivery results from successful transfers
+ */
async _getInputFiles(component) {
if (component.type === "source") {
return;
@@ -1438,6 +1494,7 @@ class Dispatcher extends EventEmitter {
const tmpDeliverRecipes = [];
for (const inputFile of component.inputFiles) {
const dstName = nunjucks.renderString(inputFile.name, this.env);
+ const mandatory = inputFile.mandatory === true;
//resolve real src
for (const src of inputFile.src) {
const srcComponent = await this._getComponent(src.srcNode);
@@ -1477,7 +1534,8 @@ class Dispatcher extends EventEmitter {
projectRootDir: this.projectRootDir,
srcRemotehostID,
fromHPCISS,
- fromHPCISStar
+ fromHPCISStar,
+ mandatory
});
}
} else if (onSameRemote) {
@@ -1523,7 +1581,8 @@ class Dispatcher extends EventEmitter {
projectRootDir: this.projectRootDir,
dstRemotehostID,
fromHPCISS,
- fromHPCISStar
+ fromHPCISStar,
+ mandatory
});
} else if (!srcIsLocal && dstIsLocal) {
//Remote → Localhost via shared storage
@@ -1562,7 +1621,8 @@ class Dispatcher extends EventEmitter {
projectRootDir: this.projectRootDir,
srcRemotehostID: srcRemotehostIDForRecipe,
fromHPCISS,
- fromHPCISStar
+ fromHPCISStar,
+ mandatory
});
} else {
//Remote → Remote
@@ -1584,7 +1644,8 @@ class Dispatcher extends EventEmitter {
srcRemotehostID,
dstRemotehostID: remotehostID,
fromHPCISS,
- fromHPCISStar
+ fromHPCISStar,
+ mandatory
});
}
} else if (!isLocal(srcComponent) && !["task", "stepjobTask", "bulkjobtask", "hpciss", "hpcisstar"].includes(srcComponent.type)) {
@@ -1603,7 +1664,8 @@ class Dispatcher extends EventEmitter {
projectRootDir: this.projectRootDir,
srcRemotehostID,
fromHPCISS,
- fromHPCISStar
+ fromHPCISStar,
+ mandatory
});
} else {
//deliver files under component directory even if destination component is storage
@@ -1634,7 +1696,8 @@ class Dispatcher extends EventEmitter {
projectRootDir: this.projectRootDir,
srcRemotehostID,
fromHPCISS,
- fromHPCISStar
+ fromHPCISStar,
+ mandatory
});
}
} else {
@@ -1653,7 +1716,8 @@ class Dispatcher extends EventEmitter {
projectRootDir: this.projectRootDir,
srcRemotehostID,
fromHPCISS,
- fromHPCISStar
+ fromHPCISStar,
+ mandatory
});
}
})
@@ -1664,13 +1728,17 @@ class Dispatcher extends EventEmitter {
await Promise.all(promises);
const deliverRecipes = [];
for (const recipe of tmpDeliverRecipes) {
- if (!deliverRecipes.some((e)=>{
+ const existingIdx = deliverRecipes.findIndex((e)=>{
return e.dstRoot === recipe.dstRoot
&& e.dstName === recipe.dstName
&& e.srcRoot === recipe.srcRoot
&& e.srcName === recipe.srcName;
- })) {
+ });
+ if (existingIdx === -1) {
deliverRecipes.push(recipe);
+ } else if (recipe.mandatory) {
+ //if any source marks this transfer as mandatory, keep it mandatory
+ deliverRecipes[existingIdx].mandatory = true;
}
}
@@ -1678,17 +1746,17 @@ class Dispatcher extends EventEmitter {
const p2 = [];
for (const recipe of deliverRecipes) {
if (recipe.fromHPCISS || recipe.fromHPCISStar) {
- p2.push(deliverFilesFromHPCISS(recipe, this.projectRootDir));
+ p2.push({ promise: deliverFilesFromHPCISS(recipe, this.projectRootDir), mandatory: recipe.mandatory });
} else if (recipe.onSameRemote) {
- p2.push(deliverFilesOnRemote(recipe));
+ p2.push({ promise: deliverFilesOnRemote(recipe), mandatory: recipe.mandatory });
} else if (recipe.betweenRemotes) {
- p2.push(deliverFilesBetweenRemotes(recipe));
+ p2.push({ promise: deliverFilesBetweenRemotes(recipe), mandatory: recipe.mandatory });
} else if (recipe.localToRemoteShared) {
- p2.push(deliverFilesLocalToRemoteShared(recipe));
+ p2.push({ promise: deliverFilesLocalToRemoteShared(recipe), mandatory: recipe.mandatory });
} else if (recipe.remoteToLocalShared) {
- p2.push(deliverFilesRemoteToLocalShared(recipe));
+ p2.push({ promise: deliverFilesRemoteToLocalShared(recipe), mandatory: recipe.mandatory });
} else if (recipe.remoteToLocal) {
- p2.push(deliverFilesFromRemote(recipe));
+ p2.push({ promise: deliverFilesFromRemote(recipe), mandatory: recipe.mandatory });
} else {
const srces = await glob(recipe.srcName, { cwd: recipe.srcRoot });
const hasGlob = hasMagic(recipe.srcName);
@@ -1703,13 +1771,30 @@ class Dispatcher extends EventEmitter {
if (hasGlob || recipe.dstName.endsWith(path.posix.sep) || recipe.dstName.endsWith(path.win32.sep)) {
newPath = path.resolve(newPath, srcFile);
}
- p2.push(deliverFile(oldPath, newPath, recipe.forceCopy));
+ p2.push({ promise: deliverFile(oldPath, newPath, recipe.forceCopy), mandatory: recipe.mandatory });
}
}
}
- const results = await Promise.all(p2);
- for (const result of results) {
- logTrace(this.projectRootDir, `${this.cwfDir}/${component.name}`, "make", result.type, "from ", result.src, "to", result.dst);
+ const settled = await Promise.allSettled(p2.map((e)=>{
+ return e.promise;
+ }));
+ const results = [];
+ const mandatoryErrors = [];
+ for (const [i, outcome] of settled.entries()) {
+ if (outcome.status === "fulfilled") {
+ logTrace(this.projectRootDir, `${this.cwfDir}/${component.name}`, "make", outcome.value.type, "from ", outcome.value.src, "to", outcome.value.dst);
+ results.push(outcome.value);
+ } else if (p2[i].mandatory) {
+ logWarn(this.projectRootDir, `${this.cwfDir}/${component.name}`, "mandatory inputFile transfer failed:", outcome.reason);
+ mandatoryErrors.push(outcome.reason);
+ } else {
+ logWarn(this.projectRootDir, `${this.cwfDir}/${component.name}`, "non-mandatory inputFile transfer failed (ignored):", outcome.reason);
+ }
+ }
+ if (mandatoryErrors.length > 0) {
+ throw new Error(`mandatory inputFile transfer failed: ${mandatoryErrors.map((e)=>{
+ return e.message;
+ }).join(", ")}`);
}
if (component.type === "viewer") {
component.files = results;
diff --git a/server/app/core/executerManager.js b/server/app/core/executerManager.js
index dddedb0c8..cd617045e 100644
--- a/server/app/core/executerManager.js
+++ b/server/app/core/executerManager.js
@@ -8,7 +8,7 @@ import childProcess from "child_process";
import axios from "axios";
import { getAccessToken } from "./webAPI.js";
import SBS from "simple-batch-system";
-import { remoteHost, jobScheduler, numJobOnLocal, defaultTaskRetryCount } from "../db/db.js";
+import { remoteHost, jobScheduler, numLocalJob, defaultTaskRetryCount } from "../db/db.js";
import { addX } from "./fileUtils.js";
import { evalCondition } from "./dispatchUtils.js";
import { getDateString } from "../lib/utility.js";
@@ -24,7 +24,7 @@ const _internal = {
executers: new Map(),
getSshHostinfo,
jobScheduler,
- numJobOnLocal,
+ numLocalJob,
remoteHost
};
@@ -553,7 +553,7 @@ function getExecutersKey(task) {
*/
function getMaxNumJob(hostinfo) {
if (hostinfo === null) {
- return _internal.numJobOnLocal;
+ return _internal.numLocalJob;
}
if (!Number.isNaN(parseInt(hostinfo.numJob, 10))) {
return Math.max(parseInt(hostinfo.numJob, 10), 1);
@@ -686,4 +686,4 @@ export {
_internal
};
-export { numJobOnLocal };
+export { numLocalJob };
diff --git a/server/app/core/exportComponent.js b/server/app/core/exportComponent.js
index 810fb988e..bab76ff2e 100644
--- a/server/app/core/exportComponent.js
+++ b/server/app/core/exportComponent.js
@@ -11,6 +11,7 @@ import { createTempd } from "./tempd.js";
import { getComponentDir } from "./componentJsonIO.js";
import { gitClone } from "./gitOperator2.js";
import { componentJsonFilename } from "../db/db.js";
+import { baseURL } from "./global.js";
const { create } = tar;
@@ -141,7 +142,6 @@ async function exportComponent(projectRootDir, componentID) {
[componentBasename]
);
- const baseURL = process.env.WHEEL_BASE_URL || "";
//Get path relative to tempdRoot, then prepend /exportComponent/
const relativePath = path.relative(root, archiveFilename);
const url = `${baseURL}/exportComponent/${relativePath}`;
diff --git a/server/app/core/exportProject.js b/server/app/core/exportProject.js
index d8190560e..d9f797c3b 100644
--- a/server/app/core/exportProject.js
+++ b/server/app/core/exportProject.js
@@ -9,6 +9,7 @@ import * as tar from "tar";
import { createTempd } from "./tempd.js";
import { readJsonGreedy } from "./fileUtils.js";
import { projectJsonFilename } from "../db/db.js";
+import { baseURL } from "./global.js";
import { gitAdd, gitClone, gitCommit, gitConfig, gitRemoveOrigin } from "./gitOperator2.js";
import { setComponentStateR } from "./componentState.js";
@@ -77,7 +78,6 @@ async function exportProject(projectRootDir, name = null, mail = null, memo = nu
[`${projectJson.name}.wheel`]
);
- const baseURL = process.env.WHEEL_BASE_URL || "";
const url = `${baseURL}/${path.join(path.relative(path.dirname(dir), archiveFilename))}`;
return url;
}
diff --git a/server/app/core/global.js b/server/app/core/global.js
index eec345264..8f8e04699 100644
--- a/server/app/core/global.js
+++ b/server/app/core/global.js
@@ -6,7 +6,7 @@
"use strict";
import debugLib from "debug";
const debug = debugLib("wheel");
-const baseURL = process.env.WHEEL_BASE_URL || "/";
+export { baseURL } from "../db/db.js";
const parentDirs = new Map(); //workflow path which is displayed on graphview
const eventEmitters = new Map(); //event emitter object which is used to communicate while running project
const watchers = new Map(); //result file watcher
@@ -38,6 +38,5 @@ export {
watchers,
checkWritePermissions,
setSio,
- getSio,
- baseURL
+ getSio
};
diff --git a/server/app/core/migrationHelper.js b/server/app/core/migrationHelper.js
new file mode 100644
index 000000000..ac39a144b
--- /dev/null
+++ b/server/app/core/migrationHelper.js
@@ -0,0 +1,107 @@
+/*
+ * Copyright (c) Center for Computational Science, RIKEN All rights reserved.
+ * Copyright (c) Research Institute for Information Technology(RIIT), Kyushu University. All rights reserved.
+ * See License in the project root for the license information.
+ */
+import os from "os";
+import path from "path";
+import fs from "fs-extra";
+
+/**
+ * Map of old server.json property names to their new names.
+ * Used by migrateConfigFile to rewrite user config files at startup.
+ */
+const PROP_RENAMES = {
+ numJobOnLocal: "numLocalJob"
+};
+
+/**
+ * Map of deprecated environment variable names to their replacements.
+ * Used by warnDeprecatedEnvVars to alert users at startup.
+ * These variables no longer map to any config property and have no effect.
+ */
+const DEPRECATED_ENV_VARS = {
+ WHEEL_LOGLEVEL: "WHEEL_LOG_LEVEL"
+};
+
+/**
+ * Migrate old property names in a JSON config file to new names.
+ * Skips silently if the file does not exist (ENOENT).
+ * If both the old and new property are present, keeps the new value and removes the old.
+ * @param {string} filePath - absolute path to the JSON config file
+ * @returns {Promise}
+ */
+async function migrateConfigFile(filePath) {
+ let raw;
+ try {
+ raw = await fs.readFile(filePath, "utf8");
+ } catch (e) {
+ if (e.code === "ENOENT") {
+ return;
+ }
+ throw e;
+ }
+
+ let obj;
+ try {
+ obj = JSON.parse(raw);
+ } catch (e) {
+ console.warn(`[WHEEL migration] Warning: could not parse ${filePath} — skipping migration: ${e.message}`);
+ return;
+ }
+
+ let changed = false;
+ for (const [oldProp, newProp] of Object.entries(PROP_RENAMES)) {
+ if (Object.prototype.hasOwnProperty.call(obj, oldProp)) {
+ if (Object.prototype.hasOwnProperty.call(obj, newProp)) {
+ console.warn(`[WHEEL migration] ${filePath}: both "${oldProp}" and "${newProp}" found. Keeping "${newProp}" and removing "${oldProp}".`);
+ } else {
+ console.warn(`[WHEEL migration] ${filePath}: renamed property "${oldProp}" → "${newProp}".`);
+ obj[newProp] = obj[oldProp];
+ }
+ delete obj[oldProp];
+ changed = true;
+ }
+ }
+
+ if (changed) {
+ await fs.writeFile(filePath, `${JSON.stringify(obj, null, 2)}\n`, "utf8");
+ }
+}
+
+/**
+ * Check environment variables for deprecated WHEEL_ names and warn via console.warn.
+ * These variables are no longer recognized by the auto-mapper and have no effect.
+ * @returns {void}
+ */
+function warnDeprecatedEnvVars() {
+ for (const [oldVar, newVar] of Object.entries(DEPRECATED_ENV_VARS)) {
+ if (Object.prototype.hasOwnProperty.call(process.env, oldVar)) {
+ console.warn(`[WHEEL migration] Deprecated environment variable "${oldVar}" is set but has no effect. Use "${newVar}" instead.`);
+ }
+ }
+}
+
+/**
+ * Run all startup migrations:
+ * 1. Migrate server.json property names in user config directories.
+ * 2. Warn about deprecated environment variables.
+ * Looks for server.json in WHEEL_CONFIG_DIR (if set) and ~/.wheel/.
+ * @returns {Promise}
+ */
+async function runMigrations() {
+ const configDirs = [
+ path.resolve(os.homedir(), ".wheel")
+ ];
+ if (typeof process.env.WHEEL_CONFIG_DIR === "string") {
+ configDirs.push(path.resolve(process.env.WHEEL_CONFIG_DIR));
+ }
+
+ for (const dir of configDirs) {
+ await migrateConfigFile(path.join(dir, "server.json"));
+ }
+
+ warnDeprecatedEnvVars();
+}
+
+export { migrateConfigFile, warnDeprecatedEnvVars, runMigrations, PROP_RENAMES, DEPRECATED_ENV_VARS };
diff --git a/server/app/core/sshManager.js b/server/app/core/sshManager.js
index e74faa79c..08d040567 100644
--- a/server/app/core/sshManager.js
+++ b/server/app/core/sshManager.js
@@ -5,6 +5,7 @@
*/
import SshClientWrapper from "ssh-client-wrapper";
import { emitAll } from "../handlers/commUtils.js";
+import { verboseSsh } from "../db/db.js";
const _internal = {
db: new Map(),
@@ -204,7 +205,7 @@ async function createSsh(projectRootDir, remoteHostName, hostinfo, clientID, isS
if (hostinfo.readyTimeout) {
hostinfo.ConnectTimeout = Math.floor(hostinfo.readyTimeout / 1000);
}
- if (process.env.WHEEL_VERBOSE_SSH) {
+ if (verboseSsh) {
hostinfo.sshOpt = ["-vvv"];
}
if (hostinfo.username) {
diff --git a/server/app/core/versionInfo.js b/server/app/core/versionInfo.js
index 45c330f48..ed9196c42 100644
--- a/server/app/core/versionInfo.js
+++ b/server/app/core/versionInfo.js
@@ -6,6 +6,7 @@
import versionData from "../db/version.json" with { type: "json" };
const { version } = versionData;
import { getLogger } from "../logSettings.js";
+import { port, baseURL, useHttp, acceptAddress, logLevel, verboseSsh, numLocalJob, enableWebApi, enableAuth } from "../db/db.js";
/**
* print version and environment variables
@@ -13,22 +14,21 @@ import { getLogger } from "../logSettings.js";
*/
function aboutWheel(projectRootDir) {
const logger = getLogger(projectRootDir);
- const baseURL = process.env.WHEEL_BASE_URL || "/";
logger.info(`starting WHEEL server (version ${version})`);
logger.info("base URL = ", baseURL);
- logger.info("environment variables");
+ logger.info("effective configuration");
logger.info(`WHEEL_TEMPD = ${process.env.WHEEL_TEMPD}`);
logger.info(`WHEEL_CONFIG_DIR = ${process.env.WHEEL_CONFIG_DIR}`);
- logger.info(`WHEEL_BASE_URL= ${process.env.WHEEL_USE_HTTP}`);
- logger.info(`WHEEL_USE_HTTP = ${process.env.WHEEL_USE_HTTP}`);
- logger.info(`WHEEL_PORT = ${process.env.WHEEL_PORT}`);
- logger.info(`WHEEL_ACCEPT_ADDRESS = ${process.env.WHEEL_ACCEPT_ADDRESS}`);
- logger.info(`WHEEL_LOGLEVEL = ${process.env.WHEEL_LOGLEVEL}`);
- logger.info(`WHEEL_VERBOSE_SSH = ${process.env.WHEEL_VERBOSE_SSH}`);
- logger.info(`WHEEL_INTERVAL = ${process.env.WHEEL_INTERVAL}`);
- logger.info(`WHEEL_NUM_LOCAL_JOB = ${process.env.WHEEL_NUM_LOCAL_JOB}`);
- logger.info(`WHEEL_ENABLE_WEB_API = ${process.env.WHEEL_ENABLE_WEB_API}`);
+ logger.info(`WHEEL_BASE_URL = ${baseURL}`);
+ logger.info(`WHEEL_USE_HTTP = ${useHttp}`);
+ logger.info(`WHEEL_PORT = ${port}`);
+ logger.info(`WHEEL_ACCEPT_ADDRESS = ${acceptAddress}`);
+ logger.info(`WHEEL_LOG_LEVEL = ${logLevel}`);
+ logger.info(`WHEEL_VERBOSE_SSH = ${verboseSsh}`);
+ logger.info(`WHEEL_NUM_LOCAL_JOB = ${numLocalJob}`);
+ logger.info(`WHEEL_ENABLE_WEB_API = ${enableWebApi}`);
+ logger.info(`WHEEL_ENABLE_AUTH = ${enableAuth}`);
}
export {
diff --git a/server/app/db/db.js b/server/app/db/db.js
index afa605afc..1597f2901 100644
--- a/server/app/db/db.js
+++ b/server/app/db/db.js
@@ -6,8 +6,10 @@
import os from "os";
import path from "path";
import fs from "fs-extra";
+import { loadConfig } from "c12";
import JsonArrayManager from "./jsonArrayManager.js";
import { fileURLToPath } from "url";
+import { runMigrations } from "../core/migrationHelper.js";
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
@@ -76,7 +78,71 @@ function getConfigFile(filename, failIfNotFound) {
}
/**
- * return value or alternate value if it is nudefined
+ * convert UPPER_SNAKE_CASE string to camelCase
+ * @param {string} str - UPPER_SNAKE_CASE string
+ * @returns {string} - camelCase string
+ */
+function toCamelCase(str) {
+ return str.toLowerCase().replace(/_([a-z])/g, (_, c)=>{
+ return c.toUpperCase();
+ });
+}
+
+/**
+ * coerce a string value to boolean, number, string, or undefined as appropriate.
+ * Returns undefined for empty or whitespace-only strings so that callers can
+ * fall back to configured defaults instead of overriding them with an empty value.
+ * @param {string} value - string value from environment variable
+ * @returns {boolean|number|string|undefined} - coerced value, or undefined for blank strings
+ */
+function coerce(value) {
+ if (value === "true") {
+ return true;
+ }
+ if (value === "false") {
+ return false;
+ }
+ const trimmed = value.trim();
+ if (trimmed === "") {
+ return undefined;
+ }
+ if (!Number.isNaN(Number(trimmed))) {
+ return Number(trimmed);
+ }
+ return value;
+}
+
+/**
+ * env vars excluded from auto-mapping (infrastructure/path settings needed before config loads,
+ * or vars that conflict with existing TLS cert export names)
+ */
+const INFRASTRUCTURE_ENV_VARS = new Set([
+ "WHEEL_CONFIG_DIR",
+ "WHEEL_TEMPD",
+ "WHEEL_USER_DB_DIR",
+ "WHEEL_SESSION_DB_DIR",
+ "WHEEL_CLEAR_SESSION_DB",
+ "WHEEL_CERT_FILENAME",
+ "WHEEL_CERT_PASSPHRASE"
+]);
+
+/**
+ * extract WHEEL_ prefixed environment variables and map them to camelCase config overrides.
+ * UPPER_SNAKE_CASE after stripping WHEEL_ prefix is converted to camelCase.
+ * String values are auto-coerced to boolean or number where appropriate.
+ * @returns {object} - config overrides from environment variables
+ */
+function extractWheelEnvOverrides() {
+ const overrides = {};
+ for (const [key, value] of Object.entries(process.env)) {
+ if (key.startsWith("WHEEL_") && !INFRASTRUCTURE_ENV_VARS.has(key)) {
+ overrides[toCamelCase(key.slice(6))] = coerce(value);
+ }
+ }
+ return overrides;
+}
+
+/**
* @param {*} target - variable to be checked
* @param {*} alt - alternate value
* @returns {*} -
@@ -106,30 +172,54 @@ function getStringVar(target, alt) {
}
/**
- * read default and userdefined config file and merge them
- * @param {string} filename - config file's name
- * @returns {Promise