File path error when running through singularity/apptainer

[steob92]

When defining temporary file paths, we're not checking if for if the analysis will be ran through apptainer:

Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.anasum_sub.sh

Lines 209 to 213 in 7694c0a

	cp -v "$FLIST" "$TEMPDIR"
	FLIST="${TEMPDIR}/$(basename $FLIST)"
	cp -v "$RUNP" "$TEMPDIR"
	cp -v $(dirname $RUNP)/$(grep TIMEMASKFILE $RUNP | awk '{print $3}') "$TEMPDIR"
	RUNP="${TEMPDIR}/$(basename $RUNP)"

As a result apptainer will search /opt/TEMPDIR/path/to/local/tmp/dir/ for temporary run parameters and runlists.

Need to check before running anasum with something like:

if [ -n "$EVNDISP_APPTAINER" ]; then
    RUNP="/opt/TEMPDIR/$(basename $RUNP)"
    FLIST="/opt/TEMPDIR/$(basename $FLIST)"
fi

There might be similar issues for evndisp and mscw stage processing.
ToDo:

• Check helper scripts for similar issues.

[steob92]

This seems to resolve the anasum issue: 4cba8aa

mscw stage also seems to run ok, no changes needed. I am getting errors with evndisp relating to:
VERITAS_DATA_DIR_2=/lustre/fs23/group/veritas/data

This looks related to DESY preprocessing and it is getting filled from:

Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.evndisp.sh

Line 224 in 4cba8aa

-e "s|VTS_2DATA_DIR|/lustre/fs23/group/veritas/data|" \

Since I don't want to break preprocessing, I've added a check to see if this directory exists:

Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.evndisp_sub.sh

Lines 38 to 40 in 012c3e8

	if [ -d "$VERITAS_DATA_DIR_2" ]; then
	APPTAINER_MOUNT+=" --bind ${VERITAS_DATA_DIR_2}:/opt/VERITAS_DATA_DIR_2 "
	fi

I have some DB issue at the moment, but at least the pre-DB stage of evndisp is running ok. @GernotMaier are the DB dumps available anywhere for direct download? Also let me know if you have a better solution for my VERITAS_DATA_DIR_2 issue.

[GernotMaier]

I am trying to understand the issue with TEMPDIR: this is set as variable both in the script and in the apptainer environment. I really want to avoid if statements about apptainers in the body of the script - we should try and handle this add the top where we bind the directories. There must be another issue or this is implemented incorrectly. What is your error message?

[steob92]

The job submits and starts on the cluster just fine, however this is the output file I get:

VERITAS Analysis Summary (University of Delaware & DESY)  (version v.4.90)
==========================================================================

sequentiell analysis

no file found to read run parameters: /opt/VERITAS_EVNDISP_AUX_DIR//ParameterFiles///raid/jacko/obriens/VERITAS/tmpdir/MSCWDISP-5431c997-03e3-41b3-8016-963f60dc35b8/ANASUM.runparameter

You can see that the temp runparameter is getting created where I point my $TMP to, but the filename that is passed to the container is /opt/VERITAS_EVNDISP_AUX_DIR//ParameterFiles/ (which is the bind point) + /raid/jacko/obriens/VERITAS/tmpdir/MSCWDISP-5431c997-03e3-41b3-8016-963f60dc35b8/ANASUM.runparameter (which is where ANASUM.runparameter sits on the host.

Instead it should be:
/opt/TEMPDIR/ANASUM.runparameter

Alternatively if $VERITAS_EVNDISP_AUX_DIR is bound to /opt/VERITAS_EVNDISP_AUX_DIR then only the basename needs to be passed rather than the full path.

Fixing this, I get a similar issue for FLIST.

[GernotMaier]

Reading through this, it looks to me it is a mixup of env variables defined on the host system with those in the container.

Reading through https://singularity-userdoc.readthedocs.io/en/latest/environment_and_metadata.html , it looks like singulariy has also a --cleanenv command line option. Do you want to try this?

[steob92]

I tried with --cleanenv by aliasing alias apptainer="singularity --cleanenv" at the top of the script.

I'm not sure the issue is with the host system. Looking here:

Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.evndisp_sub.sh

Lines 249 to 258 in 73fdf0e

	# run eventdisplay
	if [[ $CALIB != "5" ]]; then
	LOGFILE="$LOGDIR/$RUN.log"
	rm -f "$LOGDIR/$RUN.log"
	$EVNDISPSYS/bin/evndisp \
	-runnumber="$RUN" \
	-reconstructionparameter "$ACUTS" \
	-outputfile "$TEMPDIR/$RUN.root" \
	"${OPT[@]}" \
	-calibrationdirectory "$CALDIR" &> "$LOGFILE"

$TEMPDIR/$RUN.root is being explicitly passed to evndisp however it is getting set to a path on the host, without getting reset to somewhere within the container:

Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.evndisp_sub.sh

Lines 25 to 31 in 73fdf0e

	if [[ -n $TMPDIR ]]; then
	TEMPDIR=$TMPDIR/$RUN
	else
	TEMPDIR="$VERITAS_USER_DATA_DIR/TMPDIR"
	fi
	echo "Scratch dir: $TEMPDIR"
	mkdir -p "$TEMPDIR"

[GernotMaier]

Yes - this is by design. TEMPDIR is defined both in the script environment and in the container. Important is to have th "", so that the variable is not expanded.