Job Object

The Job object is an instance of the JobData class. It holds all information relevant to a given PanDA job. Below is a list of the data members of the JobData class along with explanations.

Data member	Type	Meaning
accessmode	String	Direct access instruction derived from jobparams. 'copy' means copy-to-scratch; 'direct' means direct access (remote I/O). Set to 'direct' when prodDBlockToken (storage_token) is not 'local' for non-lib data files - which is the signal from JEDI that direct access is permitted (e.g. when the user submitted with --forceDirectIO). Also set to 'direct' when transfertype implies direct I/O and storage_token != 'local', or for user analysis jobs when storage_token != 'local'. Can be explicitly overridden via --accessmode=direct or --accessmode=copy in jobparams, with copy-mode flags taking priority. Default is 'copy'.
actualcorecount	Integer	Number of cores actually used by the payload, as measured during execution.
allownooutput	String	Comma-separated list of output file name patterns that may be missing without causing the job to fail. If empty, all declared output files are required.
alrbuserplatform	String	ALRB_USER_PLATFORM value encoded in the platform/cmtconfig field using @ as a separator (e.g. x86_64-centos7-gcc8-opt@centos7). If present, the value after @ is extracted and exported as thePlatform in the ALRB container setup.
altstageout	Boolean or None	Controls whether alternative stage-out is permitted. True enables it, False (sent by PanDA as altStageOut=off) disables it unconditionally, None defers to the queue-level setting. Used e.g. to prevent alt stage-out for pre-merged jobs.
attemptnr	Integer	Job attempt number. Not used by the pilot internally; included in job definition and forwarded in each server status update.
batchid	String	Batch system job ID (e.g. SLURM job ID). Informational; intended for future removal from the job object.
batchtype	String	Batch system type (e.g. 'slurm'). Informational; intended for future removal from the job object.
checkinputsize	Boolean	If True (default), the pilot verifies input file sizes after stage-in. Set to False when the mv copytool is used and input files reside on non-local disks.
command	String	The full payload command string, assembled by the pilot. Set for container jobs.
completed	Boolean	Set to True when the job has finished or failed. Used by https.send_update() to avoid sending further updates.
containeroptions	Dictionary	Options from the job definition used in container setup. In ATLAS, the following keys are used: containerImage (exported as thePlatform), containerExec (the command to execute inside the container), and execArgs (additional arguments).
coprocess	Dictionary	Command to execute concurrently with the payload, in the form {'command': '..', 'args': '..'}.
corecount	Integer	Number of cores requested by the task. Overridden by ATHENA_PROC_NUMBER if that environment variable is set. Default is 1.
corecounts	List	Time-ordered list of all actual core count measurements taken during the job, used for averaging.
cpuconsumptiontime	Integer	Total CPU consumption time in seconds, reported to the server at job completion.
cpuconsumptionunit	String	Unit for CPU consumption time. Default is 's' (seconds).
cpuconversionfactor	Integer	Conversion factor applied to CPU consumption time when reporting to the server. Default is 1.
cpufrequencies	List	List of CPU frequency measurements (in kHz) taken during the job by the memory monitor or similar utility.
currentsize	Integer	Current size in bytes of the job object itself (used for monitoring memory growth of the pilot).
datasetin	String	Input dataset name. Deprecated - this information has been moved to FileSpec objects in indata.
dask_scheduler_ip	String	IP address of the Dask scheduler for Dask-enabled jobs. Set from the scheduler_ip field in the job definition.
dbdata	Any	Database data payload from the job report (ATLAS-specific).
dbtime	Any	Database timing information from the job report (ATLAS-specific).
debug	Boolean	If True, the pilot has received a debug command from the server via the backchannel and will send additional diagnostic information.
debug_command	String	The specific debug command received from the server (e.g. 'debug', 'debugoff').
destinationdblock	String	Destination data block for output files. Intended for future migration to FileSpec.
exeerrorcode	Integer	Error code from the payload execution layer (transformation), distinct from the transformation's own exit code.
exeerrordiag	String	Diagnostic message corresponding to exeerrorcode.
exitcode	Integer	Final exit code reported to the server, derived from piloterrorcode, transexitcode, or exeerrorcode.
exitmsg	String	Human-readable exit message reported to the server at job completion.
fileinfo	Dictionary	File information dictionary used during stage-out, populated from the job report.
homepackage	String	Home package string for the transformation (TRF), providing additional payload release information (e.g. 'Athena/22.0.107').
imagename	String	Container image name, extracted from jobparams (takes precedence) or from the job definition field container_name.
imagename_jobdef	String	Container image name as received directly from the job definition (container_name field), before any jobparams-based override.
indata	List	List of FileSpec objects describing the input files for this job. Populated by prepare_infiles() from inFiles, ddmEndPointIn, scopeIn, prodDBlockToken, etc.
infilesguids	String	Comma-separated GUIDs for input files. Deprecated - this information has been moved to FileSpec objects in indata.
is_eventservice	Boolean	True for Event Service (ES) jobs, which process events individually rather than as a full job.
is_eventservicemerge	Boolean	True for Event Service merge jobs, which combine the output of multiple ES jobs.
is_hpo	Boolean	True for Hyper-Parameter Optimisation (HPO) jobs.
jobdefinitionid	Integer	Job definition ID, included in trace reports.
jobid	Integer	Unique PanDA job identifier (PandaID), e.g. 4993552533.
jobparams	String	Raw job parameters string passed to the payload as command-line options. Parsed by the pilot to extract settings such as accessmode, imagename, zipmap, and overwrite_queuedata.
jobsetid	String	Job set ID, grouping related jobs within a task.
logdata	List	List of FileSpec objects describing the log file(s) for this job.
looping_check	Boolean	If True (default), the pilot performs the looping job check and will terminate a job that has not produced output within the configured time limit. Can be disabled from the job definition via loopingCheck=False.
lsetuptime	Float	Time in seconds spent on payload setup (lsetup), recorded for bookkeeping.
maxcpucount	Integer	Maximum CPU count (seconds) defining the looping job limit. Currently unused; the pilot uses internal configuration instead.
maxwalltime	Integer	Maximum wall-clock time in seconds allowed for this job, as set by PanDA.
memorymonitor	String	Name of the memory monitor utility to use (e.g. 'prmon').
metadata	Dictionary	Payload metadata extracted from the job report (e.g. ATLAS job report JSON), used for constructing the final server update.
minramcount	Integer	Minimum RAM in MB required by the payload, as declared in the job definition.
nevents	Integer	Number of events processed by the payload, as reported in the job report.
neventsw	Integer	Number of events written to output by the payload, as reported in the job report.
noexecstrcnv	Boolean	If True, the pilot uses the payload setup string from jobparams directly without converting it. Set by the server via noExecStrCnv.
nucleus	String	Name of the nucleus site for this job (used in Harvester/PUSH mode).
outdata	List	List of FileSpec objects describing the output files for this job.
overwrite_queuedata	Dictionary	Custom queue settings extracted from --overwriteQueueData in jobparams. Values here take precedence over the queue's own settings.
overwrite_storagedata	Dictionary	Custom storage settings extracted from --overwriteStorageData in jobparams. Values here take precedence over storage configuration.
pandasecrets	String	User-defined secrets string passed from the job definition (PanDA secrets field), forwarded to the payload environment.
payload	String	Payload name (e.g. 'Athena').
pgrp	Integer	Process group ID of the payload process, used to send signals to the entire process tree.
pid	Integer	Process ID of the payload process.
piloterrorcode	Integer	Current pilot error code. The last code set takes precedence and is reported to the server.
piloterrorcodes	List	Ordered list of all pilot error codes accumulated during the job, for diagnostic purposes.
piloterrordiag	String	Current pilot error diagnostic message, corresponding to piloterrorcode.
piloterrordiags	List	Ordered list of all pilot error diagnostic messages accumulated during the job.
pilotsecrets	Dictionary	Real-time logging secrets (e.g. credentials for a logging endpoint), received from the server via pilotSecrets.
platform	String	Software platform / cmtconfig value from the task definition (e.g. 'x86_64-centos7-gcc8-opt'). Any @-encoded ALRB_USER_PLATFORM suffix is stripped and stored separately in alrbuserplatform.
postprocess	Dictionary	Command to execute after the payload completes, in the form {'command': '..', 'args': '..'}.
preprocess	Dictionary	Command to execute before the payload starts, in the form {'command': '..', 'args': '..'}.
processingtype	String	Processing type string from the task definition (e.g. 'nightlies', 'reprocessing').
prodproxy	String	Path to the production proxy on unified queues, tracked so it can be cleaned up after the job.
produserid	String	The DN (Distinguished Name) of the user who submitted the job, included in trace reports.
requestid	Integer	Request ID associated with the job, from the reqID field in the job definition.
resourcetype	String	Resource type requested by the task (e.g. 'SCORE', 'MCORE', 'SCORE_BACKFILL').
runningstart	Any	Timestamp at which the payload entered the running state, used internally for monitoring. Not the official job start time.
serverstate	String	Job state as last reported to the PanDA server. Valid values: 'starting', 'running', 'finished', 'holding', 'failed'.
setup	String	Full payload setup string, used when constructing the postprocess command.
sizes	Dictionary	Snapshot history of job object sizes, keyed by timestamp: {timestamp: size_in_bytes, ..}. Used for pilot memory monitoring.
stageout	String	Stage-out identifier indicating which files are being staged out at a given moment (e.g. 'log').
starttime	Any	Epoch time in seconds at which the job officially started running, as reported to the server.
state	String	Internal pilot job state. Valid values: 'running', 'failed', 'finished', 'holding', 'stagein', 'stageout'.
status	Dictionary	Job status dictionary used for tracking specific sub-states. Format: {key: value, ..}, e.g. {'LOG_TRANSFER': 'NOT_DONE'}.
subprocesses	List	List of PIDs of subprocesses spawned by the payload (e.g. forked workers), used for cleanup.
swrelease	String	Software release string from the job definition (e.g. 'Atlas-22.0.107').
t0	Any	Payload startup timestamp, used for on-the-fly CPU consumption time measurements during job monitoring.
taskid	String	Unique task identifier for the task this job belongs to. Numeric string (e.g. '24342519') or 'NULL' if not set by the server.
transexitcode	Integer	Exit code returned by the payload transformation (TRF) process.
transfertype	String	Direct access instruction from the PanDA server, received via the job definition (transferType field). Values that imply direct I/O: 'direct', 'root', 'davs', or comma-separated combinations thereof. 'file' means Rucio copy via POSIX link (not remote I/O).
transformation	String	Name or URL of the payload transformation script to execute (e.g. 'Athena.py' or an https:// URL for user analysis jobs).
use_vp	Boolean	True for Versatile Pilot (VP) jobs, which use a special execution model.
usecontainer	Boolean	True if the payload is to be executed inside a container. Set by the pilot based on imagename and queue configuration.
utilities	Dictionary	Utility processes running alongside the payload, e.g. memory monitors. Format: {name: [process_handle, launch_count, command_string], ..}.
workdir	String	Absolute path to the working directory created for this job.
workdirsizes	List	Time-ordered list of working directory size measurements in bytes, taken periodically during the job.
writetofile	String	Instruction to write certain input file lists to a file rather than passing them on the command line (used when argument lists become too long).
zipmap	String	ZIP map values extracted from jobparams, used to map zipped input files to their logical names.
zombies	List	List of zombie process IDs left behind by the payload, collected for cleanup at job completion.
_rawdata	Dictionary	Placeholder for the raw job definition dictionary as received from the server. Used internally to preserve backward compatibility during the migration to typed attributes.