Batch nodes that can't run jobs exaggerate batch availability

[pbasting]

• I noticed that when checking batch availability, there appeared to be a number of nodes with the [batch] designation that have much higher memory availability than other nodes

$ node_stat.py -q batch
       ---------- CPU ----------     ---------- MEM ----------
n310 [ ||||||||||||||||||||||||| ] [ ||||||||||||||||||||||||| ] CPU:  0/32  MEM:   7/503  GB	Idle
n309 [ ||||||||||||||||||||||||| ] [ ||||||||||||||||||||||||| ] CPU:  0/32  MEM:   7/503  GB	Idle
n308 [ ||||||||||||||||||||||||| ] [ ||||||||||||||||||||||||| ] CPU:  0/32  MEM:   7/503  GB	Idle
n307 [ ||||||||||||||||||||||||| ] [ ||||||||||||||||||||||||| ] CPU:  0/32  MEM:   7/503  GB	Idle

• I tried submitting a job that used 32 threads and 200GB memory to batch but it stayed in the queue and was never assigned to these available nodes.
• Then specified one of these nodes in my PBS header

#PBS -l nodes=n310:ppn=32 -l mem=200gb -l walltime=400:00:00

• This also stayed queued and never ran.
• when looking at these nodes using mdiag -v -n I noticed that these have two queue designations, [batch][pb_q]

$ mdiag -v -n n310
compute node summary
Name                    State   Procs      Memory         Disk          Swap      Speed          Opsys   Arch Par   Load Classes                        Features

n310                     Idle   32:32   507746:515553 855306:901183 511842:519649  1.00          linux      - dis   0.08 [batch][pb_q]                  pbdev,ephi,AMD,EDR,EPYC,AMD_EDR  GRES=lscratch:875833360,socket:1,numanode:4,core:32,thread:32 NODEINDEX=310
-----                     ---   32:32   507746:515553 855306:901183 511842:519649

Total Nodes: 1  (Active: 0  Idle: 1  Down: 0)

• I'm not sure what the pb_q designation means but it appears to be a separate queue that has a subset of nodes tagged as both pb_q and batch though it appears jobs cannot be submitted to these nodes through the batch queue.
• As these contain the [batch] tag in the mdiag information, node_stat.py will display these as batch nodes, which makes the queue appear to have much higher resource availability then what can actually be used via batch submission

• I should filter the pb_q nodes out of the batch results so the availability is reported accurately.
• I should also look into these nodes with multiple queue assignments as they could be affecting the reporting of other queues