OSPool Resource Status

Refreshes every 5 minutes
100
Resources
79
All Passing
7
Failing
14
Warnings
0
Not Available
Resource Last Updated Glideins Singularity SIF CVMFS Pelican Probes
AGLT2 2026-04-03 17:02 234
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
AGLT2 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.014ms pmtu 1500 1: no reply 2: no reply 3: no reply 4: chi-710n-mx1-lo0-229.aglt2.org 6.274ms 5: no reply 6: chic-cr6--uwmadison_se-203.es.net 5.756ms 7: uwmadison_se-203--chic-cr6.es.net 23.154ms 8: perfsonar02.hep.wisc.edu 23.015ms reached Resume: pmtu 1500 hops 8 back 8
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_1803595/glide_cQafW9/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_1803595/glide_cQafW9/execute/dir_2055370/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2055370 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_1803595/glide_cQafW9/cred.d/idtokens/credential_OSPool.main_HCC_US_Michigan_gate02.idtoken GLIDEIN_ClusterId=5681790 GLIDEIN_Country=US GLIDEIN_DUNESite=US_Michigan GLIDEIN_Entry_Name=HCC_US_Michigan_gate02 GLIDEIN_Entry_Signature=815f1c9494fcd1a2fa493bcdf877605c15585fdc GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=gate02.aglt2.org gate02.aglt2.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_BulGwG GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_Nickname=AGLT2 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=AGLT2 GLIDEIN_SEs=head01.aglt2.org GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=Michigan GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_1803595/glide_cQafW9/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Michigan PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_1803667_244819047@c-212-19.aglt2.org$ "; unset PROMPT_COMMAND PS1=slot1@glidein_1803667_244819047@c-212-19.aglt2.org$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_1803595/glide_cQafW9/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el9 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/dir_1803595/glide_cQafW9/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_1803595/glide_cQafW9/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_1803595/glide_cQafW9/execute/dir_2055370/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.16.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
2073.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_sys-lv_condor 2451014720 152688276 2173747424 7% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
AMNH-HEL-CE1 2026-04-03 11:01 32
0.0%
0.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
AMNH-HEL-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: hel-head 0.255ms 1: hel-head 0.157ms 2: 172.16.108.1 0.592ms asymm 59 3: core-er2.sci-user-asa.transit.internal.amnh.org 1.032ms 4: fw.wan-asa.transit.internal.amnh.org 0.775ms asymm 61 5: sdmz.amnh.org 0.905ms 6: nyc-9208-amnh.nysernet.net 1.028ms 7: nyc32-55a1-nyc32-9208.nysernet.net 1.500ms 8: syr-57c3-nyc32-55a1.nysernet.net 7.221ms 9: buf-55a1-syr-57c3.nysernet.net 10.137ms 10: buf-9208-buf-55a1.nysernet.net 9.739ms 11: i2-clev-buf-9208.nysernet.net 15.074ms 12: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 22.534ms asymm 14 13: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 21.830ms 14: 144.92.233.112 24.299ms 15: perfsonar02.hep.wisc.edu 23.312ms reached Resume: pmtu 1500 hops 15 back 16
pass env
APPTAINER_CACHEDIR=/scratch_space/glide_jWEwcS/execute/dir_13061 BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*` } BATCH_SYSTEM=HTCondor BLAHPD_LOCATION=/home/osg01/bosco_slate/glite CONDORCE_COLLECTOR_HOST=amnh-hel-ce1.svc.opensciencegrid.org:9619 CONDOR_CONFIG=/scratch_space/glide_jWEwcS/condor_config CONDOR_PARENT_ID=hel-n0003:13061:1775214082 CONDOR_PROCD_ADDRESS=/scratch_space/glide_jWEwcS/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/scratch_space/glide_jWEwcS/log/procd_address CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 ENVIRONMENT=BATCH GFAL_CONFIG_DIR=/home/osg01/bosco-osg-wn-client/etc/gfal2.d/ GFAL_PLUGIN_DIR=/home/osg01/bosco-osg-wn-client/usr/lib64/gfal2-plugins/ GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch_space/glide_jWEwcS/cred.d/idtokens/credential_OSPool.main_COVID19_US_AMNH-HEL-CE1.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=295803 GLIDEIN_Entry_Name=COVID19_US_AMNH-HEL-CE1 GLIDEIN_Entry_Signature=0be38f2bfee485148b9e5f8fac852e6a5574a157 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=amnh-hel-ce1.svc.opensciencegrid.org amnh-hel-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_6aYd62 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=osdf1.newy32aoa.net.internet2.edu:9002 GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=AMNH-HEL-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=1a97d6060d26940eba08ee679d09a38f5c43d5c2 GLIDEIN_Site=AMNH GLIDEIN_Tmp_Dir=/scratch_space/glide_jWEwcS/tmp GLIDEIN_UUID=b0b94507-430b-422a-9ef8-a41f4e0170fd GLITE_LOCATION=/home/osg01/bosco_slate/glite GLOBUS_LOCATION=/home/osg01/bosco-osg-wn-client/usr GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_PATH=/scratch_space/glide_jWEwcS/.gwms.d/bin: GWMS_SINGULARITY_STATUS=PREFERRED HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HISTCONTROL=ignoredups HISTSIZE=1000 HOLD_GRACE_TIME=0 HOME=/scratch_space/glide_jWEwcS/execute/dir_13061 HOSTNAME=hel-n0003 JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_COVID19_US_AMNH-HEL-CE1/tokens.tgz JULIA_NUM_THREADS=1 KDEDIRS=/usr LANG=C LD_LIBRARY_PATH=/home/osg01/bosco-osg-wn-client/lib64:/home/osg01/bosco-osg-wn-client/lib:/home/osg01/bosco-osg-wn-client/usr/lib64:/home/osg01/bosco-osg-wn-client/usr/lib:/home/osg01/bosco-osg-wn-client/usr/lib64/dcap:/home/osg01/bosco-osg-wn-client/usr/lib64/lcgdm LESSOPEN=||/usr/bin/lesspipe.sh %s LOADEDMODULES= LOGNAME=osg01 MAIL=/var/spool/mail/osg01 MANPATH=/home/osg01/bosco-osg-wn-client/usr/share/man::/opt/pbs/share/man:/opt/pbs/share/man MKL_NUM_THREADS=1 MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles MODULESHOME=/usr/share/Modules MY_HOSTS=pvm_hostfile NODE_COUNT=8 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=UNAVAILABLE OSG_DATA=UNAVAILABLE OSG_GRID=/home/osg01/bosco-osg-wn-client OSG_HOSTNAME=localhost OSG_LOCATION=/home/osg01/bosco-osg-wn-client OSG_SITE_NAME=AMNH-HEL-CE1 OSG_SQUID_LOCATION=osdf1.newy32aoa.net.internet2.edu:9002 OSG_WN_TMP=/scratch_space/glide_jWEwcS/execute/dir_13061/.local-tmp PATH=/scratch_space/glide_jWEwcS/.gwms.d/bin::/home/osg01/bosco-osg-wn-client/usr/bin:/home/osg01/bosco-osg-wn-client/usr/sbin:/usr/local/sbin/:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/software/Apptainer/apptainer-1.1.5/bin:/usr/local/software/Singularity/singularity-3.5.3/bin PERL5LIB=/home/osg01/bosco-osg-wn-client/usr/share/perl5/vendor_perl:/home/osg01/bosco-osg-wn-client/usr/share/perl5:/home/osg01/bosco-osg-wn-client/usr/lib64/perl5/vendor_perl:/home/osg01/bosco-osg-wn-client/usr/lib64/perl5:/home/osg01/perl5/lib/perl5: PERL_LOCAL_LIB_ROOT=:/home/osg01/perl5 PERL_MB_OPT=--install_base /home/osg01/perl5 PERL_MM_OPT=INSTALL_BASE=/home/osg01/perl5 PREEMPT_GRACE_TIME=10000000 PVM_ARCH=LINUX64 PVM_BIN=/usr/local/software/pvm/pvm3/bin PVM_ROOT=/usr/local/software/pvm/pvm3 PVM_RSH=/usr/bin/ssh PWD=/scratch_space/glide_jWEwcS/execute/dir_13061 PYTHONPATH=/home/osg01/bosco-osg-wn-client/usr/lib/python2.7/site-packages:/home/osg01/bosco-osg-wn-client/usr/lib64/python2.7/site-packages PYTHON_CPU_COUNT=1 QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include QTLIB=/usr/lib64/qt-3.3/lib QT_GRAPHICSSYSTEM_CHECKED=1 QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins ROOT_MAX_THREADS=1 SHELL=/bin/bash SHLVL=6 SINGULARITY_CACHEDIR=/scratch_space/glide_jWEwcS/execute/dir_13061 SLOTS_LAYOUT=partitionable SLURMD_NODENAME=hel-n0003 SLURM_CLUSTER_NAME=hel SLURM_CONF=/etc/slurm/slurm.conf SLURM_CPUS_ON_NODE=8 SLURM_GTIDS=0 SLURM_JOBID=1257433 SLURM_JOB_ACCOUNT=root SLURM_JOB_CPUS_PER_NODE=8 SLURM_JOB_GID=1083 SLURM_JOB_ID=1257433 SLURM_JOB_NAME=bl_v3Tafx SLURM_JOB_NODELIST=hel-n0003 SLURM_JOB_NUM_NODES=1 SLURM_JOB_PARTITION=batch-amd SLURM_JOB_QOS=normal SLURM_JOB_UID=1001 SLURM_JOB_USER=osg01 SLURM_LOCALID=0 SLURM_MEM_PER_NODE=65536 SLURM_NNODES=1 SLURM_NODEID=0 SLURM_NODELIST=hel-n0003 SLURM_NODE_ALIASES=(null) SLURM_NPROCS=8 SLURM_NTASKS=8 SLURM_NTASKS_PER_NODE=8 SLURM_PRIO_PROCESS=0 SLURM_PROCID=0 SLURM_SUBMIT_DIR=/home/osg01/bosco_slate/sandbox/7848/784834f0/amnh-hel-ce1.svc.opensciencegrid.org_9619_amnh-hel-ce1.svc.opensciencegrid.org_1392784.0_1775014594 SLURM_SUBMIT_HOST=hel-head SLURM_TASKS_PER_NODE=8 SLURM_TASK_PID=56901 SLURM_TOPOLOGY_ADDR=hel-n0003 SLURM_TOPOLOGY_ADDR_PATTERN=node SLURM_WORKING_CLUSTER=hel:hel-head:6817:8960:101 SSH_CLIENT=128.105.82.84 64808 22 SSH_CONNECTION=128.105.82.84 64808 172.16.108.19 22 TEMP=/scratch_space/glide_jWEwcS/execute/dir_13061 TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/scratch_space/glide_jWEwcS/execute/dir_13061 TMPDIR=/scratch_space/glide_jWEwcS/execute/dir_13061/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime USER=osg01 VOMS_USERCONF=/home/osg01/bosco-osg-wn-client/etc/vomses X509_CERT_DIR=/home/osg01/bosco-osg-wn-client/etc/grid-security/certificates X509_VOMS_DIR=/home/osg01/bosco-osg-wn-client/etc/grid-security/vomsdir XDG_DATA_DIRS=/home/osg01/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_13061=13721:1775214082:3106917558 _CONDOR_ANCESTOR_25707=25787:1775015196:4051770613 _CONDOR_ANCESTOR_25787=13061:1775214081:598403636 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch_space/glide_jWEwcS/main/condor/bin _CONDOR_CHIRP_CONFIG=/scratch_space/glide_jWEwcS/execute/dir_13061/.chirp.config _CONDOR_CREDS=/scratch_space/glide_jWEwcS/execute/dir_13061/.condor_creds _CONDOR_EXECUTE=/scratch_space/glide_jWEwcS/execute _CONDOR_JOB_AD=/scratch_space/glide_jWEwcS/execute/dir_13061/.job.ad _CONDOR_JOB_IWD=/scratch_space/glide_jWEwcS/execute/dir_13061 _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/scratch_space/glide_jWEwcS/execute/dir_13061/.machine.ad _CONDOR_SCRATCH_DIR=/scratch_space/glide_jWEwcS/execute/dir_13061 _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/scratch_space/glide_jWEwcS/execute/dir_13061/.job_wrapper_failure
warning kernel
3.10.0-1062.18.1.el7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1677.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 1922182564 65239716 1759278176 4% /scratch_space
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy osdf1.newy32aoa.net.internet2.edu:9002
AMNH-Huxley-CE1 2026-04-03 17:01 71
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
AMNH-Huxley-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: huxley-head-new 1.827ms 1: huxley-head-new 2.213ms 2: 216.73.255.65 14.236ms 3: 216.73.255.1 2.170ms 4: nyc-9208-amnh.nysernet.net 2.162ms 5: nyc32-55a1-nyc32-9208.nysernet.net 3.648ms 6: syr-57c3-nyc32-55a1.nysernet.net 8.204ms 7: buf-55a1-syr-57c3.nysernet.net 11.021ms 8: buf-9208-buf-55a1.nysernet.net 11.370ms 9: i2-clev-buf-9208.nysernet.net 15.943ms 10: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 23.497ms asymm 12 11: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 23.724ms 12: 144.92.233.112 24.212ms 13: perfsonar02.hep.wisc.edu 24.570ms reached Resume: pmtu 1500 hops 13 back 14
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch_space/glide_ow43HW/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch_space/glide_ow43HW/execute/dir_1592371/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1592371 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch_space/glide_ow43HW/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch_space/glide_ow43HW/cred.d/idtokens/credential_OSPool.main_OSG_US_AMNH-Huxley-CE1.idtoken GLIDEIN_CPUS=2 GLIDEIN_ClusterId=305607 GLIDEIN_Entry_Name=OSG_US_AMNH-Huxley-CE1 GLIDEIN_Entry_Signature=06ab0aa2ad2178027b9aa870dffb081ddac7213e GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=amnh-huxley-ce1.svc.osg-htc.org amnh-huxley-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_dH0lX0 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=AMNH-Huxley-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=AMNH GLIDEIN_Tmp_Dir=/scratch_space/glide_ow43HW/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch_space/glide_ow43HW/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch_space/glide_ow43HW/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=AMNH-Huxley-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_1132524_794394561@huxley-n0001$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_1132524_794394561@huxley-n0001$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch_space/glide_ow43HW/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/scratch_space/glide_ow43HW/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/scratch_space/glide_ow43HW/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch_space/glide_ow43HW/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch_space/glide_ow43HW/execute/dir_1592371/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.56.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
70.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 117162092 42801052 74361040 37% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
AMNH-Mendel-CE1 2026-04-03 17:02 27
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
AMNH-Mendel-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: mendel-head.mendel.sdmz.amnh.org 0.127ms 1: mendel-head.mendel.sdmz.amnh.org 0.191ms 2: 216.73.255.65 15.994ms 3: 216.73.255.1 0.490ms 4: nyc-9208-amnh.nysernet.net 0.492ms 5: nyc32-55a1-nyc32-9208.nysernet.net 0.732ms 6: syr-57c3-nyc32-55a1.nysernet.net 6.472ms 7: buf-55a1-syr-57c3.nysernet.net 9.575ms 8: buf-9208-buf-55a1.nysernet.net 9.141ms 9: i2-clev-buf-9208.nysernet.net 14.245ms 10: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 20.785ms asymm 12 11: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 22.636ms 12: 144.92.233.112 22.712ms 13: perfsonar02.hep.wisc.edu 22.662ms reached Resume: pmtu 1500 hops 13 back 14
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch_space/glide_lmaRax/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch_space/glide_lmaRax/execute/dir_2484748/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2484748 CONTAINERSW_FULL_VERSION=apptainer version 1.2.5+433-gca1e06012 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/local/software/Apptainer/apptainer-1.2.5/bin/apptainer CONTAINERSW_VERSION=1.2.5+433-gca1e06012 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch_space/glide_lmaRax/cred.d/idtokens/credential_OSPool.main_OSG_US_AMNH-Mendel_osg.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=287642 GLIDEIN_Entry_Name=OSG_US_AMNH-Mendel_osg GLIDEIN_Entry_Signature=562273605da7e0cedaa8d5875d443063f21c63c6 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=amnh-mendel-ce1.svc.osg-htc.org amnh-mendel-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_NBo1HJ GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=osdf1.newy32aoa.net.internet2.edu:9002 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=AMNH-Mendel-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=1a97d6060d26940eba08ee679d09a38f5c43d5c2 GLIDEIN_Site=AMNH GLIDEIN_Tmp_Dir=/scratch_space/glide_lmaRax/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.2.5+433-gca1e06012 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/local/software/Apptainer/apptainer-1.2.5/bin/apptainer GWMS_CONTAINERSW_VERSION=1.2.5+433-gca1e06012 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/local/software/Apptainer/apptainer-1.2.5/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.2.5+433-gca1e06012 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=AMNH-Mendel-CE1 OSG_SQUID_LOCATION=osdf1.newy32aoa.net.internet2.edu:9002 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_3546991_707980570@mendel-b0002.mendel.sdmz.amnh.org$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_3546991_707980570@mendel-b0002.mendel.sdmz.amnh.org$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch_space/glide_lmaRax/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.2.5+433-gca1e06012 SINGULARITY_IMAGES_DICT=default:/scratch_space/glide_lmaRax/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/local/software/Apptainer/apptainer-1.2.5/bin/apptainer SINGULARITY_VERSION=1.2.5+433-gca1e06012 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch_space/glide_lmaRax/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch_space/glide_lmaRax/execute/dir_2484748/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-147.el8.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
756.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 875775880 38536488 792682644 5% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy osdf1.newy32aoa.net.internet2.edu:9002
ASU-Sol-CE1 2026-04-03 17:49 3
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
ASU-Sol-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.079ms 1: _gateway 0.088ms 2: 206.206.198.2 0.217ms 3: im-sl-gw.netmgmt.asu.edu 10.871ms 4: im-core-pe-gw2.netmgmt.asu.edu 7.953ms 5: 172.29.7.41 2.954ms 6: 149.169.2.33 9.885ms 7: 206.207.226.103 10.863ms 8: 206.207.226.103 1.239ms asymm 7 9: hundredge-0-0-0-24.1010.core1.phoe.net.internet2.edu 10.871ms asymm 8 10: fourhundredge-0-0-0-0.4079.core1.tucs.net.internet2.edu 45.932ms asymm 22 11: fourhundredge-0-0-0-0.4079.core1.elpa.net.internet2.edu 49.946ms asymm 21 12: fourhundredge-0-0-0-2.4079.core2.denv.net.internet2.edu 45.948ms asymm 19 13: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 45.958ms asymm 18 14: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 45.956ms asymm 17 15: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 45.945ms asymm 16 16: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 44.943ms 17: 144.92.233.112 47.945ms 18: perfsonar02.hep.wisc.edu 50.832ms reached Resume: pmtu 1500 hops 18 back 19
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg/osg01/glide_O3FQlz/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg/osg01/glide_O3FQlz/execute/dir_2257043/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2257043 CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.3-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg/osg01/glide_O3FQlz/cred.d/idtokens/credential_OSPool.main_OSG_US_ASU-Sol-CE1.idtoken GLIDEIN_CPUS=24 GLIDEIN_ClusterId=232431 GLIDEIN_Entry_Name=OSG_US_ASU-Sol-CE1 GLIDEIN_Entry_Signature=232b33666a44bb96c4619092ad77ee6bf60e54bb GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=asu-sol-ce1.svc.opensciencegrid.org asu-sol-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_CKIzZl GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=ASU-Sol-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=ASU-Sol GLIDEIN_Tmp_Dir=/scratch/osg/osg01/glide_O3FQlz/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.3-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.3-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=ASU-Sol-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_9@glidein_2038290_152777427@sg236$ "; unset PROMPT_COMMAND PS1=slot1_9@glidein_2038290_152777427@sg236$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg/osg01/glide_O3FQlz/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.3-1.el8 SINGULARITY_IMAGES_DICT=default:/scratch/osg/osg01/glide_O3FQlz/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.3-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg/osg01/glide_O3FQlz/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg/osg01/glide_O3FQlz/execute/dir_2257043/scratch _CONDOR_SLOT=slot1_9 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.77.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
910895.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on beegfs_nodev 3996579659776 3041436716032 955142943744 77% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Alabama-CHPC-CE1 2026-04-03 17:01 50
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
Alabama-CHPC-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: chpc-head001.cm.cluster 0.082ms 1: chpc-head001.cm.cluster 0.079ms 2: 10.8.96.2 2.102ms 3: no reply 4: 192.168.9.17 2.691ms asymm 5 5: no reply 6: EDGE-ASR9K-PE-9.oit.ua.edu 2.643ms 7: 192.168.9.14 4.025ms 8: 192.168.13.252 4.790ms 9: TCL-ASRK9-R1-4001.oit.ua.edu 5.379ms 10: 130.160.15.233 13.288ms 11: core1-to-core2-re.56mar.sox.net 12.178ms 12: sidco-to-56mar-al2s.sox.net 15.858ms asymm 10 13: i2-to-sox-re.sidco.sox.net 17.102ms asymm 11 14: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 25.699ms asymm 18 15: fourhundredge-0-0-0-23.4079.core2.cinc.net.internet2.edu 27.770ms asymm 17 16: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 44.935ms 17: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 25.923ms asymm 15 18: 144.92.233.112 27.679ms asymm 16 19: perfsonar02.hep.wisc.edu 27.943ms reached Resume: pmtu 1500 hops 19 back 18
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_KOhZGt/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_KOhZGt/execute/dir_158956/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=158956 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/tmp/glide_KOhZGt/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_KOhZGt/cred.d/idtokens/credential_OSPool.main_OSG_US_Alabama-CHPC-CE1_mc.idtoken GLIDEIN_CPUS=16 GLIDEIN_ClusterId=298461 GLIDEIN_Entry_Name=OSG_US_Alabama-CHPC-CE1_mc GLIDEIN_Entry_Signature=fa6d6ced6c05169d7f0f9b040c22008260887773 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=ua-hpc-ce1.svc.osg-htc.org ua-hpc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_RpSPVG GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Alabama-CHPC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=Alabama-CHPC GLIDEIN_Tmp_Dir=/tmp/glide_KOhZGt/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/tmp/glide_KOhZGt/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/tmp/glide_KOhZGt/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Alabama-CHPC-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_12@glidein_1126827_675888005@chpc-compute014.cm.cluster$ "; unset PROMPT_COMMAND PS1=slot1_12@glidein_1126827_675888005@chpc-compute014.cm.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_KOhZGt/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/tmp/glide_KOhZGt/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/tmp/glide_KOhZGt/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_KOhZGt/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_KOhZGt/execute/dir_158956/scratch _CONDOR_SLOT=slot1_12 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-372.9.1.el8.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
14512.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda7 15418689536 201048672 15217640864 2% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
BNL-SDCC-SPOOLCE01 2026-04-03 17:25 4
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
BNL-SDCC-SPOOLCE01 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.014ms pmtu 1500 1: nr0725-p1-l40.s207-192.sdcc.bnl.gov 0.405ms 1: nr0725-p1-l40.s207-192.sdcc.bnl.gov 0.475ms 2: 2620:0:210:e013::78 0.279ms 3: 2620:0:210:e043::229 0.294ms 4: fddd:7d33:c9b4:ff::51 1.539ms asymm 5 5: fddd:7d33:c9b4:ff::43 0.555ms 6: no reply 7: no reply 8: newy32aoa-cr6--wash-bb-e.igp.es.net 21.610ms asymm 16 9: wash-cr6--eqxdc4-bb-d.igp.es.net 21.297ms asymm 15 10: eqxdc4-cr6--eqxch2-bb-e.igp.es.net 21.421ms asymm 14 11: eqxch2-cr6--chic-bb-c.igp.es.net 21.417ms asymm 13 12: no reply 13: uwmadison_se-203--chic-cr6.es.net 38.853ms 14: perfsonar02.hep.wisc.edu 38.670ms reached Resume: pmtu 1500 hops 14 back 14
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/home/condor/local/sdcc/execute/dir_477825/glide_hMR32R/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/home/condor/local/sdcc/execute/dir_477825/glide_hMR32R/execute/dir_3655736/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3655736 CONTAINERSW_FULL_VERSION=apptainer version 1.4.4-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.4-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/home/condor/local/sdcc/execute/dir_477825/glide_hMR32R/cred.d/idtokens/credential_OSPool.main_DUNE_US_BNL_spoolce01.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=291185 GLIDEIN_Country=US GLIDEIN_DUNESite=US_BNL GLIDEIN_Entry_Name=DUNE_US_BNL_spoolce01 GLIDEIN_Entry_Signature=bebc34f47e1988963b51320d963c8030b64da28e GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=spoolce01.sdcc.bnl.gov spoolce01.sdcc.bnl.gov:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_BzZY9N GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=BNL-SDCC-SPOOLCE01 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=BNL-SDCC GLIDEIN_Tmp_Dir=/home/condor/local/sdcc/execute/dir_477825/glide_hMR32R/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.4-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.4-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.4-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=BNL-SDCC PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_477844_212921436@spool1488.sdcc.bnl.gov$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_477844_212921436@spool1488.sdcc.bnl.gov$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/home/condor/local/sdcc/execute/dir_477825/glide_hMR32R/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.4-1.el9 SINGULARITY_IMAGES_DICT=default:/home/condor/local/sdcc/execute/dir_477825/glide_hMR32R/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.4-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/home/condor/local/sdcc/execute/dir_477825/glide_hMR32R/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/home/condor/local/sdcc/execute/dir_477825/glide_hMR32R/execute/dir_3655736/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-570.19.1.el9_6.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
6076.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/sysvg-scratch 7048416776 321554128 6371567416 5% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
BNL-SDCC-SPOOLCE02 2026-04-03 17:12 4
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
BNL-SDCC-SPOOLCE02 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.013ms pmtu 1500 1: ar0725-p1-l19.s201-128.sdcc.bnl.gov 0.227ms 1: ar0725-p1-l19.s201-128.sdcc.bnl.gov 0.127ms 2: 2620:0:210:e033::36 0.133ms 3: 2620:0:210:e013::229 0.205ms 4: fddd:7d33:c9b4:ff::51 0.282ms asymm 5 5: fddd:7d33:c9b4:ff::42 0.489ms 6: no reply 7: no reply 8: newy32aoa-cr6--wash-bb-e.igp.es.net 21.282ms asymm 16 9: wash-cr6--eqxdc4-bb-d.igp.es.net 21.229ms asymm 15 10: eqxdc4-cr6--eqxch2-bb-e.igp.es.net 21.302ms asymm 14 11: eqxch2-cr6--chic-bb-c.igp.es.net 21.261ms asymm 13 12: no reply 13: uwmadison_se-203--chic-cr6.es.net 38.686ms 14: perfsonar02.hep.wisc.edu 38.678ms reached Resume: pmtu 1500 hops 14 back 14
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/home/condor/local/sdcc/execute/dir_2358508/glide_8kVaGq/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/home/condor/local/sdcc/execute/dir_2358508/glide_8kVaGq/execute/dir_2265994/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2265994 CONTAINERSW_FULL_VERSION=apptainer version 1.4.4-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.4-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/home/condor/local/sdcc/execute/dir_2358508/glide_8kVaGq/cred.d/idtokens/credential_OSPool.main_DUNE_US_BNL_spoolce02.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=5609664 GLIDEIN_Country=US GLIDEIN_DUNESite=US_BNL GLIDEIN_Entry_Name=DUNE_US_BNL_spoolce02 GLIDEIN_Entry_Signature=ab5f9659891dc53262d403656371681a7a7df10f GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=spoolce02.sdcc.bnl.gov spoolce02.sdcc.bnl.gov:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_XYHjFZ GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=BNL-SDCC-SPOOLCE02 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=BNL-SDCC GLIDEIN_Tmp_Dir=/home/condor/local/sdcc/execute/dir_2358508/glide_8kVaGq/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.4-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.4-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.4-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=BNL-SDCC PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_2359144_123655872@spool0117.sdcc.bnl.gov$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_2359144_123655872@spool0117.sdcc.bnl.gov$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/home/condor/local/sdcc/execute/dir_2358508/glide_8kVaGq/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.4-1.el9 SINGULARITY_IMAGES_DICT=default:/home/condor/local/sdcc/execute/dir_2358508/glide_8kVaGq/images/htc__rocky__8.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.4-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/home/condor/local/sdcc/execute/dir_2358508/glide_8kVaGq/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/home/condor/local/sdcc/execute/dir_2358508/glide_8kVaGq/execute/dir_2265994/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-570.19.1.el9_6.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
5183.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/sysvg-scratch 7048416776 1257862088 5435259456 19% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Boise-State-Borah-CE1 2026-04-03 16:01 14
0.0%
0.0%
0.0%
100.0%
all
Boise-State-Borah-CE1 — probe output
warning all
Job failed with exit code 0; no results transferred
CHTC 2026-04-03 17:01 229
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
CHTC — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.006ms pmtu 1500 1: 2607:f388:2200:100::3 0.874ms 1: 2607:f388:2200:100::3 0.860ms 2: 2607:f388:0:33f::2 0.327ms 3: perfsonar02.hep.wisc.edu 0.182ms reached Resume: pmtu 1500 hops 3 back 3
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/slot1/dir_2274485/scratch/glide_RJYTsr/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/slot1/dir_2274485/scratch/glide_RJYTsr/execute/dir_235335/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=235335 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 FTP_PROXY=http://squid-cs-2360.chtc.wisc.edu:3128 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/slot1/dir_2274485/scratch/glide_RJYTsr/cred.d/idtokens/credential_OSPool.main_HCCHTPC_US_Wisconsin_osg01_rhel7.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=292587 GLIDEIN_Country=US GLIDEIN_DUNESite=US_Wisconsin GLIDEIN_Entry_Name=HCCHTPC_US_Wisconsin_osg01_rhel7 GLIDEIN_Entry_Signature=d5a632f13ba77d83bd2511f2fe4129c95f6d4e95 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=osggrid01.hep.wisc.edu osggrid01.hep.wisc.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_slot1_77_9F6yM5 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=osgsquid.hep.wisc.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=CHTC GLIDEIN_SE_VONAME_LOWERCASE=srm://cmssrm.hep.wisc.edu:8443/srm/v2/server?SFN=/hdfs/osg/vo/ GLIDEIN_SEs=cmssrm.hep.wisc.edu GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=547752992760a694c32ac9c7748d22e0e3c3660c GLIDEIN_Site=CHTC GLIDEIN_Tmp_Dir=/var/lib/condor/execute/slot1/dir_2274485/scratch/glide_RJYTsr/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=osggrid01.hep.wisc.edu OSG_SITE_NAME=GLOW-OSG OSG_SQUID_LOCATION=osgsquid.hep.wisc.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_6@glidein_2274925_160595904@e2559.chtc.wisc.edu$ "; unset PROMPT_COMMAND PS1=slot1_6@glidein_2274925_160595904@e2559.chtc.wisc.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/slot1/dir_2274485/scratch/glide_RJYTsr/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/slot1/dir_2274485/scratch/glide_RJYTsr/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/slot1/dir_2274485/scratch/glide_RJYTsr/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/slot1/dir_2274485/scratch/glide_RJYTsr/execute/dir_235335/scratch _CONDOR_SLOT=slot1_6 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://squid-cs-2360.chtc.wisc.edu:3128
pass kernel
5.14.0-375.el9.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1067.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p2 1272933648 88675580 1119522836 8% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy http://squid-cs-2360.chtc.wisc.edu:3128
CHTC-DEV-PROD-EP-mwestphall 2026-04-03 11:01 1
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
CHTC-DEV-PROD-EP-mwestphall — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.090ms 1: _gateway 0.059ms 2: 128.104.100.2 0.571ms 3: 146.151.164.1 0.429ms 4: perfsonar02.hep.wisc.edu 0.418ms reached Resume: pmtu 1500 hops 4 back 4
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-gjynnW/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-gjynnW/execute/dir_7692 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=7692 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=CHTC PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_1@CHTC-DEV-PROD-EP-mwestphall.e1936bc0ca03$ "; unset PROMPT_COMMAND PS1=slot1_1@CHTC-DEV-PROD-EP-mwestphall.e1936bc0ca03$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-gjynnW/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-gjynnW/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-gjynnW/execute/dir_7692 _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass kernel
5.14.0-639.el9.aarch64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
72.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 263112772 174108396 75566220 70% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
CHTC-INTERACTIVE-OSPOOL-EP 2026-04-03 16:01 1
100.0%
100.0%
100.0%
100.0%
all
CHTC-INTERACTIVE-OSPOOL-EP — probe output
warning all
Job failed with exit code 0; no results transferred
CHTC-Jupyter-User-EP 2026-04-03 04:15 1
100.0%
100.0%
100.0%
100.0%
all
CHTC-Jupyter-User-EP — probe output
warning all
Job failed with exit code 0; no results transferred
CHTC-Spark-CE1 2026-04-03 17:01 29
3.4%
3.4%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
CHTC-Spark-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 4092 1: _gateway 0.076ms 1: _gateway 0.067ms 2: _gateway 0.065ms pmtu 1500 2: 128.104.100.2 0.510ms 3: 146.151.164.1 0.342ms 4: perfsonar02.hep.wisc.edu 0.308ms reached Resume: pmtu 1500 hops 4 back 4
pass env
APPTAINER_CACHEDIR=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch BASH_ENV=/usr/share/lmod/lmod/init/bash BASH_FUNC_ml%%=() { eval "$($LMOD_DIR/ml_cmd "$@")" } BASH_FUNC_module%%=() { if [ -z "${LMOD_SH_DBG_ON+x}" ]; then case "$-" in *v*x*) __lmod_sh_dbg='vx' ;; *v*) __lmod_sh_dbg='v' ;; *x*) __lmod_sh_dbg='x' ;; esac; fi; if [ -n "${__lmod_sh_dbg:-}" ]; then set +$__lmod_sh_dbg; echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output" 1>&2; fi; eval "$($LMOD_CMD shell "$@")" && eval "$(${LMOD_SETTARG_CMD:-:} -s sh)"; __lmod_my_status=$?; if [ -n "${__lmod_sh_dbg:-}" ]; then echo "Shell debugging restarted" 1>&2; set -$__lmod_sh_dbg; fi; unset __lmod_sh_dbg; return $__lmod_my_status } BASH_FUNC_which%%=() { ( alias; eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@ } BATCH_SYSTEM=HTCondor BLAHPD_LOCATION=/home/osg01/bosco/glite CONDORCE_COLLECTOR_HOST=chtc-spark-ce1.svc.osg-htc.org:9619 CONDOR_CONFIG=/var/lib/condor/execute/osg01/glide_7zROko/condor_config CONDOR_PARENT_ID=spark-agpu225:1990095:1775235672 CONDOR_PROCD_ADDRESS=/var/lib/condor/execute/osg01/glide_7zROko/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/var/lib/condor/execute/osg01/glide_7zROko/log/procd_address CONDOR_STARTER_PID=1990095 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/4977/bus DEBUGINFOD_URLS=https://debuginfod.centos.org/ ENVIRONMENT=BATCH GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=2c5a85deb24f1f4623e86c3eea56d0538d0b4afa GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/osg01/glide_7zROko/cred.d/idtokens/credential_OSPool.gpu_OSG_US_CHTC-Spark-CE1-gpu.idtoken GLIDEIN_CPUS=16 GLIDEIN_ClusterId=231034 GLIDEIN_Entry_Name=OSG_US_CHTC-Spark-CE1-gpu GLIDEIN_Entry_Signature=6a8a40ff6b1493b4db5ee3af399d679026c0e528 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=chtc-spark-ce1.svc.osg-htc.org chtc-spark-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_wBpFHB GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=CHTC-Spark-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=1a97d6060d26940eba08ee679d09a38f5c43d5c2 GLIDEIN_Site=CHTC-Spark GLIDEIN_Tmp_Dir=/var/lib/condor/execute/osg01/glide_7zROko/tmp GLIDEIN_UUID=90d3a68c-3e51-4dde-afee-85c12d4278bc GLITE_LOCATION=/home/osg01/bosco/glite GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_PATH=/var/lib/condor/execute/osg01/glide_7zROko/.gwms.d/bin: GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HISTCONTROL=ignoredups HISTSIZE=1000 HOLD_GRACE_TIME=0 HOME=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch HOSTNAME=spark-agpu225 HYDRA_BOOTSTRAP=slurm HYDRA_LAUNCHER_EXTRA_ARGS=--external-launcher I_MPI_HYDRA_BOOTSTRAP=slurm I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_OSG_US_CHTC-Spark-CE1-gpu/tokens.tgz JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LESSOPEN=||/usr/bin/lesspipe.sh %s LMOD_CMD=/usr/share/lmod/lmod/libexec/lmod LMOD_DIR=/usr/share/lmod/lmod/libexec LMOD_PKG=/usr/share/lmod/lmod LMOD_ROOT=/usr/share/lmod LMOD_SETTARG_FULL_SUPPORT=no LMOD_VERSION=8.7.32 LMOD_sys=Linux LOGNAME=osg01 MAIL=/var/spool/mail/osg01 MANPATH=/usr/share/lmod/lmod/share/man::/opt/puppetlabs/puppet/share/man MKL_NUM_THREADS=1 MODULEPATH=/software/chtc/el9/modules/zen3/:/software/chtc/manual/el9/modules:/etc/modulefiles:/usr/share/modulefiles MODULEPATH_ROOT=/usr/share/modulefiles MODULESHOME=/usr/share/lmod/lmod NODE_COUNT=16 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OLDPWD=/var/lib/condor/execute/osg01/glide_7zROko/logs/90d3a68c-3e51-4dde-afee-85c12d4278bc OMPI_MCA_plm_slurm_args=--external-launcher OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=UNAVAILABLE OSG_DATA=UNAVAILABLE OSG_GRID=/root OSG_HOSTNAME=localhost OSG_SITE_NAME=CHTC-Spark-CE1 OSG_WN_TMP=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch/.local-tmp PATH=/var/lib/condor/execute/osg01/glide_7zROko/.gwms.d/bin::/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin PREEMPT_GRACE_TIME=10000000 PRTE_MCA_plm_slurm_args=--external-launcher PWD=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch PYTHON_CPU_COUNT=1 ROCR_VISIBLE_DEVICES=0 ROOT_MAX_THREADS=1 SHELL=/bin/bash SHLVL=0 SINGULARITY_CACHEDIR=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9-cuda-12.6.0 SINGULARITY_MODE=unprivileged SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 SLOTS_LAYOUT=partitionable SLURMD_NODENAME=spark-agpu225 SLURM_CLUSTER_NAME=spark_el9 SLURM_CONF=/etc/slurm/slurm.conf SLURM_CPUS_ON_NODE=16 SLURM_GPUS_ON_NODE=1 SLURM_GTIDS=0 SLURM_JOBID=896883 SLURM_JOB_ACCOUNT=osg SLURM_JOB_CPUS_PER_NODE=16 SLURM_JOB_END_TIME=1775276501 SLURM_JOB_GID=4977 SLURM_JOB_GPUS=1 SLURM_JOB_ID=896883 SLURM_JOB_NAME=bl_2rWNlO SLURM_JOB_NODELIST=spark-agpu225 SLURM_JOB_NUM_NODES=1 SLURM_JOB_PARTITION=gpu SLURM_JOB_QOS=normal SLURM_JOB_START_TIME=1775017301 SLURM_JOB_UID=4977 SLURM_JOB_USER=osg01 SLURM_LOCALID=0 SLURM_MEM_PER_NODE=126500 SLURM_NNODES=1 SLURM_NODEID=0 SLURM_NODELIST=spark-agpu225 SLURM_NPROCS=16 SLURM_NTASKS=16 SLURM_NTASKS_PER_NODE=16 SLURM_PRIO_PROCESS=0 SLURM_PROCID=0 SLURM_SUBMIT_DIR=/home/osg01/bosco/sandbox/a054/a0545609/chtc-spark-ce1.svc.osg-htc.org_9619_chtc-spark-ce1.svc.osg-htc.org_365407.0_1775017273 SLURM_SUBMIT_HOST=spark-login SLURM_TASKS_PER_NODE=16 SLURM_TASK_PID=818222 SLURM_TOPOLOGY_ADDR=S[1-2].L4.spark-agpu225 SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node SSH_CLIENT=128.105.82.84 51367 22 SSH_CONNECTION=128.105.82.84 51367 10.130.203.250 22 TEMP=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch TMPDIR=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime USER=osg01 XDG_DATA_DIRS=/home/osg01/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share XDG_RUNTIME_DIR=/run/user/4977 XDG_SESSION_CLASS=user XDG_SESSION_ID=c1874 XDG_SESSION_TYPE=tty ZE_AFFINITY_MASK=0 _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_1017537=1017540:1775036344:3679031635 _CONDOR_ANCESTOR_1017540=1990095:1775235672:1158712940 _CONDOR_ANCESTOR_1990095=1990464:1775235672:2927033024 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/osg01/glide_7zROko/main/condor/bin _CONDOR_CHIRP_CONFIG=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch/.chirp.config _CONDOR_CREDS=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch/.condor_creds _CONDOR_EXECUTE=/var/lib/condor/execute/osg01/glide_7zROko/execute _CONDOR_JOB_AD=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch/.job.ad _CONDOR_JOB_IWD=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/var/lib/condor/execute/osg01/glide_7zROko/execute/dir_1990095/scratch/.job_wrapper_failure which_declare=declare -f
pass kernel
5.14.0-511.el9.x86_64
fail nrp_images
Failed to download https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt: <urlopen error _ssl.c:1116: The handshake operation timed out>
pass df
2770.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme1n1p2 3075412536 14531768 2904583708 1% /var/lib/condor/execute
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
CHTC-ce2000 2026-04-03 17:01 7
100.0%
100.0%
71.4%
100.0%
mtu env kernel nrp_images df http_proxy
CHTC-ce2000 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.002ms pmtu 1500 1: 2607:f388:2200:100::2 1.280ms 1: 2607:f388:2200:100::2 1.085ms 2: 2607:f388:0:377::2 0.684ms 3: perfsonar02.hep.wisc.edu 0.514ms reached Resume: pmtu 1500 hops 3 back 3
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/slot1/dir_2124230/scratch/glide_F5KK2I/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/slot1/dir_2124230/scratch/glide_F5KK2I/execute/dir_2851048/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2851048 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 FTP_PROXY=http://squid-cs-2360.chtc.wisc.edu:3128 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/slot1/dir_2124230/scratch/glide_F5KK2I/cred.d/idtokens/credential_OSPool.main_OSG_US_CHTC-ce2000.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=6282739 GLIDEIN_Entry_Name=OSG_US_CHTC-ce2000 GLIDEIN_Entry_Signature=6ae1b3c22b6a7150dc8eb1879b4a637415217f53 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=ce2000.chtc.wisc.edu ce2000.chtc.wisc.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgvopilot_pGZPyQ GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=squid-cs-2360.chtc.wisc.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=CHTC-ce2000 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=CHTC GLIDEIN_Tmp_Dir=/var/lib/condor/execute/slot1/dir_2124230/scratch/glide_F5KK2I/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ce2000.chtc.wisc.edu OSG_SITE_NAME=CHTC-ce2000 OSG_SQUID_LOCATION=squid-cs-2360.chtc.wisc.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_2124338_59057775@oconnor4000.chtc.wisc.edu$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_2124338_59057775@oconnor4000.chtc.wisc.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/slot1/dir_2124230/scratch/glide_F5KK2I/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/slot1/dir_2124230/scratch/glide_F5KK2I/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/slot1/dir_2124230/scratch/glide_F5KK2I/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/slot1/dir_2124230/scratch/glide_F5KK2I/execute/dir_2851048/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://squid-cs-2360.chtc.wisc.edu:3128
pass kernel
5.14.0-375.el9.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
61.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/condor_vg-backfill1_1+1073709_7674 66002476 1681932 64304160 3% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy http://squid-cs-2360.chtc.wisc.edu:3128
CHTC-ce2001 2026-04-03 17:07 7
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
CHTC-ce2001 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.008ms pmtu 1500 1: 2607:f388:2200:100::3 1.020ms 1: 2607:f388:2200:100::3 0.991ms 2: 2607:f388:0:33f::2 0.344ms 3: perfsonar02.hep.wisc.edu 0.266ms reached Resume: pmtu 1500 hops 3 back 3
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/slot1/dir_1325457/scratch/glide_QLMId1/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/slot1/dir_1325457/scratch/glide_QLMId1/execute/dir_3805102/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3805102 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 FTP_PROXY=http://squid-cs-2360.chtc.wisc.edu:3128 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/slot1/dir_1325457/scratch/glide_QLMId1/cred.d/idtokens/credential_OSPool.main_OSG_US_CHTC-ce2001.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=235299 GLIDEIN_Entry_Name=OSG_US_CHTC-ce2001 GLIDEIN_Entry_Signature=a22932609216d97c38e298685884359010ea44dd GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=ce2001.chtc.wisc.edu ce2001.chtc.wisc.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgvopilot_fHXBz7 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=squid-cs-2360.chtc.wisc.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=CHTC-ce2001 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=19a3aab0fae658d7866b25a2d8ffcbedc8d5e443 GLIDEIN_Site=CHTC GLIDEIN_Tmp_Dir=/var/lib/condor/execute/slot1/dir_1325457/scratch/glide_QLMId1/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ce2000.chtc.wisc.edu OSG_SITE_NAME=CHTC-ce2000 OSG_SQUID_LOCATION=squid-cs-2360.chtc.wisc.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_7@glidein_1325459_796156844@e2486.chtc.wisc.edu$ "; unset PROMPT_COMMAND PS1=slot1_7@glidein_1325459_796156844@e2486.chtc.wisc.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/slot1/dir_1325457/scratch/glide_QLMId1/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/slot1/dir_1325457/scratch/glide_QLMId1/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/slot1/dir_1325457/scratch/glide_QLMId1/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/slot1/dir_1325457/scratch/glide_QLMId1/execute/dir_3805102/scratch _CONDOR_SLOT=slot1_7 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://squid-cs-2360.chtc.wisc.edu:3128
pass kernel
5.14.0-375.el9.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
579.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sdb2 657607136 16213752 607914320 3% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy http://squid-cs-2360.chtc.wisc.edu:3128
CU-Boulder-CS-Cloud-EP 2026-04-03 17:01 1
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
CU-Boulder-CS-Cloud-EP — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: host.containers.internal 0.111ms 1: host.containers.internal 0.158ms 2: 10.224.144.1 0.403ms 3: 10.224.52.253 0.577ms 4: sn-campus-tcom.colorado.edu 0.276ms 5: biot-tcom.colorado.edu 0.274ms 6: ucbcomp-biot.colorado.edu 1.510ms asymm 7 7: compmx-ucbcomp.colorado.edu 1.727ms 8: tcommx-compmx.colorado.edu 1.908ms 9: core-1850.frgp.net 2.326ms 10: fourhundredge-0-0-0-3.3468.core2.denv.net.internet2.edu 5.374ms 11: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 29.625ms asymm 15 12: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 27.197ms asymm 14 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 27.530ms 14: 144.92.233.112 30.226ms 15: perfsonar02.hep.wisc.edu 28.889ms reached Resume: pmtu 1500 hops 15 back 16
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-OIkjC6/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-OIkjC6/execute/dir_137580 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=137580 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=CU Boulder Comp Sci OSG_SQUID_LOCATION=osg-frontier-squid.cs.colorado.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_18@CU-Boulder-CS-Cloud-EP.eaf0058d3bdf$ "; unset PROMPT_COMMAND PS1=slot1_18@CU-Boulder-CS-Cloud-EP.eaf0058d3bdf$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-OIkjC6/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-OIkjC6/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-OIkjC6/execute/dir_137580 _CONDOR_SLOT=slot1_18 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=osg-frontier-squid.cs.colorado.edu:3128
pass kernel
4.18.0-553.92.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
212.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 250991596 27740312 223251284 12% /srv
fail http_proxy
Failed to download http://cm-1.ospool.osg-htc.org/overview/ via proxy osg-frontier-squid.cs.colorado.edu:3128: HTTP Error 403: Forbidden
Cincinnati-ARC-OSG-CE1 2026-04-03 17:03 8
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
Cincinnati-ARC-OSG-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: arc-osg 0.235ms 1: arc-osg 0.235ms 2: msbb-00g95-hpc.manage.uc.edu 0.617ms 3: 10.0.27.17 0.633ms 4: 10.0.26.253 0.722ms 5: 10.0.0.2 1.189ms 6: 129.137.1.3 1.987ms 7: 129.137.1.5 2.765ms 8: 192.148.242.29 2.486ms asymm 9 9: cncno-r5-et-4-3-0s735.core.oar.net 2.468ms asymm 8 10: hundredge-0-0-0-25.1260.core1.cinc.net.internet2.edu 4.394ms asymm 9 11: fourhundredge-0-0-0-21.4079.core2.cinc.net.internet2.edu 10.104ms asymm 14 12: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 9.215ms asymm 13 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 10.900ms asymm 12 14: 144.92.233.112 12.015ms asymm 13 15: perfsonar02.hep.wisc.edu 11.794ms reached Resume: pmtu 1500 hops 15 back 15
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch-local/glide_LAVpdL/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/scratch-local/glide_LAVpdL/execute/dir_1776953/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1776953 CONTAINERSW_FULL_VERSION=apptainer version 1.3.6-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.3.6-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch-local/glide_LAVpdL/cred.d/idtokens/credential_OSPool.main_OSG_US_Cincinnati-ARC-OSG-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=5671384 GLIDEIN_Entry_Name=OSG_US_Cincinnati-ARC-OSG-CE1 GLIDEIN_Entry_Signature=fe2fbd566c690700bd910eb3c994021892f425f1 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=cincinnati-arc-osg-ce1.svc.opensciencegrid.org cincinnati-arc-osg-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_4OY2j1 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Cincinnati-ARC-OSG-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=5552c705dd9f3b999102cc42d5c5bdce18bba000 GLIDEIN_Site=Cincinnati-ARC-OSG GLIDEIN_Tmp_Dir=/scratch-local/glide_LAVpdL/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.3.6-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.3.6-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.3.6-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Cincinnati-ARC-OSG-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_950532_67188198@compute-34.localdomain$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_950532_67188198@compute-34.localdomain$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch-local/glide_LAVpdL/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.3.6-1.el8 SINGULARITY_IMAGES_DICT=default:/scratch-local/glide_LAVpdL/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.3.6-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch-local/glide_LAVpdL/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch-local/glide_LAVpdL/execute/dir_1776953/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.30.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
375.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/localscratch-lv_local_scratch 460365864 43089152 393817896 10% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Clemson-Palmetto-CE1 2026-04-03 17:01 274
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
Clemson-Palmetto-CE1 — probe output
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: _gateway 12.032ms 1: _gateway 31.971ms 2: 205-186-62-81.generic.c-light.net 0.628ms 3: xe-1-1-1-816-t01-cle.culr.net 0.965ms 4: xe-1-1-1-816-t01-sox.culr.net 11.786ms asymm 9 5: sidco-to-56mar-al2s.sox.net 17.571ms asymm 10 6: i2-to-sox-re.sidco.sox.net 19.158ms asymm 9 7: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 26.067ms asymm 13 8: fourhundredge-0-0-0-21.4079.core2.cinc.net.internet2.edu 26.978ms asymm 12 9: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 24.800ms asymm 11 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 25.854ms 11: 144.92.233.112 27.358ms 12: perfsonar02.hep.wisc.edu 27.746ms reached Resume: pmtu 9000 hops 12 back 13
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local_scratch/slurm.12285848/glide_U6YKqj/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/local_scratch/slurm.12285848/glide_U6YKqj/execute/dir_75601/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=75601 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local_scratch/slurm.12285848/glide_U6YKqj/cred.d/idtokens/credential_OSPool.main_OSG_US_Clemson-Palmetto-CE1.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=342485 GLIDEIN_Entry_Name=OSG_US_Clemson-Palmetto-CE1 GLIDEIN_Entry_Signature=63eab2392affc5b2204573e5c248a97d722e39f3 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=clemson-palmetto-ce1.svc.opensciencegrid.org clemson-palmetto-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg001_U3JTGQ GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Clemson-Palmetto-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=1a97d6060d26940eba08ee679d09a38f5c43d5c2 GLIDEIN_Site=Clemson-Palmetto GLIDEIN_Tmp_Dir=/local_scratch/slurm.12285848/glide_U6YKqj/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Clemson-Palmetto-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_7@glidein_3122361_169082172@node1825.palmetto.clemson.edu$ "; unset PROMPT_COMMAND PS1=slot1_7@glidein_3122361_169082172@node1825.palmetto.clemson.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/local_scratch/slurm.12285848/glide_U6YKqj/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el8 SINGULARITY_IMAGES_DICT=default:/local_scratch/slurm.12285848/glide_U6YKqj/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local_scratch/slurm.12285848/glide_U6YKqj/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local_scratch/slurm.12285848/glide_U6YKqj/execute/dir_75601/scratch _CONDOR_SLOT=slot1_7 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.50.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
400.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda6 433644668 13875344 419769324 4% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Colgate-CCARE-EP 2026-04-03 17:19 2
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
Colgate-CCARE-EP — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.071ms 1: _gateway 0.127ms 2: 149.43.3.1 0.540ms 3: 172.16.1.11 1.490ms 4: alb-9204-colgate.nysernet.net 3.764ms 5: syr-57c3-alb-9204.nysernet.net 4.884ms 6: buf-55a1-syr-57c3.nysernet.net 7.647ms 7: buf-9208-buf-55a1.nysernet.net 7.688ms asymm 5 8: i2-clev-buf-9208.nysernet.net 12.682ms asymm 6 9: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 19.709ms 10: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 20.149ms asymm 8 11: 144.92.233.112 21.097ms asymm 9 12: perfsonar02.hep.wisc.edu 20.957ms reached Resume: pmtu 1500 hops 12 back 11
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-vS1dj6/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-vS1dj6/execute/dir_326514 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=326514 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Colgate ITS OSG_SQUID_LOCATION=http://osg.research.colgate.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_8@Colgate-CCARE-EP.c3bcaf819c19$ "; unset PROMPT_COMMAND PS1=slot1_8@Colgate-CCARE-EP.c3bcaf819c19$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-vS1dj6/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-vS1dj6/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-vS1dj6/execute/dir_326514 _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=http://osg.research.colgate.edu:3128
pass kernel
6.8.0-49-generic
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1211.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/ccare--g04--vg-ccare--g04--lv 1837860872 473918812 1270510048 28% /srv
fail http_proxy
Failed to download http://cm-1.ospool.osg-htc.org/overview/ via proxy http://osg.research.colgate.edu:3128: <urlopen error timed out>
DESY-ZN 2026-04-03 16:01 956
100.0%
100.0%
100.0%
100.0%
all
DESY-ZN — probe output
warning all
Job failed with exit code 0; no results transferred
Duke-NCShare-CE1 2026-04-03 17:04 129
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
Duke-NCShare-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.560ms asymm 2 1: _gateway 0.418ms asymm 2 2: rtp-gw-to-scidmz-gw-1.ncren.net 0.903ms 3: bundle-ether250.613.core1.rale.net.internet2.edu 4.747ms 4: fourhundredge-0-0-0-6.4079.core1.ashb.net.internet2.edu 22.381ms asymm 12 5: fourhundredge-0-0-0-18.4079.core2.ashb.net.internet2.edu 21.227ms asymm 11 6: fourhundredge-0-0-0-1.4079.core2.clev.net.internet2.edu 22.752ms asymm 10 7: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 21.977ms asymm 9 8: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 22.338ms 9: 144.92.233.112 23.896ms 10: perfsonar02.hep.wisc.edu 23.822ms reached Resume: pmtu 1500 hops 10 back 11
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_iGxSts/images/htc__almalinux__10.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__almalinux__10.sif APPTAINER_TMPDIR=/scratch/glide_iGxSts/execute/dir_2552821/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2552821 CONTAINERSW_FULL_VERSION=apptainer version 1.4.3 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.3 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_iGxSts/cred.d/idtokens/credential_OSPool.main_OSG_US_Duke-NCShare-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=303450 GLIDEIN_Entry_Name=OSG_US_Duke-NCShare-CE1 GLIDEIN_Entry_Signature=aecab05b7abd316dbe2f4a1bc24d92548f319c6b GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=duke-ncshare-ce1.svc.osg-htc.org duke-ncshare-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_nosg1_yh1LQz GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Duke-NCShare-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=Duke-NCShare GLIDEIN_Tmp_Dir=/scratch/glide_iGxSts/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.3 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.3 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.3 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Duke-NCShare-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2998566_839009921@compute-07$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2998566_839009921@compute-07$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_iGxSts/images/htc__almalinux__10.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.3 SINGULARITY_IMAGES_DICT=default:/scratch/glide_iGxSts/images/htc__almalinux__10.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__almalinux__10.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.3 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_iGxSts/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_iGxSts/execute/dir_2552821/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
6.8.0-85-generic
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
642.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/ubuntu--vg-lstor--lv 824572824 109335936 673277464 14% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
FANDM-ITS-CE1 2026-04-03 17:01 32
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
FANDM-ITS-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: cv01.cluster 0.225ms 1: cv01.cluster 0.158ms 2: 10.68.128.1 8.453ms 3: 10.68.0.9 18.376ms 4: 10.68.0.14 0.396ms 5: 10.255.255.25 0.959ms 6: 10.255.255.34 0.960ms asymm 5 7: 164.153.140.76 11.997ms asymm 14 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: 144.92.233.112 25.170ms asymm 17 14: perfsonar02.hep.wisc.edu 25.165ms reached Resume: pmtu 1500 hops 14 back 19
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_ElitgI/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_ElitgI/execute/dir_186910/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=186910 CONTAINERSW_FULL_VERSION=apptainer version 1.4.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_ElitgI/cred.d/idtokens/credential_OSPool.main_OSG_US_FANDM-ITS-CE1.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=6107637 GLIDEIN_Entry_Name=OSG_US_FANDM-ITS-CE1 GLIDEIN_Entry_Signature=0fea0a9f7fc4dbeccde3e62a6ef1b1888d394dba GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=fandm-its-ce1.svc.opensciencegrid.org fandm-its-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_hXCH1G GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=FANDM-ITS-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-1.osg-htc.org GLIDEIN_Signature=5552c705dd9f3b999102cc42d5c5bdce18bba000 GLIDEIN_Site=FANDM-ITS GLIDEIN_Tmp_Dir=/scratch/glide_ElitgI/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=FANDM-ITS-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_2641043_319318328@n14.cluster$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_2641043_319318328@n14.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_ElitgI/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.0-1.el8 SINGULARITY_IMAGES_DICT=default:/scratch/glide_ElitgI/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_ElitgI/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_ElitgI/execute/dir_186910/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.50.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1023.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p5 1092553500 18895424 1073658076 2% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
FIU-IRCC-CE1 2026-04-03 17:04 15
0.0%
0.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
FIU-IRCC-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: hpcosgce1.panther.net 0.131ms 1: hpcosgce1.panther.net 0.125ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass env
APPTAINER_CACHEDIR=/scratch/osg/glide_Q27nSn/execute/dir_22586 BASH_ENV=/home/share/Modules/4.1.3/init/bash BASH_FUNC_module%%=() { unset _mlre _mlIFS _mlshdbg; if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then case "$-" in *v*x*) set +vx; _mlshdbg='vx' ;; *v*) set +v; _mlshdbg='v' ;; *x*) set +x; _mlshdbg='x' ;; *) _mlshdbg='' ;; esac; fi; if [ -n "${IFS+x}" ]; then _mlIFS=$IFS; fi; IFS=' '; for _mlv in ${MODULES_RUN_QUARANTINE:-}; do if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' "; fi; _mlrv="MODULES_RUNENV_${_mlv}"; _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' "; fi; done; if [ -n "${_mlre:-}" ]; then eval `eval ${_mlre}/usr/bin/tclsh /home/share/Modules/4.1.3/libexec/modulecmd.tcl bash '"$@"'`; else eval `/usr/bin/tclsh /home/share/Modules/4.1.3/libexec/modulecmd.tcl bash "$@"`; fi; _mlstatus=$?; if [ -n "${_mlIFS+x}" ]; then IFS=$_mlIFS; else unset IFS; fi; if [ -n "${_mlshdbg:-}" ]; then set -$_mlshdbg; fi; unset _mlre _mlv _mlrv _mlIFS _mlshdbg; return $_mlstatus } BASH_FUNC_module()=() { unset _mlre _mlIFS _mlshdbg; if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then case "$-" in *v*x*) set +vx; _mlshdbg='vx' ;; *v*) set +v; _mlshdbg='v' ;; *x*) set +x; _mlshdbg='x' ;; *) _mlshdbg='' ;; esac; fi; if [ -n "${IFS+x}" ]; then _mlIFS=$IFS; fi; IFS=' '; for _mlv in ${MODULES_RUN_QUARANTINE:-}; do if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' "; fi; _mlrv="MODULES_RUNENV_${_mlv}"; _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' "; fi; done; if [ -n "${_mlre:-}" ]; then eval `eval ${_mlre}/usr/bin/tclsh /home/share/Modules/4.1.3/libexec/modulecmd.tcl bash '"$@"'`; else eval `/usr/bin/tclsh /home/share/Modules/4.1.3/libexec/modulecmd.tcl bash "$@"`; fi; _mlstatus=$?; if [ -n "${_mlIFS+x}" ]; then IFS=$_mlIFS; else unset IFS; fi; if [ -n "${_mlshdbg:-}" ]; then set -$_mlshdbg; fi; unset _mlre _mlv _mlrv _mlIFS _mlshdbg; return $_mlstatus } BASH_FUNC_switchml%%=() { typeset swfound=1; if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then typeset swname='main'; if [ -e /home/share/Modules/4.1.3/libexec/modulecmd.tcl ]; then typeset swfound=0; unset MODULES_USE_COMPAT_VERSION; fi; else typeset swname='compatibility'; if [ -e /home/share/Modules/4.1.3/libexec/modulecmd-compat ]; then typeset swfound=0; MODULES_USE_COMPAT_VERSION=1; export MODULES_USE_COMPAT_VERSION; fi; fi; if [ $swfound -eq 0 ]; then echo "Switching to Modules $swname version"; source /home/share/Modules/4.1.3/init/bash; else echo "Cannot switch to Modules $swname version, command not found"; return 1; fi } BASH_FUNC_switchml()=() { typeset swfound=1; if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then typeset swname='main'; if [ -e /home/share/Modules/4.1.3/libexec/modulecmd.tcl ]; then typeset swfound=0; unset MODULES_USE_COMPAT_VERSION; fi; else typeset swname='compatibility'; if [ -e /home/share/Modules/4.1.3/libexec/modulecmd-compat ]; then typeset swfound=0; MODULES_USE_COMPAT_VERSION=1; export MODULES_USE_COMPAT_VERSION; fi; fi; if [ $swfound -eq 0 ]; then echo "Switching to Modules $swname version"; source /home/share/Modules/4.1.3/init/bash; else echo "Cannot switch to Modules $swname version, command not found"; return 1; fi } BASH_FUNC_which%%=() { ( alias; eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@" } BATCH_SYSTEM=HTCondor BLAHPD_LOCATION=/home/osg/bosco/glite CMAKE_PREFIX_PATH=/home/share/slurm/current CMAKE_PREFIX_PATH_modshare=/home/share/slurm/current:1 CONDORCE_COLLECTOR_HOST=fiu-ircc-ce1.svc.osg-htc.org:9619 CONDOR_CONFIG=/scratch/osg/glide_Q27nSn/condor_config CONDOR_PARENT_ID=n013:22586:1775235788 CONDOR_PROCD_ADDRESS=/scratch/osg/glide_Q27nSn/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/scratch/osg/glide_Q27nSn/log/procd_address CPATH=/home/share/slurm/current/include CPATH_modshare=/home/share/slurm/current/include:1 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 ENV=/home/share/Modules/4.1.3/init/profile.sh ENVIRONMENT=BATCH GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg/glide_Q27nSn/cred.d/idtokens/credential_OSPool.main_OSG_US_FIU-IRCC.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=5611813 GLIDEIN_Entry_Name=OSG_US_FIU-IRCC GLIDEIN_Entry_Signature=7868ff89a2ec0d3089299d9d8671204818aec99b GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=fiu-ircc-ce1.svc.osg-htc.org fiu-ircc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_AbU6kQ GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=FIU-IRCC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=FIU-IRCC GLIDEIN_Tmp_Dir=/scratch/osg/glide_Q27nSn/tmp GLIDEIN_UUID=162829ec-c918-407f-a78c-b16f6b18d62c GLITE_LOCATION=/home/osg/bosco/glite GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_PATH=/scratch/osg/glide_Q27nSn/.gwms.d/bin: GWMS_SINGULARITY_STATUS=PREFERRED HAS_SINGULARITY=False HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HISTCONTROL=ignoredups HISTSIZE=1000 HOLD_GRACE_TIME=0 HOME=/scratch/osg/glide_Q27nSn/execute/dir_22586 HOSTNAME=n013 JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_OSG_US_FIU-IRCC/tokens.tgz JULIA_NUM_THREADS=1 LANG=C LD_LIBRARY_PATH=/home/share/slurm/current/lib LD_LIBRARY_PATH_modshare=/home/share/slurm/current/lib:1 LESSOPEN=||/usr/bin/lesspipe.sh %s LIBRARY_PATH=/home/share/slurm/current/lib LIBRARY_PATH_modshare=/home/share/slurm/current/lib:1 LOADEDMODULES=slurm LOADEDMODULES_modshare=slurm:1 LOGNAME=osg MAIL=/var/spool/mail/osg MANPATH=/home/share/Modules/4.1.3/share/man MKL_NUM_THREADS=1 MODULEPATH=/home/share/Modules/4.1.3/modulefiles/linux-centos7-x86_64 MODULEPATH_modshare=/home/share/Modules/4.1.3/modulefiles/linux-centos7-x86_64:1 MODULESHOME=/home/share/Modules/4.1.3 MODULES_CMD=/home/share/Modules/4.1.3/libexec/modulecmd.tcl NODE_COUNT=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=UNAVAILABLE OSG_DATA=UNAVAILABLE OSG_GRID=/cvmfs/oasis.opensciencegrid.org/osg-software/osg-wn-client/current/el7-x86_64/ OSG_HOSTNAME=localhost OSG_SITE_NAME=FIU-IRCC-CE1 OSG_WN_TMP=/scratch/osg/glide_Q27nSn/execute/dir_22586/.local-tmp PATH=/scratch/osg/glide_Q27nSn/.gwms.d/bin::/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/share/slurm/current/bin:/home/share/Modules/4.1.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/osg/bin PATH_modshare=/usr/sbin:1:/home/share/Modules/4.1.3/bin:1:/usr/bin:1:/home/share/slurm/current/bin:1:/usr/local/sbin:1:/usr/local/bin:1 PREEMPT_GRACE_TIME=10000000 PWD=/scratch/osg/glide_Q27nSn/execute/dir_22586 PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SHELL=/bin/bash SHLVL=6 SINGULARITY_CACHEDIR=/scratch/osg/glide_Q27nSn/execute/dir_22586 SLOTS_LAYOUT=fixed SLURMD_NODENAME=n013 SLURM_CLUSTER_NAME=fiuhpcslurm SLURM_CONF=/home/share/slurm/22.05.6/etc/slurm.conf SLURM_CPUS_ON_NODE=2 SLURM_GTIDS=0 SLURM_JOBID=2750354 SLURM_JOB_ACCOUNT=osg SLURM_JOB_CPUS_PER_NODE=2 SLURM_JOB_GID=109 SLURM_JOB_ID=2750354 SLURM_JOB_NAME=bl_X15vWj SLURM_JOB_NODELIST=n013 SLURM_JOB_NUM_NODES=1 SLURM_JOB_PARTITION=osg SLURM_JOB_QOS=osg SLURM_JOB_UID=84159 SLURM_JOB_USER=osg SLURM_LOCALID=0 SLURM_MEM_PER_NODE=8192 SLURM_NNODES=1 SLURM_NODEID=0 SLURM_NODELIST=n013 SLURM_NODE_ALIASES=(null) SLURM_NPROCS=1 SLURM_NTASKS=1 SLURM_NTASKS_PER_NODE=1 SLURM_PRIO_PROCESS=0 SLURM_PROCID=0 SLURM_SUBMIT_DIR=/home/osg/bosco/sandbox/35bb/35bb529d/fiu-ircc-ce1.svc.osg-htc.org_9619_fiu-ircc-ce1.svc.osg-htc.org_14888.0_1775225845 SLURM_SUBMIT_HOST=hpcosgce1.panther.net SLURM_TASKS_PER_NODE=1 SLURM_TASK_PID=26966 SLURM_TOPOLOGY_ADDR=n013 SLURM_TOPOLOGY_ADDR_PATTERN=node SLURM_WORKING_CLUSTER=fiuhpcslurm:ms1:6817:9728:101 SSH_CLIENT=128.105.82.84 59707 22 SSH_CONNECTION=128.105.82.84 59707 131.94.55.228 22 S_COLORS=auto TEMP=/scratch/osg/glide_Q27nSn/execute/dir_22586 TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/scratch/osg/glide_Q27nSn/execute/dir_22586 TMPDIR=/scratch/osg/glide_Q27nSn/execute/dir_22586/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime USER=osg _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_22586=8826:1775235790:4240784779 _CONDOR_ANCESTOR_3311=8048:1775227100:3407179689 _CONDOR_ANCESTOR_8048=22586:1775235765:2076907510 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg/glide_Q27nSn/main/condor/bin _CONDOR_CHIRP_CONFIG=/scratch/osg/glide_Q27nSn/execute/dir_22586/.chirp.config _CONDOR_CREDS=/scratch/osg/glide_Q27nSn/execute/dir_22586/.condor_creds _CONDOR_EXECUTE=/scratch/osg/glide_Q27nSn/execute _CONDOR_JOB_AD=/scratch/osg/glide_Q27nSn/execute/dir_22586/.job.ad _CONDOR_JOB_IWD=/scratch/osg/glide_Q27nSn/execute/dir_22586 _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/scratch/osg/glide_Q27nSn/execute/dir_22586/.machine.ad _CONDOR_SCRATCH_DIR=/scratch/osg/glide_Q27nSn/execute/dir_22586 _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/scratch/osg/glide_Q27nSn/execute/dir_22586/.job_wrapper_failure _LMFILES_=/home/share/Modules/4.1.3/modulefiles/linux-centos7-x86_64/slurm _LMFILES__modshare=/home/share/Modules/4.1.3/modulefiles/linux-centos7-x86_64/slurm:1 which_declare=declare -f
warning kernel
3.10.0-1127.el7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
41051.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 192.168.232.21@tcp:192.168.232.22@tcp:/scratch 50398964480 6843090280 43045744136 14% /scratch
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
FSU-HNPGRID 2026-04-03 17:02 46
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
FSU-HNPGRID — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: vlan99.6506.ken.fsu.edu 0.974ms 1: vlan99.6506.ken.fsu.edu 0.797ms 2: no reply 3: no reply 4: no reply 5: irb100.rsb-br.net.fsu.edu 1.475ms 6: 128.186.253.197 1.907ms 7: 128.186.253.192 2.093ms 8: tlh-flrcore-asr9010-1-te0203-1902.net.flrnet.org 5.758ms 9: tlh-bb-rtr-2-fh0001-1.rtr.net.flrnet.org 7.495ms asymm 11 10: jax-bb-rtr-1-fh0001-1.rtr.net.flrnet.org 8.325ms 11: prov-i2-atla-renet-1070.net.flrnet.org 7.704ms 12: fourhundredge-0-0-0-6.4079.core1.atla.net.internet2.edu 31.179ms asymm 24 13: fourhundredge-0-0-0-22.4079.core2.atla.net.internet2.edu 31.445ms asymm 23 14: fourhundredge-0-0-0-2.4079.core2.nash.net.internet2.edu 30.246ms asymm 22 15: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 31.174ms asymm 21 16: fourhundredge-0-0-0-21.4079.core2.cinc.net.internet2.edu 31.214ms asymm 20 17: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 30.490ms asymm 19 18: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 30.462ms 19: 144.92.233.112 32.625ms 20: perfsonar02.hep.wisc.edu 32.554ms reached Resume: pmtu 1500 hops 20 back 21
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_5rPDIX/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_5rPDIX/execute/dir_2289053/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2289053 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_5rPDIX/cred.d/idtokens/credential_OSPool.main_GLUEX_US_FSU-HNPGRID.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=302715 GLIDEIN_Entry_Name=GLUEX_US_FSU-HNPGRID GLIDEIN_Entry_Signature=dc6a2109f1581b4f868e81061531813cf15c5d43 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=fsu-hnpgrid-ce1.svc.osg-htc.org fsu-hnpgrid-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_n0oV1P GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=FSU-HNPGRID GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=19a3aab0fae658d7866b25a2d8ffcbedc8d5e443 GLIDEIN_Site=OSG_US_FSU_HNPGRID GLIDEIN_Tmp_Dir=/scratch/glide_5rPDIX/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=FSU-HNPGRID PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_6@glidein_2937635_432026273@scigrid2.physics.fsu.edu$ "; unset PROMPT_COMMAND PS1=slot1_6@glidein_2937635_432026273@scigrid2.physics.fsu.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_5rPDIX/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el9 SINGULARITY_IMAGES_DICT=default:/scratch/glide_5rPDIX/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_5rPDIX/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_5rPDIX/execute/dir_2289053/scratch _CONDOR_SLOT=slot1_6 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.27.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
899.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda2 1056680372 59927904 943048996 6% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
GP-ARGO-ksu-backfill 2026-04-03 17:05 1
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
GP-ARGO-ksu-backfill — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.113ms 1: _gateway 0.061ms 2: ptx-bb-ksu-hale-et0-0-18-0.bb.net.kanren.net 0.259ms 3: ptx-bb-ku-ellsworth-et0-0-0-0.bb.net.kanren.net 1.998ms 4: ptx-bb-ku-price-et0-0-0-0.bb.net.kanren.net 2.136ms 5: ptx-bb-kc-walnut-et0-0-0-0.bb.net.kanren.net 3.111ms 6: fourhundredge-0-0-0-10.2100.core2.kans.net.internet2.edu 5.223ms 7: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 14.571ms asymm 9 8: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 16.449ms 9: 144.92.233.112 17.130ms 10: perfsonar02.hep.wisc.edu 17.065ms reached Resume: pmtu 1500 hops 10 back 11
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-8gxSzX/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-8gxSzX/execute/dir_280767 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=280767 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Kansas State University OSG_SQUID_LOCATION=172.17.0.1:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_4@GP-ARGO-ksu-backfill.b4d530169f4b$ "; unset PROMPT_COMMAND PS1=slot1_4@GP-ARGO-ksu-backfill.b4d530169f4b$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-8gxSzX/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-8gxSzX/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-8gxSzX/execute/dir_280767 _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=172.17.0.1:3128
pass kernel
4.18.0-477.27.1.el8_8.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
8910.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md0 9764216768 420551384 9343665384 5% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy 172.17.0.1:3128
GP-ARGO-uams-backfill 2026-04-03 17:03 1
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
GP-ARGO-uams-backfill — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.086ms 1: _gateway 0.153ms 2: 144.30.235.3 0.361ms 3: no reply 4: 144.30.32.44 0.708ms 5: 144.30.32.35 8.814ms 6: no reply 7: 208.90.110.92 9.019ms 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: 144.92.233.112 38.220ms asymm 25 18: perfsonar02.hep.wisc.edu 37.987ms reached Resume: pmtu 1500 hops 18 back 25
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-it8frX/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-it8frX/execute/dir_2027941 BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=University of Arkansas for Medical Sciences OSG_SQUID_LOCATION=172.17.0.1:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_42@GP-ARGO-uams-backfill.35b6fa46d466$ "; unset PROMPT_COMMAND PS1=slot1_42@GP-ARGO-uams-backfill.35b6fa46d466$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-it8frX/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.3-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-it8frX/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.3-1.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-it8frX/execute/dir_2027941 _CONDOR_SLOT=slot1_42 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=172.17.0.1:3128
pass kernel
4.18.0-477.27.1.el8_8.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
9028.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md0 9764216768 296970948 9467245820 4% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy 172.17.0.1:3128
GP-ARGO-wichita-backfill 2026-04-03 17:08 1
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
GP-ARGO-wichita-backfill — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.061ms 1: _gateway 0.027ms 2: 156.26.198.35 1.027ms 3: 10.201.0.121 0.410ms 4: 10.201.0.126 0.927ms 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: perfsonar02.hep.wisc.edu 20.288ms reached Resume: pmtu 1500 hops 14 back 15
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-7q9osU/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-7q9osU/execute/dir_273203 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=273203 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Wichita State University OSG_SQUID_LOCATION=172.17.0.1:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_3@GP-ARGO-wichita-backfill.d6ed6946ca85$ "; unset PROMPT_COMMAND PS1=slot1_3@GP-ARGO-wichita-backfill.d6ed6946ca85$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-7q9osU/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-7q9osU/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-7q9osU/execute/dir_273203 _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=172.17.0.1:3128
pass kernel
4.18.0-477.27.1.el8_8.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
9016.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md0 9764216768 310009324 9454207444 4% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy 172.17.0.1:3128
GSU-Adonis-EP 2026-04-03 17:07 1
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
GSU-Adonis-EP — probe output
pass mtu
pmtu=1430 1?: [LOCALHOST] pmtu 1430 1: 10-245-19-162.prometheus-prometheus-node-exporter.monitoring.svc.cluster.local 0.077ms 1: 10-245-19-162.prometheus-prometheus-node-exporter.monitoring.svc.cluster.local 0.067ms 2: 10.245.18.2 0.381ms 3: rtcdfdrs-rtlsdc02rs-gw.gsu.edu 0.600ms 4: fwcdfxrs-rtcdfars-rtcdfdrs-gw.gsu.edu 0.176ms 5: fwcdfrs-rtcdfdrs.gsunet 0.641ms 6: rtcdfdedge-rtcdfdrs-gw.gsu.edu 0.405ms 7: SOX-10G-Secondary-TO-GSU.gsu.edu 2.447ms asymm 8 8: sidco-to-56mar-al2s.sox.net 8.886ms 9: i2-to-sox-re.sidco.sox.net 9.561ms 10: i2-to-sox-re.sidco.sox.net 9.212ms asymm 9 11: fourhundredge-0-0-0-21.4079.core2.cinc.net.internet2.edu 20.076ms asymm 15 12: fourhundredge-0-0-0-22.4079.core2.cinc.net.internet2.edu 19.571ms asymm 15 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 20.994ms 14: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 19.879ms asymm 13 15: perfsonar02.hep.wisc.edu 22.040ms reached Resume: pmtu 1430 hops 15 back 16
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-BsIw2j/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-BsIw2j/execute/dir_3828376 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3828376 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=GSU-Adonis OSG_SQUID_LOCATION=http://arcosg04.rs.gsu.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_4@GSU-Adonis-EP.osg-endpoint-74b7c7597f-fbcpp$ "; unset PROMPT_COMMAND PS1=slot1_4@GSU-Adonis-EP.osg-endpoint-74b7c7597f-fbcpp$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-BsIw2j/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-BsIw2j/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-BsIw2j/execute/dir_3828376 _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=http://arcosg04.rs.gsu.edu:3128
pass kernel
6.8.0-79-lowlatency
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
43.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on tmpfs 67108864 21912512 45196352 33% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy http://arcosg04.rs.gsu.edu:3128
GWU-Viper-CE1 2026-04-03 10:00 17
100.0%
23.5%
0.0%
23.5%
all
GWU-Viper-CE1 — probe output
warning all
Job failed with exit code 0; no results transferred
Harrisburg-HT24A01-EP 2026-04-03 17:03 1
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
Harrisburg-HT24A01-EP — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.173ms 1: _gateway 0.224ms 2: 172.16.107.5 0.488ms 3: 172.30.191.251 0.800ms 4: c-50-173-121-1.unallocated.comcastbusiness.net 0.922ms 5: te-0-3-0-1-3989-sur03.lowerpaxton.pa.pitt.comcast.net 1.982ms 6: be-5-sur02.lowerpaxton.pa.pitt.comcast.net 3.034ms 7: be-84-rar01.lancaster.pa.pitt.comcast.net 3.348ms 8: be-34-ar01.mckeesport.pa.pitt.comcast.net 12.136ms 9: be-31611-cs01.pittsburgh.pa.ibone.comcast.net 12.054ms 10: be-4113-pe13.ashburn.va.ibone.comcast.net 19.702ms asymm 9 11: 192.205.32.45 20.900ms asymm 10 12: 32.130.89.0 30.372ms asymm 21 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: 12.76.93.22 42.813ms asymm 16 20: r-uwmadison-cssc-ae4-3455.uwsys.net 42.406ms asymm 17 21: 143.235.41.129 33.587ms asymm 18 22: perfsonar02.hep.wisc.edu 33.589ms reached Resume: pmtu 1500 hops 22 back 20
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-FUoldJ/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-FUoldJ/execute/dir_19811 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=19811 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Harrisburg - CARC PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_2@Harrisburg-HT24A01-EP.510786f98024$ "; unset PROMPT_COMMAND PS1=slot1_2@Harrisburg-HT24A01-EP.510786f98024$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-FUoldJ/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-FUoldJ/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-FUoldJ/execute/dir_19811 _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass kernel
5.14.0-427.35.1.el9_4.aarch64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
7020.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme1n1 7499379032 138315948 7361063084 2% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Hawaii-Koa-CE1 2026-04-03 17:01 300
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
Hawaii-Koa-CE1 — probe output
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: _gateway 0.144ms 1: _gateway 0.126ms 2: et-1-2-0-2500-scidmz-re0.uhnet.net 0.225ms 3: et-0-2-0-35-ohelo-re0.uhnet.net 0.212ms 4: internet2-uhm-losa.uhnet.net 49.512ms 5: fourhundredge-0-0-0-2.4079.core2.salt.net.internet2.edu 93.477ms asymm 13 6: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 94.829ms asymm 12 7: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 95.197ms asymm 11 8: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 94.506ms asymm 10 9: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 94.999ms 10: 144.92.233.112 96.274ms 11: perfsonar02.hep.wisc.edu 96.223ms reached Resume: pmtu 9000 hops 11 back 12
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/mnt/nfs/scratch/osg01/glide_rSgjdN/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/mnt/nfs/scratch/osg01/glide_rSgjdN/execute/dir_849990/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=849990 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/mnt/nfs/scratch/osg01/glide_rSgjdN/cred.d/idtokens/credential_OSPool.main_OSG_US_Hawaii-Koa-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6114702 GLIDEIN_Entry_Name=OSG_US_Hawaii-Koa-CE1 GLIDEIN_Entry_Signature=738234d0e94d0eee059e7764d175de9426595102 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=hawaii-koa-ce1.svc.osg-htc.org hawaii-koa-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_npLmWR GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Hawaii-Koa-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=Hawaii-Koa GLIDEIN_Tmp_Dir=/mnt/nfs/scratch/osg01/glide_rSgjdN/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Hawaii-Koa-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_124943_26111246@gn-09-21-00.mana.ci.its.hawaii.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_124943_26111246@gn-09-21-00.mana.ci.its.hawaii.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/mnt/nfs/scratch/osg01/glide_rSgjdN/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/mnt/nfs/scratch/osg01/glide_rSgjdN/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/mnt/nfs/scratch/osg01/glide_rSgjdN/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/mnt/nfs/scratch/osg01/glide_rSgjdN/execute/dir_849990/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.41.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
9445.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 10.64.0.99:/datastore/scratch/osg01/glide_rSgjdN/execute/dir_849990 10585177088 681119744 9904057344 7% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
IRISHEP-SSL-RIVER-DEV-BACKFILL 2026-04-03 17:01 55
100.0%
100.0%
98.2%
100.0%
mtu env kernel nrp_images df http_proxy
IRISHEP-SSL-RIVER-DEV-BACKFILL — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.016ms pmtu 1500 1: 2605:9a00:10:200c::2 0.284ms 1: 2605:9a00:10:200c::2 0.292ms 2: a08-s004-u3-s250b-to-b65-ll129-s500a.p2p.uchicago.net 0.377ms 3: no reply 4: b65-ll129-s500-to-h01-391-l21-s500.p2p.uchicago.net 0.392ms asymm 2 5: perfsonar02.hep.wisc.edu 17.919ms reached Resume: pmtu 1500 hops 5 back 5
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-Lmfk15/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-Lmfk15/execute/dir_1922400 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1922400 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=IRISHEP-SSL-UCHICAGO OSG_SQUID_LOCATION=squid.grid.uchicago.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_23@IRISHEP-SSL-RIVER-DEV-BACKFILL.river-c007.ssl-hep.org$ "; unset PROMPT_COMMAND PS1=slot1_23@IRISHEP-SSL-RIVER-DEV-BACKFILL.river-c007.ssl-hep.org$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-Lmfk15/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-Lmfk15/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-Lmfk15/execute/dir_1922400 _CONDOR_SLOT=slot1_23 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=squid.grid.uchicago.edu:3128
pass kernel
5.14.0-611.20.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
565.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 765870024 133726716 593165824 19% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy squid.grid.uchicago.edu:3128
ISI 2026-04-03 17:33 5
60.0%
60.0%
60.0%
100.0%
mtu env kernel nrp_images df http_proxy
ISI — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.027ms pmtu 1500 1: 2606:d5c0:0:1::1 7.969ms 1: 2606:d5c0:0:1::1 15.476ms 2: 2606:d5c0::1 16.078ms 3: 2001:1878:15::1 8.748ms 4: lax-agg10--losa4-agg-01--400g--01.cenic.net 1.928ms 5: lax-agg10-hpr--lax-agg10-dc.cenic.net 9.039ms 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_3074381/scratch/glide_6KNhvR/images/htc__debian__13.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__debian__13.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_3074381/scratch/glide_6KNhvR/execute/dir_3591049/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3591049 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_3074381/scratch/glide_6KNhvR/cred.d/idtokens/credential_OSPool.main_OSG_US_ISI_osg.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=5603052 GLIDEIN_Country=US GLIDEIN_Entry_Name=OSG_US_ISI_osg GLIDEIN_Entry_Signature=496f3d8981f644a4daba09849ff6c2a49e8224a7 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=osg.isi.edu osg.isi.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_M2VyRE GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=workflow.isi.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=ISI GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=5552c705dd9f3b999102cc42d5c5bdce18bba000 GLIDEIN_Site=ISI GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_3074381/scratch/glide_6KNhvR/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=osg.isi.edu OSG_SITE_NAME=ISI OSG_SQUID_LOCATION=workflow.isi.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_3074383_203179265@compute-1.isi.edu$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_3074383_203179265@compute-1.isi.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_3074381/scratch/glide_6KNhvR/images/htc__debian__13.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/dir_3074381/scratch/glide_6KNhvR/images/htc__debian__13.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__debian__13.sif SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_3074381/scratch/glide_6KNhvR/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_3074381/scratch/glide_6KNhvR/execute/dir_3591049/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=workflow.isi.edu:3128
pass kernel
4.18.0-553.109.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
74.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/rl-root 101697536 23409672 78287864 24% /srv
fail http_proxy
Download timed out after 30s: http://cm-1.ospool.osg-htc.org/overview/ via proxy workflow.isi.edu:3128
IU-Jetstream2-Backfill 2026-04-03 17:00 101
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
IU-Jetstream2-Backfill — probe output
pass mtu
pmtu=8850 1?: [LOCALHOST] pmtu 8850 1: _gateway 0.065ms 1: _gateway 0.035ms 2: host-192-168-0-1.js2local 0.744ms 3: host-192-168-3-22.js2local 0.929ms asymm 2 4: 149.165.159.1 0.263ms asymm 3 5: 172.16.11.140 0.835ms asymm 4 6: bl-nas2.hps.iu.edu 0.439ms asymm 5 7: 10.9.213.8 1.798ms asymm 6 8: 10.9.213.9 2.075ms asymm 6 9: ae-9.0.rtr.ll.indiana.gigapop.net 2.640ms asymm 8 10: ae-9.0.rtr.ll.indiana.gigapop.net 2.375ms asymm 8 11: 144.92.233.118 23.588ms asymm 10 12: perfsonar02.hep.wisc.edu 23.378ms reached Resume: pmtu 8850 hops 12 back 11
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3 APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-sxChSD/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-sxChSD/execute/dir_7337 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=7337 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Pervasive Technology Institute PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_1@IU-Jetstream2-Backfill.blue-9b4ccc9f6-bgdt2$ "; unset PROMPT_COMMAND PS1=slot1_1@IU-Jetstream2-Backfill.blue-9b4ccc9f6-bgdt2$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3 SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-sxChSD/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-sxChSD/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-sxChSD/execute/dir_7337 _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass kernel
6.1.0-38-cloud-amd64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
177.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 206376368 12006564 185944180 7% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
JSU-Tigers-CE1 2026-04-03 11:42 2
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
JSU-Tigers-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.474ms 1: _gateway 0.427ms 2: 173.235.100.112 5.779ms 3: 173.235.100.7 9.633ms asymm 2 4: 173.235.100.17 19.036ms asymm 3 5: fourhundredge-0-0-0-23.4079.core2.atla.net.internet2.edu 31.150ms asymm 14 6: fourhundredge-0-0-0-2.4079.core2.nash.net.internet2.edu 32.728ms asymm 13 7: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 38.110ms asymm 12 8: fourhundredge-0-0-0-21.4079.core2.cinc.net.internet2.edu 36.461ms asymm 11 9: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 33.125ms asymm 10 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 32.346ms asymm 9 11: 144.92.233.112 34.640ms asymm 10 12: perfsonar02.hep.wisc.edu 38.639ms reached Resume: pmtu 1500 hops 12 back 12
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_2490810/glide_sNhCa4/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_2490810/glide_sNhCa4/execute/dir_2805413/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2805413 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_2490810/glide_sNhCa4/cred.d/idtokens/credential_OSPool.main_OSG_US_JSU-Tigers-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=294018 GLIDEIN_Entry_Name=OSG_US_JSU-Tigers-CE1 GLIDEIN_Entry_Signature=cb42e24ea8015e304efa84296b7dd9b26d12ac41 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=jsu-tigers-ce1.svc.opensciencegrid.org jsu-tigers-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_sqSNDz GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=JSU-Tigers-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=JSU-Tigers GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_2490810/glide_sNhCa4/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=JSU-Tigers-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2490812_224664003@chtc.ep0.jsums.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2490812_224664003@chtc.ep0.jsums.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_2490810/glide_sNhCa4/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/dir_2490810/glide_sNhCa4/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_2490810/glide_sNhCa4/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_2490810/glide_sNhCa4/execute/dir_2805413/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-615.el9.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1674.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/condor_vg-condor_lv 1844816532 3323752 1755724980 1% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
KENT-STATE-RESEARCH-EP 2026-04-03 17:01 4
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
KENT-STATE-RESEARCH-EP — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.198ms 1: _gateway 0.333ms 2: gw2.research.cs.kent.edu 0.785ms 3: 10.255.255.1 0.712ms 4: akrnq-r4-et-0-2-1s3112.core.oar.net 0.988ms 5: wstra-r5-et-2-1-0s100.core.oar.net 3.279ms 6: clmbn-r4-et-2-0-1s100.core.oar.net 6.443ms 7: 199.218.39.146 8.792ms 8: cncno-r5-et-4-3-0s735.core.oar.net 9.194ms 9: hundredge-0-0-0-25.1260.core1.cinc.net.internet2.edu 12.976ms asymm 10 10: fourhundredge-0-0-0-23.4079.core2.cinc.net.internet2.edu 16.524ms asymm 12 11: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 13.264ms 12: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 15.433ms asymm 10 13: 144.92.233.112 16.606ms asymm 11 14: perfsonar02.hep.wisc.edu 16.561ms reached Resume: pmtu 1500 hops 14 back 13
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-e89Gml/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-e89Gml/execute/dir_2103460 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2103460 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Kent_State_Research PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_51@KENT-STATE-RESEARCH-EP.90ddc0dfa4d5$ "; unset PROMPT_COMMAND PS1=slot1_51@KENT-STATE-RESEARCH-EP.90ddc0dfa4d5$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-e89Gml/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-e89Gml/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-e89Gml/execute/dir_2103460 _CONDOR_SLOT=slot1_51 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass kernel
5.14.0-688.el9.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
41614.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg--main-home 45986104216 41425340 43636117572 1% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
KSU-Beocat-CE1 2026-04-03 17:03 12
8.3%
8.3%
8.3%
100.0%
mtu env kernel nrp_images df http_proxy
KSU-Beocat-CE1 — probe output
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: no reply 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 9000 Resume: pmtu 9000
pass env
APPTAINER_CACHEDIR=/tmp/glide_rSVGBd/execute/dir_945570/scratch BASH_ENV=/usr/share/lmod/lmod/init/bash BASH_FUNC_ml%%=() { eval "$($LMOD_DIR/ml_cmd "$@")" } BASH_FUNC_module%%=() { if [ -z "${LMOD_SH_DBG_ON+x}" ]; then case "$-" in *v*x*) __lmod_sh_dbg='vx' ;; *v*) __lmod_sh_dbg='v' ;; *x*) __lmod_sh_dbg='x' ;; esac; fi; if [ -n "${__lmod_sh_dbg:-}" ]; then set +$__lmod_sh_dbg; echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output" 1>&2; fi; eval "$($LMOD_CMD shell "$@")" && eval "$(${LMOD_SETTARG_CMD:-:} -s sh)"; __lmod_my_status=$?; if [ -n "${__lmod_sh_dbg:-}" ]; then echo "Shell debugging restarted" 1>&2; set -$__lmod_sh_dbg; fi; unset __lmod_sh_dbg; return $__lmod_my_status } BASH_FUNC_which%%=() { ( alias; eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@ } BATCH_SYSTEM=HTCondor BLAHPD_LOCATION=/homes/osgsubmit/bosco_ce1/glite COBBLER_SERVER=10.5.1.42 CONDORCE_COLLECTOR_HOST=ksu-beocat-ce1.svc.opensciencegrid.org:9619 CONDOR_CONFIG=/tmp/glide_rSVGBd/condor_config CONDOR_PARENT_ID=dwarf31:945570:1775235675 CONDOR_PROCD_ADDRESS=/tmp/glide_rSVGBd/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/tmp/glide_rSVGBd/log/procd_address CONDOR_STARTER_PID=945570 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/3838/bus DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima: ENVIRONMENT=BATCH FPATH=/usr/share/lmod/lmod/init/ksh_funcs GFAL_CONFIG_DIR=/homes/osgsubmit/bosco-osg-wn-client/etc/gfal2.d/ GFAL_PLUGIN_DIR=/homes/osgsubmit/bosco-osg-wn-client/usr/lib64/gfal2-plugins/ GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_rSVGBd/cred.d/idtokens/credential_OSPool.main_OSG_US_KSU-Beocat-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=303517 GLIDEIN_Entry_Name=OSG_US_KSU-Beocat-CE1 GLIDEIN_Entry_Signature=c54bb490a39b71a07a54e33e6c4a7f7fc249747e GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=ksu-beocat-ce1.svc.opensciencegrid.org ksu-beocat-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgsubmit_shFR6g GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=squid.beocat.ksu.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=KSU-Beocat-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=BEOCAT-SLATE GLIDEIN_Tmp_Dir=/tmp/glide_rSVGBd/tmp GLIDEIN_UUID=3452ba48-ca8e-4fde-89dc-a0520088696e GLITE_LOCATION=/homes/osgsubmit/bosco_ce1/glite GLOBUS_LOCATION=/homes/osgsubmit/bosco-osg-wn-client/usr GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_PATH=/tmp/glide_rSVGBd/.gwms.d/bin: GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HISTCONTROL=ignoredups HISTSIZE=1000 HOLD_GRACE_TIME=0 HOME=/tmp/glide_rSVGBd/execute/dir_945570/scratch HOSTNAME=dwarf31 HYDRA_BOOTSTRAP=slurm HYDRA_LAUNCHER_EXTRA_ARGS=--external-launcher I_MPI_HYDRA_BOOTSTRAP=slurm I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_OSG_US_KSU-Beocat-CE1/tokens.tgz JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/homes/osgsubmit/bosco-osg-wn-client/lib64:/homes/osgsubmit/bosco-osg-wn-client/lib:/homes/osgsubmit/bosco-osg-wn-client/usr/lib64:/homes/osgsubmit/bosco-osg-wn-client/usr/lib:/homes/osgsubmit/bosco-osg-wn-client/usr/lib64/dcap:/homes/osgsubmit/bosco-osg-wn-client/usr/lib64/lcgdm LESSOPEN=||/usr/bin/lesspipe.sh %s LMOD_CMD=/usr/share/lmod/lmod/libexec/lmod LMOD_DIR=/usr/share/lmod/lmod/libexec LMOD_IGNORE_CACHE=1 LMOD_PACKAGE_PATH=/etc/lmod LMOD_PKG=/usr/share/lmod/lmod LMOD_ROOT=/usr/share/lmod LMOD_SETTARG_FULL_SUPPORT=no LMOD_SYSTEM_DEFAULT_MODULES=StdEnv/default LMOD_VERSION=8.7.65 LMOD_sys=Linux LOADEDMODULES=beocat-scripts/default:StdEnv/default LOGNAME=osgsubmit MAIL=/var/spool/mail/osgsubmit MANPATH=/homes/osgsubmit/bosco-osg-wn-client/usr/share/man:/usr/share/lmod/lmod/share/man: MKL_NUM_THREADS=1 MODULEPATH=/opt/software/modulefiles/all:/opt/beocat/custom_modules MODULEPATH_ROOT=/usr/share/modulefiles MODULESHOME=/usr/share/lmod/lmod MOTD_SHOWN=pam NODE_COUNT=4 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OLDPWD=/tmp/glide_rSVGBd/logs/3452ba48-ca8e-4fde-89dc-a0520088696e OMPI_MCA_plm_slurm_args=--external-launcher OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=UNAVAILABLE OSG_DATA=UNAVAILABLE OSG_GRID=/homes/osgsubmit/bosco-osg-wn-client OSG_HOSTNAME=localhost OSG_LOCATION=/homes/osgsubmit/bosco-osg-wn-client OSG_SITE_NAME=KSU-Beocat-CE1 OSG_SQUID_LOCATION=squid.beocat.ksu.edu:3128 OSG_WN_TMP=/tmp/glide_rSVGBd/execute/dir_945570/scratch/.local-tmp PATH=/tmp/glide_rSVGBd/.gwms.d/bin::/homes/osgsubmit/bosco-osg-wn-client/usr/bin:/homes/osgsubmit/bosco-osg-wn-client/usr/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/beocat/shared/beocat-scripts:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/homes/osgsubmit/.local/bin:/homes/osgsubmit/bin PERL5LIB=/homes/osgsubmit/bosco-osg-wn-client/usr/share/perl5/vendor_perl:/homes/osgsubmit/bosco-osg-wn-client/usr/share/perl5:/homes/osgsubmit/bosco-osg-wn-client/usr/lib64/perl5/vendor_perl:/homes/osgsubmit/bosco-osg-wn-client/usr/lib64/perl5 PREEMPT_GRACE_TIME=10000000 PRTE_MCA_plm_slurm_args=--external-launcher PWD=/tmp/glide_rSVGBd/execute/dir_945570/scratch PYTHONPATH=/homes/osgsubmit/bosco-osg-wn-client/usr/lib/python3.9/site-packages:/homes/osgsubmit/bosco-osg-wn-client/usr/lib64/python3.9/site-packages PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SHELL=/bin/bash SHLVL=0 SINGULARITY_CACHEDIR=/tmp/glide_rSVGBd/execute/dir_945570/scratch SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_MODE=unprivileged SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 SLOTS_LAYOUT=partitionable SLURMD_NODENAME=dwarf31 SLURM_CLUSTER_NAME=beocat SLURM_CONF=/etc/slurm/slurm.conf SLURM_CPUS_ON_NODE=4 SLURM_GTIDS=0 SLURM_JOBID=7400597 SLURM_JOB_ACCOUNT=beodefault SLURM_JOB_CPUS_PER_NODE=4 SLURM_JOB_END_TIME=1775316783 SLURM_JOB_GID=3838 SLURM_JOB_ID=7400597 SLURM_JOB_NAME=bl_mizzSU SLURM_JOB_NODELIST=dwarf31 SLURM_JOB_NUM_NODES=1 SLURM_JOB_PARTITION=osg.q SLURM_JOB_QOS=normal SLURM_JOB_START_TIME=1775230383 SLURM_JOB_UID=3838 SLURM_JOB_USER=osgsubmit SLURM_LOCALID=0 SLURM_MEM_PER_NODE=8000 SLURM_NNODES=1 SLURM_NODEID=0 SLURM_NODELIST=dwarf31 SLURM_NPROCS=4 SLURM_NTASKS=4 SLURM_NTASKS_PER_NODE=4 SLURM_PRIO_PROCESS=0 SLURM_PROCID=0 SLURM_SUBMIT_DIR=/homes/osgsubmit/bosco_ce1/sandbox/912a/912afc48/ksu-beocat-ce1.svc.opensciencegrid.org_9619_ksu-beocat-ce1.svc.opensciencegrid.org_1575017.0_1775230277 SLURM_SUBMIT_HOST=icr-helios SLURM_TASKS_PER_NODE=4 SLURM_TASK_PID=792438 SLURM_TOPOLOGY_ADDR=due1121-prod-core-40g-a1,due1121-prod-core-100g-c1.due1121-prod-sw-40g-a9.dwarf31 SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass SSH_CLIENT=128.105.82.96 1151 22 SSH_CONNECTION=128.105.82.96 1151 10.5.1.6 22 SSH_ORIGINAL_COMMAND=/bin/bash -l -c 'BLAHPD_LOCATION=~/bosco_ce1/glite GLITE_LOCATION=~/bosco_ce1/glite ~/bosco_ce1/glite/bin/blahpd ' TEMP=/tmp/glide_rSVGBd/execute/dir_945570/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/tmp/glide_rSVGBd/execute/dir_945570/scratch TMPDIR=/tmp/glide_rSVGBd/execute/dir_945570/scratch/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime USER=osgsubmit VOMS_USERCONF=/homes/osgsubmit/bosco-osg-wn-client/etc/vomses X509_CERT_DIR=/homes/osgsubmit/bosco-osg-wn-client/etc/grid-security/certificates X509_VOMS_DIR=/homes/osgsubmit/bosco-osg-wn-client/etc/grid-security/vomsdir XDG_DATA_DIRS=/homes/osgsubmit/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share XDG_RUNTIME_DIR=/run/user/3838 XDG_SESSION_CLASS=user XDG_SESSION_ID=14723 XDG_SESSION_TYPE=tty _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_844801=845488:1775230512:3336080787 _CONDOR_ANCESTOR_845488=945570:1775235672:2750319847 _CONDOR_ANCESTOR_945570=946149:1775235675:3613059059 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_rSVGBd/main/condor/bin _CONDOR_CHIRP_CONFIG=/tmp/glide_rSVGBd/execute/dir_945570/scratch/.chirp.config _CONDOR_CREDS=/tmp/glide_rSVGBd/execute/dir_945570/scratch/.condor_creds _CONDOR_EXECUTE=/tmp/glide_rSVGBd/execute _CONDOR_JOB_AD=/tmp/glide_rSVGBd/execute/dir_945570/scratch/.job.ad _CONDOR_JOB_IWD=/tmp/glide_rSVGBd/execute/dir_945570/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/tmp/glide_rSVGBd/execute/dir_945570/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/tmp/glide_rSVGBd/execute/dir_945570/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/tmp/glide_rSVGBd/execute/dir_945570/scratch/.job_wrapper_failure _LMFILES_=/opt/beocat/custom_modules/beocat-scripts/default.lua:/opt/beocat/custom_modules/StdEnv/default.lua _ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0ge30sCm1UID0gewpTdGRFbnYgPSB7CmZuID0gIi9vcHQvYmVvY2F0L2N1c3RvbV9tb2R1bGVzL1N0ZEVudi9kZWZhdWx0Lmx1YSIsCmZ1bGxOYW1lID0gIlN0ZEVudi9kZWZhdWx0IiwKbG9hZE9yZGVyID0gMiwKcHJvcFQgPSB7fSwKc3RhY2tEZXB0aCA9IDAsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJTdGRFbnYvZGVmYXVsdCIsCndWID0gIipkZWZhdWx0Lip6ZmluYWwiLAp9LApbImJlb2NhdC1zY3JpcHRzIl0gPSB7CmZuID0gIi9vcHQvYmVvY2F0L2N1c3RvbV9tb2R1bGVzL2Jlb2Nh _ModuleTable002_=dC1zY3JpcHRzL2RlZmF1bHQubHVhIiwKZnVsbE5hbWUgPSAiYmVvY2F0LXNjcmlwdHMvZGVmYXVsdCIsCmxvYWRPcmRlciA9IDEsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAxLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAiYmVvY2F0LXNjcmlwdHMvZGVmYXVsdCIsCndWID0gIipkZWZhdWx0Lip6ZmluYWwiLAp9LAp9LAptcGF0aEEgPSB7CiIvb3B0L3NvZnR3YXJlL21vZHVsZWZpbGVzL2FsbCIsICIvb3B0L2Jlb2NhdC9jdXN0b21fbW9kdWxlcyIsCn0sCnN5c3RlbUJhc2VNUEFUSCA9ICIvb3B0L3NvZnR3YXJlL21vZHVsZWZpbGVzL2FsbDovb3B0L2Jlb2NhdC9jdXN0b21fbW9kdWxlcyIsCn0K _ModuleTable_Sz_=2 __Init_Default_Modules=1 __LMOD_REF_COUNT_MODULEPATH=/opt/software/modulefiles/all:1;/opt/beocat/custom_modules:1 __LMOD_REF_COUNT_PATH=/opt/beocat/shared/beocat-scripts:1;/usr/local/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1 which_declare=declare -f
pass kernel
5.14.0-611.36.1.el9_7.x86_64
fail nrp_images
Failed to download https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt: <urlopen error _ssl.c:1130: The handshake operation timed out>
pass df
887.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg00-root 968898324 38719824 930178500 4% /tmp
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy squid.beocat.ksu.edu:3128
LIGO-WA-CE1 2026-04-03 17:08 25
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
LIGO-WA-CE1 — probe output
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: pluto.dcs.ligo-wa.caltech.edu 0.313ms 1: pluto.dcs.ligo-wa.caltech.edu 0.325ms 2: 198.129.208.1 0.426ms asymm 3 3: no reply 4: bois-cr6--salt-bb-c.igp.es.net 43.818ms asymm 13 5: salt-cr6--denv-bb-b.igp.es.net 43.708ms asymm 12 6: denv-cr6--kans-bb-e.igp.es.net 43.633ms asymm 11 7: kans-cr6--chic-bb-d.igp.es.net 43.765ms asymm 10 8: no reply 9: uwmadison_se-377--chic-cr6.es.net 60.872ms asymm 10 10: perfsonar02.hep.wisc.edu 60.945ms reached Resume: pmtu 9000 hops 10 back 11
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_2034381/scratch/glide_a5lb0Z/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_2034381/scratch/glide_a5lb0Z/execute/dir_392931/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=392931 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_2034381/scratch/glide_a5lb0Z/cred.d/idtokens/credential_OSPool.main_LIGO_US_Caltech_ldas_grid-wa_4C.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=221790 GLIDEIN_Entry_Name=LIGO_US_Caltech_ldas_grid-wa_4C GLIDEIN_Entry_Signature=b56d10c7ed3c68f253b94a39f2717c810308563f GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=ligo-hanford-ce1.svc.osg-htc.org ligo-hanford-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_u2fjmq GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=frontier.dcs.ligo-wa.caltech.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=LIGO-WA-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=547752992760a694c32ac9c7748d22e0e3c3660c GLIDEIN_Site=LIGO-WA GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_2034381/scratch/glide_a5lb0Z/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=LIGO-WA-CE1 OSG_SQUID_LOCATION=frontier.dcs.ligo-wa.caltech.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_487_768507600@node299.dcs.ligo-wa.caltech.edu$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_487_768507600@node299.dcs.ligo-wa.caltech.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_2034381/scratch/glide_a5lb0Z/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/dir_2034381/scratch/glide_a5lb0Z/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_2034381/scratch/glide_a5lb0Z/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_2034381/scratch/glide_a5lb0Z/execute/dir_392931/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.105.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
63.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/node299-slot1_1+1764935_274 72195192 5333304 66845504 8% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy frontier.dcs.ligo-wa.caltech.edu:3128
LSUHSC-Tigerfish-CE2 2026-04-03 17:01 6
100.0%
0.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
LSUHSC-Tigerfish-CE2 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: tigerfish.cluster 0.126ms 1: tigerfish.cluster 0.242ms 2: 155.58.254.1 0.857ms 3: rcb-bdr-r1.lsuhsc.edu 0.942ms 4: 208.100.102.142 1.289ms 5: 10.250.25.153 4.014ms asymm 11 6: 10.250.24.121 3.640ms asymm 10 7: 10.250.25.69 3.726ms asymm 9 8: 10.250.24.126 3.647ms 9: 208.100.66.98 6.490ms 10: fourhundredge-0-0-0-2.4079.core1.houh.net.internet2.edu 30.946ms asymm 20 11: fourhundredge-0-0-0-2.4079.core1.hous.net.internet2.edu 30.971ms asymm 19 12: fourhundredge-0-0-0-0.4079.core2.dall.net.internet2.edu 31.386ms asymm 18 13: fourhundredge-0-0-0-2.4079.core2.kans.net.internet2.edu 30.738ms asymm 17 14: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 30.864ms asymm 16 15: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 32.627ms 16: 144.92.233.112 33.439ms 17: perfsonar02.hep.wisc.edu 33.395ms reached Resume: pmtu 1500 hops 17 back 18
pass env
APPTAINER_CACHEDIR=/srv/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=862286 CONTAINERSW_FULL_VERSION=singularity version 3.7.4-1.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/singularity CONTAINERSW_VERSION=3.7.4-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/mnt/beegfs/scratch/glide_lLNvoY/cred.d/idtokens/credential_OSPool.main_OSG_US_LSUHSC-Tigerfish-CE2.idtoken GLIDEIN_CPUS=40 GLIDEIN_ClusterId=351424 GLIDEIN_Entry_Name=OSG_US_LSUHSC-Tigerfish-CE2 GLIDEIN_Entry_Signature=5cddfa6858c1ef6a2627c2f35cde65064028eb3d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=lsuhsc-tf-ce2.svc.opensciencegrid.org lsuhsc-tf-ce2.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_giDFPs GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=LSUHSC-Tigerfish-CE2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=LSUHSC-Tigerfish GLIDEIN_Tmp_Dir=/mnt/beegfs/scratch/glide_lLNvoY/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity version 3.7.4-1.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/singularity GWMS_CONTAINERSW_VERSION=3.7.4-1.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.7.4-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=LSUHSC-Tigerfish-CE2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_3356589_145639680@node01.cluster$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_3356589_145639680@node01.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/mnt/beegfs/scratch/glide_lLNvoY/execute/dir_862286:/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/mnt/beegfs/scratch/glide_lLNvoY/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity version 3.7.4-1.el8 SINGULARITY_IMAGES_DICT=default:/mnt/beegfs/scratch/glide_lLNvoY/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/singularity SINGULARITY_TMPDIR=/mnt/beegfs/scratch/glide_lLNvoY/execute/dir_862286/scratch SINGULARITY_VERSION=3.7.4-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/mnt/beegfs/scratch/glide_lLNvoY/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/mnt/beegfs/scratch/glide_lLNvoY/execute/dir_862286/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-240.15.1.el8_3.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
451168.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on beegfs_nodev 596209287168 123125010944 473084276224 21% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Lafayette-Firebird-CE1 2026-04-03 17:02 17
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
Lafayette-Firebird-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: firebird.cluster 0.315ms 1: firebird.cluster 0.330ms 2: fw-comsvc-colo.lafayette.edu 7.936ms 3: 139.147.1.113 8.186ms asymm 4 4: 172.31.0.33 1.230ms 5: 172.31.0.3 15.710ms 6: gw0-g0-3-ibgp.prd.lafayette.edu 15.755ms asymm 5 7: 164.153.140.160 19.406ms asymm 6 8: hundredge-0-0-0-28.78.core1.phil.net.internet2.edu 22.756ms asymm 7 9: fourhundredge-0-0-0-2.4079.core2.ashb.net.internet2.edu 36.343ms asymm 15 10: fourhundredge-0-0-0-1.4079.core2.clev.net.internet2.edu 37.413ms asymm 14 11: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 37.260ms asymm 13 12: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 36.287ms 13: 144.92.233.112 39.508ms 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg01/glide_ifG7vu/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg01/glide_ifG7vu/execute/dir_1685215/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1685215 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01/glide_ifG7vu/cred.d/idtokens/credential_OSPool.main_OSG_US_Lafayette-firebird-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=5677201 GLIDEIN_Entry_Name=OSG_US_Lafayette-firebird-CE1 GLIDEIN_Entry_Signature=5db28e2bf2cab512785d2a7f4d8fbe24972d2568 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=lafayette-firebird-ce1.svc.osg-htc.org lafayette-firebird-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_m4E54Y GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Lafayette-Firebird-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=436230a2fef517d351fbb58e714fa5f5fa541af0 GLIDEIN_Site=Lafayette-Firebird GLIDEIN_Tmp_Dir=/scratch/osg01/glide_ifG7vu/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Lafayette-Firebird-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_1542716_36234520@node01.cluster$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_1542716_36234520@node01.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg01/glide_ifG7vu/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el9 SINGULARITY_IMAGES_DICT=default:/scratch/osg01/glide_ifG7vu/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01/glide_ifG7vu/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01/glide_ifG7vu/execute/dir_1685215/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.16.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
7024.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/disk0-scratch 7439005696 73685116 7365320580 1% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Langston-Lion-CE1 2026-04-03 17:52 2
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
Langston-Lion-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: lionvmhost.ccre.langston.edu 0.165ms 1: lionvmhost.ccre.langston.edu 0.165ms 2: 164.58.29.113 0.391ms 3: 198.71.46.45 2.339ms asymm 6 4: fourhundredge-0-0-0-9.302.core1.tuls.net.internet2.edu 5.735ms asymm 6 5: fourhundredge-0-0-0-2.4079.core1.kans.net.internet2.edu 19.456ms asymm 11 6: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 18.098ms asymm 10 7: fourhundredge-0-0-0-22.4079.core1.chic.net.internet2.edu 19.630ms asymm 9 8: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 18.466ms 9: 144.92.233.112 20.558ms 10: perfsonar02.hep.wisc.edu 20.527ms reached Resume: pmtu 1500 hops 10 back 11
pass env
APPTAINER_CACHEDIR=/srv/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1275014 CONTAINERSW_FULL_VERSION=singularity-ce version 4.3.3-3.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/singularity CONTAINERSW_VERSION=4.3.3-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osgccre/glide_kGY5uj/cred.d/idtokens/credential_OSPool.main_OSG_US_Langston-Lion-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=236064 GLIDEIN_Entry_Name=OSG_US_Langston-Lion-CE1 GLIDEIN_Entry_Signature=e2326c3ff00d69dcfc3026c315f69fefedb12f6a GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=langston-lion-ce1.svc.osg-htc.org langston-lion-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgccre_7X6G4R GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Langston-Lion-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=Langston-Lion GLIDEIN_Tmp_Dir=/scratch/osgccre/glide_kGY5uj/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity-ce version 4.3.3-3.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/singularity GWMS_CONTAINERSW_VERSION=4.3.3-3.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=4.3.3-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Langston-Lion-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_749632_126847917@c000.ccre.langston.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_749632_126847917@c000.ccre.langston.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/scratch/osgccre/glide_kGY5uj/execute/dir_1275014:/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/scratch/osgccre/glide_kGY5uj/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity-ce version 4.3.3-3.el9 SINGULARITY_IMAGES_DICT=default:/scratch/osgccre/glide_kGY5uj/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/singularity SINGULARITY_TMPDIR=/scratch/osgccre/glide_kGY5uj/execute/dir_1275014/scratch SINGULARITY_VERSION=4.3.3-3.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osgccre/glide_kGY5uj/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osgccre/glide_kGY5uj/execute/dir_1275014/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-570.49.1.el9_6.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
884.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 936705024 9511380 927193644 2% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Lehigh_-_Hawk 2026-04-03 17:01 515
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
Lehigh_-_Hawk — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 192.168.7.251 0.373ms 1: 192.168.7.251 0.365ms 2: borderfw-e.cc.lehigh.edu 0.074ms 3: pack35-out.cc.lehigh.edu 0.166ms 4: pack34-out.cc.lehigh.edu 0.152ms asymm 3 5: 164.153.140.133 2.669ms asymm 4 6: hundredge-0-0-0-28.78.core1.phil.net.internet2.edu 5.349ms asymm 5 7: fourhundredge-0-0-0-2.4079.core2.ashb.net.internet2.edu 21.091ms asymm 13 8: fourhundredge-0-0-0-1.4079.core2.clev.net.internet2.edu 21.730ms asymm 12 9: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 21.259ms asymm 11 10: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 21.428ms 11: 144.92.233.112 22.827ms 12: perfsonar02.hep.wisc.edu 22.774ms reached Resume: pmtu 1500 hops 12 back 13
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg/13543024/glide_XCR7Pb/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg/13543024/glide_XCR7Pb/execute/dir_2602899/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2602899 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg/13543024/glide_XCR7Pb/cred.d/idtokens/credential_OSPool.main_OSG_US_Lehigh-Hawk_osg-hawk-ce.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6125224 GLIDEIN_Country=US GLIDEIN_Entry_Name=OSG_US_Lehigh-Hawk_osg-hawk-ce GLIDEIN_Entry_Signature=29ab12b1989860efe3b17c3cf7d381f95503636e GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=osg-hawk-ce.cc.lehigh.edu osg-hawk-ce.cc.lehigh.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_06Vvud GLIDEIN_MaxMemMBs=7000 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=osg-squid.cc.lehigh.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Lehigh - Hawk GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=Lehigh - Hawk GLIDEIN_Tmp_Dir=/scratch/osg/13543024/glide_XCR7Pb/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv HTTPS_PROXY=osg-squid.cc.lehigh.edu:3128 JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=osg-hawk-ce.cc.lehigh.edu OSG_SITE_NAME=LEHIGH-HAWK-CE OSG_SQUID_LOCATION=osg-squid.cc.lehigh.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2120560_497486712@hawk-a119.cc.lehigh.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2120560_497486712@hawk-a119.cc.lehigh.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg/13543024/glide_XCR7Pb/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el9 SINGULARITY_IMAGES_DICT=default:/scratch/osg/13543024/glide_XCR7Pb/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TERM=linux TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg/13543024/glide_XCR7Pb/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg/13543024/glide_XCR7Pb/execute/dir_2602899/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=osg-squid.cc.lehigh.edu:3128
pass kernel
5.14.0-611.27.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
382.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/almalinux-scratch 874028092 428787520 400768664 52% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy osg-squid.cc.lehigh.edu:3128
MI-HORUS-CE2 2026-04-03 17:01 302
75.5%
59.9%
17.2%
65.2%
mtu env kernel nrp_images df http_proxy
MI-HORUS-CE2 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: wsu-rtr-fe.osris.org 0.357ms 1: wsu-rtr-fe.osris.org 0.328ms 2: 10.0.8.17 0.512ms 3: 10.0.8.5 0.297ms 4: ae1x160.dtrt-wsudc-c1.mich.net 0.839ms 5: et-3-1-0x3.gdrp-eberhard-c1.mich.net 3.742ms 6: ae100x0.gdrp-gvsu-eberhard-c1.mich.net 3.503ms 7: ae102x0.chcg-lumen-600w-c1.mich.net 7.238ms asymm 8 8: 192.122.182.34 9.194ms asymm 9 9: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 9.832ms 10: 144.92.233.112 11.162ms 11: perfsonar02.hep.wisc.edu 11.010ms reached Resume: pmtu 1500 hops 11 back 12
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_ortJet/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/scratch/glide_ortJet/execute/dir_23710/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=23710 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_ortJet/cred.d/idtokens/credential_OSPool.main_OSG_US_MI-HORUS-CE2_lg.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=351943 GLIDEIN_Entry_Name=OSG_US_MI-HORUS-CE2_lg GLIDEIN_Entry_Signature=288467fbd7048c8018f1cd57215bed6c6a9cf81d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=mi-horus-ce2.svc.osg-htc.org mi-horus-ce2.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_KhSRKY GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=MI-HORUS-CE2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=MI-HORUS GLIDEIN_Tmp_Dir=/scratch/glide_ortJet/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=MI-HORUS-CE2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3812421_216657210@wsu-lg02.osris.org$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3812421_216657210@wsu-lg02.osris.org$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_ortJet/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el9 SINGULARITY_IMAGES_DICT=default:/scratch/glide_ortJet/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_ortJet/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_ortJet/execute/dir_23710/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-503.11.1.el9_5.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
554.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_scratch-scratch 937231232 355965548 581265684 38% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
MSU-DataMachine-CE1 2026-04-03 17:01 120
100.0%
90.8%
100.0%
90.8%
mtu env kernel nrp_images df http_proxy
MSU-DataMachine-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: crdc-am11-icer-sw2-hpcc-unsecured-dmz-irb-1799.rtr.net.msu.edu 3.722ms 1: crdc-am11-icer-sw2-hpcc-unsecured-dmz-irb-1799.rtr.net.msu.edu 4.354ms 2: crdc-todc1-hpcc-unsecured-dmz-et-0-0-42-1799.rtr.net.msu.edu 2.173ms asymm 3 3: dc-fw-hpcc-unsecured-dmz-reth1-3671.rtr.net.msu.edu 8.328ms 4: no reply 5: cc-border-mx-core-lo0-1000.rtr.net.msu.edu 12.584ms 6: border-palo-ae1-1484.net.msu.edu 16.842ms 7: 35.12.102.102 17.399ms 8: ae23x110.msu6.mich.net 17.089ms 9: ae102x0.msu6.mich.net 24.675ms asymm 8 10: ae102x0.chcg-lumen-600w-c1.mich.net 22.074ms asymm 11 11: 192.122.182.34 25.568ms asymm 13 12: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 26.246ms asymm 13 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 26.560ms 14: 144.92.233.112 27.800ms 15: perfsonar02.hep.wisc.edu 27.572ms reached Resume: pmtu 1500 hops 15 back 16
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/osg01/glide_y9wIb9/images/htc__debian__13.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__debian__13.sif APPTAINER_TMPDIR=/tmp/osg01/glide_y9wIb9/execute/dir_423444/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=423444 CONTAINERSW_FULL_VERSION=apptainer version 1.3.3-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.3.3-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/osg01/glide_y9wIb9/cred.d/idtokens/credential_OSPool.main_OSG_US_MSU-DataMachine-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=237054 GLIDEIN_Entry_Name=OSG_US_MSU-DataMachine-CE1 GLIDEIN_Entry_Signature=046f2c0d9b7dd3f0726583009b594de09a89a330 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=msu-datamachine-ce1.svc.opensciencegrid.org msu-datamachine-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_nobody_VbPYg4 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=7 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=MSU-DataMachine-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=MSU-DataMachine GLIDEIN_Tmp_Dir=/tmp/osg01/glide_y9wIb9/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.3.3-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.3.3-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.3.3-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=MSU-DataMachine-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3616275_357122095@nal-005-docker$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3616275_357122095@nal-005-docker$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/osg01/glide_y9wIb9/images/htc__debian__13.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.3.3-1.el8 SINGULARITY_IMAGES_DICT=default:/tmp/osg01/glide_y9wIb9/images/htc__debian__13.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__debian__13.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.3.3-1.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/osg01/glide_y9wIb9/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/osg01/glide_y9wIb9/execute/dir_423444/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.15.0-157-generic
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
24170.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md0 26252153856 907503112 25344650744 4% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
MTState-Tempest-CE1 2026-04-03 17:01 304
99.7%
99.7%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
MTState-Tempest-CE1 — probe output
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: _gateway 0.154ms 1: _gateway 0.128ms 2: hsrp203.253.msu.montana.edu 0.820ms 3: no reply 4: rnedge-prodfw.msu.montana.edu 1.863ms 5: ae13--539.icar-sttl1-2.infra.pnw-gigapop.net 15.703ms 6: lo0--210.icar-sttl1-1.infra.pnw-gigapop.net 15.623ms 7: 209.124.181.167 17.139ms 8: fourhundredge-0-0-0-18.4079.core1.seat.net.internet2.edu 53.283ms asymm 12 9: fourhundredge-0-0-0-0.4079.core1.salt.net.internet2.edu 48.979ms asymm 11 10: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 49.473ms 11: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 48.816ms asymm 9 12: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 46.782ms asymm 8 13: fourhundredge-0-0-0-22.4079.core1.chic.net.internet2.edu 47.246ms asymm 7 14: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 47.565ms asymm 6 15: 144.92.233.112 49.960ms asymm 7 16: perfsonar02.hep.wisc.edu 50.195ms reached Resume: pmtu 9000 hops 16 back 9
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/jobs/3821641/glide_dJxJfw/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/jobs/3821641/glide_dJxJfw/execute/dir_440754/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=440754 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/jobs/3821641/glide_dJxJfw/cred.d/idtokens/credential_OSPool.main_OSG_US_MTState-Tempest-CE1.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=292970 GLIDEIN_Entry_Name=OSG_US_MTState-Tempest-CE1 GLIDEIN_Entry_Signature=ceda3c330309bea3ef8cef64d1c516b0746a402e GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=mtstate-tempest-ce1.svc.opensciencegrid.org mtstate-tempest-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_20Gdfa GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=8 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=MTState-Tempest-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=MTState-Tempest GLIDEIN_Tmp_Dir=/tmp/jobs/3821641/glide_dJxJfw/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=MTState-Tempest-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_684360_93991376@epyc033.rci.montana.edu$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_684360_93991376@epyc033.rci.montana.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/jobs/3821641/glide_dJxJfw/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/tmp/jobs/3821641/glide_dJxJfw/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/jobs/3821641/glide_dJxJfw/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/jobs/3821641/glide_dJxJfw/execute/dir_440754/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.89.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
715.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1 937234648 186610992 750623656 20% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
MWT2 2026-04-03 17:01 215
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
MWT2 — probe output
pass mtu
pmtu=9000 1?: [LOCALHOST] 0.009ms pmtu 9000 1: 2001:18e8:c02:5::252 0.248ms 1: 2001:18e8:c02:5::252 0.219ms 2: vlan-4082.bordera1.ictc.net.rt.iu.edu 0.224ms asymm 3 3: et-9-3-0.2015.rtr.ictc.indiana.gigapop.net 0.969ms 4: 2001:18e8:ff00:2c::2 2.455ms asymm 3 5: lo-0.2.rtr2.chic.indiana.gigapop.net 3.808ms 6: no reply 7: uwmadison_se-203--chic-cr6.es.net 21.066ms 8: uwmadison_se-203--chic-cr6.es.net 21.217ms asymm 7 9: perfsonar02.hep.wisc.edu 21.025ms reached Resume: pmtu 9000 hops 9 back 8
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_1072546/glide_eQhIBo/images/htc__debian__13.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__debian__13.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_1072546/glide_eQhIBo/execute/dir_195917/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=195917 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_1072546/glide_eQhIBo/cred.d/idtokens/credential_OSPool.main_Engage_US_MWT2_uiuc_condce_mcore.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=233883 GLIDEIN_Country=US GLIDEIN_DUNESite=US_UChicago GLIDEIN_Entry_Name=Engage_US_MWT2_uiuc_condce_mcore GLIDEIN_Entry_Signature=cdea3d15620f8fbd1f2375ed8ccc9fa64e0080c0 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=uiuc-gk.mwt2.org uiuc-gk.mwt2.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_skvnjI GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=MWT2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=UChicago GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_1072546/glide_eQhIBo/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=uiuc-gk.mwt2.org OSG_SITE_NAME=MWT2_CE_UIUC OSG_SQUID_LOCATION=iut2-slate.mwt2.org:32200 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_8@glidein_1072548_145491455@iut2-c297.iu.edu$ "; unset PROMPT_COMMAND PS1=slot1_8@glidein_1072548_145491455@iut2-c297.iu.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_1072546/glide_eQhIBo/images/htc__debian__13.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/dir_1072546/glide_eQhIBo/images/htc__debian__13.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__debian__13.sif SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_1072546/glide_eQhIBo/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_1072546/glide_eQhIBo/execute/dir_195917/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-570.60.1.el9_6.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1221.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 1842548380 467856160 1281022128 27% /srv
fail http_proxy
Failed to download http://cm-1.ospool.osg-htc.org/overview/ via proxy iut2-slate.mwt2.org:32200: <urlopen error [Errno 113] No route to host>
Maine-Penobscot-CE1 2026-04-03 17:04 20
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
Maine-Penobscot-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: login1.localdomain 0.210ms 1: login1.localdomain 0.195ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass env
APPTAINER_CACHEDIR=/srv/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2453696 CONTAINERSW_FULL_VERSION=singularity-ce version 3.11.5-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/singularity CONTAINERSW_VERSION=3.11.5-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01p/glide_Kr2frb/cred.d/idtokens/credential_OSPool.main_OSG_US_Maine-Penobscot-CE1_epyc.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=6113165 GLIDEIN_Entry_Name=OSG_US_Maine-Penobscot-CE1_epyc GLIDEIN_Entry_Signature=2a26f5fc075e294b322338aaa2159875e2e2a95f GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=maine-penobscot-ce1.svc.opensciencegrid.org maine-penobscot-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01p_RP5Try GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Maine-Penobscot-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=5552c705dd9f3b999102cc42d5c5bdce18bba000 GLIDEIN_Site=Maine-Penobscot GLIDEIN_Tmp_Dir=/scratch/osg01p/glide_Kr2frb/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity-ce version 3.11.5-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/singularity GWMS_CONTAINERSW_VERSION=3.11.5-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.11.5-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Maine-Penobscot-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_3647358_884514845@node-159.localdomain$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_3647358_884514845@node-159.localdomain$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/scratch/osg01p/glide_Kr2frb/execute/dir_2453696:/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/scratch/osg01p/glide_Kr2frb/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity-ce version 3.11.5-1.el9 SINGULARITY_IMAGES_DICT=default:/scratch/osg01p/glide_Kr2frb/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/singularity SINGULARITY_TMPDIR=/scratch/osg01p/glide_Kr2frb/execute/dir_2453696/scratch SINGULARITY_VERSION=3.11.5-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01p/glide_Kr2frb/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01p/glide_Kr2frb/execute/dir_2453696/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-284.30.1.el9_2.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1368.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p1 1537170196 24324044 1434705532 2% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Missouri-Hellbender-CE1 2026-04-03 17:02 384
100.0%
99.7%
0.0%
99.7%
mtu env kernel nrp_images df http_proxy
Missouri-Hellbender-CE1 — probe output
not_available mtu
tracepath not found on this worker node
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/scratch/glide_7marmD/images/htc__ubuntu__24.04.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__ubuntu__24.04.sif APPTAINER_TMPDIR=/local/scratch/glide_7marmD/execute/dir_1549236/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1549236 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/scratch/glide_7marmD/cred.d/idtokens/credential_OSPool.main_OSG_US_Missouri-Hellbender-CE1.idtoken GLIDEIN_CPUS=2 GLIDEIN_ClusterId=6274594 GLIDEIN_Entry_Name=OSG_US_Missouri-Hellbender-CE1 GLIDEIN_Entry_Signature=5cb055fd63403d3d18d464174455fe71a116125c GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=missouri-hellbender-ce1.svc.opensciencegrid.org missouri-hellbender-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_s2c_osg_01_lqeEFP GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Missouri-Hellbender-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=5552c705dd9f3b999102cc42d5c5bdce18bba000 GLIDEIN_Site=Missouri-Hellbender GLIDEIN_Tmp_Dir=/local/scratch/glide_7marmD/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Missouri-Hellbender-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_3570962_297177600@c217.mgmt.hellbender$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_3570962_297177600@c217.mgmt.hellbender$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/local/scratch/glide_7marmD/images/htc__ubuntu__24.04.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/local/scratch/glide_7marmD/images/htc__ubuntu__24.04.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__ubuntu__24.04.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/scratch/glide_7marmD/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/scratch/glide_7marmD/execute/dir_1549236/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.8.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1200.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme1n1p1 1562049324 303046148 1259003176 20% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
NCSA-Delta-CE1 2026-04-03 16:01 2
100.0%
100.0%
100.0%
100.0%
all
NCSA-Delta-CE1 — probe output
warning all
Job failed with exit code 0; no results transferred
NCSU-OSG-CE1 2026-04-03 12:46 2
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
NCSU-OSG-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: vl2140-dc2-6509-1-fwsm-2.ncstate.net 8.509ms 1: vl2140-dc2-6509-1-fwsm-2.ncstate.net 8.965ms 2: 10.250.142.112 9.421ms asymm 3 3: 10.250.133.69 8.985ms 4: 128.109.1.245 5.981ms 5: rtp-gw-to-ws-gw.ncren.net 17.005ms 6: bundle-ether250.613.core1.rale.net.internet2.edu 19.468ms 7: fourhundredge-0-0-0-6.4079.core1.ashb.net.internet2.edu 37.771ms asymm 15 8: fourhundredge-0-0-0-16.4079.core2.ashb.net.internet2.edu 39.637ms asymm 14 9: fourhundredge-0-0-0-1.4079.core2.clev.net.internet2.edu 37.593ms asymm 13 10: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 38.819ms asymm 12 11: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 37.296ms 12: 144.92.233.112 40.640ms 13: perfsonar02.hep.wisc.edu 39.823ms reached Resume: pmtu 1500 hops 13 back 14
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_bXZXhl/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_bXZXhl/execute/dir_1401132/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1401132 CONTAINERSW_FULL_VERSION=apptainer version 1.4.1-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.1-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_bXZXhl/cred.d/idtokens/credential_OSPool.main_OSG_US_NCSU-OSG-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=351113 GLIDEIN_Entry_Name=OSG_US_NCSU-OSG-CE1 GLIDEIN_Entry_Signature=56f3ae3a53a275df81b74777d28a94dd37b12b4e GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=ncsu-osg-ce1.svc.osg-htc.org ncsu-osg-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_OSG01_2Ewzfi GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=NCSU-OSG-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=NCSU-OSG GLIDEIN_Tmp_Dir=/scratch/glide_bXZXhl/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.1-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.1-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.1-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=NCSU-OSG-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_1377035_802272159@vcledch0201.hpc.ncsu.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_1377035_802272159@vcledch0201.hpc.ncsu.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_bXZXhl/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.1-1.el9 SINGULARITY_IMAGES_DICT=default:/scratch/glide_bXZXhl/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.1-1.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_bXZXhl/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_bXZXhl/execute/dir_1401132/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-570.23.1.el9_6.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
3473.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p7 3671331348 29296852 3642034496 1% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
ND-CAML-GPU-CE1 2026-04-03 16:01 4
50.0%
50.0%
100.0%
100.0%
all
ND-CAML-GPU-CE1 — probe output
warning all
Job failed with exit code 0; no results transferred
NMSU-Discovery-CE1 2026-04-03 17:01 58
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
NMSU-Discovery-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.88.37.254 0.276ms 1: 10.88.37.254 0.290ms 2: dc-core-vlan-88-dc.nmsu.edu 1.209ms 3: 10.123.129.137 0.793ms asymm 4 4: 10.123.128.49 1.252ms asymm 6 5: 10.123.128.41 6.784ms asymm 7 6: 192.65.78.197 1.240ms 7: 208.77.78.113 10.759ms 8: bundle-ether160.3466.core1.denv.net.internet2.edu 15.768ms 9: bundle-ether160.3466.core1.denv.net.internet2.edu 16.903ms asymm 8 10: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 43.896ms asymm 15 11: fourhundredge-0-0-0-21.4079.core1.chic.net.internet2.edu 44.951ms asymm 13 12: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 44.907ms 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 44.908ms asymm 12 14: 144.92.233.112 44.903ms asymm 13 15: perfsonar02.hep.wisc.edu 44.756ms reached Resume: pmtu 1500 hops 15 back 15
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/fs1/osg/osg01/glide_gMQle3/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/fs1/osg/osg01/glide_gMQle3/execute/dir_4074488/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=4074488 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/fs1/osg/osg01/glide_gMQle3/cred.d/idtokens/credential_OSPool.main_OSG_US_NMSU-Discovery-CE1_large.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=236301 GLIDEIN_Entry_Name=OSG_US_NMSU-Discovery-CE1_large GLIDEIN_Entry_Signature=0d77aea544ff84f5d79187c39d6359857aabdbd7 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=nmsu-discovery-ce1.svc.osg-htc.org nmsu-discovery-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_UornZg GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=192.168.211.16:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=NMSU-Discovery-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=NMSU_DISCOVERY GLIDEIN_Tmp_Dir=/fs1/osg/osg01/glide_gMQle3/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=NMSU-Discovery-CE1 OSG_SQUID_LOCATION=192.168.211.16:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_8@glidein_2429835_5653681@discovery-c16.cluster.local$ "; unset PROMPT_COMMAND PS1=slot1_8@glidein_2429835_5653681@discovery-c16.cluster.local$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/fs1/osg/osg01/glide_gMQle3/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el8 SINGULARITY_IMAGES_DICT=default:/fs1/osg/osg01/glide_gMQle3/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/fs1/osg/osg01/glide_gMQle3/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/fs1/osg/osg01/glide_gMQle3/execute/dir_4074488/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.92.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
35430.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on fs1 37580963840 429539328 37151424512 2% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy 192.168.211.16:3128
NWICG_NDCMS 2026-04-03 11:03 4
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
NWICG_NDCMS — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: r64cisco9508-85-secondary.crc.nd.edu 0.495ms 1: r64cisco9508-85-secondary.crc.nd.edu 0.463ms 2: crc-def2compute-3.gw.nd.edu 0.509ms 3: 172.21.0.29 0.388ms asymm 4 4: man-pa-node-vip.fw.nd.edu 0.627ms 5: jbr0-inside.gw.nd.edu 8.417ms 6: et-4-2-1.401.rtr.ictc.indiana.gigapop.net 13.932ms 7: ae-9.0.rtr.ll.indiana.gigapop.net 13.925ms 8: ae-7.1.rtr2.chic.indiana.gigapop.net 17.333ms 9: 144.92.233.118 34.405ms 10: perfsonar02.hep.wisc.edu 34.253ms reached Resume: pmtu 1500 hops 10 back 10
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/cms,/users APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/condor/execute/dir_940646/glide_cTWdtZ/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/condor/execute/dir_940646/glide_cTWdtZ/execute/dir_1002854/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1002854 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_NotreDame GLIDEIN_CONDOR_TOKEN=/var/condor/execute/dir_940646/glide_cTWdtZ/cred.d/idtokens/credential_OSPool.main_CMSHTPC_T3_US_NotreDame_deepthought.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=6120251 GLIDEIN_Country=US GLIDEIN_DUNESite=US_NotreDame GLIDEIN_Entry_Name=CMSHTPC_T3_US_NotreDame_deepthought GLIDEIN_Entry_Signature=a0b2959eb6a14604261f2c2a56a341e3b77f6b10 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=deepthought.crc.nd.edu deepthought.crc.nd.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_4KsFp1 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=NWICG_NDCMS GLIDEIN_SEs=deepthought.crc.nd.edu GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=436230a2fef517d351fbb58e714fa5f5fa541af0 GLIDEIN_Site=NotreDame GLIDEIN_Tmp_Dir=/var/condor/execute/dir_940646/glide_cTWdtZ/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=NotreDame PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_940648_96675070@d12chas569.crc.nd.edu$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_940648_96675070@d12chas569.crc.nd.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/cms,/users SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/condor/execute/dir_940646/glide_cTWdtZ/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-1.el9 SINGULARITY_IMAGES_DICT=default:/var/condor/execute/dir_940646/glide_cTWdtZ/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/condor/execute/dir_940646/glide_cTWdtZ/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/condor/execute/dir_940646/glide_cTWdtZ/execute/dir_1002854/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.16.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1743.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda2 1949168640 120987832 1828180808 7% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Nebraska 2026-04-03 17:01 18
100.0%
0.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
Nebraska — probe output
pass mtu
pmtu=9000 1?: [LOCALHOST] 0.016ms pmtu 9000 1: 2600:900:6:1102::28 0.614ms 1: 2600:900:6:1102::28 0.599ms 2: 2600:900:6:1048::1 0.754ms 3: no reply 4: chic-cr6--uwmadison_se-203.es.net 16.068ms 5: uwmadison_se-203--chic-cr6.es.net 33.286ms 6: perfsonar02.hep.wisc.edu 33.168ms reached Resume: pmtu 9000 hops 6 back 6
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/var/lib/condor/execute/dir_159545,/cvmfs,/etc/cvmfs,/etc/grid-security,/var/lib/cvmfs,/var/lib/sss/pipes,/var/lib/sss/mc,/etc/nsswitch.conf,/dev/fuse,/dev/log APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=rocky:9 APPTAINER_TMPDIR=/var/lib/condor/execute/dir_159545/glide_Aaxwif/execute/dir_70957/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=70957 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0~rc.1-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer CONTAINERSW_VERSION=1.5.0~rc.1-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T2_US_Nebraska GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_159545/glide_Aaxwif/cred.d/idtokens/credential_OSPool.main_CMS_T2_US_Nebraska_Red_whole_op.idtoken GLIDEIN_CPUS=32 GLIDEIN_ClusterId=230785 GLIDEIN_Country=US GLIDEIN_DUNESite=US_Nebraska GLIDEIN_ESTIMATED_CPUS=40 GLIDEIN_Entry_Name=CMS_T2_US_Nebraska_Red_whole_op GLIDEIN_Entry_Signature=061cc0d5a84d70abb25d9b6b0296962a7c082d1f GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=red.unl.edu red.unl.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_UueDCR GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=red-squid.unl.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Nebraska GLIDEIN_SE_VONAME_LOWERCASE=srm://srm.unl.edu:8443/srm/v2/server?SFN=/mnt/hadoop/user/ GLIDEIN_SEs=srm.unl.edu GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=1a97d6060d26940eba08ee679d09a38f5c43d5c2 GLIDEIN_Site=Nebraska GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_159545/glide_Aaxwif/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0~rc.1-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0~rc.1-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0~rc.1-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=red.unl.edu OSG_SITE_NAME=Red OSG_SQUID_LOCATION=red-squid.unl.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_32@glidein_13_64830920@red-c5105.unl.edu$ "; unset PROMPT_COMMAND PS1=slot1_32@glidein_13_64830920@red-c5105.unl.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/var/lib/condor/execute/dir_159545,/cvmfs,/etc/cvmfs,/etc/grid-security,/var/lib/cvmfs,/var/lib/sss/pipes,/var/lib/sss/mc,/etc/nsswitch.conf,/dev/fuse,/dev/log SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0~rc.1-1.el8 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_MODE=unprivileged SINGULARITY_NAME=rocky:9 SINGULARITY_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer SINGULARITY_VERSION=1.5.0~rc.1-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_159545/glide_Aaxwif/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_159545/glide_Aaxwif/execute/dir_70957/scratch _CONDOR_SLOT=slot1_32 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=red-squid.unl.edu:3128
pass kernel
6.18.3-1.el8.elrepo.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
826.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_system-lv_var 962155504 95938796 866216708 10% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy red-squid.unl.edu:3128
ORU-Titan-CE1 2026-04-03 17:01 20
100.0%
0.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
ORU-Titan-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.304ms asymm 2 1: _gateway 0.213ms asymm 2 2: 156.110.41.145 0.534ms 3: 156.110.31.225 2.373ms 4: 198.71.46.45 1.197ms asymm 5 5: fourhundredge-0-0-0-9.302.core1.tuls.net.internet2.edu 4.298ms 6: fourhundredge-0-0-0-2.4079.core1.kans.net.internet2.edu 16.241ms asymm 12 7: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 34.029ms asymm 11 8: fourhundredge-0-0-0-21.4079.core1.chic.net.internet2.edu 17.257ms asymm 10 9: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 18.308ms 10: 144.92.233.112 19.696ms 11: perfsonar02.hep.wisc.edu 19.570ms reached Resume: pmtu 1500 hops 11 back 12
pass env
APPTAINER_CACHEDIR=/srv/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2122593 CONTAINERSW_FULL_VERSION=singularity version 3.7.1-5.1.ohpc.2.1 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/opt/ohpc/pub/libs/singularity/3.7.1/bin/singularity CONTAINERSW_VERSION=3.7.1-5.1.ohpc.2.1 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osgsubmit/glide_bFkoLn/cred.d/idtokens/credential_OSPool.main_OSG_US_ORU-Titan-CE1.idtoken GLIDEIN_CPUS=40 GLIDEIN_ClusterId=295958 GLIDEIN_Entry_Name=OSG_US_ORU-Titan-CE1 GLIDEIN_Entry_Signature=ec2d5c7027ab195f653360a0efad9bc2f1afa734 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=oru-titan-ce1.svc.opensciencegrid.org oru-titan-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgsubmit_9oark1 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=ORU-Titan-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=ORU-Titan GLIDEIN_Tmp_Dir=/scratch/osgsubmit/glide_bFkoLn/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity version 3.7.1-5.1.ohpc.2.1 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/opt/ohpc/pub/libs/singularity/3.7.1/bin/singularity GWMS_CONTAINERSW_VERSION=3.7.1-5.1.ohpc.2.1 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/opt/ohpc/pub/libs/singularity/3.7.1/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.7.1-5.1.ohpc.2.1 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=ORU-Titan-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_8@glidein_1890295_206461072@c006.orca.oru.edu$ "; unset PROMPT_COMMAND PS1=slot1_8@glidein_1890295_206461072@c006.orca.oru.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/scratch/osgsubmit/glide_bFkoLn/execute/dir_2122593:/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/scratch/osgsubmit/glide_bFkoLn/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity version 3.7.1-5.1.ohpc.2.1 SINGULARITY_IMAGES_DICT=default:/scratch/osgsubmit/glide_bFkoLn/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/opt/ohpc/pub/libs/singularity/3.7.1/bin/singularity SINGULARITY_TMPDIR=/scratch/osgsubmit/glide_bFkoLn/execute/dir_2122593/scratch SINGULARITY_VERSION=3.7.1-5.1.ohpc.2.1 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osgsubmit/glide_bFkoLn/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osgsubmit/glide_bFkoLn/execute/dir_2122593/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.30.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
471357.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on beegfs_nodev 777312396288 283058491392 494253904896 37% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
OU-Sooner-CE1 2026-04-03 16:01 2
100.0%
100.0%
100.0%
100.0%
all
OU-Sooner-CE1 — probe output
warning all
Job failed with exit code 0; no results transferred
PDX-Coeus-CE1 2026-04-03 17:04 11
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
PDX-Coeus-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 9000 1: _gateway 0.747ms asymm 2 1: _gateway 0.742ms asymm 2 2: _gateway 0.634ms asymm 3 3: 10.252.100.1 0.216ms pmtu 1500 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_PPClnO/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/tmp/glide_PPClnO/execute/dir_7329/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=7329 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/tmp/glide_PPClnO/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=2c5a85deb24f1f4623e86c3eea56d0538d0b4afa GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_PPClnO/cred.d/idtokens/credential_OSPool.gpu_OSG_US_PDX-Coeus-CE1.idtoken GLIDEIN_CPUS=32 GLIDEIN_ClusterId=6274112 GLIDEIN_Entry_Name=OSG_US_PDX-Coeus-CE1 GLIDEIN_Entry_Signature=e9c0809ee16674db26a7020079d1e22e6c0fac55 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=pdx-coeus-ce1.svc.osg-htc.org pdx-coeus-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_bVsTMa GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=coeus.cluster:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=PDX-Coeus-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=5552c705dd9f3b999102cc42d5c5bdce18bba000 GLIDEIN_Site=PDX-Coeus GLIDEIN_Tmp_Dir=/tmp/glide_PPClnO/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/tmp/glide_PPClnO/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/tmp/glide_PPClnO/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=PDX-Coeus-CE1 OSG_SQUID_LOCATION=coeus.cluster:3128 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_6@glidein_2193128_486188614@gpu09$ "; unset PROMPT_COMMAND PS1=slot1_6@glidein_2193128_486188614@gpu09$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_PPClnO/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/tmp/glide_PPClnO/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SINGULARITY_PATH=/tmp/glide_PPClnO/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_PPClnO/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_PPClnO/execute/dir_7329/scratch _CONDOR_SLOT=slot1_6 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.36.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1523.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p1 1748318720 62060580 1597374620 4% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy coeus.cluster:3128
PDX-Orca-CE1 2026-04-03 17:06 6
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
PDX-Orca-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 9000 1: _gateway 0.782ms asymm 2 1: _gateway 0.546ms asymm 2 2: _gateway 0.622ms asymm 3 3: 10.252.100.1 0.183ms pmtu 1500 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_NSfwtD/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_NSfwtD/execute/dir_572926/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=572926 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/tmp/glide_NSfwtD/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_NSfwtD/cred.d/idtokens/credential_OSPool.main_OSG_US_PDX-Orca-CE1.idtoken GLIDEIN_CPUS=32 GLIDEIN_ClusterId=5603092 GLIDEIN_Entry_Name=OSG_US_PDX-Orca-CE1 GLIDEIN_Entry_Signature=cdb979e6f8ebcef5958e6faedd91db632ed6e8e1 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=pdx-orca-ce1.svc.osg-htc.org pdx-orca-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_VOFkbL GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=PDX-Orca-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=5552c705dd9f3b999102cc42d5c5bdce18bba000 GLIDEIN_Site=PDX-Orca GLIDEIN_Tmp_Dir=/tmp/glide_NSfwtD/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/tmp/glide_NSfwtD/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/tmp/glide_NSfwtD/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=PDX-Orca-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_1547350_857634432@orcaga22$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_1547350_857634432@orcaga22$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_NSfwtD/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/tmp/glide_NSfwtD/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/tmp/glide_NSfwtD/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_NSfwtD/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_NSfwtD/execute/dir_572926/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.13.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
380.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sdb1 460367788 38410720 398498156 9% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
PSU-LIGO 2026-04-03 11:02 39
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
PSU-LIGO — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 13.628ms 1: _gateway 20.097ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/localscratch/condor/execute/dir_890928/scratch/glide_G4jCAn/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/localscratch/condor/execute/dir_890928/scratch/glide_G4jCAn/execute/dir_2681541/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2681541 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/localscratch/condor/execute/dir_890928/scratch/glide_G4jCAn/cred.d/idtokens/credential_OSPool.main_OSG_US_PSU-LIGO.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=5609489 GLIDEIN_Entry_Name=OSG_US_PSU-LIGO GLIDEIN_Entry_Signature=9e01795dbf061eeb098c7cb1eb9bcadbec97de1c GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=psu-ligo-ce1.svc.opensciencegrid.org psu-ligo-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_miMrqg GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=ligo-squid-01.gwave.ics.psu.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=PSU-LIGO GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=68376560144ff7f968c1f9a47b56d2289fce3db9 GLIDEIN_Site=PSU-LIGO GLIDEIN_Tmp_Dir=/localscratch/condor/execute/dir_890928/scratch/glide_G4jCAn/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=PSU-LIGO OSG_SQUID_LOCATION=ligo-squid-01.gwave.ics.psu.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_890944_47510399@comp-cc-0445$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_890944_47510399@comp-cc-0445$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/localscratch/condor/execute/dir_890928/scratch/glide_G4jCAn/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/localscratch/condor/execute/dir_890928/scratch/glide_G4jCAn/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/localscratch/condor/execute/dir_890928/scratch/glide_G4jCAn/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/localscratch/condor/execute/dir_890928/scratch/glide_G4jCAn/execute/dir_2681541/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.111.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1094.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p3 1366833076 218815384 1148017692 17% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy ligo-squid-01.gwave.ics.psu.edu:3128
Purdue-Anvil-CE1 2026-04-03 17:01 22
100.0%
63.6%
0.0%
63.6%
mtu env kernel nrp_images df http_proxy
Purdue-Anvil-CE1 — probe output
pass mtu
pmtu=4092 1?: [LOCALHOST] pmtu 4092 1: 172.17.3.179 0.409ms 1: 172.17.3.179 0.434ms 2: math-g190-c7710-02-e1-10.tcom.purdue.edu 0.608ms 3: math-g190-ar7280dr3a-02-et50-1.tcom.purdue.edu 0.210ms 4: math-g190-ar7280dr3a-01-vl2500.tcom.purdue.edu 0.364ms 5: math-g190-ar7280dr3a-02-et54-1.tcom.purdue.edu 0.320ms asymm 4 6: research-gigapop-ll-lhcone-400g.tcom.purdue.edu 2.214ms asymm 5 7: lo-0.2.rtr2.chic.indiana.gigapop.net 5.911ms asymm 6 8: no reply 9: uwmadison_se-203--chic-cr6.es.net 22.924ms asymm 8 10: perfsonar02.hep.wisc.edu 22.757ms reached Resume: pmtu 4092 hops 10 back 9
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_tPsng1/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/tmp/glide_tPsng1/execute/dir_3436742/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3436742 CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.3-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_tPsng1/cred.d/idtokens/credential_OSPool.main_OSG_US_Purdue-Anvil-CE1.idtoken GLIDEIN_CPUS=128 GLIDEIN_ClusterId=301335 GLIDEIN_Entry_Name=OSG_US_Purdue-Anvil-CE1 GLIDEIN_Entry_Signature=0cc5f657a6e142cf255b47b011f3ba93383d63a9 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=purdue-anvil-ce1.svc.osg-htc.org purdue-anvil-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_x-ob_m0Y7rD GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=squid1.rcac.purdue.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Purdue-Anvil-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=Purdue-Anvil GLIDEIN_Tmp_Dir=/tmp/glide_tPsng1/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.3-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.3-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Purdue-Anvil-CE1 OSG_SQUID_LOCATION=squid1.rcac.purdue.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_1827999_169263348@a916.anvil.rcac.purdue.edu$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_1827999_169263348@a916.anvil.rcac.purdue.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_tPsng1/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.3-1.el8 SINGULARITY_IMAGES_DICT=default:/tmp/glide_tPsng1/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.3-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_tPsng1/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_tPsng1/execute/dir_3436742/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.81.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
288.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda2 374917264 72703216 302214048 20% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy squid1.rcac.purdue.edu:3128
Purdue-Geddes 2026-04-03 17:02 10
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
Purdue-Geddes — probe output
pass mtu
pmtu=8950 1?: [LOCALHOST] pmtu 8950 1: 172-21-160-11.csi-rbdplugin-metrics.rook-ceph.svc.cluster.local 0.055ms 1: 172-21-160-11.csi-rbdplugin-metrics.rook-ceph.svc.cluster.local 0.044ms 2: math-g109-dz9264f-01-vlan1160.tcom.purdue.edu 0.381ms 3: math-g190-c7710-01-eth-3-15.tcom.purdue.edu 0.537ms 4: math-g190-ar7280dr3a-02-et50-1.tcom.purdue.edu 0.160ms 5: math-g190-ar7280dr3a-01-vl2500.tcom.purdue.edu 0.372ms 6: math-g190-ar7280dr3a-02-et54-1.tcom.purdue.edu 0.367ms asymm 5 7: research-gigapop-ll-lhcone-400g.tcom.purdue.edu 2.192ms asymm 6 8: lo-0.2.rtr2.chic.indiana.gigapop.net 5.796ms asymm 7 9: no reply 10: uwmadison_se-203--chic-cr6.es.net 23.015ms asymm 9 11: perfsonar02.hep.wisc.edu 22.777ms reached Resume: pmtu 8950 hops 11 back 10
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-wMtDhj/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-wMtDhj/execute/dir_488148 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=488148 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Purdue Geddes OSG_SQUID_LOCATION=http://squid1.rcac.purdue.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_2@Purdue-Geddes.osg-geddes-backfill-a-8$ "; unset PROMPT_COMMAND PS1=slot1_2@Purdue-Geddes.osg-geddes-backfill-a-8$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-wMtDhj/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-wMtDhj/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-wMtDhj/execute/dir_488148 _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=http://squid1.rcac.purdue.edu:3128
pass kernel
4.18.0-553.22.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
105.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/rbd4 123268672 12503692 110748596 11% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy http://squid1.rcac.purdue.edu:3128
Rhodes-HPC-CE1 2026-04-03 17:01 10
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
Rhodes-HPC-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: lotus 0.185ms 1: lotus 0.133ms 2: 10.70.0.1 0.232ms 3: xg-1-1-35-3909-soag12.hornlake.tn.malt.comcast.net 1.676ms 4: no reply 5: no reply 6: be-288-ar01.hornlake.tn.malt.comcast.net 1.956ms 7: 162.151.119.229 1.927ms 8: be-34912-cs01.dallas.tx.ibone.comcast.net 10.529ms 9: be-3211-pe11.1950stemmons.tx.ibone.comcast.net 10.950ms 10: no reply 11: kanc-bb2-link.ip.twelve99.net 26.690ms asymm 12 12: chi-bb2-link.ip.twelve99.net 26.975ms asymm 10 13: chi-b24-link.ip.twelve99.net 26.698ms asymm 11 14: wiscnet-ic-308898.ip.twelve99-cust.net 26.538ms asymm 13 15: r-uwmilwaukee-ems-ae1-3473.uwsys.net 29.279ms asymm 14 16: r-uwmadison-cssc-ae4-3455.uwsys.net 31.274ms asymm 15 17: 143.235.41.129 31.696ms asymm 16 18: perfsonar02.hep.wisc.edu 31.603ms reached Resume: pmtu 1500 hops 18 back 18
pass env
APPTAINER_CACHEDIR=/srv/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=835030 CONTAINERSW_FULL_VERSION=singularity-ce version 4.2.2-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/singularity CONTAINERSW_VERSION=4.2.2-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_auFNgB/cred.d/idtokens/credential_OSPool.main_OSG_US_Rhodes-HPC-CE1.idtoken GLIDEIN_CPUS=48 GLIDEIN_ClusterId=299927 GLIDEIN_Entry_Name=OSG_US_Rhodes-HPC-CE1 GLIDEIN_Entry_Signature=f4e669788121899b8d8ac734d1e5bc6da4f6ce80 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=rhodes-hpc-ce1.svc.osg-htc.org rhodes-hpc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_zqbyO9 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=10.0.0.1:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Rhodes-HPC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=Rhodes-HPC GLIDEIN_Tmp_Dir=/scratch/glide_auFNgB/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity-ce version 4.2.2-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/singularity GWMS_CONTAINERSW_VERSION=4.2.2-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=4.2.2-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Rhodes-HPC-CE1 OSG_SQUID_LOCATION=10.0.0.1:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_17@glidein_1402595_63269688@compute05$ "; unset PROMPT_COMMAND PS1=slot1_17@glidein_1402595_63269688@compute05$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/scratch/glide_auFNgB/execute/dir_835030:/srv,/etc/hosts,/etc/localtime,/cvmfs:/cvmfs:ro SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/scratch/glide_auFNgB/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity-ce version 4.2.2-1.el9 SINGULARITY_IMAGES_DICT=default:/scratch/glide_auFNgB/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/singularity SINGULARITY_TMPDIR=/scratch/glide_auFNgB/execute/dir_835030/scratch SINGULARITY_VERSION=4.2.2-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_auFNgB/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_auFNgB/execute/dir_835030/scratch _CONDOR_SLOT=slot1_17 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-503.22.1.el9_5.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
136.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 499862048 356605492 143256556 72% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy 10.0.0.1:3128
SDSC-PRP-OSPool-Provisioner 2026-04-03 17:02 1157
100.0%
98.4%
98.4%
100.0%
mtu env kernel nrp_images df http_proxy
SDSC-PRP-OSPool-Provisioner — probe output
pass mtu
pmtu=8930 1?: [LOCALHOST] pmtu 8930 1: hcc-nrp-pki-c1703.unl.edu 0.449ms 1: hcc-nrp-pki-c1703.unl.edu 0.589ms 2: 129.93.153.206 0.652ms 3: 129.93.241.41 1.165ms 4: 129.93.6.29 5.067ms asymm 6 5: 10.161.1.14 6.678ms 6: fourhundredge-0-0-0-10.1410.core2.kans.net.internet2.edu 10.611ms 7: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 17.547ms asymm 9 8: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 19.903ms 9: 144.92.233.112 20.662ms 10: perfsonar02.hep.wisc.edu 27.919ms reached Resume: pmtu 8930 hops 10 back 8
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-s8py0n/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-s8py0n/execute/dir_9579 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=9579 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el8 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=SDSC-PRP OSG_SQUID_LOCATION=10.110.90.167:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1@SDSC-PRP-OSPool-Provisioner.osg-direct-69b05150-013c12-6s86k$ "; unset PROMPT_COMMAND PS1=slot1@SDSC-PRP-OSPool-Provisioner.osg-direct-69b05150-013c12-6s86k$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-s8py0n/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el8 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-s8py0n/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el8 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-s8py0n/execute/dir_9579 _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=10.110.90.167:3128
pass kernel
6.8.0-100-generic
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
6014.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 7498373120 1191337172 6307035948 16% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy 10.110.90.167:3128
SLU-Libra-CE1 2026-04-03 11:57 2
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
SLU-Libra-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: head01.cm.cluster 0.178ms 1: head01.cm.cluster 0.179ms 2: 10.108.121.254 17.048ms 3: 10.248.100.81 17.215ms 4: 165.134.248.113 1.190ms 5: 10.248.101.30 17.405ms 6: 10.248.100.20 25.156ms 7: 165.134.223.27 25.144ms 8: fourhundredge-0-0-0-10.23.core2.kans.net.internet2.edu 38.437ms 9: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 45.998ms asymm 11 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 45.195ms 11: 144.92.233.112 47.491ms 12: perfsonar02.hep.wisc.edu 44.626ms reached Resume: pmtu 1500 hops 12 back 13
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg01/glide_vf9Lp1/images/htc__almalinux__10.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__almalinux__10.sif APPTAINER_TMPDIR=/scratch/osg01/glide_vf9Lp1/execute/dir_3565415/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3565415 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/osg01/glide_vf9Lp1/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01/glide_vf9Lp1/cred.d/idtokens/credential_OSPool.main_OSG_US_SLU-Libra-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6113697 GLIDEIN_Entry_Name=OSG_US_SLU-Libra-CE1 GLIDEIN_Entry_Signature=18a6814839ab2c45c5229f09b00a2352571abe43 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=slu-libra-ce1.svc.osg-htc.org slu-libra-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_YwFlJ5 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=SLU-Libra-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=SLU-Libra GLIDEIN_Tmp_Dir=/scratch/osg01/glide_vf9Lp1/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/osg01/glide_vf9Lp1/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/osg01/glide_vf9Lp1/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=SLU-Libra-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3109547_352789340@comp02$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3109547_352789340@comp02$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg01/glide_vf9Lp1/images/htc__almalinux__10.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/scratch/osg01/glide_vf9Lp1/images/htc__almalinux__10.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__almalinux__10.sif SINGULARITY_PATH=/scratch/osg01/glide_vf9Lp1/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01/glide_vf9Lp1/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01/glide_vf9Lp1/execute/dir_3565415/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.38.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
145544.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 10.149.6.1:/data01/scratch/osg01/glide_vf9Lp1/execute/dir_3565415 173455855616 20841342976 152614512640 13% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
SU-ITS-CE2 2026-04-03 17:01 65
100.0%
0.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
SU-ITS-CE2 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.5.255.253 0.806ms 1: 10.5.255.253 0.921ms 2: 10.30.29.35 0.823ms 3: 10.30.29.46 1.213ms asymm 2 4: 128.230.61.33 1.667ms asymm 3 5: syr-9208-su.nysernet.net 1.054ms asymm 4 6: buf-9208-syr-9208.nysernet.net 4.974ms asymm 5 7: i2-clev-buf-9208.nysernet.net 10.689ms asymm 6 8: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 15.964ms asymm 9 9: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 17.083ms asymm 8 10: 144.92.233.112 18.221ms asymm 9 11: perfsonar02.hep.wisc.edu 17.992ms reached Resume: pmtu 1500 hops 11 back 11
pass env
APPTAINER_CACHEDIR=/srv BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=singularity version 3.6.0-rc.4 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/bin/singularity CONTAINERSW_VERSION=3.6.0-rc.4 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_18311/glide_yRg57N/cred.d/idtokens/credential_OSPool.main_Glow_US_Syracuse_condor-ce2_whole.idtoken GLIDEIN_CPUS=40 GLIDEIN_ClusterId=232988 GLIDEIN_Country=US GLIDEIN_ESTIMATED_CPUS=40 GLIDEIN_Entry_Name=Glow_US_Syracuse_condor-ce2_whole GLIDEIN_Entry_Signature=e48649b25b70a32b99319bf8fb822a2e013d526d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=its-condor-ce2.syr.edu its-condor-ce2.syr.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_nobody_gYqaqv GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=10.5.0.41:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=SU-ITS-CE2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=SU-ITS GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_18311/glide_yRg57N/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity version 3.6.0-rc.4 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/bin/singularity GWMS_CONTAINERSW_VERSION=3.6.0-rc.4 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.6.0-rc.4 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=its-condor-ce2.syr.edu OSG_SITE_NAME=SU-ITS-CE2 OSG_SQUID_LOCATION=10.5.0.41:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_30@glidein_18313_756147492@CRUSH-OSG-C7-10-5-198-182$ "; unset PROMPT_COMMAND PS1=slot1_30@glidein_18313_756147492@CRUSH-OSG-C7-10-5-198-182$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_CACHEDIR=/srv SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity version 3.6.0-rc.4 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_MODE=privileged SINGULARITY_NAME=rocky:9 SINGULARITY_PATH=/bin/singularity SINGULARITY_TMPDIR=/var/lib/condor/execute/dir_18311/glide_yRg57N/execute/dir_7687 SINGULARITY_VERSION=3.6.0-rc.4 SLOTS_LAYOUT=partitionable TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_18311/glide_yRg57N/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_18311/glide_yRg57N/execute/dir_7687 _CONDOR_SLOT=slot1_30 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
warning kernel
3.10.0-1160.66.1.el7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
167.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/centos-root 206542340 30764160 175778180 15% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy 10.5.0.41:3128
SU-ITS-CE3 2026-04-03 17:01 220
100.0%
0.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
SU-ITS-CE3 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.5.255.253 1.066ms 1: 10.5.255.253 1.937ms 2: 10.30.29.35 1.395ms 3: 10.30.29.44 1.225ms asymm 2 4: 128.230.61.33 1.805ms asymm 3 5: syr-9208-su.nysernet.net 0.830ms asymm 4 6: buf-9208-syr-9208.nysernet.net 4.553ms asymm 5 7: i2-clev-buf-9208.nysernet.net 9.709ms asymm 6 8: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 17.312ms asymm 9 9: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 16.382ms asymm 8 10: 144.92.233.112 17.763ms asymm 9 11: perfsonar02.hep.wisc.edu 17.968ms reached Resume: pmtu 1500 hops 11 back 11
pass env
APPTAINER_CACHEDIR=/srv BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=singularity version 3.6.0-rc.4 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/bin/singularity CONTAINERSW_VERSION=3.6.0-rc.4 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_16184/glide_qaKQ66/cred.d/idtokens/credential_OSPool.main_Glow_US_Syracuse_condor-ce3_whole.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=296831 GLIDEIN_Country=US GLIDEIN_ESTIMATED_CPUS=40 GLIDEIN_Entry_Name=Glow_US_Syracuse_condor-ce3_whole GLIDEIN_Entry_Signature=7d0e752cc05a682736f13770c957326626ca1730 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=its-condor-ce3.syr.edu its-condor-ce3.syr.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_nobody_OrN8lq GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=10.5.0.41:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=SU-ITS-CE3 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=1a97d6060d26940eba08ee679d09a38f5c43d5c2 GLIDEIN_Site=SU-ITS GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_16184/glide_qaKQ66/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity version 3.6.0-rc.4 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/bin/singularity GWMS_CONTAINERSW_VERSION=3.6.0-rc.4 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.6.0-rc.4 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=its-condor-ce3.syr.edu OSG_SITE_NAME=SU-ITS-CE3 OSG_SQUID_LOCATION=10.5.0.41:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_14@glidein_16186_331411828@CRUSH-OSG-C7-10-5-202-218$ "; unset PROMPT_COMMAND PS1=slot1_14@glidein_16186_331411828@CRUSH-OSG-C7-10-5-202-218$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_CACHEDIR=/srv SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity version 3.6.0-rc.4 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_MODE=privileged SINGULARITY_NAME=rocky:9 SINGULARITY_PATH=/bin/singularity SINGULARITY_TMPDIR=/var/lib/condor/execute/dir_16184/glide_qaKQ66/execute/dir_20511 SINGULARITY_VERSION=3.6.0-rc.4 SLOTS_LAYOUT=partitionable TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_16184/glide_qaKQ66/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_16184/glide_qaKQ66/execute/dir_20511 _CONDOR_SLOT=slot1_14 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
warning kernel
3.10.0-1160.66.1.el7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
177.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/centos-root 206542340 20802140 185740200 11% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy 10.5.0.41:3128
Swan-CE1 2026-04-03 11:16 2
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
Swan-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: c3838.pki.hcc.unl.edu 0.422ms 1: _gateway 0.207ms 1: c3838.pki.hcc.unl.edu 2.802ms 2: c3838.pki.hcc.unl.edu 0.248ms asymm 1 3: hcc-pki-mlxe.unl.edu 0.365ms asymm 2 4: 129.93.241.41 0.829ms asymm 3 5: 129.93.6.29 3.910ms asymm 4 6: 10.161.1.14 4.723ms asymm 5 7: fourhundredge-0-0-0-10.1410.core2.kans.net.internet2.edu 8.996ms asymm 6 8: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 19.464ms asymm 9 9: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 19.891ms asymm 8 10: 144.92.233.112 20.775ms asymm 9 11: perfsonar02.hep.wisc.edu 18.060ms reached Resume: pmtu 1500 hops 11 back 10
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_Co5dSy/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_Co5dSy/execute/dir_4192703/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=4192703 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T2_US_Nebraska GLIDEIN_CONDOR_TOKEN=/scratch/glide_Co5dSy/cred.d/idtokens/credential_OSPool.main_HCC_US_Omaha_swan.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6072122 GLIDEIN_Country=US GLIDEIN_DUNESite=US_Swan GLIDEIN_Entry_Name=HCC_US_Omaha_swan GLIDEIN_Entry_Signature=58bad4df49838655dac32dadddc0395b9c59a77b GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=swan-gw1.unl.edu swan-gw1.unl.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_GK351q GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Swan-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=469d5d2c20f293a978a86b5899e85ed0454773e1 GLIDEIN_Site=Swan GLIDEIN_Tmp_Dir=/scratch/glide_Co5dSy/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=swan-gw1.unl.edu OSG_SITE_NAME=Swan OSG_SQUID_LOCATION=red-squid.unl.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_4066148_349202040@c1424.swan.hcc.unl.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_4066148_349202040@c1424.swan.hcc.unl.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_Co5dSy/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el8 SINGULARITY_IMAGES_DICT=default:/scratch/glide_Co5dSy/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TERM=vt220 TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_Co5dSy/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_Co5dSy/execute/dir_4192703/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.89.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
3522.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md3 3748646536 55604252 3693042284 2% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy red-squid.unl.edu:3128
Swarthmore-Firebird-CE1 2026-04-03 17:05 14
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
Swarthmore-Firebird-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: firebird.cluster 0.291ms 1: firebird.cluster 0.275ms 2: fw-comsvc-colo.lafayette.edu 8.143ms 3: 139.147.1.113 8.110ms asymm 4 4: 172.31.0.33 1.194ms 5: 172.31.0.3 15.773ms 6: gw0-g0-3-ibgp.prd.lafayette.edu 15.979ms asymm 5 7: 164.153.140.160 19.024ms asymm 6 8: hundredge-0-0-0-28.78.core1.phil.net.internet2.edu 19.985ms asymm 7 9: fourhundredge-0-0-0-2.4079.core2.ashb.net.internet2.edu 38.112ms asymm 15 10: fourhundredge-0-0-0-1.4079.core2.clev.net.internet2.edu 37.751ms asymm 14 11: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 37.019ms asymm 13 12: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 38.246ms 13: 144.92.233.112 39.126ms 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg01/glide_4auJd5/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg01/glide_4auJd5/execute/dir_1693368/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1693368 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01/glide_4auJd5/cred.d/idtokens/credential_OSPool.main_OSG_US_Swarthmore-firebird-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=300533 GLIDEIN_Entry_Name=OSG_US_Swarthmore-firebird-CE1 GLIDEIN_Entry_Signature=60cd0fad4cbb7e3d5057067218ef778210b3a413 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=swarthmore-firebird-ce1.svc.osg-htc.org swarthmore-firebird-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_0xf9ei GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Swarthmore-Firebird-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=Swarthmore-Firebird GLIDEIN_Tmp_Dir=/scratch/osg01/glide_4auJd5/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Swarthmore-Firebird-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_1542979_513441328@node01.cluster$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_1542979_513441328@node01.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg01/glide_4auJd5/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el9 SINGULARITY_IMAGES_DICT=default:/scratch/osg01/glide_4auJd5/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01/glide_4auJd5/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01/glide_4auJd5/execute/dir_1693368/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.16.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
7024.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/disk0-scratch 7439005696 73384308 7365621388 1% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
TCNJ-ELSA-CE2 2026-04-03 17:01 43
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
TCNJ-ELSA-CE2 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: ohpc-sms 0.234ms 1: ohpc-sms 0.178ms 2: hpc-firewall-common.hpc.tcnj.edu 0.926ms 3: hpc-firewall.hpc.tcnj.edu 0.365ms 4: 159.91.51.251 0.565ms 5: 159.91.58.161 0.577ms 6: 159.91.58.18 0.909ms 7: 159.91.58.142 0.995ms 8: 130.156.252.77 3.872ms asymm 14 9: 130.156.35.126 5.822ms asymm 13 10: 130.156.35.161 4.012ms asymm 14 11: bundle-ether1.104.core1.phil.net.internet2.edu 4.285ms asymm 13 12: fourhundredge-0-0-0-2.4079.core2.ashb.net.internet2.edu 22.178ms asymm 20 13: fourhundredge-0-0-0-1.4079.core2.clev.net.internet2.edu 21.654ms asymm 21 14: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 21.217ms asymm 18 15: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 22.720ms asymm 19 16: 144.92.233.112 23.028ms asymm 18 17: perfsonar02.hep.wisc.edu 24.071ms reached Resume: pmtu 1500 hops 17 back 22
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/scratch/glide_aZQtXW/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/local/scratch/glide_aZQtXW/execute/dir_2581145/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2581145 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/scratch/glide_aZQtXW/cred.d/idtokens/credential_OSPool.main_OSG_US_TCNJ-ELSA-CE2.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=305626 GLIDEIN_Entry_Name=OSG_US_TCNJ-ELSA-CE2 GLIDEIN_Entry_Signature=6e70ca8d5c46a5a607c4a31036270797ca7a21a5 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=tcnj-elsa-ce2.svc.opensciencegrid.org tcnj-elsa-ce2.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_54ao2X GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=6 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=TCNJ-ELSA-CE2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=ELSA GLIDEIN_Tmp_Dir=/local/scratch/glide_aZQtXW/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=TCNJ-ELSA-CE2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2316880_387760500@node028.localdomain$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2316880_387760500@node028.localdomain$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/local/scratch/glide_aZQtXW/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el8 SINGULARITY_IMAGES_DICT=default:/local/scratch/glide_aZQtXW/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/scratch/glide_aZQtXW/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/scratch/glide_aZQtXW/execute/dir_2581145/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.89.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
136.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda6 191178636 38298580 143095892 22% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
TIGER-OSG-BACKFILL-PROD 2026-04-03 17:00 1
0.0%
0.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
TIGER-OSG-BACKFILL-PROD — probe output
pass mtu
pmtu=1480 1?: [LOCALHOST] 0.009ms pmtu 1480 1: tiger0038.chtc.wisc.edu 0.170ms 1: tiger0038.chtc.wisc.edu 0.052ms 2: 2607:f388:2200:c0::2 0.467ms 3: perfsonar02.hep.wisc.edu 0.262ms reached Resume: pmtu 1480 hops 3 back 3
pass env
APPTAINER_CACHEDIR=/pilot/osgvo-pilot-eClvnE/execute/dir_8112 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=8112 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main-container GLIDEIN_ResourceName=TIGER-OSG-BACKFILL-PROD GLIDEIN_Site=CHTC GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_DIR=/pilot/osgvo-pilot-eClvnE GWMS_SINGULARITY_STATUS=PREFERRED HAS_SINGULARITY=False HOME=/home/osg JULIA_NUM_THREADS=1 LC_CTYPE=C.UTF-8 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=TIGER-OSG-BACKFILL-PROD OSG_SQUID_LOCATION= OSG_WN_TMP=/pilot/osgvo-pilot-eClvnE/execute/dir_8112/.local-tmp PATH=/usr/local/bin:/usr/bin PWD=/pilot/osgvo-pilot-eClvnE/execute/dir_8112 PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SHLVL=0 SINGULARITY_CACHEDIR=/pilot/osgvo-pilot-eClvnE/execute/dir_8112 TEMP=/pilot/osgvo-pilot-eClvnE/execute/dir_8112 TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/pilot/osgvo-pilot-eClvnE/execute/dir_8112 TMPDIR=/pilot/osgvo-pilot-eClvnE/execute/dir_8112/.local-tmp TZ=UTC _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_3061=3068:1775234453:3490248809 _CONDOR_ANCESTOR_3068=8112:1775235653:2123493922 _CONDOR_ANCESTOR_8112=8477:1775235653:2193508440 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/pilot/osgvo-pilot-eClvnE/execute/dir_8112/.chirp.config _CONDOR_CREDS=/pilot/osgvo-pilot-eClvnE/execute/dir_8112/.condor_creds _CONDOR_JOB_AD=/pilot/osgvo-pilot-eClvnE/execute/dir_8112/.job.ad _CONDOR_JOB_IWD=/pilot/osgvo-pilot-eClvnE/execute/dir_8112 _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/pilot/osgvo-pilot-eClvnE/execute/dir_8112/.machine.ad _CONDOR_SCRATCH_DIR=/pilot/osgvo-pilot-eClvnE/execute/dir_8112 _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/pilot/osgvo-pilot-eClvnE/execute/dir_8112/.job_wrapper_failure
pass kernel
5.14.0-375.el9.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
183.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 976283900 783466176 192817724 81% /
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
TNTech-Warp1-CE1 2026-04-03 17:01 2
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
TNTech-Warp1-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.155ms 1: _gateway 0.196ms 2: 149.149.251.249 0.474ms 3: 149.149.248.1 5.670ms 4: i2-to-sox-re.sidco.sox.net 6.025ms 5: no reply 6: fourhundredge-0-0-0-22.4079.core2.cinc.net.internet2.edu 16.794ms asymm 10 7: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 17.043ms asymm 9 8: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 17.089ms 9: 144.92.233.112 18.632ms 10: perfsonar02.hep.wisc.edu 18.531ms reached Resume: pmtu 1500 hops 10 back 11
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/osg01/glide_22r1pA/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/osg01/glide_22r1pA/execute/dir_368137/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=368137 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/tmp/osg01/glide_22r1pA/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/osg01/glide_22r1pA/cred.d/idtokens/credential_OSPool.main_OSG_US_TNTech-Warp1-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6280915 GLIDEIN_Entry_Name=OSG_US_TNTech-Warp1-CE1 GLIDEIN_Entry_Signature=35a77ebc26205402bb0ea1b4c03ec2a924152fb2 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=tntech-warp1-ce1.svc.osg-htc.org tntech-warp1-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_obfZez GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=TNTech-Warp1-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=TNTech-Warp1 GLIDEIN_Tmp_Dir=/tmp/osg01/glide_22r1pA/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/tmp/osg01/glide_22r1pA/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/tmp/osg01/glide_22r1pA/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=TNTech-Warp1-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3993097_860849175@gpunode009.localdomain$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3993097_860849175@gpunode009.localdomain$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/osg01/glide_22r1pA/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/tmp/osg01/glide_22r1pA/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/tmp/osg01/glide_22r1pA/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/osg01/glide_22r1pA/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/osg01/glide_22r1pA/execute/dir_368137/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-513.24.1.el8_9.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
3332.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 3689451536 7925588 3494037388 1% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UA-LR-ITS-EP 2026-04-03 17:01 30
100.0%
100.0%
100.0%
90.0%
mtu env kernel nrp_images df http_proxy
UA-LR-ITS-EP — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.083ms 1: _gateway 0.114ms 2: 144.167.99.3 0.188ms 3: FH213-Firewall-Campus.net.ualr.edu 7.990ms 4: rtr-fh213-intf1-0.mgt.ualr.edu 1.165ms 5: 100.64.0.5 8.312ms asymm 4 6: 208.90.110.230 8.665ms asymm 5 7: no reply 8: no reply 9: no reply 10: 208.90.110.99 20.432ms asymm 11 11: 208.90.110.99 19.819ms 12: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 46.405ms asymm 17 13: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 35.817ms asymm 17 14: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 34.814ms asymm 15 15: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 35.886ms 16: 144.92.233.112 36.922ms 17: perfsonar02.hep.wisc.edu 36.688ms reached Resume: pmtu 1500 hops 17 back 18
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-a8trMz/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-a8trMz/execute/dir_422061 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=422061 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=UA Little Rock ITS OSG_SQUID_LOCATION=144.167.99.254:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_5@UA-LR-ITS-EP.fdd52a9019a8$ "; unset PROMPT_COMMAND PS1=slot1_5@UA-LR-ITS-EP.fdd52a9019a8$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-a8trMz/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-a8trMz/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-a8trMz/execute/dir_422061 _CONDOR_SLOT=slot1_5 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=144.167.99.254:3128
pass kernel
4.18.0-553.6.1.el8.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
2667.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 2901188512 103650328 2797538184 4% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy 144.167.99.254:3128
UAH-Voyager-CE1 2026-04-03 17:09 32
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
UAH-Voyager-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: stor001 0.463ms 1: stor001 0.285ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/data001/OSG/glide_SlZHOZ/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/data001/OSG/glide_SlZHOZ/execute/dir_3469239/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3469239 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/data001/OSG/glide_SlZHOZ/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/data001/OSG/glide_SlZHOZ/cred.d/idtokens/credential_OSPool.main_OSG_US_UAH-Voyager-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=5611116 GLIDEIN_Entry_Name=OSG_US_UAH-Voyager-CE1 GLIDEIN_Entry_Signature=b95da1e759e3c7d0c4e2b2196660477e714d9241 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=uah-voyager-ce1.svc.osg-htc.org uah-voyager-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_1bgQPW GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UAH-Voyager-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=UAH-Voyager GLIDEIN_Tmp_Dir=/data001/OSG/glide_SlZHOZ/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/data001/OSG/glide_SlZHOZ/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/data001/OSG/glide_SlZHOZ/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=UAH-Voyager-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_1887139_503015490@node019$ "; unset PROMPT_COMMAND PS1=slot1@glidein_1887139_503015490@node019$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/data001/OSG/glide_SlZHOZ/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/data001/OSG/glide_SlZHOZ/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/data001/OSG/glide_SlZHOZ/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/data001/OSG/glide_SlZHOZ/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/data001/OSG/glide_SlZHOZ/execute/dir_3469239/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-503.40.1.el9_5.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
379465.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 172.17.170.1:/zpool/raid001/data001/OSG/glide_SlZHOZ/execute/dir_3469239 397984927744 86392832 397898534912 1% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UCMerced-Pinnacles-CE1 2026-04-03 17:07 64
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
UCMerced-Pinnacles-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: pinnacles.cluster 0.723ms 1: pinnacles.cluster 0.641ms 2: 169.236.74.3 1.076ms 3: 10.7.1.57 1.132ms 4: pa-core-0-1.ucmerced.edu 1.439ms 5: edge-1-3.ucmerced.edu 4.522ms 6: frg-agg4--ucmerced--2nd-10g.cenic.net 2.640ms 7: sacr2-agg-01--ferg1-agg-01--300g--01.cenic.net 15.553ms asymm 12 8: rive1-agg-01--sacr2-agg-01--800g--01.cenic.net 15.185ms asymm 11 9: dc-lax-agg10--rive1-agg-01-800g--01.cenic.net 15.310ms 10: lax-agg10-hpr--lax-agg10-dc.cenic.net 22.206ms asymm 14 11: hpr-i2--lax-agg10.cenic.net 25.615ms asymm 15 12: fourhundredge-0-0-0-2.4079.core2.salt.net.internet2.edu 70.678ms asymm 24 13: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 70.018ms asymm 23 14: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 69.449ms asymm 22 15: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 70.072ms asymm 21 16: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 69.079ms asymm 20 17: 144.92.233.112 71.203ms asymm 21 18: perfsonar02.hep.wisc.edu 71.336ms reached Resume: pmtu 1500 hops 18 back 23
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/localscratch/osg01/glide_oTTCeT/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/localscratch/osg01/glide_oTTCeT/execute/dir_3007880/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3007880 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/localscratch/osg01/glide_oTTCeT/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/localscratch/osg01/glide_oTTCeT/cred.d/idtokens/credential_OSPool.main_OSG_US_UCMerced-Pinnacles-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=6273452 GLIDEIN_Entry_Name=OSG_US_UCMerced-Pinnacles-CE1 GLIDEIN_Entry_Signature=db401f4da317eecc8cfa45c7d67de705767a1984 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=ucmerced-pinnacles-ce1.svc.opensciencegrid.org ucmerced-pinnacles-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_kXhfEa GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UCMerced-Pinnacles-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=5552c705dd9f3b999102cc42d5c5bdce18bba000 GLIDEIN_Site=UCMerced-Pinnacles GLIDEIN_Tmp_Dir=/localscratch/osg01/glide_oTTCeT/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/localscratch/osg01/glide_oTTCeT/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/localscratch/osg01/glide_oTTCeT/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=UCMerced-Pinnacles-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_2475098_129282736@node079.cluster$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_2475098_129282736@node079.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/localscratch/osg01/glide_oTTCeT/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/localscratch/osg01/glide_oTTCeT/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/localscratch/osg01/glide_oTTCeT/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/localscratch/osg01/glide_oTTCeT/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/localscratch/osg01/glide_oTTCeT/execute/dir_3007880/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.5.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
841.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme1n1p1 937233280 54944648 882288632 6% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UCR-HPCC-CE1 2026-04-03 11:22 2
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
UCR-HPCC-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: nat-clust.clust.hpcc.ucr.edu 0.161ms 1: nat-clust.clust.hpcc.ucr.edu 0.105ms 2: d-genom-1120d-1-v31.ucr.edu 0.688ms 3: 172.23.254.250 0.667ms 4: 172.30.254.9 0.549ms 5: 172.30.254.8 0.446ms 6: b-som-bn-1-v2020.ucr.edu 1.495ms asymm 7 7: 137.164.26.128 12.766ms asymm 13 8: hpr-i2--lax-agg10.cenic.net 14.484ms asymm 14 9: fourhundredge-0-0-0-2.4079.core2.salt.net.internet2.edu 58.708ms asymm 23 10: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 61.159ms asymm 22 11: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 60.346ms asymm 21 12: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 58.338ms asymm 20 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 59.800ms asymm 19 14: 144.92.233.112 61.319ms asymm 20 15: perfsonar02.hep.wisc.edu 62.491ms reached Resume: pmtu 1500 hops 15 back 22
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_G07EVT/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/scratch/glide_G07EVT/execute/dir_3064560/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3064560 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/glide_G07EVT/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_G07EVT/cred.d/idtokens/credential_OSPool.main_OSG_US_UCR-HPCC-CE1.idtoken GLIDEIN_CPUS=10 GLIDEIN_ClusterId=6281832 GLIDEIN_Entry_Name=OSG_US_UCR-HPCC-CE1 GLIDEIN_Entry_Signature=677ddf66568064162612351aac74e2cf68cfa68d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=ucr-hpcc-ce1.svc.osg-htc.org ucr-hpcc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_y57CpB GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UCR-HPCC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=UCR-HPCC GLIDEIN_Tmp_Dir=/scratch/glide_G07EVT/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/glide_G07EVT/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/glide_G07EVT/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=UCR-HPCC-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_3018741_276476895@c17.ib.hpcc.ucr.edu$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_3018741_276476895@c17.ib.hpcc.ucr.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_G07EVT/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/scratch/glide_G07EVT/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/scratch/glide_G07EVT/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_G07EVT/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_G07EVT/execute/dir_3064560/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.104.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
439.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda6 465561308 4421076 461140232 1% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UCSDT2 2026-04-03 16:01 15
100.0%
100.0%
100.0%
100.0%
all
UCSDT2 — probe output
warning all
Job failed with exit code 0; no results transferred
UChicago_Pile_Backfill 2026-04-03 17:03 41
100.0%
100.0%
85.4%
100.0%
mtu env kernel nrp_images df http_proxy
UChicago_Pile_Backfill — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 9000 1: 192.170.197.2 0.340ms 1: 192.170.197.2 0.381ms 2: h01-389-300-to-d16-10-200.vroz.p2p.uchicago.net 0.284ms 3: borderfw-to-h01-389-300.vroz.p2p.uchicago.net 0.380ms 4: borderfw-to-b65-ll129-300.vroz.p2p.uchicago.net 0.338ms pmtu 1500 4: 10.5.10.253 0.498ms 5: borderfw-to-b65-ll129-500.p2p.uchicago.net 1.176ms 6: r-chicago-600w-ae1-2295.uwsys.net 7.686ms 7: r-chicago-710-ae0-3506.uwsys.net 6.778ms asymm 11 8: r-uwmilwaukee-ems-ae1-3473.uwsys.net 4.714ms asymm 7 9: r-uwmadison-cssc-ae4-3455.uwsys.net 6.735ms asymm 8 10: 143.235.41.129 6.419ms asymm 9 11: perfsonar02.hep.wisc.edu 6.450ms reached Resume: pmtu 1500 hops 11 back 11
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-DdYsq7/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-DdYsq7/execute/dir_1954246 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1954246 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=UChicago_Pile OSG_SQUID_LOCATION=head05.pile.uchicago.edu:30244 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_2@UChicago-Pile-Backfill.c040.pile.uchicago.edu$ "; unset PROMPT_COMMAND PS1=slot1_2@UChicago-Pile-Backfill.c040.pile.uchicago.edu$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-DdYsq7/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-DdYsq7/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-DdYsq7/execute/dir_1954246 _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=head05.pile.uchicago.edu:30244
pass kernel
5.14.0-611.26.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1601.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 1842548380 69679804 1679198484 4% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy head05.pile.uchicago.edu:30244
UConn-OSG 2026-04-03 17:01 161
100.0%
93.8%
97.5%
93.8%
mtu env kernel nrp_images df http_proxy
UConn-OSG — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.489ms 1: _gateway 0.398ms 2: 137.99.254.89 0.389ms 3: mx-hbl-core-po38.net.uconn.edu 1.400ms 4: vss-core-border-inside-po52-640.net.uconn.edu 0.952ms 5: fw-msb-i-lag1.666.net.uconn.edu 8.210ms 6: vrf-border-outside-po21-664.net.uconn.edu 9.235ms 7: mx-hbl-border-outside-ae51-628.net.uconn.edu 8.680ms 8: enrt064hhh-mx-ae6-634.net.cen.ct.gov 9.584ms 9: 10.172.96.168 9.753ms 10: bundle-ether2.101.core1.hart2.net.internet2.edu 10.830ms 11: fourhundredge-0-0-0-0.4079.core1.alba.net.internet2.edu 31.642ms asymm 18 12: fourhundredge-0-0-0-2.4079.core2.clev.net.internet2.edu 33.290ms asymm 17 13: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 33.913ms asymm 16 14: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 33.380ms asymm 15 15: 144.92.233.112 34.653ms asymm 16 16: perfsonar02.hep.wisc.edu 34.751ms reached Resume: pmtu 1500 hops 16 back 18
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/execute/dir_3193013/glide_UYmo3b/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/local/execute/dir_3193013/glide_UYmo3b/execute/dir_3014387/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3014387 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/execute/dir_3193013/glide_UYmo3b/cred.d/idtokens/credential_OSPool.main_OSG_US_UConn_gluskap_op.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=5603457 GLIDEIN_Country=US GLIDEIN_DUNESite=US_UConn GLIDEIN_Entry_Name=OSG_US_UConn_gluskap_op GLIDEIN_Entry_Signature=04539a0120e34039432e245175e16a12041e541c GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=gluskap.phys.uconn.edu gluskap.phys.uconn.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_prod05_Is2RdO GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=9 GLIDEIN_Proxy_URL=gryphn.phys.uconn.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UConn-OSG GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=5552c705dd9f3b999102cc42d5c5bdce18bba000 GLIDEIN_Site=UConn GLIDEIN_Tmp_Dir=/local/execute/dir_3193013/glide_UYmo3b/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=gluskap.phys.uconn.edu OSG_SITE_NAME=UConn-OSG_CE OSG_SQUID_LOCATION=gryphn.phys.uconn.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_3193037_396234202@nod31.phys.uconn.edu$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_3193037_396234202@nod31.phys.uconn.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/local/execute/dir_3193013/glide_UYmo3b/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/local/execute/dir_3193013/glide_UYmo3b/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/execute/dir_3193013/glide_UYmo3b/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/execute/dir_3193013/glide_UYmo3b/execute/dir_3014387/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.89.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
873.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 937231232 21018860 916212372 3% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy gryphn.phys.uconn.edu:3128
UIUC-TGI-RAILS-CE1 2026-04-03 17:03 20
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
UIUC-TGI-RAILS-CE1 — probe output
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: tgi-rails-pub-gw.gw.ncsa.edu 0.454ms 1: tgi-rails-pub-gw.gw.ncsa.edu 0.329ms 2: npcf-core-west-tgi-rails-data.gw.ncsa.edu 0.275ms 3: npcf-exit-east-npcf-core-west.gw.ncsa.edu 12.793ms 4: 198.17.196.86 0.531ms 5: 72.36.80.70 3.399ms 6: r-uwmilwaukee-ems-ae1-3473.uwsys.net 7.966ms 7: r-uwmadison-cssc-ae4-3455.uwsys.net 11.631ms 8: 143.235.41.129 8.150ms 9: perfsonar02.hep.wisc.edu 7.989ms reached Resume: pmtu 9000 hops 9 back 10
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/projects/bcxl/svcosguser/glide_0aeCfx/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/projects/bcxl/svcosguser/glide_0aeCfx/execute/dir_3599321/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3599321 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/projects/bcxl/svcosguser/glide_0aeCfx/cred.d/idtokens/credential_OSPool.main_OSG_US_UIUC-TGI-RAILS-CE1_cpu-small.idtoken GLIDEIN_CPUS=24 GLIDEIN_ClusterId=6273198 GLIDEIN_Entry_Name=OSG_US_UIUC-TGI-RAILS-CE1_cpu-small GLIDEIN_Entry_Signature=4aea1d74f564770e75115d275b593e4d110d6058 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=uiuc-tgi-rails-ce1.svc.osg-htc.org uiuc-tgi-rails-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_svcosguser_RHlxGI GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UIUC-TGI-RAILS-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=5552c705dd9f3b999102cc42d5c5bdce18bba000 GLIDEIN_Site=UIUC-TGI-RAILS GLIDEIN_Tmp_Dir=/projects/bcxl/svcosguser/glide_0aeCfx/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=UIUC-TGI-RAILS-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_7@glidein_2548544_157924092@rails01$ "; unset PROMPT_COMMAND PS1=slot1_7@glidein_2548544_157924092@rails01$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/projects/bcxl/svcosguser/glide_0aeCfx/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el8 SINGULARITY_IMAGES_DICT=default:/projects/bcxl/svcosguser/glide_0aeCfx/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/projects/bcxl/svcosguser/glide_0aeCfx/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/projects/bcxl/svcosguser/glide_0aeCfx/execute/dir_3599321/scratch _CONDOR_SLOT=slot1_7 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.105.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
76168.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on pool1.railsvast.internal.ncsa.edu:/projects/bcxl/svcosguser/glide_0aeCfx/execute/dir_3599321 80530274304 662206464 79868067840 1% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UND-Talon-CE1 2026-04-03 17:01 33
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
UND-Talon-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.128.128.1 8.138ms 1: 10.128.128.1 7.700ms 2: 172.29.10.26 8.088ms 3: 172.29.10.17 8.452ms asymm 4 4: w.av25.3/22/1.ndus.edu 0.836ms asymm 5 5: 165.234.8.181 15.664ms 6: 134.129.0.37 16.266ms 7: 165.234.21.1 26.825ms asymm 8 8: telecomb-gr-01-hu0-5-0-3.northernlights.gigapop.net 34.402ms asymm 9 9: r-minneapolis-511-ae0-110.uwsys.net 34.267ms asymm 11 10: r-uweauclaire-centennial-ae0-3470.uwsys.net 35.047ms asymm 17 11: r-uwmadison-animal-ae3-3507.uwsys.net 38.398ms asymm 13 12: r-uwmadison-cssc-ae3-3439.uwsys.net 35.076ms asymm 14 13: 143.235.41.129 39.773ms asymm 15 14: perfsonar02.hep.wisc.edu 37.805ms reached Resume: pmtu 1500 hops 14 back 15
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_D83E3u/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_D83E3u/execute/dir_3565841/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3565841 CONTAINERSW_FULL_VERSION=apptainer version 1.1.9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/cm/local/apps/apptainer/current/bin/apptainer CONTAINERSW_VERSION=1.1.9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_D83E3u/cred.d/idtokens/credential_OSPool.main_OSG_US_UND-Talon-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=5681872 GLIDEIN_Entry_Name=OSG_US_UND-Talon-CE1 GLIDEIN_Entry_Signature=de44112ad85d0e546011a1cc9a0d7c61ab2da3a3 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=und-talon-ce1.svc.opensciencegrid.org und-talon-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_oZumz2 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UND-Talon-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=UND-Talon GLIDEIN_Tmp_Dir=/tmp/glide_D83E3u/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.1.9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/cm/local/apps/apptainer/current/bin/apptainer GWMS_CONTAINERSW_VERSION=1.1.9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/cm/local/apps/apptainer/current/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.1.9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=UND-Talon-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3510335_75776218@talon05.cm.cluster$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3510335_75776218@talon05.cm.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_D83E3u/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.1.9 SINGULARITY_IMAGES_DICT=default:/tmp/glide_D83E3u/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/cm/local/apps/apptainer/current/bin/apptainer SINGULARITY_VERSION=1.1.9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_D83E3u/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_D83E3u/execute/dir_3565841/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-427.42.1.el9_4.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
782.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 920325060 99858256 820466804 11% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UNL-PATH-EP 2026-04-03 17:01 24
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
UNL-PATH-EP — probe output
pass mtu
pmtu=1480 1?: [LOCALHOST] pmtu 1480 1: 129-93-179-141.kubelet.kube-system.svc.cluster.local 0.122ms 1: 129-93-179-141.node-exporter.monitoring.svc.cluster.local 0.050ms 2: 129.93.179.190 0.267ms 3: hcc-shor-m1436-vl48.unl.edu 0.710ms 4: no reply 5: chic-cr6--uwmadison_se-203.es.net 15.936ms 6: uwmadison_se-203--chic-cr6.es.net 33.276ms 7: perfsonar02.hep.wisc.edu 33.292ms reached Resume: pmtu 1480 hops 7 back 7
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-RYY5sc/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-RYY5sc/execute/dir_2784419 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2784419 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=UNL-PATH PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_42@UNL-PATH-EP.osgvo-docker-pilot-ospool-bdc6cf4c6-7mspr$ "; unset PROMPT_COMMAND PS1=slot1_42@UNL-PATH-EP.osgvo-docker-pilot-ospool-bdc6cf4c6-7mspr$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-RYY5sc/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-RYY5sc/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-RYY5sc/execute/dir_2784419 _CONDOR_SLOT=slot1_42 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass kernel
4.18.0-489.el8.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
391.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/openebs-pvc--8aa8be44--fc7f--43a9--acb0--d0655dda3d9b 442688776 32697988 409974404 8% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UNR-CC-CE1 2026-04-03 17:01 12
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
UNR-CC-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: cc-head.10g 0.122ms 1: cc-head.10g 0.110ms 2: 134.197.34.254 0.761ms 3: 10.251.1.250 0.916ms 4: 134.197.0.9 0.634ms 5: scs-unr-link1.nevada.net 1.550ms 6: i2re-slc-nshe.nevada.net 3.826ms 7: fourhundredge-0-0-0-1.4079.core1.salt.net.internet2.edu 45.184ms asymm 17 8: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 45.632ms asymm 16 9: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 44.891ms asymm 15 10: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 45.121ms asymm 14 11: fourhundredge-0-0-0-22.4079.core1.chic.net.internet2.edu 44.311ms asymm 13 12: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 44.439ms 13: 144.92.233.112 46.914ms 14: perfsonar02.hep.wisc.edu 46.900ms reached Resume: pmtu 1500 hops 14 back 15
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg01/glide_jW434d/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg01/glide_jW434d/execute/dir_2419855/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2419855 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/osg01/glide_jW434d/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01/glide_jW434d/cred.d/idtokens/credential_OSPool.main_OSG_US_UNR-CC-CE1.idtoken GLIDEIN_CPUS=32 GLIDEIN_ClusterId=5603016 GLIDEIN_Entry_Name=OSG_US_UNR-CC-CE1 GLIDEIN_Entry_Signature=e8214b7958407d30bc4b738d8ba23a50382ca286 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=unr-cc-ce1.svc.osg-htc.org unr-cc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_qepzMN GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=cc.engr.unr.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UNR-CC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=5552c705dd9f3b999102cc42d5c5bdce18bba000 GLIDEIN_Site=UNR-CC GLIDEIN_Tmp_Dir=/scratch/osg01/glide_jW434d/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/osg01/glide_jW434d/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/osg01/glide_jW434d/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=UNR-CC-CE1 OSG_SQUID_LOCATION=cc.engr.unr.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_27@glidein_3307025_46803420@nvbaisec2.ib$ "; unset PROMPT_COMMAND PS1=slot1_27@glidein_3307025_46803420@nvbaisec2.ib$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg01/glide_jW434d/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/scratch/osg01/glide_jW434d/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/scratch/osg01/glide_jW434d/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01/glide_jW434d/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01/glide_jW434d/execute/dir_2419855/scratch _CONDOR_SLOT=slot1_27 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-503.16.1.el9_5.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1232.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md0 1401788772 37742440 1292765592 3% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy cc.engr.unr.edu:3128
USC-CARC-Artemis-Backfill 2026-04-03 17:01 7
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
USC-CARC-Artemis-Backfill — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: host.containers.internal 0.060ms 1: host.containers.internal 0.039ms 2: artemis-osg 0.226ms 3: 68.181.11.126 0.378ms 4: 68.181.25.18 0.406ms 5: 10.250.9.113 8.169ms 6: 207.151.53.252 8.658ms 7: 207.151.53.144 8.417ms asymm 6 8: hpr--lax-agg10--csuf--10g--01.cenic.net 8.523ms asymm 7 9: hpr-i2--lax-agg10.cenic.net 12.238ms asymm 8 10: fourhundredge-0-0-0-2.4079.core2.salt.net.internet2.edu 55.036ms asymm 17 11: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 55.731ms asymm 16 12: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 54.713ms asymm 15 13: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 55.380ms asymm 14 14: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 55.973ms asymm 13 15: 144.92.233.112 57.451ms asymm 14 16: perfsonar02.hep.wisc.edu 57.516ms reached Resume: pmtu 1500 hops 16 back 16
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-P0hcbO/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-P0hcbO/execute/dir_266001 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=266001 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Center for Advanced Research Computing OSG_SQUID_LOCATION=68.181.11.8:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_12@USC-CARC-Artemis-Backfill.00ecb9142e1f$ "; unset PROMPT_COMMAND PS1=slot1_12@USC-CARC-Artemis-Backfill.00ecb9142e1f$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-P0hcbO/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-P0hcbO/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-P0hcbO/execute/dir_266001 _CONDOR_SLOT=slot1_12 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=68.181.11.8:3128
pass kernel
4.18.0-372.16.1.el8_6.0.1.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
41.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on tmpfs 65980400 22037588 43942812 34% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy 68.181.11.8:3128
USCMS-FNAL-WC1 2026-04-03 11:07 1
0.0%
0.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
USCMS-FNAL-WC1 — probe output
pass mtu
pmtu=9000 1?: [LOCALHOST] 0.009ms pmtu 9000 1: r-cms-fcc2-2-vlan191.fnal.gov 1.330ms 1: r-cms-fcc2-2-vlan191.fnal.gov 1.326ms 2: r-border-fcc2-1-2513-1.fnal.gov 0.313ms 3: no reply 4: fnalgcc-cr6--chic-bb-e.igp.es.net 2.124ms asymm 6 5: no reply 6: uwmadison_se-203--chic-cr6.es.net 19.386ms 7: perfsonar02.hep.wisc.edu 19.292ms reached Resume: pmtu 9000 hops 7 back 7
pass env
APPTAINER_CACHEDIR=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch BATCH_SYSTEM=HTCondor CONDORCE_COLLECTOR_HOST=cmsosgce-opp1.fnal.gov:9619 CONDOR_CONFIG=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/condor_config CONDOR_PARENT_ID=osg-55231-0-cmswn5016:24172:1775214461 CONDOR_PROCD_ADDRESS=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/log/procd_address CONDOR_STARTER_PID=24172 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 FERMIHTC_APPTAINER_ARGS=--pid --ipc --contain FERMIHTC_APPTAINER_BIN=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer FERMIHTC_APPTAINER_BIND=/cvmfs /etc/hosts /etc/grid-security FERMIHTC_APPTAINER_CMD=exec FERMIHTC_APPTAINER_PWD=/srv GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/cred.d/idtokens/credential_OSPool.main_CMSHTPC_T1_US_FNAL_condce_opp1_whole.idtoken GLIDEIN_CPUS=128 GLIDEIN_ClusterId=287239 GLIDEIN_Country=US GLIDEIN_DUNESite=US_FNAL-T1 GLIDEIN_ESTIMATED_CPUS=64 GLIDEIN_Entry_Name=CMSHTPC_T1_US_FNAL_condce_opp1_whole GLIDEIN_Entry_Signature=8d3912dc0ad8c7729b5b223c819d1630ad153469 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=cmsosgce-opp1.fnal.gov cmsosgce-opp1.fnal.gov:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_q1vIM2 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_ProcId=5 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=USCMS-FNAL-WC1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=1a97d6060d26940eba08ee679d09a38f5c43d5c2 GLIDEIN_Site=FNAL GLIDEIN_Tmp_Dir=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/tmp GLIDEIN_UUID=02a6f9af-2e6d-4187-adb4-6138b056cac4 GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_PATH=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/.gwms.d/bin: GWMS_SINGULARITY_STATUS=PREFERRED HAS_SINGULARITY=False HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch HOSTNAME=osg-55231-0-cmswn5016.fnal.gov JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_CMSHTPC_T1_US_FNAL_condce_opp1_whole/tokens.tgz JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OLDPWD=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/logs/02a6f9af-2e6d-4187-adb4-6138b056cac4 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=/cvmfs/oasis.opensciencegrid.org OSG_DATA=UNAVAILABLE OSG_GRID=/etc/osg/wn-client/ OSG_HOSTNAME=cmsosgce-opp1.fnal.gov OSG_SITE_NAME=USCMS-FNAL-WC1-OPP1 OSG_SITE_READ= OSG_SITE_WRITE= OSG_SQUID_LOCATION=squid.fnal.gov:3128 OSG_WN_TMP=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch/.local-tmp PATH=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/.gwms.d/bin::/bin:/usr/bin:/sbin:/usr/sbin PREEMPT_GRACE_TIME=10000000 PWD=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SHLVL=0 SINGULARITY_CACHEDIR=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch SLOTS_LAYOUT=partitionable TEMP=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch TMPDIR=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_20723=20726:1775213984:973416413 _CONDOR_ANCESTOR_20726=24172:1775214461:3437546755 _CONDOR_ANCESTOR_24172=24703:1775214468:2496961533 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/main/condor/bin _CONDOR_CHIRP_CONFIG=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch/.chirp.config _CONDOR_CREDS=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch/.condor_creds _CONDOR_EXECUTE=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute _CONDOR_JOB_AD=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch/.job.ad _CONDOR_JOB_IWD=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/storage/local/data1/condor/execute/dir_3151236/glide_OHKSHj/execute/dir_24172/scratch/.job_wrapper_failure _JAVA_OPTIONS=-Xmx1024m
pass kernel
5.14.0-611.42.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
7027.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 7499377664 130338040 7369039624 2% /storage/local/data1/condor/execute/dir_3151236
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy squid.fnal.gov:3128
USD-Lawrence-CE1 2026-04-03 17:01 50
100.0%
0.0%
0.0%
0.0%
mtu env kernel nrp_images df http_proxy
USD-Lawrence-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: login01.lawrence 0.317ms 1: login01.lawrence 0.268ms 2: 206.209.0.250 0.551ms 3: 206.176.0.181 2.262ms 4: 206.71.32.5 0.547ms 5: 206.71.32.241 1.937ms 6: 206.71.32.162 12.534ms 7: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 22.448ms asymm 9 8: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 23.901ms 9: 144.92.233.112 24.649ms 10: perfsonar02.hep.wisc.edu 23.405ms reached Resume: pmtu 1500 hops 10 back 13
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/job_1172834/glide_88PbVe/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/job_1172834/glide_88PbVe/execute/dir_3522450/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3522450 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/job_1172834/glide_88PbVe/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/job_1172834/glide_88PbVe/cred.d/idtokens/credential_OSPool.main_OSG_US_USD-Lawrence-CE1.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=348762 GLIDEIN_Entry_Name=OSG_US_USD-Lawrence-CE1 GLIDEIN_Entry_Signature=1d05ebcfc013c1da425fd8319e44af08dea6061e GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=usd-lawrence-ce1.svc.opensciencegrid.org usd-lawrence-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01@usd.local_EP7qOy GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=USD-Lawrence-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=19a3aab0fae658d7866b25a2d8ffcbedc8d5e443 GLIDEIN_Site=USD-Lawrence GLIDEIN_Tmp_Dir=/scratch/job_1172834/glide_88PbVe/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/job_1172834/glide_88PbVe/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/job_1172834/glide_88PbVe/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=USD-Lawrence-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_2644700_436684976@node012.lawrence$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_2644700_436684976@node012.lawrence$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/job_1172834/glide_88PbVe/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/scratch/job_1172834/glide_88PbVe/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/scratch/job_1172834/glide_88PbVe/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/job_1172834/glide_88PbVe/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/job_1172834/glide_88PbVe/execute/dir_3522450/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-372.26.1.el8_6.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
132.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda6 207528956 68757172 138771784 34% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UTC-Epyc 2026-04-03 17:05 4
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
UTC-Epyc — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.687ms 1: _gateway 0.572ms 2: 10.249.200.157 8.095ms 3: 192.30.208.30 8.428ms 4: no reply 5: no reply 6: peer-as26827.pr01.iad3.tfbnw.net 27.765ms asymm 5 7: no reply 8: telia.iad02.atlas.cogentco.com 27.928ms asymm 12 9: rest-bb1-link.ip.twelve99.net 24.017ms asymm 12 10: nyk-bb5-link.ip.twelve99.net 30.417ms asymm 13 11: chi-bb1-link.ip.twelve99.net 42.180ms asymm 14 12: chi-b24-link.ip.twelve99.net 43.127ms asymm 13 13: wiscnet-ic-308898.ip.twelve99-cust.net 64.772ms 14: r-uwmilwaukee-ems-ae1-3473.uwsys.net 49.830ms 15: r-uwmadison-cssc-ae4-3455.uwsys.net 46.083ms asymm 14 16: 143.235.41.129 47.354ms asymm 15 17: perfsonar02.hep.wisc.edu 47.224ms reached Resume: pmtu 1500 hops 17 back 17
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/mnt/internal/osg01/glide_gdcZOT/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/mnt/internal/osg01/glide_gdcZOT/execute/dir_3111892/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3111892 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/mnt/internal/osg01/glide_gdcZOT/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=2c5a85deb24f1f4623e86c3eea56d0538d0b4afa GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/mnt/internal/osg01/glide_gdcZOT/cred.d/idtokens/credential_OSPool.gpu_OSG_US_UTC-Epyc.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=5678375 GLIDEIN_Entry_Name=OSG_US_UTC-Epyc GLIDEIN_Entry_Signature=b58e2ade9ac2f5f8e9f721694ea3fcf726eb0cfc GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=utc-epyc-ce1.svc.opensciencegrid.org utc-epyc-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_vgfMFs GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UTC-Epyc GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=68376560144ff7f968c1f9a47b56d2289fce3db9 GLIDEIN_Site=UTC-Epyc GLIDEIN_Tmp_Dir=/mnt/internal/osg01/glide_gdcZOT/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/mnt/internal/osg01/glide_gdcZOT/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/mnt/internal/osg01/glide_gdcZOT/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=UTC-Epyc PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_10@glidein_267511_574553378@epyc05$ "; unset PROMPT_COMMAND PS1=slot1_10@glidein_267511_574553378@epyc05$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/mnt/internal/osg01/glide_gdcZOT/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/mnt/internal/osg01/glide_gdcZOT/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SINGULARITY_PATH=/mnt/internal/osg01/glide_gdcZOT/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/mnt/internal/osg01/glide_gdcZOT/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/mnt/internal/osg01/glide_gdcZOT/execute/dir_3111892/scratch _CONDOR_SLOT=slot1_10 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.26.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
824.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p1 976284608 112246900 864037708 12% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UW-IT-CE1 2026-04-03 17:01 51
100.0%
100.0%
35.3%
100.0%
mtu env kernel nrp_images df http_proxy
UW-IT-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: klone-dip1.hyak.local 0.291ms 1: klone-dip1.hyak.local 0.283ms 2: ae3--20.uwar-uwtc-1.infra.washington.edu 0.492ms 3: lo0--20.uwcr-ads-2.infra.washington.edu 0.853ms 4: ae4--308.uwcr-ads-2.infra.washington.edu 0.869ms 5: no reply 6: ae16--500.icar-sttl2-2.infra.pnw-gigapop.net 0.921ms 7: lo0--210.icar-sttl1-1.infra.pnw-gigapop.net 0.982ms 8: 209.124.181.167 2.416ms 9: fourhundredge-0-0-0-18.4079.core1.seat.net.internet2.edu 51.821ms asymm 19 10: fourhundredge-0-0-0-0.4079.core1.salt.net.internet2.edu 51.438ms asymm 18 11: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 51.731ms asymm 17 12: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 52.350ms asymm 16 13: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 50.539ms asymm 15 14: fourhundredge-0-0-0-21.4079.core1.chic.net.internet2.edu 50.511ms 15: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 53.273ms asymm 13 16: 144.92.233.112 53.473ms asymm 14 17: perfsonar02.hep.wisc.edu 53.352ms reached Resume: pmtu 1500 hops 17 back 16
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_w7xJht/images/htc__ubuntu__24.04.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__ubuntu__24.04.sif APPTAINER_TMPDIR=/tmp/glide_w7xJht/execute/dir_39666/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=39666 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_w7xJht/cred.d/idtokens/credential_OSPool.main_OSG_US_UW-IT-CE1.idtoken GLIDEIN_CPUS=40 GLIDEIN_ClusterId=6115949 GLIDEIN_Entry_Name=OSG_US_UW-IT-CE1 GLIDEIN_Entry_Signature=b632cfa161c429dd6b5a39e28110040b9b747e8c GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=uw-it-ce1.svc.osg-htc.org uw-it-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_oscigrid_oQn1JX GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UW-IT-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=UW-IT GLIDEIN_Tmp_Dir=/tmp/glide_w7xJht/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=UW-IT-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_3083_512559163@n3004.hyak.local$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_3083_512559163@n3004.hyak.local$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_w7xJht/images/htc__ubuntu__24.04.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/tmp/glide_w7xJht/images/htc__ubuntu__24.04.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__ubuntu__24.04.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_w7xJht/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_w7xJht/execute/dir_39666/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://klone-dip1:3128 no_proxy=localhost,hyak.local,hyakm.washington.edu
pass kernel
4.18.0-513.18.1.el8_9.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
354.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 374896792 3611632 371285160 1% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy http://klone-dip1:3128
UWEC-BOSE-CE1 2026-04-03 17:12 4
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
UWEC-BOSE-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 7.982ms 1: _gateway 8.002ms 2: r-uweauclaire-cvtc-ae2-0.uwsys.net 8.316ms 3: r-uweauclaire-centennial-et-0-1-9-3503.uwsys.net 12.246ms asymm 8 4: r-uwmadison-animal-ae3-3507.uwsys.net 4.729ms 5: r-uwmadison-cssc-ae3-3439.uwsys.net 14.280ms 6: 143.235.41.129 12.197ms 7: perfsonar02.hep.wisc.edu 12.129ms reached Resume: pmtu 1500 hops 7 back 6
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/scratch/job_360611/glide_tKBHZn/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/local/scratch/job_360611/glide_tKBHZn/execute/dir_1781338/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1781338 CONTAINERSW_FULL_VERSION=apptainer version 1.2.5 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/data/software/default/apptainer/1.2.5/bin/apptainer CONTAINERSW_VERSION=1.2.5 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/scratch/job_360611/glide_tKBHZn/cred.d/idtokens/credential_OSPool.main_OSG_US_UWEC-BOSE-CE1.idtoken GLIDEIN_CONTAINER_ENV=keepall GLIDEIN_CPUS=16 GLIDEIN_ClusterId=234912 GLIDEIN_Entry_Name=OSG_US_UWEC-BOSE-CE1 GLIDEIN_Entry_Signature=15b15d07a4ea270b5314d88e487be64e7bc61c69 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=uwec-bose-ce1.svc.osg-htc.org uwec-bose-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_THOWEZ GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UWEC-BOSE-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=UWEC-BOSE GLIDEIN_Tmp_Dir=/local/scratch/job_360611/glide_tKBHZn/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.2.5 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/data/software/default/apptainer/1.2.5/bin/apptainer GWMS_CONTAINERSW_VERSION=1.2.5 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/data/software/default/apptainer/1.2.5/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.2.5 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=UWEC-BOSE-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_6@glidein_3436913_520863746@cn02.head.bose.hpc.cluster$ "; unset PROMPT_COMMAND PS1=slot1_6@glidein_3436913_520863746@cn02.head.bose.hpc.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIN=/data/software/default/apptainer/1.2.5/bin SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/local/scratch/job_360611/glide_tKBHZn/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.2.5 SINGULARITY_IMAGES_DICT=default:/local/scratch/job_360611/glide_tKBHZn/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/data/software/default/apptainer/1.2.5/bin/apptainer SINGULARITY_VERSION=1.2.5 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/scratch/job_360611/glide_tKBHZn/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/scratch/job_360611/glide_tKBHZn/execute/dir_1781338/scratch _CONDOR_SLOT=slot1_6 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-348.el8.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
752.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda8 837052400 48354644 788697756 6% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UWL-Test-EP 2026-04-03 17:02 14
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
UWL-Test-EP — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.125ms 1: _gateway 0.167ms 2: 138.49.30.1 0.884ms asymm 3 3: 10.159.96.1 0.633ms 4: 138.49.0.2 1.108ms asymm 5 5: 138.49.0.1 0.830ms 6: r-uwlacrosse-wing-ae3-182.uwsys.net 1.062ms asymm 5 7: r-uwmadison-animal-et-0-1-4-3504.uwsys.net 4.069ms asymm 6 8: 143.235.41.129 3.719ms 9: perfsonar02.hep.wisc.edu 3.503ms reached Resume: pmtu 1500 hops 9 back 8
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-coCa4L/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-coCa4L/execute/dir_530542 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=530542 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=UW-La Crosse CSCPE PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_11@UWL-Test-EP.11b8feafab7e$ "; unset PROMPT_COMMAND PS1=slot1_11@UWL-Test-EP.11b8feafab7e$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-coCa4L/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-coCa4L/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-coCa4L/execute/dir_530542 _CONDOR_SLOT=slot1_11 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass kernel
5.14.0-611.16.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
138.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 235337452 89998636 145338816 39% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UWM-Mortimer-CE1 2026-04-03 17:01 168
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
UWM-Mortimer-CE1 — probe output
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: _gateway 46.469ms 1: _gateway 46.844ms 2: no reply 3: r-uwmilwaukee-ems-ae9-160.uwsys.net 47.037ms 4: r-uwmadison-cssc-ae4-3455.uwsys.net 4.514ms 5: 143.235.41.129 49.151ms 6: perfsonar02.hep.wisc.edu 49.578ms reached Resume: pmtu 9000 hops 6 back 7
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_1vkcPc/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_1vkcPc/execute/dir_2556683/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2556683 CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.3-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_1vkcPc/cred.d/idtokens/credential_OSPool.main_OSG_US_UWM-Mortimer-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=6277174 GLIDEIN_Entry_Name=OSG_US_UWM-Mortimer-CE1 GLIDEIN_Entry_Signature=a8a49b504c39daab47dba8fbd664505459435786 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=uwm-mortimer-ce1.svc.osg-htc.org uwm-mortimer-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_LUGdCN GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=uwm-proxy.uwm.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UWM-Mortimer-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=436230a2fef517d351fbb58e714fa5f5fa541af0 GLIDEIN_Site=UWM-Mortimer GLIDEIN_Tmp_Dir=/tmp/glide_1vkcPc/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.3-1.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.3-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=UWM-Mortimer-CE1 OSG_SQUID_LOCATION=uwm-proxy.uwm.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_3663763_118828544@execute-122.mortimer.hpc.uwm.edu$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_3663763_118828544@execute-122.mortimer.hpc.uwm.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_1vkcPc/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.3-1.el8 SINGULARITY_IMAGES_DICT=default:/tmp/glide_1vkcPc/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.3-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_1vkcPc/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_1vkcPc/execute/dir_2556683/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.78.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
323.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/VolGroup-tmp 366991360 27737416 339253944 8% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy uwm-proxy.uwm.edu:3128
Utah-Granite-CE1 2026-04-03 17:01 98
100.0%
95.9%
0.0%
95.9%
mtu env kernel nrp_images df http_proxy
Utah-Granite-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.242.160.3 0.141ms 1: 10.242.160.3 0.143ms 2: rt-inscc-ddc-core-102.chpc.utah.edu 0.522ms asymm 3 3: rt-park-inscc-ul.chpc.utah.edu 0.692ms asymm 4 4: 155.99.130.13 0.589ms asymm 5 5: 155.99.130.58 0.524ms asymm 6 6: 199.104.93.30 1.966ms 7: 199.104.93.21 1.591ms 8: 140.197.253.138 2.019ms 9: 140.197.248.40 1.957ms asymm 11 10: lv3-beibr-b-182-int.uen.net 2.544ms 11: fourhundredge-0-0-0-8.1013.core1.salt.net.internet2.edu 3.463ms 12: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 36.377ms asymm 20 13: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 35.888ms asymm 19 14: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 34.980ms asymm 18 15: fourhundredge-0-0-0-21.4079.core1.chic.net.internet2.edu 45.273ms asymm 17 16: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 35.674ms 17: 144.92.233.112 38.625ms 18: perfsonar02.hep.wisc.edu 38.909ms reached Resume: pmtu 1500 hops 18 back 19
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/local/osguserg/1021835/glide_rKkjlO/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/local/osguserg/1021835/glide_rKkjlO/execute/dir_1392071/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1392071 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/local/osguserg/1021835/glide_rKkjlO/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/local/osguserg/1021835/glide_rKkjlO/cred.d/idtokens/credential_OSPool.main_OSG_US_Utah-Granite-CE1.idtoken GLIDEIN_CPUS=16 GLIDEIN_ClusterId=5610251 GLIDEIN_Entry_Name=OSG_US_Utah-Granite-CE1 GLIDEIN_Entry_Signature=bc29a3b97a630c48b521ea84bac77d39ddea6d41 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=utah-granite-ce1.svc.opensciencegrid.org utah-granite-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osguserg_vgBVAz GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Utah-Granite-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=Utah-Granite GLIDEIN_Tmp_Dir=/scratch/local/osguserg/1021835/glide_rKkjlO/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/local/osguserg/1021835/glide_rKkjlO/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/local/osguserg/1021835/glide_rKkjlO/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Utah-Granite-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_7@glidein_2231273_39040040@grn052.int.chpc.utah.edu$ "; unset PROMPT_COMMAND PS1=slot1_7@glidein_2231273_39040040@grn052.int.chpc.utah.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/local/osguserg/1021835/glide_rKkjlO/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/scratch/local/osguserg/1021835/glide_rKkjlO/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/scratch/local/osguserg/1021835/glide_rKkjlO/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/local/osguserg/1021835/glide_rKkjlO/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/local/osguserg/1021835/glide_rKkjlO/execute/dir_1392071/scratch _CONDOR_SLOT=slot1_7 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.89.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
5747.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_peaks_crypt-lv_scratch 6114895872 88337432 6026558440 2% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Utah-Kingspeak-CE1 2026-04-03 17:01 36
100.0%
86.1%
0.0%
86.1%
mtu env kernel nrp_images df http_proxy
Utah-Kingspeak-CE1 — probe output
pass mtu
pmtu=2044 1?: [LOCALHOST] pmtu 2044 1: 10.242.154.2 0.086ms 1: 10.242.154.2 0.066ms 2: rt-inscc-ddc-core-101.chpc.utah.edu 0.444ms asymm 3 3: 10.2.0.1 0.515ms asymm 4 4: 155.99.130.11 0.493ms asymm 5 5: 155.99.130.64 0.598ms asymm 6 6: 199.104.93.34 1.299ms 7: no reply 8: 140.197.253.138 2.229ms 9: 140.197.248.40 2.269ms asymm 11 10: lv3-beibr-b-182-int.uen.net 2.474ms 11: fourhundredge-0-0-0-8.1013.core1.salt.net.internet2.edu 3.989ms 12: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 37.886ms asymm 20 13: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 36.513ms asymm 19 14: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 35.337ms asymm 18 15: fourhundredge-0-0-0-23.4079.core1.chic.net.internet2.edu 37.748ms asymm 17 16: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 38.065ms 17: 144.92.233.112 38.362ms 18: perfsonar02.hep.wisc.edu 38.529ms reached Resume: pmtu 2044 hops 18 back 19
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/local/osguserk/18505509/glide_dCFAlA/images/htc__ubuntu__24.04.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__ubuntu__24.04.sif APPTAINER_TMPDIR=/scratch/local/osguserk/18505509/glide_dCFAlA/execute/dir_3157281/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3157281 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/local/osguserk/18505509/glide_dCFAlA/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/local/osguserk/18505509/glide_dCFAlA/cred.d/idtokens/credential_OSPool.main_OSG_US_Utah-Kingspeak-CE1.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=305628 GLIDEIN_Entry_Name=OSG_US_Utah-Kingspeak-CE1 GLIDEIN_Entry_Signature=1e20161ce31cd0eb225eaa5c686cf2084778f38a GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=utah-kingspeak-ce1.svc.opensciencegrid.org utah-kingspeak-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osguserk_hzSDql GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Utah-Kingspeak-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=68a8d9fe2b016de27e3e6d38ff6089bced869176 GLIDEIN_Site=Utah-Kingspeak GLIDEIN_Tmp_Dir=/scratch/local/osguserk/18505509/glide_dCFAlA/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/local/osguserk/18505509/glide_dCFAlA/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/local/osguserk/18505509/glide_dCFAlA/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Utah-Kingspeak-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_3113742_48124150@kp005.ipoib.kingspeak.peaks$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_3113742_48124150@kp005.ipoib.kingspeak.peaks$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/local/osguserk/18505509/glide_dCFAlA/images/htc__ubuntu__24.04.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/scratch/local/osguserk/18505509/glide_dCFAlA/images/htc__ubuntu__24.04.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__ubuntu__24.04.sif SINGULARITY_PATH=/scratch/local/osguserk/18505509/glide_dCFAlA/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/local/osguserk/18505509/glide_dCFAlA/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/local/osguserk/18505509/glide_dCFAlA/execute/dir_3157281/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.89.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
443.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_peaks_crypt-lv_scratch 471358596 6450512 464908084 2% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Utah-Lonepeak-CE1 2026-04-03 17:01 100
100.0%
93.0%
0.0%
93.0%
mtu env kernel nrp_images df http_proxy
Utah-Lonepeak-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: gw-peaks-2.chpc.utah.edu 0.151ms 1: gw-peaks-2.chpc.utah.edu 0.142ms 2: rt-inscc-ddc-core-101.chpc.utah.edu 0.636ms asymm 3 3: 10.2.0.1 0.520ms asymm 4 4: 155.99.130.11 0.668ms asymm 5 5: 155.99.130.64 0.647ms asymm 6 6: 199.104.93.34 1.325ms 7: 199.104.93.21 2.132ms 8: 140.197.253.138 1.988ms 9: 140.197.248.40 2.191ms asymm 11 10: lv3-beibr-b-182-int.uen.net 2.385ms 11: fourhundredge-0-0-0-8.1013.core1.salt.net.internet2.edu 5.962ms 12: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 36.965ms asymm 20 13: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 37.476ms asymm 19 14: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 36.693ms asymm 18 15: fourhundredge-0-0-0-23.4079.core1.chic.net.internet2.edu 37.170ms asymm 17 16: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 37.806ms 17: 144.92.233.112 38.564ms 18: perfsonar02.hep.wisc.edu 38.944ms reached Resume: pmtu 1500 hops 18 back 19
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/local/osguserl/9038321/glide_SPFxX8/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/scratch/local/osguserl/9038321/glide_SPFxX8/execute/dir_1697799/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1697799 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/local/osguserl/9038321/glide_SPFxX8/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/local/osguserl/9038321/glide_SPFxX8/cred.d/idtokens/credential_OSPool.main_OSG_US_Utah-Lonepeak-CE1.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=234765 GLIDEIN_Entry_Name=OSG_US_Utah-Lonepeak-CE1 GLIDEIN_Entry_Signature=66d9cc826b323ab9ee0a02a02f36597b077e679b GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=utah-lonepeak-ce1.svc.opensciencegrid.org utah-lonepeak-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osguserl_J2JQZA GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Utah-Lonepeak-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=Utah-Lonepeak GLIDEIN_Tmp_Dir=/scratch/local/osguserl/9038321/glide_SPFxX8/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/local/osguserl/9038321/glide_SPFxX8/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/local/osguserl/9038321/glide_SPFxX8/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Utah-Lonepeak-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_4060586_473100534@lp079.lonepeak.peaks$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_4060586_473100534@lp079.lonepeak.peaks$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/local/osguserl/9038321/glide_SPFxX8/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/scratch/local/osguserl/9038321/glide_SPFxX8/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/scratch/local/osguserl/9038321/glide_SPFxX8/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/local/osguserl/9038321/glide_SPFxX8/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/local/osguserl/9038321/glide_SPFxX8/execute/dir_1697799/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.89.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
482.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_peaks_crypt-lv_scratch 518644224 12813188 505831036 3% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Utah-Notchpeak-CE1 2026-04-03 17:01 98
100.0%
92.9%
0.0%
92.9%
mtu env kernel nrp_images df http_proxy
Utah-Notchpeak-CE1 — probe output
pass mtu
pmtu=2044 1?: [LOCALHOST] pmtu 2044 1: 10.242.154.2 0.134ms 1: 10.242.154.2 0.055ms 2: rt-inscc-ddc-core-101.chpc.utah.edu 0.559ms asymm 3 3: 10.2.0.1 0.532ms asymm 4 4: 155.99.130.11 0.494ms asymm 5 5: 155.99.130.64 0.593ms asymm 6 6: 199.104.93.34 1.375ms 7: no reply 8: 140.197.253.138 6.320ms 9: 140.197.248.40 2.474ms asymm 11 10: lv3-beibr-b-182-int.uen.net 2.706ms 11: fourhundredge-0-0-0-8.1013.core1.salt.net.internet2.edu 4.391ms 12: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 36.686ms asymm 20 13: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 36.669ms asymm 19 14: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 36.336ms asymm 18 15: fourhundredge-0-0-0-23.4079.core1.chic.net.internet2.edu 35.665ms asymm 17 16: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 37.022ms 17: 144.92.233.112 38.568ms 18: perfsonar02.hep.wisc.edu 41.617ms reached Resume: pmtu 2044 hops 18 back 19
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/local/osgusern/11693655/glide_3W5SPz/images/htc__debian__13.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__debian__13.sif APPTAINER_TMPDIR=/scratch/local/osgusern/11693655/glide_3W5SPz/execute/dir_2962067/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2962067 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/local/osgusern/11693655/glide_3W5SPz/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/local/osgusern/11693655/glide_3W5SPz/cred.d/idtokens/credential_OSPool.main_OSG_US_Utah-Notchpeak-CE1.idtoken GLIDEIN_CPUS=16 GLIDEIN_ClusterId=6281171 GLIDEIN_Entry_Name=OSG_US_Utah-Notchpeak-CE1 GLIDEIN_Entry_Signature=8f175b5c8483e3c24b2de0f1226561f7109f51a4 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=utah-notchpeak-ce1.svc.opensciencegrid.org utah-notchpeak-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgusern_WlzBuq GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Utah-Notchpeak-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=Utah-Notchpeak GLIDEIN_Tmp_Dir=/scratch/local/osgusern/11693655/glide_3W5SPz/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/local/osgusern/11693655/glide_3W5SPz/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/local/osgusern/11693655/glide_3W5SPz/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=Utah-Notchpeak-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_2088497_387081242@notch041.ipoib.int.chpc.utah.edu$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_2088497_387081242@notch041.ipoib.int.chpc.utah.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/local/osgusern/11693655/glide_3W5SPz/images/htc__debian__13.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/scratch/local/osgusern/11693655/glide_3W5SPz/images/htc__debian__13.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__debian__13.sif SINGULARITY_PATH=/scratch/local/osgusern/11693655/glide_3W5SPz/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/local/osgusern/11693655/glide_3W5SPz/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/local/osgusern/11693655/glide_3W5SPz/execute/dir_2962067/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
4.18.0-553.89.1.el8_10.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
1786.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_peaks_crypt-lv_scratch 1902232068 29002660 1873229408 2% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
VU-AUGIE-CE1 2026-04-03 17:02 2
100.0%
0.0%
0.0%
0.0%
mtu env kernel nrp_images df http_proxy
VU-AUGIE-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: clusterhn.cluster.pssclabs.com 0.212ms 1: clusterhn.cluster.pssclabs.com 0.167ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_7y8umI/images/htc__ubuntu__24.04.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__ubuntu__24.04.sif APPTAINER_TMPDIR=/scratch/glide_7y8umI/execute/dir_813979/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=813979 CONTAINERSW_FULL_VERSION=apptainer version 1.1.8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.1.8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_7y8umI/cred.d/idtokens/credential_OSPool.main_VU-AUGIE-CE1.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=301054 GLIDEIN_Entry_Name=VU-AUGIE-CE1 GLIDEIN_Entry_Signature=5065e9a24254716edc92682d326ef6da3ad9b1eb GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=vu-augie-ce1.svc.opensciencegrid.org vu-augie-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_NoMlkc GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=augie.villanova.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=VU-AUGIE-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=VU-AUGIE GLIDEIN_Tmp_Dir=/scratch/glide_7y8umI/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.1.8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.1.8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.1.8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=VU-AUGIE-CE1 OSG_SQUID_LOCATION=augie.villanova.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_16746_534309003@n007.cluster.com$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_16746_534309003@n007.cluster.com$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_7y8umI/images/htc__ubuntu__24.04.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.1.8 SINGULARITY_IMAGES_DICT=default:/scratch/glide_7y8umI/images/htc__ubuntu__24.04.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__ubuntu__24.04.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.1.8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_7y8umI/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_7y8umI/execute/dir_813979/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.15.0-57-generic
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
714.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p4 760688044 11175228 749512816 2% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy augie.villanova.edu:3128
WISC-PATH-EP 2026-04-03 17:02 4
100.0%
100.0%
25.0%
100.0%
mtu env kernel nrp_images df http_proxy
WISC-PATH-EP — probe output
fail mtu
Could not determine pmtu tracepath: perfsonar02.hep.wisc.edu: Name or service not known
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-n7wckK/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-n7wckK/execute/dir_947998 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=947998 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=WISC-PATH PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_57@WISC-PATH-EP.osgvo-docker-pilot-ospool-6964fd8986-ltmvv$ "; unset PROMPT_COMMAND PS1=slot1_57@WISC-PATH-EP.osgvo-docker-pilot-ospool-6964fd8986-ltmvv$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-n7wckK/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-n7wckK/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-n7wckK/execute/dir_947998 _CONDOR_SLOT=slot1_57 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=http://frontier-squid:3128
pass kernel
4.18.0-517.el8.x86_64
fail nrp_images
Failed to download https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt: <urlopen error [Errno -2] Name or service not known>
pass df
382.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/openebs-pvc--0bde29e8--25a4--4eb5--8ef9--b2bd5b2558b7 442688776 41829480 400842912 10% /srv
fail http_proxy
Failed to download http://cm-1.ospool.osg-htc.org/overview/ via proxy http://frontier-squid:3128: <urlopen error timed out>
WSU-GRID-CE1 2026-04-03 11:02 13
100.0%
100.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
WSU-GRID-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: gwha1.nas.grid.wayne.edu 0.632ms 1: gwha1.nas.grid.wayne.edu 0.179ms 2: 141.217.136.1 0.550ms 3: 193-cr-c6509e-1-r515-fo-8-1-v98.core.infra 0.619ms 4: 193-br-dcs-7280sr3k-1-po-1.core.infra 0.949ms 5: ae1x160.dtrt-wsudc-c1.mich.net 8.701ms 6: et-3-1-0x3.gdrp-eberhard-c1.mich.net 12.121ms 7: no reply 8: ae102x0.chcg-lumen-600w-c1.mich.net 15.316ms asymm 9 9: 192.122.182.34 16.772ms asymm 10 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 17.646ms 11: 144.92.233.112 19.694ms 12: perfsonar02.hep.wisc.edu 19.977ms reached Resume: pmtu 1500 hops 12 back 13
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/wsu/tmp/glide_8mQED6/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/wsu/tmp/glide_8mQED6/execute/dir_18041 BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=apptainer version 1.3.1-1.el7 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/wsu/tmp/glide_8mQED6/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.3.1-1.el7 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/wsu/tmp/glide_8mQED6/cred.d/idtokens/credential_OSPool.main_OSG_US_WSU-GRID_himem.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=235248 GLIDEIN_Entry_Name=OSG_US_WSU-GRID_himem GLIDEIN_Entry_Signature=e0e27af9adbff1a59e127d1dea242a1c0f5008a8 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=wsu-grid-ce1.svc.opensciencegrid.org wsu-grid-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_HHetwv GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=vsquid1.grid.wayne.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=WSU-GRID-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=GRID_ce2 GLIDEIN_Tmp_Dir=/wsu/tmp/glide_8mQED6/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.3.1-1.el7 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/wsu/tmp/glide_8mQED6/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.3.1-1.el7 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/wsu/tmp/glide_8mQED6/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.3.1-1.el7 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=WSU-GRID-CE1 OSG_SQUID_LOCATION=vsquid1.grid.wayne.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_5@glidein_14469_18806830@sfx6$ "; unset PROMPT_COMMAND PS1=slot1_5@glidein_14469_18806830@sfx6$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/wsu/tmp/glide_8mQED6/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.3.1-1.el7 SINGULARITY_IMAGES_DICT=default:/wsu/tmp/glide_8mQED6/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/wsu/tmp/glide_8mQED6/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.3.1-1.el7 SLOTS_LAYOUT=partitionable TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/wsu/tmp/glide_8mQED6/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/wsu/tmp/glide_8mQED6/execute/dir_18041 _CONDOR_SLOT=slot1_5 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass kernel
5.15.80-200.el7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
46.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on panfs://10.3.140.1/wsu 48828112 64960 48763152 1% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy vsquid1.grid.wayne.edu:3128
WY-MedicineBow-CE1 2026-04-03 17:31 34
100.0%
100.0%
0.0%
100.0%
mtu env kernel nrp_images df http_proxy
WY-MedicineBow-CE1 — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 9000 1: no reply 2: _gateway 19.373ms pmtu 1500 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/project/openscigrid/glide_rrKZBU/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/project/openscigrid/glide_rrKZBU/execute/dir_2879533/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2879533 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/project/openscigrid/glide_rrKZBU/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/project/openscigrid/glide_rrKZBU/cred.d/idtokens/credential_OSPool.main_OSG_US_WY-MedicineBow-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6282962 GLIDEIN_Entry_Name=OSG_US_WY-MedicineBow-CE1 GLIDEIN_Entry_Signature=c1c736ead1a4d25a5c043f119dde09a96a611876 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=wy-medicinebow-ce1.svc.opensciencegrid.org wy-medicinebow-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_yKH9ew GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=WY-MedicineBow-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=91d4d224d06dccc6aa0de4dc1335cbb48438bb67 GLIDEIN_Site=WY-MedicineBow GLIDEIN_Tmp_Dir=/project/openscigrid/glide_rrKZBU/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/project/openscigrid/glide_rrKZBU/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/project/openscigrid/glide_rrKZBU/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=localhost OSG_SITE_NAME=WY-MedicineBow-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2763546_280825264@mbcpu-001.cluster$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2763546_280825264@mbcpu-001.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/project/openscigrid/glide_rrKZBU/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/project/openscigrid/glide_rrKZBU/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/project/openscigrid/glide_rrKZBU/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/project/openscigrid/glide_rrKZBU/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/scratch/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/project/openscigrid/glide_rrKZBU/execute/dir_2879533/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass kernel
5.14.0-611.16.1.el9_7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
5031.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on vast-eth:/cluster/medbow/project/openscigrid/glide_rrKZBU/execute/dir_2879533 5368709120 92575744 5276133376 2% /srv
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
uprm-cms 2026-04-03 17:58 54
100.0%
0.0%
100.0%
100.0%
mtu env kernel nrp_images df http_proxy
uprm-cms — probe output
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: cms-hn.local 0.261ms 1: cms-hn.local 0.271ms 2: 136.145.77.1 0.595ms 3: 136.145.230.114 0.454ms 4: R8500.uprm.edu 0.632ms 5: hub787-rum-a.upr.edu 5.206ms 6: et-0-0-1-819.rt04.be.ampath.net 35.040ms 7: et-0-0-1-67.rt05.bb.ampath.net 35.170ms 8: hundredge-0-0-0-26.1804.core1.jack.net.internet2.edu 43.652ms 9: fourhundredge-0-0-0-6.4079.core1.atla.net.internet2.edu 69.190ms asymm 21 10: fourhundredge-0-0-0-23.4079.core2.atla.net.internet2.edu 68.292ms asymm 20 11: fourhundredge-0-0-0-2.4079.core2.nash.net.internet2.edu 67.345ms asymm 19 12: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 68.804ms asymm 18 13: fourhundredge-0-0-0-23.4079.core2.cinc.net.internet2.edu 68.298ms asymm 17 14: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 67.886ms asymm 16 15: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 67.913ms 16: 144.92.233.112 69.327ms 17: perfsonar02.hep.wisc.edu 69.457ms reached Resume: pmtu 1500 hops 17 back 18
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=rocky:9 APPTAINER_TMPDIR=/var/lib/condor/execute/dir_2191929/glide_mM0Er9/execute/dir_2370365 BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=apptainer version 1.2.5-1.el7 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.2.5-1.el7 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=3988edbcce83b48dea24ce3d85d988ba8960e058 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=de94f2f98e5288872fceba107dfeeeedadd527a4 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_PuertoRico GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_2191929/glide_mM0Er9/cred.d/idtokens/credential_OSPool.main_CMS_T3_US_PuertoRico_UPRM.idtoken GLIDEIN_ClusterId=233507 GLIDEIN_Country=US GLIDEIN_DUNESite=US_PuertoRico GLIDEIN_Entry_Name=CMS_T3_US_PuertoRico_UPRM GLIDEIN_Entry_Signature=d9fafb4dc99a23036616a6f2f392232152ea47a4 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=cms-grid0.hep.uprm.edu cms-grid0.hep.uprm.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_9TUHul GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=5 GLIDEIN_Proxy_URL=squid-0-0-0.local:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=uprm-cms GLIDEIN_SEs=cms-se.hep.uprm.edu GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=e2967ec701f0f334aca432cb68900f8c01f94561 GLIDEIN_Site=PuertoRico GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_2191929/glide_mM0Er9/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.2.5-1.el7 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.2.5-1.el7 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.2.5-1.el7 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=cms-grid0.hep.uprm.edu OSG_SITE_NAME=uprm-cms-ce OSG_SQUID_LOCATION=squid-0-0-0.local:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2192551_177049635@sandy-0-2.local$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2192551_177049635@sandy-0-2.local$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.2.5-1.el7 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_MODE=unprivileged SINGULARITY_NAME=rocky:9 SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.2.5-1.el7 SLOTS_LAYOUT=fixed TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_2191929/glide_mM0Er9/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_2191929/glide_mM0Er9/execute/dir_2370365 _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
warning kernel
3.10.0-1160.105.1.el7.x86_64
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260403T053001Z.sif
pass df
152.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 206292968 36241860 159548964 19% /srv
pass http_proxy
Downloaded http://cm-1.ospool.osg-htc.org/overview/ successfully via proxy squid-0-0-0.local:3128