OSPool Resource Status

Refreshes every 5 minutes
81
Resources
75
All Passing
1
Failing
5
Warnings
0
Not Available
Resource Last Updated Glideins Singularity SIF CVMFS Pelican Probes
AGLT2 8 hours ago 542
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
AGLT2 — probe output
pass kernel
5.14.0-611.49.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_3397471/glide_u844a4/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_3397471/glide_u844a4/execute/dir_3937085/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3937085 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_3397471/glide_u844a4/cred.d/idtokens/credential_OSPool.main_HCC_US_Michigan_gate02.idtoken GLIDEIN_ClusterId=479453 GLIDEIN_Country=US GLIDEIN_DUNESite=US_Michigan GLIDEIN_Entry_Name=HCC_US_Michigan_gate02 GLIDEIN_Entry_Signature=f564e7f81311d09e038633bc9db1c9877a723e0d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=gate02.aglt2.org gate02.aglt2.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_tIXkr5 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_Nickname=AGLT2 GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=7 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=AGLT2 GLIDEIN_SEs=head01.aglt2.org GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=Michigan GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_3397471/glide_u844a4/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Michigan PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3397494_600464868@c7-11-15.aglt2.org$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3397494_600464868@c7-11-15.aglt2.org$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_3397471/glide_u844a4/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/dir_3397471/glide_u844a4/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_3397471/glide_u844a4/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_3397471/glide_u844a4/execute/dir_3937085/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
3170.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sdb1 3688677440 176749448 3324478752 6% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] 0.012ms pmtu 9000 1: 2001:48a8:68f7:1:fa6b:d9ff:fe2d:b1cb 0.786ms 1: 2001:48a8:68f7:1:7ead:4fff:fe6b:b017 0.771ms 2: 2001:48a8:68f7:ff06:: 1.052ms 3: l3-pereseb1-perencas1.pe-re-seb-1.umnet.umich.edu 1.065ms asymm 2 4: chic-cr6--uwmadison_se-203.es.net 5.785ms 5: uwmadison_se-203--chic-cr6.es.net 23.178ms 6: uwmadison_se-203--chic-cr6.es.net 23.156ms asymm 5 7: perfsonar02.hep.wisc.edu 22.983ms reached Resume: pmtu 9000 hops 7 back 6
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
AMNH-HEL-CE1 2 hours ago 32
0.0%
0.0%
100.0%
78.1%
kernel env df firewall mtu nrp_images http_proxy gpu
AMNH-HEL-CE1 — probe output
warning kernel
3.10.0-1062.18.1.el7.x86_64
pass env
APPTAINER_CACHEDIR=/scratch_space/glide_ViYSr2/execute/dir_63437 BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*` } BATCH_SYSTEM=HTCondor BLAHPD_LOCATION=/home/osg01/bosco_slate/glite CONDORCE_COLLECTOR_HOST=amnh-hel-ce1.svc.opensciencegrid.org:9619 CONDOR_CONFIG=/scratch_space/glide_ViYSr2/condor_config CONDOR_PARENT_ID=hel-n0003:63437:1780408204 CONDOR_PROCD_ADDRESS=/scratch_space/glide_ViYSr2/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/scratch_space/glide_ViYSr2/log/procd_address CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 ENVIRONMENT=BATCH GFAL_CONFIG_DIR=/home/osg01/bosco-osg-wn-client/etc/gfal2.d/ GFAL_PLUGIN_DIR=/home/osg01/bosco-osg-wn-client/usr/lib64/gfal2-plugins/ GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch_space/glide_ViYSr2/cred.d/idtokens/credential_OSPool.main_COVID19_US_AMNH-HEL-CE1.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=5887751 GLIDEIN_Entry_Name=COVID19_US_AMNH-HEL-CE1 GLIDEIN_Entry_Signature=b5f10c564ca2d7e53e57c2e1c2b66608bc866a56 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=amnh-hel-ce1.svc.opensciencegrid.org amnh-hel-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_kkTMVu GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=osdf1.newy32aoa.net.internet2.edu:9002 GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=AMNH-HEL-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=AMNH GLIDEIN_Tmp_Dir=/scratch_space/glide_ViYSr2/tmp GLIDEIN_UUID=b017ba19-baca-4982-b16d-5410c125b74d GLITE_LOCATION=/home/osg01/bosco_slate/glite GLOBUS_LOCATION=/home/osg01/bosco-osg-wn-client/usr GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_PATH=/scratch_space/glide_ViYSr2/.gwms.d/bin: GWMS_SINGULARITY_STATUS=PREFERRED HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HISTCONTROL=ignoredups HISTSIZE=1000 HOLD_GRACE_TIME=0 HOME=/scratch_space/glide_ViYSr2/execute/dir_63437 HOSTNAME=hel-n0003 JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_COVID19_US_AMNH-HEL-CE1/tokens.tgz JULIA_NUM_THREADS=1 KDEDIRS=/usr LANG=C LD_LIBRARY_PATH=/home/osg01/bosco-osg-wn-client/lib64:/home/osg01/bosco-osg-wn-client/lib:/home/osg01/bosco-osg-wn-client/usr/lib64:/home/osg01/bosco-osg-wn-client/usr/lib:/home/osg01/bosco-osg-wn-client/usr/lib64/dcap:/home/osg01/bosco-osg-wn-client/usr/lib64/lcgdm LESSOPEN=||/usr/bin/lesspipe.sh %s LOADEDMODULES= LOGNAME=osg01 MAIL=/var/spool/mail/osg01 MANPATH=/home/osg01/bosco-osg-wn-client/usr/share/man::/opt/pbs/share/man:/opt/pbs/share/man MKL_NUM_THREADS=1 MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles MODULESHOME=/usr/share/Modules MY_HOSTS=pvm_hostfile NODE_COUNT=8 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=UNAVAILABLE OSG_DATA=UNAVAILABLE OSG_GRID=/home/osg01/bosco-osg-wn-client OSG_HOSTNAME=amnh-hel-ce1.svc.opensciencegrid.org OSG_LOCATION=/home/osg01/bosco-osg-wn-client OSG_SITE_NAME=AMNH-HEL-CE1 OSG_SQUID_LOCATION=osdf1.newy32aoa.net.internet2.edu:9002 OSG_WN_TMP=/scratch_space/glide_ViYSr2/execute/dir_63437/.local-tmp PATH=/scratch_space/glide_ViYSr2/.gwms.d/bin::/home/osg01/bosco-osg-wn-client/usr/bin:/home/osg01/bosco-osg-wn-client/usr/sbin:/usr/local/sbin/:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/software/Apptainer/apptainer-1.1.5/bin:/usr/local/software/Singularity/singularity-3.5.3/bin PERL5LIB=/home/osg01/bosco-osg-wn-client/usr/share/perl5/vendor_perl:/home/osg01/bosco-osg-wn-client/usr/share/perl5:/home/osg01/bosco-osg-wn-client/usr/lib64/perl5/vendor_perl:/home/osg01/bosco-osg-wn-client/usr/lib64/perl5:/home/osg01/perl5/lib/perl5: PERL_LOCAL_LIB_ROOT=:/home/osg01/perl5 PERL_MB_OPT=--install_base /home/osg01/perl5 PERL_MM_OPT=INSTALL_BASE=/home/osg01/perl5 PREEMPT_GRACE_TIME=10000000 PVM_ARCH=LINUX64 PVM_BIN=/usr/local/software/pvm/pvm3/bin PVM_ROOT=/usr/local/software/pvm/pvm3 PVM_RSH=/usr/bin/ssh PWD=/scratch_space/glide_ViYSr2/execute/dir_63437 PYTHONPATH=/home/osg01/bosco-osg-wn-client/usr/lib/python2.7/site-packages:/home/osg01/bosco-osg-wn-client/usr/lib64/python2.7/site-packages PYTHON_CPU_COUNT=1 QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include QTLIB=/usr/lib64/qt-3.3/lib QT_GRAPHICSSYSTEM_CHECKED=1 QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins ROOT_MAX_THREADS=1 SHELL=/bin/bash SHLVL=6 SINGULARITY_CACHEDIR=/scratch_space/glide_ViYSr2/execute/dir_63437 SLOTS_LAYOUT=partitionable SLURMD_NODENAME=hel-n0003 SLURM_CLUSTER_NAME=hel SLURM_CONF=/etc/slurm/slurm.conf SLURM_CPUS_ON_NODE=8 SLURM_GTIDS=0 SLURM_JOBID=1268280 SLURM_JOB_ACCOUNT=root SLURM_JOB_CPUS_PER_NODE=8 SLURM_JOB_GID=1083 SLURM_JOB_ID=1268280 SLURM_JOB_NAME=bl_ToVt7B SLURM_JOB_NODELIST=hel-n0003 SLURM_JOB_NUM_NODES=1 SLURM_JOB_PARTITION=batch-amd SLURM_JOB_QOS=normal SLURM_JOB_UID=1001 SLURM_JOB_USER=osg01 SLURM_LOCALID=0 SLURM_MEM_PER_NODE=65536 SLURM_NNODES=1 SLURM_NODEID=0 SLURM_NODELIST=hel-n0003 SLURM_NODE_ALIASES=(null) SLURM_NPROCS=8 SLURM_NTASKS=8 SLURM_NTASKS_PER_NODE=8 SLURM_PRIO_PROCESS=0 SLURM_PROCID=0 SLURM_SUBMIT_DIR=/home/osg01/bosco_slate/sandbox/20f2/20f209fb/amnh-hel-ce1.svc.opensciencegrid.org_9619_amnh-hel-ce1.svc.opensciencegrid.org_8572.0_1780207136 SLURM_SUBMIT_HOST=hel-head SLURM_TASKS_PER_NODE=8 SLURM_TASK_PID=7799 SLURM_TOPOLOGY_ADDR=hel-n0003 SLURM_TOPOLOGY_ADDR_PATTERN=node SLURM_WORKING_CLUSTER=hel:hel-head:6817:8960:101 SSH_CLIENT=128.105.82.98 16106 22 SSH_CONNECTION=128.105.82.98 16106 172.16.108.19 22 TEMP=/scratch_space/glide_ViYSr2/execute/dir_63437 TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/scratch_space/glide_ViYSr2/execute/dir_63437 TMPDIR=/scratch_space/glide_ViYSr2/execute/dir_63437/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime USER=osg01 VOMS_USERCONF=/home/osg01/bosco-osg-wn-client/etc/vomses X509_CERT_DIR=/home/osg01/bosco-osg-wn-client/etc/grid-security/certificates X509_VOMS_DIR=/home/osg01/bosco-osg-wn-client/etc/grid-security/vomsdir XDG_DATA_DIRS=/home/osg01/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_33461=33564:1780213865:3960049847 _CONDOR_ANCESTOR_33564=63437:1780408203:352366803 _CONDOR_ANCESTOR_63437=65243:1780408209:3027202686 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch_space/glide_ViYSr2/main/condor/bin _CONDOR_CHIRP_CONFIG=/scratch_space/glide_ViYSr2/execute/dir_63437/.chirp.config _CONDOR_CREDS=/scratch_space/glide_ViYSr2/execute/dir_63437/.condor_creds _CONDOR_EXECUTE=/scratch_space/glide_ViYSr2/execute _CONDOR_JOB_AD=/scratch_space/glide_ViYSr2/execute/dir_63437/.job.ad _CONDOR_JOB_IWD=/scratch_space/glide_ViYSr2/execute/dir_63437 _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/scratch_space/glide_ViYSr2/execute/dir_63437/.machine.ad _CONDOR_SCRATCH_DIR=/scratch_space/glide_ViYSr2/execute/dir_63437 _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/scratch_space/glide_ViYSr2/execute/dir_63437/.job_wrapper_failure
pass df
1672.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 1922182564 70889084 1753628808 4% /scratch_space
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: hel-head 0.666ms 1: hel-head 0.209ms 2: 172.16.108.1 0.540ms asymm 59 3: core-er2.sci-user-asa.transit.internal.amnh.org 1.294ms 4: fw.wan-asa.transit.internal.amnh.org 0.743ms asymm 61 5: sdmz.amnh.org 0.888ms 6: nyc-9208-amnh.nysernet.net 1.192ms 7: nyc32-55a1-nyc32-9208.nysernet.net 1.575ms 8: syr-57c3-nyc32-55a1.nysernet.net 6.925ms 9: buf-55a1-syr-57c3.nysernet.net 10.989ms 10: buf-9208-buf-55a1.nysernet.net 9.751ms 11: i2-clev-buf-9208.nysernet.net 18.637ms 12: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 21.922ms asymm 14 13: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 22.977ms 14: 144.92.233.112 23.377ms 15: perfsonar02.hep.wisc.edu 23.499ms reached Resume: pmtu 1500 hops 15 back 16
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy osdf1.newy32aoa.net.internet2.edu:9002
pass gpu
Tue Jun 2 10:05:00 2026 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.309.01 Driver Version: 535.309.01 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 Quadro M5000 Off | 00000000:02:00.0 Off | Off | | 38% 24C P8 13W / 150W | 3MiB / 8192MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 1 Quadro M5000 Off | 00000000:82:00.0 Off | Off | | 38% 21C P8 12W / 150W | 3MiB / 8192MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | No running processes found | +---------------------------------------------------------------------------------------+
AMNH-Huxley-CE1 2 hours ago 109
99.1%
99.1%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
AMNH-Huxley-CE1 — probe output
pass kernel
4.18.0-553.56.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch_space/glide_JRx2Mt/images/htc__almalinux__10.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__almalinux__10.sif APPTAINER_TMPDIR=/scratch_space/glide_JRx2Mt/execute/dir_2509732/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2509732 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch_space/glide_JRx2Mt/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch_space/glide_JRx2Mt/cred.d/idtokens/credential_OSPool.main_OSG_US_AMNH-Huxley-CE1.idtoken GLIDEIN_CPUS=2 GLIDEIN_ClusterId=516670 GLIDEIN_Entry_Name=OSG_US_AMNH-Huxley-CE1 GLIDEIN_Entry_Signature=ebd06f297b30214a6f095b1128d2b6e38c56b5c2 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=amnh-huxley-ce1.svc.osg-htc.org amnh-huxley-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_gRfc0R GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=6 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=AMNH-Huxley-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=4a553cb760eb0536cf01a0e964943d71a876fc26 GLIDEIN_Site=AMNH GLIDEIN_Tmp_Dir=/scratch_space/glide_JRx2Mt/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch_space/glide_JRx2Mt/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch_space/glide_JRx2Mt/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=amnh-huxley-ce1.svc.osg-htc.org OSG_SITE_NAME=AMNH-Huxley-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_2406743_426962925@huxley-n0015$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_2406743_426962925@huxley-n0015$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch_space/glide_JRx2Mt/images/htc__almalinux__10.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/scratch_space/glide_JRx2Mt/images/htc__almalinux__10.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__almalinux__10.sif SINGULARITY_PATH=/scratch_space/glide_JRx2Mt/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch_space/glide_JRx2Mt/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch_space/glide_JRx2Mt/execute/dir_2509732/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
102.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 117162092 9884988 107277104 9% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: huxley-head-new 1.789ms 1: huxley-head-new 1.651ms 2: 216.73.255.65 11.454ms 3: 216.73.255.1 0.891ms 4: nyc-9208-amnh.nysernet.net 2.362ms 5: nyc32-55a1-nyc32-9208.nysernet.net 1.855ms 6: syr-57c3-nyc32-55a1.nysernet.net 6.850ms 7: buf-55a1-syr-57c3.nysernet.net 9.936ms 8: buf-9208-buf-55a1.nysernet.net 10.482ms 9: i2-clev-buf-9208.nysernet.net 15.926ms 10: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 22.475ms asymm 12 11: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 21.315ms 12: 144.92.233.112 24.313ms 13: perfsonar02.hep.wisc.edu 23.316ms reached Resume: pmtu 1500 hops 13 back 14
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
AMNH-Mendel-CE1 2 hours ago 18
100.0%
38.9%
100.0%
38.9%
kernel env df firewall mtu nrp_images http_proxy
AMNH-Mendel-CE1 — probe output
pass kernel
4.18.0-147.el8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch_space/glide_Jbsh37/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch_space/glide_Jbsh37/execute/dir_1386102/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1386102 CONTAINERSW_FULL_VERSION=apptainer version 1.2.5+433-gca1e06012 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/local/software/Apptainer/apptainer-1.2.5/bin/apptainer CONTAINERSW_VERSION=1.2.5+433-gca1e06012 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch_space/glide_Jbsh37/cred.d/idtokens/credential_OSPool.main_OSG_US_AMNH-Mendel_osg-preempt.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=513137 GLIDEIN_Entry_Name=OSG_US_AMNH-Mendel_osg-preempt GLIDEIN_Entry_Signature=686bd6a9c7f047fc60ebf46dbde42fd38a33a92a GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=amnh-mendel-ce1.svc.osg-htc.org amnh-mendel-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_UUZsf8 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=osdf1.newy32aoa.net.internet2.edu:9002 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=AMNH-Mendel-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=AMNH GLIDEIN_Tmp_Dir=/scratch_space/glide_Jbsh37/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.2.5+433-gca1e06012 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/local/software/Apptainer/apptainer-1.2.5/bin/apptainer GWMS_CONTAINERSW_VERSION=1.2.5+433-gca1e06012 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/local/software/Apptainer/apptainer-1.2.5/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.2.5+433-gca1e06012 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=amnh-mendel-ce1.svc.osg-htc.org OSG_SITE_NAME=AMNH-Mendel-CE1 OSG_SQUID_LOCATION=osdf1.newy32aoa.net.internet2.edu:9002 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_8@glidein_4147243_154684272@mendel-c0013.mendel.sdmz.amnh.org$ "; unset PROMPT_COMMAND PS1=slot1_8@glidein_4147243_154684272@mendel-c0013.mendel.sdmz.amnh.org$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch_space/glide_Jbsh37/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.2.5+433-gca1e06012 SINGULARITY_IMAGES_DICT=default:/scratch_space/glide_Jbsh37/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/local/software/Apptainer/apptainer-1.2.5/bin/apptainer SINGULARITY_VERSION=1.2.5+433-gca1e06012 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch_space/glide_Jbsh37/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch_space/glide_Jbsh37/execute/dir_1386102/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
172.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 218102872 26283156 180671020 13% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: mendel-head.mendel.sdmz.amnh.org 0.055ms 1: mendel-head.mendel.sdmz.amnh.org 0.176ms 2: 216.73.255.65 27.879ms 3: 216.73.255.1 5.185ms 4: nyc-9208-amnh.nysernet.net 0.505ms 5: nyc32-55a1-nyc32-9208.nysernet.net 0.756ms 6: syr-57c3-nyc32-55a1.nysernet.net 6.558ms 7: buf-55a1-syr-57c3.nysernet.net 9.452ms 8: buf-9208-buf-55a1.nysernet.net 9.248ms 9: i2-clev-buf-9208.nysernet.net 15.250ms 10: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 21.720ms asymm 12 11: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 20.083ms 12: 144.92.233.112 22.659ms 13: perfsonar02.hep.wisc.edu 22.626ms reached Resume: pmtu 1500 hops 13 back 14
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy osdf1.newy32aoa.net.internet2.edu:9002
Alabama-CHPC-CE1 2 hours ago 50
100.0%
96.0%
100.0%
96.0%
kernel env df firewall mtu nrp_images http_proxy
Alabama-CHPC-CE1 — probe output
pass kernel
4.18.0-372.9.1.el8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_aMuMVn/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_aMuMVn/execute/dir_3739284/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3739284 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/tmp/glide_aMuMVn/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_aMuMVn/cred.d/idtokens/credential_OSPool.main_OSG_US_Alabama-CHPC-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6278911 GLIDEIN_Entry_Name=OSG_US_Alabama-CHPC-CE1 GLIDEIN_Entry_Signature=619cd9c16f33a56771b8816975dea692cec175a8 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=ua-hpc-ce1.svc.osg-htc.org ua-hpc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_OuOtpv GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Alabama-CHPC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-1.osg-htc.org GLIDEIN_Signature=fc3215af03f5a606519dd42bc9fed8d4e3456f69 GLIDEIN_Site=Alabama-CHPC GLIDEIN_Tmp_Dir=/tmp/glide_aMuMVn/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/tmp/glide_aMuMVn/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/tmp/glide_aMuMVn/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ua-hpc-ce1.svc.osg-htc.org OSG_SITE_NAME=Alabama-CHPC-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3702404_475032122@chpc-compute017.cm.cluster$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3702404_475032122@chpc-compute017.cm.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_aMuMVn/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/tmp/glide_aMuMVn/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/tmp/glide_aMuMVn/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_aMuMVn/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_aMuMVn/execute/dir_3739284/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
14069.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda7 15418689536 665317252 14753372284 5% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: chpc-head001.cm.cluster 0.075ms 1: chpc-head001.cm.cluster 0.072ms 2: 10.8.96.2 1.852ms 3: no reply 4: 192.168.9.17 2.784ms asymm 5 5: no reply 6: EDGE-ASR9K-PE-9.oit.ua.edu 2.774ms 7: 192.168.9.14 4.151ms 8: 192.168.13.252 4.338ms 9: TCL-ASRK9-R1-4001.oit.ua.edu 5.551ms 10: 130.160.15.233 13.803ms 11: core1-to-core2-re.56mar.sox.net 11.429ms 12: sidco-to-56mar-al2s.sox.net 14.577ms asymm 10 13: i2-to-sox-re.sidco.sox.net 15.483ms asymm 11 14: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 25.757ms asymm 18 15: fourhundredge-0-0-0-23.4079.core2.cinc.net.internet2.edu 26.953ms asymm 17 16: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 25.336ms 17: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 26.024ms asymm 15 18: 144.92.233.112 27.610ms asymm 16 19: perfsonar02.hep.wisc.edu 28.477ms reached Resume: pmtu 1500 hops 19 back 18
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
BNL-SDCC-SPOOLCE01 2 hours ago 4
100.0%
75.0%
100.0%
75.0%
kernel env df firewall mtu nrp_images http_proxy
BNL-SDCC-SPOOLCE01 — probe output
pass kernel
5.14.0-611.5.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/home/condor/local/sdcc/execute/dir_3388723/glide_wqS03I/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/home/condor/local/sdcc/execute/dir_3388723/glide_wqS03I/execute/dir_2845052/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2845052 CONTAINERSW_FULL_VERSION=apptainer version 1.4.4-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.4-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/home/condor/local/sdcc/execute/dir_3388723/glide_wqS03I/cred.d/idtokens/credential_OSPool.main_DUNE_US_BNL_spoolce01.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=479098 GLIDEIN_Country=US GLIDEIN_DUNESite=US_BNL GLIDEIN_Entry_Name=DUNE_US_BNL_spoolce01 GLIDEIN_Entry_Signature=388f8a74372ad7633f181d58a24768cc7852294a GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=spoolce01.sdcc.bnl.gov spoolce01.sdcc.bnl.gov:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_pwXBdm GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=BNL-SDCC-SPOOLCE01 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=BNL-SDCC GLIDEIN_Tmp_Dir=/home/condor/local/sdcc/execute/dir_3388723/glide_wqS03I/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.4-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.4-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.4-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=BNL-SDCC PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_3388748_512891105@spool1809.sdcc.bnl.gov$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_3388748_512891105@spool1809.sdcc.bnl.gov$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/home/condor/local/sdcc/execute/dir_3388723/glide_wqS03I/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.4-1.el9 SINGULARITY_IMAGES_DICT=default:/home/condor/local/sdcc/execute/dir_3388723/glide_wqS03I/images/htc__rocky__8.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.4-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/home/condor/local/sdcc/execute/dir_3388723/glide_wqS03I/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/home/condor/local/sdcc/execute/dir_3388723/glide_wqS03I/execute/dir_2845052/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
5982.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/sysvg-scratch 7048416776 419820940 6273300604 7% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.010ms pmtu 1500 1: ar0725-p1-l53.s186-0.sdcc.bnl.local 0.225ms 1: ar0725-p1-l53.s186-0.sdcc.bnl.local 0.154ms 2: 2620:0:210:e044::24 0.222ms 3: 2620:0:210:e043::231 0.351ms 4: fddd:7d33:c9b4:ff::51 0.335ms asymm 5 5: fddd:7d33:c9b4:ff::43 0.519ms 6: no reply 7: no reply 8: newy32aoa-cr6--wash-bb-e.igp.es.net 21.361ms asymm 16 9: wash-cr6--eqxdc4-bb-d.igp.es.net 21.320ms asymm 15 10: eqxdc4-cr6--eqxch2-bb-e.igp.es.net 21.375ms asymm 14 11: eqxch2-cr6--chic-bb-c.igp.es.net 21.317ms asymm 13 12: no reply 13: uwmadison_se-203--chic-cr6.es.net 38.792ms 14: perfsonar02.hep.wisc.edu 38.864ms reached Resume: pmtu 1500 hops 14 back 14
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
BNL-SDCC-SPOOLCE02 2 hours ago 4
100.0%
50.0%
100.0%
50.0%
kernel env df firewall mtu nrp_images http_proxy
BNL-SDCC-SPOOLCE02 — probe output
pass kernel
5.14.0-611.5.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/home/condor/local/sdcc/execute/dir_703737/glide_6LonVa/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/home/condor/local/sdcc/execute/dir_703737/glide_6LonVa/execute/dir_1891124/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1891124 CONTAINERSW_FULL_VERSION=apptainer version 1.4.4-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.4-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/home/condor/local/sdcc/execute/dir_703737/glide_6LonVa/cred.d/idtokens/credential_OSPool.main_DUNE_US_BNL_spoolce02.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=487726 GLIDEIN_Country=US GLIDEIN_DUNESite=US_BNL GLIDEIN_Entry_Name=DUNE_US_BNL_spoolce02 GLIDEIN_Entry_Signature=dae55b5bff86fe306a0c818760d4aff7445b5d83 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=spoolce02.sdcc.bnl.gov spoolce02.sdcc.bnl.gov:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_IGSEen GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=BNL-SDCC-SPOOLCE02 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=BNL-SDCC GLIDEIN_Tmp_Dir=/home/condor/local/sdcc/execute/dir_703737/glide_6LonVa/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.4-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.4-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.4-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=BNL-SDCC PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_703759_557250650@spool1587.sdcc.bnl.gov$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_703759_557250650@spool1587.sdcc.bnl.gov$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/home/condor/local/sdcc/execute/dir_703737/glide_6LonVa/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.4-1.el9 SINGULARITY_IMAGES_DICT=default:/home/condor/local/sdcc/execute/dir_703737/glide_6LonVa/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.4-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/home/condor/local/sdcc/execute/dir_703737/glide_6LonVa/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/home/condor/local/sdcc/execute/dir_703737/glide_6LonVa/execute/dir_1891124/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
6093.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/sysvg-scratch 7048416776 303822820 6389298724 5% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.007ms pmtu 1500 1: nr0725-p1-l43.s151-0.sdcc.bnl.gov 0.355ms 1: nr0725-p1-l43.s151-0.sdcc.bnl.gov 0.378ms 2: 2620:0:210:e044::4 0.239ms 3: 2620:0:210:e023::229 0.201ms 4: fddd:7d33:c9b4:ff::51 0.279ms asymm 5 5: fddd:7d33:c9b4:ff::42 0.541ms 6: no reply 7: no reply 8: newy32aoa-cr6--wash-bb-e.igp.es.net 21.303ms asymm 16 9: wash-cr6--eqxdc4-bb-d.igp.es.net 21.209ms asymm 15 10: eqxdc4-cr6--eqxch2-bb-e.igp.es.net 21.334ms asymm 14 11: eqxch2-cr6--chic-bb-c.igp.es.net 21.287ms asymm 13 12: no reply 13: uwmadison_se-203--chic-cr6.es.net 38.684ms 14: perfsonar02.hep.wisc.edu 38.617ms reached Resume: pmtu 1500 hops 14 back 14
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Boise-State-Borah-CE1 3 hours ago 16
0.0%
0.0%
0.0%
93.8%
all
Boise-State-Borah-CE1 — probe output
warning all
Job failed with exit code 0; no results transferred
CHTC 2 hours ago 331
100.0%
99.7%
100.0%
99.7%
kernel env df firewall mtu nrp_images http_proxy
CHTC — probe output
pass kernel
5.14.0-620.el9.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3,/mnt/alphafold,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/slot1/dir_2720193/scratch/glide_5RNQlI/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/slot1/dir_2720193/scratch/glide_5RNQlI/execute/dir_3957376/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3957376 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/slot1/dir_2720193/scratch/glide_5RNQlI/cred.d/idtokens/credential_OSPool.main_HCCHTPC_US_Wisconsin_osg01_rhel7.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=487796 GLIDEIN_Country=US GLIDEIN_DUNESite=US_Wisconsin GLIDEIN_Entry_Name=HCCHTPC_US_Wisconsin_osg01_rhel7 GLIDEIN_Entry_Signature=e6cece3fa26c72ba0a695db404c03941399c4714 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=osggrid01.hep.wisc.edu osggrid01.hep.wisc.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_slot1_12_c3cruF GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=osgsquid.hep.wisc.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=CHTC GLIDEIN_SE_VONAME_LOWERCASE=srm://cmssrm.hep.wisc.edu:8443/srm/v2/server?SFN=/hdfs/osg/vo/ GLIDEIN_SEs=cmssrm.hep.wisc.edu GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=CHTC GLIDEIN_Tmp_Dir=/var/lib/condor/execute/slot1/dir_2720193/scratch/glide_5RNQlI/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=osggrid01.hep.wisc.edu OSG_SITE_NAME=GLOW-OSG OSG_SQUID_LOCATION=osgsquid.hep.wisc.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_8@glidein_2720199_958329077@build4001.chtc.wisc.edu$ "; unset PROMPT_COMMAND PS1=slot1_8@glidein_2720199_958329077@build4001.chtc.wisc.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3,/mnt/alphafold,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/slot1/dir_2720193/scratch/glide_5RNQlI/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/slot1/dir_2720193/scratch/glide_5RNQlI/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/slot1/dir_2720193/scratch/glide_5RNQlI/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/slot1/dir_2720193/scratch/glide_5RNQlI/execute/dir_3957376/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1031.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p2 1272933648 126346592 1081851824 11% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.008ms pmtu 1500 1: 2607:f388:2200:100::2 0.848ms 1: 2607:f388:2200:100::2 0.873ms 2: 2607:f388:0:377::2 0.639ms 3: perfsonar02.hep.wisc.edu 0.503ms reached Resume: pmtu 1500 hops 3 back 3
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy osgsquid.hep.wisc.edu:3128
CHTC-DEV-PROD-EP-mwestphall 2 hours ago 1
100.0%
0.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
CHTC-DEV-PROD-EP-mwestphall — probe output
pass kernel
5.14.0-639.el9.aarch64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-Z0pEzN/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-Z0pEzN/execute/dir_7122 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=7122 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=CHTC PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_1@CHTC-DEV-PROD-EP-mwestphall.dcfa2cdd11eb$ "; unset PROMPT_COMMAND PS1=slot1_1@CHTC-DEV-PROD-EP-mwestphall.dcfa2cdd11eb$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-Z0pEzN/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-Z0pEzN/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-Z0pEzN/execute/dir_7122 _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass df
67.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 263112772 179123496 70551120 72% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.046ms 1: _gateway 0.063ms 2: 128.104.100.2 0.306ms 3: 146.151.164.1 0.452ms 4: perfsonar02.hep.wisc.edu 0.291ms reached Resume: pmtu 1500 hops 4 back 4
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
CHTC-Spark-CE1 2 hours ago 13
100.0%
61.5%
0.0%
61.5%
kernel env df firewall mtu nrp_images http_proxy
CHTC-Spark-CE1 — probe output
pass kernel
5.14.0-511.el9.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/osg01/glide_FmqiZl/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/osg01/glide_FmqiZl/execute/dir_1244054/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1244054 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/osg01/glide_FmqiZl/cred.d/idtokens/credential_OSPool.main_OSG_US_CHTC-Spark-CE1_pre.idtoken GLIDEIN_CPUS=20 GLIDEIN_ClusterId=6270990 GLIDEIN_Entry_Name=OSG_US_CHTC-Spark-CE1_pre GLIDEIN_Entry_Signature=93b848545807308a9a556ecd0370014514fe15ef GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=chtc-spark-ce1.svc.osg-htc.org chtc-spark-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_bDDQC9 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=CHTC-Spark-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=CHTC-Spark GLIDEIN_Tmp_Dir=/var/lib/condor/execute/osg01/glide_FmqiZl/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=chtc-spark-ce1.svc.osg-htc.org OSG_SITE_NAME=CHTC-Spark-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_8@glidein_772766_116569302@spark-a218.chtc.wisc.edu$ "; unset PROMPT_COMMAND PS1=slot1_8@glidein_772766_116569302@spark-a218.chtc.wisc.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/osg01/glide_FmqiZl/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/osg01/glide_FmqiZl/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/osg01/glide_FmqiZl/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/osg01/glide_FmqiZl/execute/dir_1244054/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1269.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p1 1537155984 127984556 1331014424 9% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 4092 1: _gateway 0.119ms 1: _gateway 0.128ms 2: _gateway 0.116ms pmtu 1500 2: 128.104.100.2 0.313ms 3: 146.151.164.1 0.352ms 4: perfsonar02.hep.wisc.edu 0.301ms reached Resume: pmtu 1500 hops 4 back 4
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Cincinnati-ARC-OSG-CE1 2 hours ago 8
100.0%
87.5%
0.0%
87.5%
kernel env df firewall mtu nrp_images http_proxy
Cincinnati-ARC-OSG-CE1 — probe output
pass kernel
4.18.0-553.30.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch-local/glide_0etiW3/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch-local/glide_0etiW3/execute/dir_917149/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=917149 CONTAINERSW_FULL_VERSION=apptainer version 1.3.6-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.3.6-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch-local/glide_0etiW3/cred.d/idtokens/credential_OSPool.main_OSG_US_Cincinnati-ARC-OSG-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=5888779 GLIDEIN_Entry_Name=OSG_US_Cincinnati-ARC-OSG-CE1 GLIDEIN_Entry_Signature=b3981ce21bc3c6030fc46b748198cb05b566588b GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=cincinnati-arc-osg-ce1.svc.opensciencegrid.org cincinnati-arc-osg-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_Z0Ykhk GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Cincinnati-ARC-OSG-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=Cincinnati-ARC-OSG GLIDEIN_Tmp_Dir=/scratch-local/glide_0etiW3/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.3.6-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.3.6-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.3.6-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=cincinnati-arc-osg-ce1.svc.opensciencegrid.org OSG_SITE_NAME=Cincinnati-ARC-OSG-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_3284103_159098324@compute-4.localdomain$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_3284103_159098324@compute-4.localdomain$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch-local/glide_0etiW3/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.3.6-1.el8 SINGULARITY_IMAGES_DICT=default:/scratch-local/glide_0etiW3/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.3.6-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch-local/glide_0etiW3/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch-local/glide_0etiW3/execute/dir_917149/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
367.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/localscratch-lv_local_scratch 460365864 51505188 385401860 12% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: arc-osg 0.228ms 1: arc-osg 0.247ms 2: msbb-00g95-hpc.manage.uc.edu 0.591ms 3: 10.0.26.17 0.736ms 4: 10.0.27.253 0.607ms 5: 10.0.0.2 1.266ms 6: 129.137.1.3 1.825ms 7: 129.137.1.5 2.678ms 8: cncne-r4-et-0-2-0s812.core.oar.net 2.348ms asymm 9 9: cncno-r5-et-4-3-0s735.core.oar.net 7.446ms asymm 8 10: hundredge-0-0-0-25.1260.core1.cinc.net.internet2.edu 5.212ms asymm 9 11: fourhundredge-0-0-0-23.4079.core2.cinc.net.internet2.edu 9.507ms asymm 14 12: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 8.551ms asymm 13 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 10.114ms asymm 12 14: 144.92.233.112 11.808ms asymm 13 15: perfsonar02.hep.wisc.edu 11.808ms reached Resume: pmtu 1500 hops 15 back 15
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Clemson-Palmetto-CE1 2 hours ago 289
100.0%
72.3%
0.0%
72.3%
kernel env df firewall mtu nrp_images http_proxy
Clemson-Palmetto-CE1 — probe output
pass kernel
4.18.0-553.123.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local_scratch/slurm.13437156/glide_7HKieJ/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/local_scratch/slurm.13437156/glide_7HKieJ/execute/dir_869351/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=869351 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local_scratch/slurm.13437156/glide_7HKieJ/cred.d/idtokens/credential_OSPool.main_OSG_US_Clemson-Palmetto-CE1.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=562337 GLIDEIN_Entry_Name=OSG_US_Clemson-Palmetto-CE1 GLIDEIN_Entry_Signature=4d19b3dbab56079b0398e260f57972e91d2d3300 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=clemson-palmetto-ce1.svc.opensciencegrid.org clemson-palmetto-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg001_Y7UZsB GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Clemson-Palmetto-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=Clemson-Palmetto GLIDEIN_Tmp_Dir=/local_scratch/slurm.13437156/glide_7HKieJ/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=clemson-palmetto-ce1.svc.opensciencegrid.org OSG_SITE_NAME=Clemson-Palmetto-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_32548_958590688@node2047.palmetto.clemson.edu$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_32548_958590688@node2047.palmetto.clemson.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/local_scratch/slurm.13437156/glide_7HKieJ/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/local_scratch/slurm.13437156/glide_7HKieJ/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local_scratch/slurm.13437156/glide_7HKieJ/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local_scratch/slurm.13437156/glide_7HKieJ/execute/dir_869351/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
2748.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sdb1 2927240708 44886748 2882353960 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: _gateway 14.870ms 1: _gateway 9.685ms 2: 205-186-62-81.generic.c-light.net 0.733ms 3: xe-1-1-1-816-t01-cle.culr.net 0.936ms 4: xe-1-1-1-816-t01-sox.culr.net 12.299ms asymm 9 5: 365dc-to-56mar-re.sox.net 17.393ms asymm 10 6: i2-to-sox-re.sidco.sox.net 18.631ms asymm 9 7: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 25.648ms asymm 12 8: fourhundredge-0-0-0-23.4079.core2.cinc.net.internet2.edu 27.070ms asymm 11 9: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 25.859ms asymm 10 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 25.520ms asymm 9 11: 144.92.233.112 27.350ms asymm 10 12: perfsonar02.hep.wisc.edu 27.332ms reached Resume: pmtu 9000 hops 12 back 12
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Colgate-CCARE-EP 2 hours ago 2
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy gpu
Colgate-CCARE-EP — probe output
pass kernel
6.8.0-111-generic
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-kLC8Ca/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-kLC8Ca/execute/dir_3769 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3769 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Colgate ITS OSG_SQUID_LOCATION=http://149.43.3.11:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_5@Colgate-CCARE-EP.4dfdac37bd89$ "; unset PROMPT_COMMAND PS1=slot1_5@Colgate-CCARE-EP.4dfdac37bd89$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-kLC8Ca/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-kLC8Ca/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-kLC8Ca/execute/dir_3769 _CONDOR_SLOT=slot1_5 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=http://149.43.3.11:3128
pass df
1095.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/ccare--g04--vg-ccare--g04--lv 1837860872 595243620 1149185240 35% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.080ms 1: _gateway 0.076ms 2: 149.43.3.1 0.739ms 3: 172.16.1.11 1.164ms 4: alb-9204-colgate.nysernet.net 3.791ms 5: syr-57c3-alb-9204.nysernet.net 7.023ms 6: buf-55a1-syr-57c3.nysernet.net 10.418ms 7: buf-9208-buf-55a1.nysernet.net 9.805ms 8: i2-clev-buf-9208.nysernet.net 15.020ms 9: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 22.881ms asymm 11 10: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 21.702ms 11: 144.92.233.112 23.441ms 12: perfsonar02.hep.wisc.edu 23.250ms reached Resume: pmtu 1500 hops 12 back 13
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://149.43.3.11:3128
pass gpu
Tue Jun 2 13:50:12 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA H100 80GB HBM3 Off | 00000000:4E:00.0 Off | 0 | | N/A 54C P0 344W / 700W | 1205MiB / 81559MiB | 76% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA H100 80GB HBM3 Off | 00000000:5F:00.0 Off | 0 | | N/A 52C P0 221W / 700W | 1151MiB / 81559MiB | 80% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA H100 80GB HBM3 Off | 00000000:CB:00.0 Off | 0 | | N/A 44C P0 70W / 700W | 5MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| +-----------------------------------------------------------------------------------------+
Duke-NCShare-CE1 2 hours ago 128
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
Duke-NCShare-CE1 — probe output
pass kernel
6.8.0-111-generic
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_NuytJY/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_NuytJY/execute/dir_4089016/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=4089016 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_NuytJY/cred.d/idtokens/credential_OSPool.main_OSG_US_Duke-NCShare-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6478865 GLIDEIN_Entry_Name=OSG_US_Duke-NCShare-CE1 GLIDEIN_Entry_Signature=ed193f3ce77ed1012379155adc72875631c54c67 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=duke-ncshare-ce1.svc.osg-htc.org duke-ncshare-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_nosg1_jDHFFh GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Duke-NCShare-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=Duke-NCShare GLIDEIN_Tmp_Dir=/scratch/glide_NuytJY/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=duke-ncshare-ce1.svc.osg-htc.org OSG_SITE_NAME=Duke-NCShare-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2531308_448859872@compute-11$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2531308_448859872@compute-11$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_NuytJY/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5 SINGULARITY_IMAGES_DICT=default:/scratch/glide_NuytJY/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_NuytJY/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_NuytJY/execute/dir_4089016/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
600.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/ubuntu--vg-lstor--lv 824572824 152518136 630095264 20% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.335ms asymm 2 1: _gateway 0.247ms asymm 2 2: rtp-gw-to-scidmz-gw-1.ncren.net 0.579ms 3: bundle-ether250.613.core1.rale.net.internet2.edu 2.506ms 4: fourhundredge-0-0-0-6.4079.core1.ashb.net.internet2.edu 23.473ms asymm 10 5: fourhundredge-0-0-0-1.4079.core1.clev.net.internet2.edu 21.466ms asymm 9 6: fourhundredge-0-0-0-2.4079.core1.eqch.net.internet2.edu 21.967ms asymm 8 7: fourhundredge-0-0-0-6.4079.core1.star.net.internet2.edu 22.630ms 8: 144.92.233.112 23.730ms 9: perfsonar02.hep.wisc.edu 23.743ms reached Resume: pmtu 1500 hops 9 back 10
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
FSU-HNPGRID 2 hours ago 56
100.0%
87.5%
100.0%
87.5%
kernel env df firewall mtu nrp_images http_proxy
FSU-HNPGRID — probe output
pass kernel
5.14.0-611.49.2.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_CcDQX1/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_CcDQX1/execute/dir_2954446/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2954446 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_CcDQX1/cred.d/idtokens/credential_OSPool.main_GLUEX_US_FSU-HNPGRID.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=5894273 GLIDEIN_Entry_Name=GLUEX_US_FSU-HNPGRID GLIDEIN_Entry_Signature=e1324744f59cfc80e7d09783677eff87f98548ea GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=fsu-hnpgrid-ce1.svc.osg-htc.org fsu-hnpgrid-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_I2Gv2v GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=7 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=FSU-HNPGRID GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=OSG_US_FSU_HNPGRID GLIDEIN_Tmp_Dir=/scratch/glide_CcDQX1/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=fsu-hnpgrid-ce1.svc.osg-htc.org OSG_SITE_NAME=FSU-HNPGRID PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_8@glidein_2834025_543945186@scigrid8.physics.fsu.edu$ "; unset PROMPT_COMMAND PS1=slot1_8@glidein_2834025_543945186@scigrid8.physics.fsu.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_CcDQX1/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/scratch/glide_CcDQX1/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_CcDQX1/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_CcDQX1/execute/dir_2954446/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1524.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda4 3899867640 2301289360 1598578280 60% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: vlan99.6506.ken.fsu.edu 0.931ms 1: vlan99.6506.ken.fsu.edu 0.857ms 2: no reply 3: no reply 4: no reply 5: irb100.rsb-br.net.fsu.edu 1.719ms 6: 128.186.253.197 1.644ms 7: 128.186.253.192 5.892ms 8: tlh-flrcore-asr9010-1-te0203-1902.net.flrnet.org 4.727ms 9: tlh-bb-rtr-2-fh0001-1.rtr.net.flrnet.org 8.455ms asymm 11 10: jax-bb-rtr-1-fh0001-1.rtr.net.flrnet.org 9.700ms 11: prov-i2-atla-renet-1070.net.flrnet.org 6.024ms 12: fourhundredge-0-0-0-6.4079.core1.atla.net.internet2.edu 35.191ms asymm 24 13: fourhundredge-0-0-0-23.4079.core2.atla.net.internet2.edu 34.360ms asymm 23 14: fourhundredge-0-0-0-2.4079.core2.nash.net.internet2.edu 35.491ms asymm 22 15: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 31.827ms asymm 21 16: fourhundredge-0-0-0-21.4079.core2.cinc.net.internet2.edu 30.683ms asymm 20 17: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 34.943ms asymm 19 18: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 32.351ms 19: 144.92.233.112 36.028ms 20: perfsonar02.hep.wisc.edu 32.617ms reached Resume: pmtu 1500 hops 20 back 21
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
GP-ARGO-ksu-backfill 17 mins ago 1
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
GP-ARGO-ksu-backfill — probe output
pass kernel
4.18.0-477.27.1.el8_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-YayizL/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-YayizL/execute/dir_3712674 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3712674 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Kansas State University OSG_SQUID_LOCATION=172.17.0.1:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_20@GP-ARGO-ksu-backfill.dedd04dbaf32$ "; unset PROMPT_COMMAND PS1=slot1_20@GP-ARGO-ksu-backfill.dedd04dbaf32$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-YayizL/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-YayizL/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el8 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-YayizL/execute/dir_3712674 _CONDOR_SLOT=slot1_20 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=172.17.0.1:3128
pass df
8876.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md0 9764216768 456381920 9307834848 5% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.145ms 1: _gateway 0.113ms 2: ptx-bb-ksu-hale-et0-0-18-0.bb.net.kanren.net 0.268ms 3: ptx-bb-ku-ellsworth-et0-0-0-0.bb.net.kanren.net 2.139ms 4: ptx-bb-ku-price-et0-0-0-0.bb.net.kanren.net 2.110ms 5: ptx-bb-kc-walnut-et0-0-0-0.bb.net.kanren.net 11.932ms 6: fourhundredge-0-0-0-10.2100.core2.kans.net.internet2.edu 7.952ms 7: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 15.923ms asymm 9 8: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 15.926ms 9: 144.92.233.112 17.921ms 10: perfsonar02.hep.wisc.edu 17.850ms reached Resume: pmtu 1500 hops 10 back 11
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 172.17.0.1:3128
GP-ARGO-mst-backfill 8 hours ago 1
100.0%
0.0%
100.0%
0.0%
kernel env df firewall mtu nrp_images http_proxy
GP-ARGO-mst-backfill — probe output
pass kernel
4.18.0-477.27.1.el8_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-GTTF1S/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-GTTF1S/execute/dir_754060 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=754060 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Missouri S&T OSG_SQUID_LOCATION=172.17.0.1:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_79@GP-ARGO-mst-backfill.411b14b8e638$ "; unset PROMPT_COMMAND PS1=slot1_79@GP-ARGO-mst-backfill.411b14b8e638$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-GTTF1S/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-GTTF1S/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el8 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-GTTF1S/execute/dir_754060 _CONDOR_SLOT=slot1_79 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=172.17.0.1:3128
pass df
8923.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md0 9764216768 407650956 9356565812 5% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.088ms 1: _gateway 0.104ms 2: itdc-nccs-100-vrfdata-rtr.network.mst.edu 1.144ms 3: 172.16.2.142 0.613ms 4: 172.16.0.86 0.381ms 5: 172.16.96.66 1.388ms 6: mntn-sl-he0-0-2-2-0.mo.more.net 4.344ms 7: mntn-co-he0-1-2-2-0.mo.more.net 6.953ms 8: mntn-kc-he0-1-2-2-0.mo.more.net 11.558ms 9: 150.199.90.66 15.171ms 10: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 20.750ms asymm 12 11: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 24.633ms 12: 144.92.233.112 22.949ms 13: perfsonar02.hep.wisc.edu 25.325ms reached Resume: pmtu 1500 hops 13 back 14
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 172.17.0.1:3128
GP-ARGO-semo-backfill 2 hours ago 1
100.0%
0.0%
100.0%
0.0%
kernel env df firewall mtu nrp_images http_proxy
GP-ARGO-semo-backfill — probe output
pass kernel
4.18.0-477.27.1.el8_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-ItrbVz/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-ItrbVz/execute/dir_858990 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=858990 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Southeast Missouri State University OSG_SQUID_LOCATION=172.17.0.1:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_63@GP-ARGO-semo-backfill.475310b2f955$ "; unset PROMPT_COMMAND PS1=slot1_63@GP-ARGO-semo-backfill.475310b2f955$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-ItrbVz/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-ItrbVz/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el8 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-ItrbVz/execute/dir_858990 _CONDOR_SLOT=slot1_63 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=172.17.0.1:3128
pass df
8913.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md0 9764216768 418302584 9345914184 5% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.090ms 1: _gateway 0.076ms 2: 150.201.179.254 0.184ms 3: 150.201.223.1 2.079ms asymm 4 4: sl-dist-02-te0-0-0-4-700110.mo.more.net 6.212ms 5: sl-core-01-be3.mo.more.net 6.066ms 6: co-core-01-he0-1-0-2.mo.more.net 10.889ms asymm 5 7: kc-core-01-he0-1-0-1.mo.more.net 13.931ms asymm 6 8: fourhundredge-0-0-0-10.1440.core2.kans.net.internet2.edu 16.947ms asymm 7 9: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 33.941ms asymm 10 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 33.947ms asymm 9 11: 144.92.233.112 28.936ms asymm 10 12: perfsonar02.hep.wisc.edu 26.884ms reached Resume: pmtu 1500 hops 12 back 12
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 172.17.0.1:3128
GP-ARGO-uams-backfill 7 hours ago 1
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
GP-ARGO-uams-backfill — probe output
pass kernel
4.18.0-477.27.1.el8_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-r067Fa/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-r067Fa/execute/dir_3169571 BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=University of Arkansas for Medical Sciences OSG_SQUID_LOCATION=172.17.0.1:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_79@GP-ARGO-uams-backfill.8493b95cc2d8$ "; unset PROMPT_COMMAND PS1=slot1_79@GP-ARGO-uams-backfill.8493b95cc2d8$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-r067Fa/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.3-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-r067Fa/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.3-1.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-r067Fa/execute/dir_3169571 _CONDOR_SLOT=slot1_79 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=172.17.0.1:3128
pass df
9000.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md0 9764216768 326130268 9438086500 4% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.093ms 1: _gateway 0.130ms 2: 144.30.235.3 0.252ms 3: no reply 4: 144.30.32.44 0.955ms 5: 144.30.32.34 10.821ms 6: no reply 7: 208.90.110.92 8.996ms 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: 144.92.233.112 38.508ms asymm 25 18: perfsonar02.hep.wisc.edu 46.821ms reached Resume: pmtu 1500 hops 18 back 25
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 172.17.0.1:3128
GSU-ACIDS 2 hours ago 13
30.8%
23.1%
0.0%
53.8%
kernel env df firewall mtu nrp_images http_proxy
GSU-ACIDS — probe output
pass kernel
6.8.0-111-generic
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/users/svc-osgaccnt/glide_TCNwfh/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/users/svc-osgaccnt/glide_TCNwfh/execute/dir_2932281/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2932281 CONTAINERSW_FULL_VERSION=apptainer version 1.4.1 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/sysapps/ubuntu-applications/apptainer/1.4.1/install/bin/apptainer CONTAINERSW_VERSION=1.4.1 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/users/svc-osgaccnt/glide_TCNwfh/cred.d/idtokens/credential_OSPool.main_OSG_US_GSU-ACIDS.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=5895324 GLIDEIN_Entry_Name=OSG_US_GSU-ACIDS GLIDEIN_Entry_Signature=dab8d1d6d6e571e995e9f4cdff935b7cb6d6b894 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=gsu-acids-ce1.svc.opensciencegrid.org gsu-acids-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_svc-osgaccnt_U1qFWP GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=arcosg04.rs.gsu.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=GSU-ACIDS GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=GSU-ACIDS GLIDEIN_Tmp_Dir=/scratch/users/svc-osgaccnt/glide_TCNwfh/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.1 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/sysapps/ubuntu-applications/apptainer/1.4.1/install/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.1 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/sysapps/ubuntu-applications/apptainer/1.4.1/install/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.1 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=gsu-acids-ce1.svc.opensciencegrid.org OSG_SITE_NAME=GSU-ACIDS OSG_SQUID_LOCATION=arcosg04.rs.gsu.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_25@glidein_2394071_148029435@acidscn011.rs.gsu.edu$ "; unset PROMPT_COMMAND PS1=slot1_25@glidein_2394071_148029435@acidscn011.rs.gsu.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/users/svc-osgaccnt/glide_TCNwfh/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.1 SINGULARITY_IMAGES_DICT=default:/scratch/users/svc-osgaccnt/glide_TCNwfh/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/sysapps/ubuntu-applications/apptainer/1.4.1/install/bin/apptainer SINGULARITY_VERSION=1.4.1 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/users/svc-osgaccnt/glide_TCNwfh/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/users/svc-osgaccnt/glide_TCNwfh/execute/dir_2932281/scratch _CONDOR_SLOT=slot1_25 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
193766.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on beegfs_nodev 270018342912 66839538688 203178804224 25% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.245.10.3 0.360ms 1: 10.245.10.2 0.362ms 2: rtcdfars-rtlsdc01rs-gw.gsu.edu 0.328ms 3: fwcdfxrs-rtcdfars-rtcdfdrs-gw.gsu.edu 0.147ms 4: fwcdfrs-rtcdfdrs.gsunet 0.714ms 5: rtcdfdedge-rtcdfdrs-gw.gsu.edu 0.336ms 6: SOX-10G-Secondary-TO-GSU.gsu.edu 2.221ms asymm 7 7: core1-to-core2-re.56mar.sox.net 1.669ms asymm 6 8: 365dc-to-56mar-re.sox.net 7.002ms asymm 7 9: i2-to-sox-re.sidco.sox.net 9.057ms asymm 8 10: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 20.063ms asymm 15 11: fourhundredge-0-0-0-22.4079.core2.cinc.net.internet2.edu 20.757ms asymm 14 12: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 19.988ms asymm 13 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 19.033ms asymm 12 14: 144.92.233.112 22.229ms asymm 13 15: perfsonar02.hep.wisc.edu 22.001ms reached Resume: pmtu 1500 hops 15 back 15
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy arcosg04.rs.gsu.edu:3128
GSU-Adonis-EP 2 hours ago 1
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy gpu
GSU-Adonis-EP — probe output
pass kernel
6.8.0-79-lowlatency
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-9pM6Br/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-9pM6Br/execute/dir_1924892 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1924892 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=GSU-Adonis OSG_SQUID_LOCATION=http://arcosg04.rs.gsu.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_4@GSU-Adonis-EP.osg-endpoint-74b7c7597f-7qdgv$ "; unset PROMPT_COMMAND PS1=slot1_4@GSU-Adonis-EP.osg-endpoint-74b7c7597f-7qdgv$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-9pM6Br/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-9pM6Br/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el8 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-9pM6Br/execute/dir_1924892 _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=http://arcosg04.rs.gsu.edu:3128
pass df
46.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on tmpfs 67108864 18250356 48858508 28% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1430 1?: [LOCALHOST] pmtu 1430 1: 10-245-19-162.ceph-csi-cephfs-nodeplugin-http-metrics.ceph-csi.svc.cluster.local 0.070ms 1: 10-245-19-162.prometheus-prometheus-node-exporter.monitoring.svc.cluster.local 0.054ms 2: 10.245.18.2 0.454ms 3: rtcdfdrs-rtlsdc02rs-gw.gsu.edu 0.574ms 4: fwcdfxrs-rtcdfars-rtcdfdrs-gw.gsu.edu 0.186ms 5: fwcdfrs-rtcdfdrs.gsunet 0.599ms 6: rtcdfaedge-rtcdfars-gw.gsu.edu 0.372ms 7: SOX-10G-Primary-TO-GSU.gsu.edu 2.846ms 8: 365dc-to-56mar-re.sox.net 7.279ms 9: sidco-to-56mar-al2s.sox.net 8.458ms asymm 8 10: i2-to-sox-re.sidco.sox.net 8.998ms asymm 9 11: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 20.761ms asymm 16 12: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 19.499ms asymm 14 13: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 18.656ms asymm 14 14: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 18.980ms asymm 13 15: perfsonar02.hep.wisc.edu 21.598ms reached Resume: pmtu 1430 hops 15 back 16
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://arcosg04.rs.gsu.edu:3128
pass gpu
Tue Jun 2 13:50:19 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA H100 NVL On | 00000000:3F:00.0 Off | 0 | | N/A 27C P0 58W / 400W | 4MiB / 95830MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
GWU-Viper-CE1 1 hour ago 41
100.0%
4.9%
0.0%
4.9%
kernel env df firewall mtu nrp_images http_proxy
GWU-Viper-CE1 — probe output
pass kernel
5.14.0-570.52.1.el9_6.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg/dir_2581268/glide_eTnqLh/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg/dir_2581268/glide_eTnqLh/execute/dir_579591/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=579591 CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg/dir_2581268/glide_eTnqLh/cred.d/idtokens/credential_OSPool.main_OSG_US_GWU-Viper-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=474353 GLIDEIN_Entry_Name=OSG_US_GWU-Viper-CE1 GLIDEIN_Entry_Signature=a36323c96457b31206a69b3ba1f4d191a42acfad GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=gwu-viper-ce1.svc.osg-htc.org gwu-viper-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_STJEO6 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=GWU-Viper-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=GWU-Viper GLIDEIN_Tmp_Dir=/scratch/osg/dir_2581268/glide_eTnqLh/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=gwu-viper-ce1.svc.osg-htc.org OSG_SITE_NAME=GWU-Viper-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_2581271_635136018@node007$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_2581271_635136018@node007$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg/dir_2581268/glide_eTnqLh/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.3-1.el9 SINGULARITY_IMAGES_DICT=default:/scratch/osg/dir_2581268/glide_eTnqLh/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.3-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg/dir_2581268/glide_eTnqLh/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg/dir_2581268/glide_eTnqLh/execute/dir_579591/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
854.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 960302824 15012500 896435864 2% /srv
fail firewall
Firewall blocked connections to port(s): [80, 8000, 8443] http://ospool-network-test.osg.chtc.io:80: FAILED (timed out) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: FAILED (timed out) http://ospool-network-test.osg.chtc.io:8443: FAILED (timed out) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.008ms pmtu 1500 1: 2620:118:500f:2b8a::1 2.860ms 1: 2620:118:500f:2b8a::1 3.648ms 2: no reply 3: no reply 4: no reply 5: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 15.600ms asymm 6 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Hawaii-Koa-CE1 2 hours ago 270
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
Hawaii-Koa-CE1 — probe output
pass kernel
5.14.0-611.54.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/mnt/nfs/scratch/osg01/glide_NoKe7d/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/mnt/nfs/scratch/osg01/glide_NoKe7d/execute/dir_3235793/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3235793 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/mnt/nfs/scratch/osg01/glide_NoKe7d/cred.d/idtokens/credential_OSPool.main_OSG_US_Hawaii-Koa-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6278154 GLIDEIN_Entry_Name=OSG_US_Hawaii-Koa-CE1 GLIDEIN_Entry_Signature=628462b6e0a42565bd5e2d5088ccd76f20a7ebfc GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=hawaii-koa-ce1.svc.osg-htc.org hawaii-koa-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_4Q7m1z GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Hawaii-Koa-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-1.osg-htc.org GLIDEIN_Signature=fc3215af03f5a606519dd42bc9fed8d4e3456f69 GLIDEIN_Site=Hawaii-Koa GLIDEIN_Tmp_Dir=/mnt/nfs/scratch/osg01/glide_NoKe7d/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=hawaii-koa-ce1.svc.osg-htc.org OSG_SITE_NAME=Hawaii-Koa-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3465548_383230704@cn-15-21-01.mana.ci.its.hawaii.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3465548_383230704@cn-15-21-01.mana.ci.its.hawaii.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/mnt/nfs/scratch/osg01/glide_NoKe7d/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/mnt/nfs/scratch/osg01/glide_NoKe7d/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/mnt/nfs/scratch/osg01/glide_NoKe7d/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/mnt/nfs/scratch/osg01/glide_NoKe7d/execute/dir_3235793/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
9294.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 10.64.0.99:/datastore/scratch/osg01/glide_NoKe7d/execute/dir_3235793 10585027584 839273472 9745754112 8% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: _gateway 0.078ms 1: _gateway 0.091ms 2: et-1-2-0-2500-scidmz-re0.uhnet.net 0.205ms 3: et-0-2-0-35-ohelo-re0.uhnet.net 0.204ms 4: internet2-uhm-losa.uhnet.net 47.626ms 5: fourhundredge-0-0-0-2.4079.core2.salt.net.internet2.edu 95.162ms asymm 13 6: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 93.393ms asymm 12 7: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 103.473ms asymm 11 8: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 94.400ms asymm 10 9: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 94.183ms 10: 144.92.233.112 96.146ms 11: perfsonar02.hep.wisc.edu 96.106ms reached Resume: pmtu 9000 hops 11 back 12
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
IRISHEP-SSL-RIVER-DEV-BACKFILL 2 hours ago 63
100.0%
96.8%
100.0%
96.8%
kernel env df firewall mtu nrp_images http_proxy
IRISHEP-SSL-RIVER-DEV-BACKFILL — probe output
pass kernel
5.14.0-611.47.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-Vp6ElS/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-Vp6ElS/execute/dir_699651/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=699651 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=IRISHEP-SSL-UCHICAGO OSG_SQUID_LOCATION=squid.grid.uchicago.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_10@IRISHEP-SSL-RIVER-DEV-BACKFILL.river-c055.ssl-hep.org$ "; unset PROMPT_COMMAND PS1=slot1_10@IRISHEP-SSL-RIVER-DEV-BACKFILL.river-c055.ssl-hep.org$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/pilot/osgvo-pilot-Vp6ElS/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-Vp6ElS/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-Vp6ElS/execute/dir_699651/scratch _CONDOR_SLOT=slot1_10 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=squid.grid.uchicago.edu:3128
pass df
352.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 765870024 356807480 370085060 50% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.012ms pmtu 1500 1: 2605:9a00:10:200c::2 0.241ms 1: 2605:9a00:10:200c::2 0.159ms 2: a08-s004-u3-s250b-to-a08-s004-u3-s251b.p2p.uchicago.net 0.199ms asymm 1 3: a08-s004-u3-s251b-to-h01-391-s500a.p2p.uchicago.net 0.342ms asymm 2 4: uwmadison_se-203--chic-cr6.es.net 18.170ms 5: perfsonar02.hep.wisc.edu 18.075ms reached Resume: pmtu 1500 hops 5 back 5
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy squid.grid.uchicago.edu:3128
ISI 1 hour ago 3
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
ISI — probe output
pass kernel
4.18.0-553.124.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_671964/scratch/glide_1Y7jIM/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_671964/scratch/glide_1Y7jIM/execute/dir_1248898/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1248898 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_671964/scratch/glide_1Y7jIM/cred.d/idtokens/credential_OSPool.main_OSG_US_ISI_osg.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=5785009 GLIDEIN_Country=US GLIDEIN_Entry_Name=OSG_US_ISI_osg GLIDEIN_Entry_Signature=41602db45fec7bd0961d74f7d128392b0449bfc4 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=osg.isi.edu osg.isi.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_lklpXY GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=workflow.isi.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=ISI GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=ISI GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_671964/scratch/glide_1Y7jIM/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=osg.isi.edu OSG_SITE_NAME=ISI OSG_SQUID_LOCATION=workflow.isi.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_671968_405777166@compute-2.isi.edu$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_671968_405777166@compute-2.isi.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_671964/scratch/glide_1Y7jIM/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/dir_671964/scratch/glide_1Y7jIM/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_671964/scratch/glide_1Y7jIM/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_671964/scratch/glide_1Y7jIM/execute/dir_1248898/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=workflow.isi.edu:3128
pass df
75.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/rl-root 101697536 22670480 79027056 23% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.009ms pmtu 1500 1: 2606:d5c0:0:1::1 8.280ms 1: 2606:d5c0:0:1::1 8.072ms 2: 2606:d5c0::1 8.221ms 3: 2001:1878:15::1 8.871ms 4: lax-agg10--losa4-agg-01--400g--01.cenic.net 1.965ms 5: lax-agg10-hpr--lax-agg10-dc.cenic.net 8.930ms 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy workflow.isi.edu:3128
IU-Jetstream2-Backfill 2 hours ago 49
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
IU-Jetstream2-Backfill — probe output
pass kernel
6.12.85+deb13-cloud-amd64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3 APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-eBYqyR/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-eBYqyR/execute/dir_8724/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=8724 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Pervasive Technology Institute PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_1@IU-Jetstream2-Backfill.blue-68b8958644-hskt8$ "; unset PROMPT_COMMAND PS1=slot1_1@IU-Jetstream2-Backfill.blue-68b8958644-hskt8$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3 SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/pilot/osgvo-pilot-eBYqyR/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-eBYqyR/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-eBYqyR/execute/dir_8724/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
170.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 206376368 19050824 178899920 10% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=8850 1?: [LOCALHOST] pmtu 8850 1: _gateway 0.082ms 1: _gateway 0.036ms 2: host-192-168-0-1.js2local 0.843ms 3: host-192-168-3-22.js2local 0.949ms asymm 2 4: 149.165.159.1 0.311ms asymm 3 5: 172.16.11.140 0.488ms asymm 4 6: bl-nas2.hps.iu.edu 0.615ms asymm 5 7: 10.9.213.8 1.782ms asymm 6 8: 10.9.213.9 2.043ms asymm 6 9: 149.165.254.249 2.390ms asymm 7 10: ae-7.1.rtr2.chic.indiana.gigapop.net 6.015ms asymm 9 11: 144.92.233.118 23.194ms asymm 10 12: 144.92.233.118 22.875ms asymm 10 13: perfsonar02.hep.wisc.edu 23.117ms reached Resume: pmtu 8850 hops 13 back 11
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
JSU-Tigers-CE1 57 mins ago 2
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
JSU-Tigers-CE1 — probe output
pass kernel
5.14.0-701.el9.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_1407446/glide_QgPMsN/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_1407446/glide_QgPMsN/execute/dir_1784986/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1784986 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_1407446/glide_QgPMsN/cred.d/idtokens/credential_OSPool.main_OSG_US_JSU-Tigers-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=479230 GLIDEIN_Entry_Name=OSG_US_JSU-Tigers-CE1 GLIDEIN_Entry_Signature=d1213d2f8ceb7151a80b00c4d7204c5a0af34f12 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=jsu-tigers-ce1.svc.opensciencegrid.org jsu-tigers-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_6Z8WNk GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=JSU-Tigers-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=JSU-Tigers GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_1407446/glide_QgPMsN/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=jsu-tigers-ce1.svc.opensciencegrid.org OSG_SITE_NAME=JSU-Tigers-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_1407448_288271392@chtc.ep0.jsums.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_1407448_288271392@chtc.ep0.jsums.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_1407446/glide_QgPMsN/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/dir_1407446/glide_QgPMsN/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_1407446/glide_QgPMsN/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_1407446/glide_QgPMsN/execute/dir_1784986/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1673.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/condor_vg-condor_lv 1844816532 3787556 1755261176 1% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.349ms 1: _gateway 0.360ms 2: 173.235.100.96 5.563ms 3: 173.235.100.17 16.279ms 4: fourhundredge-0-0-0-21.4079.core2.atla.net.internet2.edu 33.954ms asymm 14 5: fourhundredge-0-0-0-2.4079.core2.nash.net.internet2.edu 32.041ms asymm 13 6: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 33.990ms asymm 12 7: fourhundredge-0-0-0-21.4079.core2.cinc.net.internet2.edu 32.624ms asymm 11 8: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 33.213ms asymm 10 9: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 33.118ms 10: 144.92.233.112 34.464ms 11: perfsonar02.hep.wisc.edu 34.535ms reached Resume: pmtu 1500 hops 11 back 12
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
KENT-STATE-RESEARCH-EP 2 hours ago 4
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
KENT-STATE-RESEARCH-EP — probe output
pass kernel
5.14.0-710.el9.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-hfovcr/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-hfovcr/execute/dir_78043 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=78043 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Kent_State_Research PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_66@KENT-STATE-RESEARCH-EP.2f4b871634f4$ "; unset PROMPT_COMMAND PS1=slot1_66@KENT-STATE-RESEARCH-EP.2f4b871634f4$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-hfovcr/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-hfovcr/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-hfovcr/execute/dir_78043 _CONDOR_SLOT=slot1_66 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass df
41558.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg--main-home 45986104216 100850600 43576692312 1% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.072ms 1: _gateway 0.130ms 2: gw2.research.cs.kent.edu 0.583ms 3: 10.255.255.1 0.696ms 4: akrnq-r4-et-0-2-1s3112.core.oar.net 0.772ms 5: athnh-r5-et-1-0-0s100.bb.oar.net 7.265ms 6: schrd-r4-et-2-1-0s100.core.oar.net 9.014ms 7: schrd-r5-et-2-0-4s100.core.oar.net 9.131ms 8: dytnq-r4-et-2-1-0s100.core.oar.net 11.346ms 9: cncno-r5-et-4-3-0s735.core.oar.net 12.794ms 10: hundredge-0-0-0-25.1260.core1.cinc.net.internet2.edu 12.761ms 11: fourhundredge-0-0-0-21.4079.core2.cinc.net.internet2.edu 14.843ms asymm 12 12: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 16.130ms asymm 11 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 16.662ms asymm 10 14: 144.92.233.112 18.371ms asymm 11 15: perfsonar02.hep.wisc.edu 18.240ms reached Resume: pmtu 1500 hops 15 back 13
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
KSU-Beocat-CE1 2 hours ago 176
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
KSU-Beocat-CE1 — probe output
pass kernel
5.14.0-611.54.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_xMAFm4/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/tmp/glide_xMAFm4/execute/dir_3859832/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3859832 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_xMAFm4/cred.d/idtokens/credential_OSPool.main_OSG_US_KSU-Beocat-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=487451 GLIDEIN_Entry_Name=OSG_US_KSU-Beocat-CE1 GLIDEIN_Entry_Signature=96136aa60d5cfbb9d49f1bccc4375dce4e36349c GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=ksu-beocat-ce1.svc.opensciencegrid.org ksu-beocat-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgsubmit_GPO8ix GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=9 GLIDEIN_Proxy_URL=squid.beocat.ksu.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=KSU-Beocat-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=BEOCAT-SLATE GLIDEIN_Tmp_Dir=/tmp/glide_xMAFm4/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ksu-beocat-ce1.svc.opensciencegrid.org OSG_SITE_NAME=KSU-Beocat-CE1 OSG_SQUID_LOCATION=squid.beocat.ksu.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_2361423_281106844@wizard26.beocat.ksu.edu$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_2361423_281106844@wizard26.beocat.ksu.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_xMAFm4/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/tmp/glide_xMAFm4/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_xMAFm4/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_xMAFm4/execute/dir_3859832/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
884.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg00-root 968898324 42001284 926897040 5% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: no reply 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 9000 Resume: pmtu 9000
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy squid.beocat.ksu.edu:3128
LIGO-WA-CE1 2 hours ago 25
100.0%
96.0%
100.0%
96.0%
kernel env df firewall mtu nrp_images http_proxy
LIGO-WA-CE1 — probe output
pass kernel
4.18.0-553.124.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=rocky:9 APPTAINER_TMPDIR=/var/lib/condor/execute/dir_345730/scratch/glide_xhWuNz/execute/dir_1883991/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1883991 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=73f2f314552aa3bd77005cafe714824f2143942e GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=0b131f1ff6ec513a999c687e746e08d86c967842 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_345730/scratch/glide_xhWuNz/cred.d/idtokens/credential_OSPool.main_LIGO_US_Caltech_ldas_grid-wa_4C.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=5772568 GLIDEIN_Entry_Name=LIGO_US_Caltech_ldas_grid-wa_4C GLIDEIN_Entry_Signature=0e98c922e14010922a3f67409ca072df14c7639a GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=ligo-hanford-ce1.svc.osg-htc.org ligo-hanford-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_4XLHZi GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=frontier.dcs.ligo-wa.caltech.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=LIGO-WA-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=bd7b1d5942e06bfef9a3321d3b9cf31ba1e84379 GLIDEIN_Site=LIGO-WA GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_345730/scratch/glide_xhWuNz/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ligo-hanford-ce1.svc.osg-htc.org OSG_SITE_NAME=LIGO-WA-CE1 OSG_SQUID_LOCATION=frontier.dcs.ligo-wa.caltech.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_345783_125458176@node476.dcs.ligo-wa.caltech.edu$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_345783_125458176@node476.dcs.ligo-wa.caltech.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_MODE=unprivileged SINGULARITY_NAME=rocky:9 SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_345730/scratch/glide_xhWuNz/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_345730/scratch/glide_xhWuNz/execute/dir_1883991/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
64.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/node476-slot1_1+2738_664 72195192 4287100 67891708 6% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: pluto.dcs.ligo-wa.caltech.edu 0.326ms 1: pluto.dcs.ligo-wa.caltech.edu 0.284ms 2: 198.129.208.1 0.416ms asymm 3 3: no reply 4: bois-cr6--salt-bb-c.igp.es.net 43.652ms asymm 13 5: salt-cr6--denv-bb-b.igp.es.net 43.652ms asymm 12 6: denv-cr6--kans-bb-e.igp.es.net 43.642ms asymm 11 7: kans-cr6--chic-bb-d.igp.es.net 43.626ms asymm 10 8: no reply 9: uwmadison_se-377--chic-cr6.es.net 61.109ms asymm 10 10: perfsonar02.hep.wisc.edu 60.896ms reached Resume: pmtu 9000 hops 10 back 11
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy frontier.dcs.ligo-wa.caltech.edu:3128
Lafayette-Firebird-CE1 2 hours ago 33
100.0%
97.0%
0.0%
97.0%
kernel env df firewall mtu nrp_images http_proxy
Lafayette-Firebird-CE1 — probe output
pass kernel
5.14.0-611.55.1.el9_7.0.3.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg01/glide_yd8WmS/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg01/glide_yd8WmS/execute/dir_959551/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=959551 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01/glide_yd8WmS/cred.d/idtokens/credential_OSPool.main_OSG_US_Lafayette-firebird-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=5888110 GLIDEIN_Entry_Name=OSG_US_Lafayette-firebird-CE1 GLIDEIN_Entry_Signature=43ba23e5d8b2b0500e3d61b421613cb2409b03d5 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=lafayette-firebird-ce1.svc.osg-htc.org lafayette-firebird-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_yB7hdQ GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Lafayette-Firebird-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=Lafayette-Firebird GLIDEIN_Tmp_Dir=/scratch/osg01/glide_yd8WmS/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=lafayette-firebird-ce1.svc.osg-htc.org OSG_SITE_NAME=Lafayette-Firebird-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_969419_23453752@node01.cluster$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_969419_23453752@node01.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg01/glide_yd8WmS/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/scratch/osg01/glide_yd8WmS/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01/glide_yd8WmS/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01/glide_yd8WmS/execute/dir_959551/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
6991.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/disk0-scratch 7439005696 108356184 7330649512 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: firebird.cluster 1.620ms 1: firebird.cluster 1.392ms 2: fw-comsvc-colo.lafayette.edu 1.716ms 3: 139.147.1.113 2.017ms asymm 4 4: 172.31.0.33 1.789ms 5: 172.31.0.3 2.605ms 6: gw0-g0-3-ibgp.prd.lafayette.edu 1.724ms asymm 5 7: 164.153.140.160 4.999ms asymm 6 8: hundredge-0-0-0-28.78.core1.phil.net.internet2.edu 6.111ms asymm 7 9: fourhundredge-0-0-0-2.4079.core2.ashb.net.internet2.edu 24.134ms asymm 14 10: fourhundredge-0-0-0-1.4079.core2.clev.net.internet2.edu 23.439ms asymm 13 11: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 24.977ms asymm 12 12: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 22.911ms asymm 11 13: 144.92.233.112 25.139ms asymm 12 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Langston-Lion-CE1 2 hours ago 2
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
Langston-Lion-CE1 — probe output
pass kernel
5.14.0-570.49.1.el9_6.x86_64
pass env
APPTAINER_CACHEDIR=/srv/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=889761 CONTAINERSW_FULL_VERSION=singularity-ce version 4.3.3-3.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/singularity CONTAINERSW_VERSION=4.3.3-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osgccre/glide_BplIei/cred.d/idtokens/credential_OSPool.main_OSG_US_Langston-Lion-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=402669 GLIDEIN_Entry_Name=OSG_US_Langston-Lion-CE1 GLIDEIN_Entry_Signature=c7c22300fec613938f11b7c8759dd95eaad5effb GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=langston-lion-ce1.svc.osg-htc.org langston-lion-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgccre_OgR30K GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Langston-Lion-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=4a553cb760eb0536cf01a0e964943d71a876fc26 GLIDEIN_Site=Langston-Lion GLIDEIN_Tmp_Dir=/scratch/osgccre/glide_BplIei/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity-ce version 4.3.3-3.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/singularity GWMS_CONTAINERSW_VERSION=4.3.3-3.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=4.3.3-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=langston-lion-ce1.svc.osg-htc.org OSG_SITE_NAME=Langston-Lion-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_775021_559841816@c000.ccre.langston.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_775021_559841816@c000.ccre.langston.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/scratch/osgccre/glide_BplIei/execute/dir_889761:/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/scratch/osgccre/glide_BplIei/images/htc__debian__13.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity-ce version 4.3.3-3.el9 SINGULARITY_IMAGES_DICT=default:/scratch/osgccre/glide_BplIei/images/htc__debian__13.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__debian__13.sif SINGULARITY_PATH=/usr/bin/singularity SINGULARITY_TMPDIR=/scratch/osgccre/glide_BplIei/execute/dir_889761/scratch SINGULARITY_VERSION=4.3.3-3.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osgccre/glide_BplIei/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osgccre/glide_BplIei/execute/dir_889761/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
884.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 936705024 9510412 927194612 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: lionvmhost.ccre.langston.edu 0.150ms 1: lionvmhost.ccre.langston.edu 0.150ms 2: 164.58.29.113 0.337ms 3: 198.71.46.45 2.079ms asymm 4 4: fourhundredge-0-0-0-9.302.core1.tuls.net.internet2.edu 5.262ms 5: fourhundredge-0-0-0-2.4079.core1.kans.net.internet2.edu 19.059ms asymm 11 6: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 18.463ms asymm 10 7: fourhundredge-0-0-0-22.4079.core1.chic.net.internet2.edu 18.239ms asymm 9 8: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 20.268ms 9: 144.92.233.112 20.534ms 10: perfsonar02.hep.wisc.edu 20.511ms reached Resume: pmtu 1500 hops 10 back 11
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Lehigh_-_Hawk 2 hours ago 147
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
Lehigh_-_Hawk — probe output
pass kernel
5.14.0-611.55.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg/13790575/glide_48wiK5/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg/13790575/glide_48wiK5/execute/dir_359645/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=359645 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg/13790575/glide_48wiK5/cred.d/idtokens/credential_OSPool.main_OSG_US_Lehigh-Hawk_osg-hawk-ce.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6368066 GLIDEIN_Country=US GLIDEIN_Entry_Name=OSG_US_Lehigh-Hawk_osg-hawk-ce GLIDEIN_Entry_Signature=cc5289b967c79eedd4d4bd2318ed46ddd45c37c6 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=osg-hawk-ce.cc.lehigh.edu osg-hawk-ce.cc.lehigh.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_OgFiIO GLIDEIN_MaxMemMBs=7000 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=osg-squid.cc.lehigh.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Lehigh - Hawk GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=Lehigh - Hawk GLIDEIN_Tmp_Dir=/scratch/osg/13790575/glide_48wiK5/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv HTTPS_PROXY=osg-squid.cc.lehigh.edu:3128 JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=osg-hawk-ce.cc.lehigh.edu OSG_SITE_NAME=LEHIGH-HAWK-CE OSG_SQUID_LOCATION=osg-squid.cc.lehigh.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_4156180_750367800@hawk-a704.cc.lehigh.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_4156180_750367800@hawk-a704.cc.lehigh.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg/13790575/glide_48wiK5/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/scratch/osg/13790575/glide_48wiK5/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TERM=linux TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg/13790575/glide_48wiK5/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg/13790575/glide_48wiK5/execute/dir_359645/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=osg-squid.cc.lehigh.edu:3128
pass df
629.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/almalinux-scratch 884349220 178935224 660417800 22% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 192.168.7.251 0.548ms 1: 192.168.7.251 0.377ms 2: borderfw-e.cc.lehigh.edu 0.070ms 3: ewfm35-out.cc.lehigh.edu 0.166ms 4: pack34-out.cc.lehigh.edu 0.147ms asymm 3 5: 164.153.140.133 2.798ms asymm 4 6: hundredge-0-0-0-28.78.core1.phil.net.internet2.edu 5.958ms asymm 5 7: fourhundredge-0-0-0-2.4079.core2.ashb.net.internet2.edu 21.763ms asymm 12 8: fourhundredge-0-0-0-1.4079.core2.clev.net.internet2.edu 20.363ms asymm 11 9: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 22.394ms asymm 10 10: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 20.530ms asymm 9 11: 144.92.233.112 22.983ms asymm 10 12: perfsonar02.hep.wisc.edu 22.804ms reached Resume: pmtu 1500 hops 12 back 12
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy osg-squid.cc.lehigh.edu:3128
MI-HORUS-CE2 2 hours ago 764
99.9%
20.7%
0.3%
20.7%
kernel env df firewall mtu nrp_images http_proxy
MI-HORUS-CE2 — probe output
pass kernel
5.14.0-611.49.2.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_666Vp6/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_666Vp6/execute/dir_1150064/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1150064 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_666Vp6/cred.d/idtokens/credential_OSPool.main_OSG_US_MI-HORUS-CE2_lg.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=568986 GLIDEIN_Entry_Name=OSG_US_MI-HORUS-CE2_lg GLIDEIN_Entry_Signature=429b53a4cb54d0b6bb0c52f0abd2f009e273ffa1 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=mi-horus-ce2.svc.osg-htc.org mi-horus-ce2.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_F7qkKN GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=MI-HORUS-CE2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=4a553cb760eb0536cf01a0e964943d71a876fc26 GLIDEIN_Site=MI-HORUS GLIDEIN_Tmp_Dir=/scratch/glide_666Vp6/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=mi-horus-ce2.svc.osg-htc.org OSG_SITE_NAME=MI-HORUS-CE2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_4128594_94535504@wsu-lg03.osris.org$ "; unset PROMPT_COMMAND PS1=slot1@glidein_4128594_94535504@wsu-lg03.osris.org$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_666Vp6/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el9 SINGULARITY_IMAGES_DICT=default:/scratch/glide_666Vp6/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_666Vp6/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_666Vp6/execute/dir_1150064/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
375.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_scratch-scratch 937231232 543490984 393740248 58% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: wsu-rtr-fe.osris.org 0.668ms 1: wsu-rtr-fe.osris.org 0.476ms 2: 10.0.8.17 0.356ms 3: 10.0.8.5 0.268ms 4: ae1x160.dtrt-wsudc-c1.mich.net 0.276ms 5: et-3-1-0x3.gdrp-eberhard-c1.mich.net 3.926ms 6: ae100x0.gdrp-gvsu-eberhard-c1.mich.net 3.664ms 7: ae102x0.chcg-lumen-600w-c1.mich.net 6.953ms asymm 8 8: 192.122.182.34 8.464ms asymm 9 9: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 10.888ms 10: 144.92.233.112 10.990ms 11: perfsonar02.hep.wisc.edu 11.153ms reached Resume: pmtu 1500 hops 11 back 12
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
MTState-Tempest-CE1 2 hours ago 459
98.3%
61.7%
0.0%
65.1%
kernel env df firewall mtu nrp_images http_proxy gpu
MTState-Tempest-CE1 — probe output
pass kernel
4.18.0-553.89.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/jobs/4032980/glide_Hp4H28/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/jobs/4032980/glide_Hp4H28/execute/dir_304524/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=304524 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/jobs/4032980/glide_Hp4H28/cred.d/idtokens/credential_OSPool.main_OSG_US_MTState-Tempest-CE1.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=6362114 GLIDEIN_Entry_Name=OSG_US_MTState-Tempest-CE1 GLIDEIN_Entry_Signature=ab8e78f4bb4a0396cb711358bfcebfcd5cfa987d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=mtstate-tempest-ce1.svc.opensciencegrid.org mtstate-tempest-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_tM18Vh GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=MTState-Tempest-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=MTState-Tempest GLIDEIN_Tmp_Dir=/tmp/jobs/4032980/glide_Hp4H28/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=mtstate-tempest-ce1.svc.opensciencegrid.org OSG_SITE_NAME=MTState-Tempest-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_7@glidein_2242843_463501108@epyc023.rci.montana.edu$ "; unset PROMPT_COMMAND PS1=slot1_7@glidein_2242843_463501108@epyc023.rci.montana.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/jobs/4032980/glide_Hp4H28/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/tmp/jobs/4032980/glide_Hp4H28/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/jobs/4032980/glide_Hp4H28/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/jobs/4032980/glide_Hp4H28/execute/dir_304524/scratch _CONDOR_SLOT=slot1_7 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
666.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda 937234648 238779148 698455500 26% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: _gateway 0.219ms 1: _gateway 0.212ms 2: hsrp203.252.msu.montana.edu 0.929ms 3: no reply 4: rnedge-prodfw.msu.montana.edu 1.406ms 5: ae13--539.icar-sttl1-2.infra.pnw-gigapop.net 15.821ms 6: lo0--210.icar-sttl1-1.infra.pnw-gigapop.net 15.947ms 7: 209.124.181.167 17.999ms 8: fourhundredge-0-0-0-16.4079.core1.seat.net.internet2.edu 48.944ms asymm 12 9: fourhundredge-0-0-0-0.4079.core1.salt.net.internet2.edu 46.810ms asymm 11 10: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 46.893ms 11: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 47.797ms asymm 9 12: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 49.572ms asymm 8 13: fourhundredge-0-0-0-23.4079.core1.chic.net.internet2.edu 47.423ms asymm 7 14: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 47.402ms asymm 6 15: 144.92.233.112 49.322ms asymm 7 16: perfsonar02.hep.wisc.edu 49.582ms reached Resume: pmtu 9000 hops 16 back 9
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass gpu
Tue Jun 2 07:50:10 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA A100-SXM4-40GB Off | 00000000:41:00.0 Off | 0 | | N/A 27C P0 52W / 400W | 4MiB / 40960MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
MWT2 2 hours ago 180
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
MWT2 — probe output
pass kernel
5.14.0-611.54.3.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_2863529/glide_v4xBqA/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_2863529/glide_v4xBqA/execute/dir_1657112/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1657112 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_2863529/glide_v4xBqA/cred.d/idtokens/credential_OSPool.main_Engage_US_MWT2_uiuc_condce_mcore.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=397238 GLIDEIN_Country=US GLIDEIN_DUNESite=US_UChicago GLIDEIN_Entry_Name=Engage_US_MWT2_uiuc_condce_mcore GLIDEIN_Entry_Signature=098d4a43a573f3682be0b2cee7cefcbd907f07a6 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=uiuc-gk.mwt2.org uiuc-gk.mwt2.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_5QlWke GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=MWT2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=UChicago GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_2863529/glide_v4xBqA/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=uiuc-gk.mwt2.org OSG_SITE_NAME=MWT2_CE_UIUC OSG_SQUID_LOCATION=uct2-squid.mwt2.org:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_8@glidein_2863537_786358223@uct2-c613.mwt2.org$ "; unset PROMPT_COMMAND PS1=slot1_8@glidein_2863537_786358223@uct2-c613.mwt2.org$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_2863529/glide_v4xBqA/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/var/lib/condor/execute/dir_2863529/glide_v4xBqA/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_2863529/glide_v4xBqA/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_2863529/glide_v4xBqA/execute/dir_1657112/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1325.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 1842548380 359134692 1389743596 21% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] 0.012ms pmtu 9000 1: 2605:9a00:10:200a::2 24.988ms 1: 2605:9a00:10:200a::2 13.910ms 2: g30-008-a11-s250a-msp-to-g30-008-a12-s250a-msp.p2p.uchicago.net 19.401ms asymm 1 3: g30-008-a12-s250a-msp-to-h01-391-s500a.p2p.uchicago.net 0.337ms asymm 2 4: h01-391-s500a-to-b65-ll129-s500a.p2p.uchicago.net 0.263ms asymm 2 5: no reply 6: uwmadison_se-203--chic-cr6.es.net 18.239ms asymm 4 7: perfsonar02.hep.wisc.edu 18.014ms reached Resume: pmtu 9000 hops 7 back 5
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy uct2-squid.mwt2.org:3128
Missouri-Hellbender-CE1 2 hours ago 380
100.0%
94.2%
0.0%
94.2%
kernel env df firewall mtu nrp_images http_proxy
Missouri-Hellbender-CE1 — probe output
pass kernel
4.18.0-553.124.4.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/scratch/glide_xZM1rH/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/local/scratch/glide_xZM1rH/execute/dir_3379988/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3379988 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/scratch/glide_xZM1rH/cred.d/idtokens/credential_OSPool.main_OSG_US_Missouri-Hellbender-CE1.idtoken GLIDEIN_CPUS=2 GLIDEIN_ClusterId=475820 GLIDEIN_Entry_Name=OSG_US_Missouri-Hellbender-CE1 GLIDEIN_Entry_Signature=d8e1b7e2300d288603c276f0650f1a07a964b039 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=missouri-hellbender-ce1.svc.opensciencegrid.org missouri-hellbender-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_s2c_osg_01_hxzm79 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Missouri-Hellbender-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=Missouri-Hellbender GLIDEIN_Tmp_Dir=/local/scratch/glide_xZM1rH/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=missouri-hellbender-ce1.svc.opensciencegrid.org OSG_SITE_NAME=Missouri-Hellbender-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_3067757_93648730@c219.mgmt.hellbender$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_3067757_93648730@c219.mgmt.hellbender$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/local/scratch/glide_xZM1rH/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/local/scratch/glide_xZM1rH/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/scratch/glide_xZM1rH/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/scratch/glide_xZM1rH/execute/dir_3379988/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1036.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme1n1p1 1562049324 475171856 1086877468 31% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: rcore2-hellbender-mgmt-v348.rnet.missouri.edu 10.632ms 1: rcore2-hellbender-mgmt-v348.rnet.missouri.edu 10.723ms 2: ncore-agg-rnet.bb.missouri.edu 10.915ms 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
ND-CAML-GPU-CE1 2 hours ago 7
100.0%
28.6%
100.0%
28.6%
kernel env df firewall mtu nrp_images http_proxy gpu
ND-CAML-GPU-CE1 — probe output
pass kernel
5.14.0-611.55.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/condor/execute/dir_628171/glide_C1n6hL/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/var/condor/execute/dir_628171/glide_C1n6hL/execute/dir_1218088/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1218088 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=f29942d112f8b61bda4863d2278e2e0af66f8785 GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_NotreDame GLIDEIN_CONDOR_TOKEN=/var/condor/execute/dir_628171/glide_C1n6hL/cred.d/idtokens/credential_OSPool.gpu_CMS_US_ND-CAML_gpu.idtoken GLIDEIN_CPUS=12 GLIDEIN_ClusterId=5896116 GLIDEIN_Entry_Name=CMS_US_ND-CAML_gpu GLIDEIN_Entry_Signature=9cb14137b1cec8c97270863b0f7126abe4dbc137 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=nd-caml-gpu-ce1.svc.osg-htc.org nd-caml-gpu-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_L0OHHu GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=ND-CAML-GPU-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=ND_CAMLGPU GLIDEIN_Tmp_Dir=/var/condor/execute/dir_628171/glide_C1n6hL/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=nd-caml-gpu-ce1.svc.osg-htc.org OSG_SITE_NAME=ND-CAML-GPU-CE1 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_628173_126968009@qa-v100-010.crc.nd.edu$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_628173_126968009@qa-v100-010.crc.nd.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/condor/execute/dir_628171/glide_C1n6hL/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/var/condor/execute/dir_628171/glide_C1n6hL/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/condor/execute/dir_628171/glide_C1n6hL/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/condor/execute/dir_628171/glide_C1n6hL/execute/dir_1218088/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
319.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda2 467572672 132744752 334827920 29% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: r64cisco9508-85-secondary.crc.nd.edu 0.600ms 1: r64cisco9508-85-secondary.crc.nd.edu 0.548ms 2: crc-def2compute-2.gw.nd.edu 0.537ms 3: 172.21.0.29 0.429ms asymm 4 4: man-pa-node-vip.fw.nd.edu 0.692ms 5: jjbr1-nside.gw.nd.edu 8.190ms 6: 149.165.254.41 13.466ms 7: ae-7.1.rtr2.chic.indiana.gigapop.net 16.877ms 8: 144.92.233.118 34.203ms 9: perfsonar02.hep.wisc.edu 34.015ms reached Resume: pmtu 1500 hops 9 back 9
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass gpu
Tue Jun 2 09:50:09 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 Quadro RTX 6000 Off | 00000000:86:00.0 Off | 0 | | N/A 62C P0 76W / 250W | 463MiB / 23040MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 Quadro RTX 6000 Off | 00000000:D8:00.0 Off | 0 | | N/A 20C P8 12W / 250W | 3MiB / 23040MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
NMSU-Discovery-CE1 2 hours ago 82
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
NMSU-Discovery-CE1 — probe output
pass kernel
4.18.0-553.92.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/fs1/osg/osg01/glide_Hnk6p2/images/htc__almalinux__10.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__almalinux__10.sif APPTAINER_TMPDIR=/fs1/osg/osg01/glide_Hnk6p2/execute/dir_2091507/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2091507 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/fs1/osg/osg01/glide_Hnk6p2/cred.d/idtokens/credential_OSPool.main_OSG_US_NMSU-Discovery-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=515834 GLIDEIN_Entry_Name=OSG_US_NMSU-Discovery-CE1 GLIDEIN_Entry_Signature=3abecd22c863d9e4314475763a3ce76f58cbaff4 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=nmsu-discovery-ce1.svc.osg-htc.org nmsu-discovery-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_ugqI2V GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=192.168.211.16:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=NMSU-Discovery-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=4a553cb760eb0536cf01a0e964943d71a876fc26 GLIDEIN_Site=NMSU_DISCOVERY GLIDEIN_Tmp_Dir=/fs1/osg/osg01/glide_Hnk6p2/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=nmsu-discovery-ce1.svc.osg-htc.org OSG_SITE_NAME=NMSU-Discovery-CE1 OSG_SQUID_LOCATION=192.168.211.16:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_617083_253667912@discovery-g10.cluster.local$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_617083_253667912@discovery-g10.cluster.local$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/fs1/osg/osg01/glide_Hnk6p2/images/htc__almalinux__10.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el8 SINGULARITY_IMAGES_DICT=default:/fs1/osg/osg01/glide_Hnk6p2/images/htc__almalinux__10.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__almalinux__10.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/fs1/osg/osg01/glide_Hnk6p2/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/fs1/osg/osg01/glide_Hnk6p2/execute/dir_2091507/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
35436.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on fs1 37580963840 423559168 37157404672 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.88.37.254 16.959ms 1: 10.88.37.254 16.868ms 2: dc-core-vlan-88-dc.nmsu.edu 16.187ms 3: 10.123.129.137 16.079ms asymm 4 4: 10.123.128.49 19.059ms asymm 6 5: 10.123.128.45 15.952ms asymm 7 6: 192.65.78.193 18.010ms 7: abqg-rgon.checs.net 24.688ms 8: 208.77.77.233 24.454ms asymm 7 9: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 58.107ms asymm 15 10: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 58.934ms asymm 14 11: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 56.053ms asymm 14 12: fourhundredge-0-0-0-22.4079.core1.chic.net.internet2.edu 51.381ms asymm 13 13: 144.92.233.112 52.625ms 14: perfsonar02.hep.wisc.edu 60.010ms reached Resume: pmtu 1500 hops 14 back 15
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 192.168.211.16:3128
NWICG_NDCMS 2 hours ago 4
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
NWICG_NDCMS — probe output
pass kernel
5.14.0-611.55.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/cms,/users APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/condor/execute/dir_4005116/glide_uYIWTQ/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/condor/execute/dir_4005116/glide_uYIWTQ/execute/dir_4110122/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=4110122 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_NotreDame GLIDEIN_CONDOR_TOKEN=/var/condor/execute/dir_4005116/glide_uYIWTQ/cred.d/idtokens/credential_OSPool.main_CMSHTPC_T3_US_NotreDame_deepthought.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=6371795 GLIDEIN_Country=US GLIDEIN_DUNESite=US_NotreDame GLIDEIN_Entry_Name=CMSHTPC_T3_US_NotreDame_deepthought GLIDEIN_Entry_Signature=cbd0b7ca7a39093a4dd268de3ddc6ac216d93f3d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=deepthought.crc.nd.edu deepthought.crc.nd.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_x1LF0g GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=NWICG_NDCMS GLIDEIN_SEs=deepthought.crc.nd.edu GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=fc3215af03f5a606519dd42bc9fed8d4e3456f69 GLIDEIN_Site=NotreDame GLIDEIN_Tmp_Dir=/var/condor/execute/dir_4005116/glide_uYIWTQ/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=deepthought.crc.nd.edu OSG_SITE_NAME=NWICG_NDCMS OSG_SQUID_LOCATION=eddie.crc.nd.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_4005126_133127400@d12chas554.crc.nd.edu$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_4005126_133127400@d12chas554.crc.nd.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/cms,/users SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/var/condor/execute/dir_4005116/glide_uYIWTQ/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/var/condor/execute/dir_4005116/glide_uYIWTQ/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/condor/execute/dir_4005116/glide_uYIWTQ/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/condor/execute/dir_4005116/glide_uYIWTQ/execute/dir_4110122/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1764.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda2 1949168640 98639828 1850528812 6% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: r64cisco9508-85-secondary.crc.nd.edu 0.531ms 1: r64cisco9508-85-secondary.crc.nd.edu 0.583ms 2: crc-def2compute-1.gw.nd.edu 0.467ms 3: 172.21.0.29 0.390ms asymm 4 4: man-pa-node-vip.fw.nd.edu 0.651ms 5: jjbr1-nside.gw.nd.edu 47.455ms 6: 149.165.254.41 13.521ms 7: ae-7.1.rtr2.chic.indiana.gigapop.net 16.688ms 8: 144.92.233.118 34.199ms 9: perfsonar02.hep.wisc.edu 33.813ms reached Resume: pmtu 1500 hops 9 back 9
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy eddie.crc.nd.edu:3128
Nebraska 2 hours ago 20
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
Nebraska — probe output
pass kernel
5.14.0-611.54.6.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/var/lib/condor/execute/dir_1509353,/cvmfs,/etc/cvmfs,/etc/grid-security,/var/lib/cvmfs,/var/lib/sss/pipes,/var/lib/sss/mc,/etc/nsswitch.conf,/dev/fuse,/dev/log APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:8 APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=rocky:8 APPTAINER_TMPDIR=/var/lib/condor/execute/dir_1509353/glide_5nVCjD/execute/dir_1607591/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1607591 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T2_US_Nebraska GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_1509353/glide_5nVCjD/cred.d/idtokens/credential_OSPool.main_CMS_T2_US_Nebraska_Red_test_apptainer.idtoken GLIDEIN_CPUS=48 GLIDEIN_ClusterId=6468024 GLIDEIN_Country=US GLIDEIN_ESTIMATED_CPUS=36 GLIDEIN_Entry_Name=CMS_T2_US_Nebraska_Red_test_apptainer GLIDEIN_Entry_Signature=ea6efba251cb4967b8374f8bf1c787a78bc69f48 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=red.unl.edu red.unl.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_i8sZiw GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=red-squid.unl.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Nebraska GLIDEIN_SE_VONAME_LOWERCASE=srm://srm.unl.edu:8443/srm/v2/server?SFN=/mnt/hadoop/user/ GLIDEIN_SEs=srm.unl.edu GLIDEIN_SINGULARITY_BINARY_OVERRIDE=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=Nebraska GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_1509353/glide_5nVCjD/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=red.unl.edu OSG_SITE_NAME=Red OSG_SQUID_LOCATION=red-squid.unl.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_18@glidein_14_217230984@red-c7126.unl.edu$ "; unset PROMPT_COMMAND PS1=slot1_18@glidein_14_217230984@red-c7126.unl.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/var/lib/condor/execute/dir_1509353,/cvmfs,/etc/cvmfs,/etc/grid-security,/var/lib/cvmfs,/var/lib/sss/pipes,/var/lib/sss/mc,/etc/nsswitch.conf,/dev/fuse,/dev/log SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:8 SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:8 SINGULARITY_MODE=unprivileged SINGULARITY_NAME=rocky:8 SINGULARITY_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_1509353/glide_5nVCjD/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_1509353/glide_5nVCjD/execute/dir_1607591/scratch _CONDOR_SLOT=slot1_18 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=red-squid.unl.edu:3128
pass df
802.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg0-lv_var 923606400 82522528 841083872 9% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] 0.004ms pmtu 9000 1: 2600:900:6:1102::28 0.437ms 1: 2600:900:6:1102::28 0.404ms 2: 2600:900:6:1048::1 0.622ms 3: no reply 4: chic-cr6--uwmadison_se-203.es.net 15.795ms 5: uwmadison_se-203--chic-cr6.es.net 33.092ms 6: perfsonar02.hep.wisc.edu 32.953ms reached Resume: pmtu 9000 hops 6 back 6
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy red-squid.unl.edu:3128
ORU-Titan-CE1 2 hours ago 18
100.0%
88.9%
0.0%
88.9%
kernel env df firewall mtu nrp_images http_proxy
ORU-Titan-CE1 — probe output
pass kernel
4.18.0-553.40.1.el8_10.x86_64
pass env
APPTAINER_CACHEDIR=/srv/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=129685 CONTAINERSW_FULL_VERSION=singularity version 3.7.1-5.1.ohpc.2.1 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/opt/ohpc/pub/libs/singularity/3.7.1/bin/singularity CONTAINERSW_VERSION=3.7.1-5.1.ohpc.2.1 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osgsubmit/glide_Hb60y5/cred.d/idtokens/credential_OSPool.main_OSG_US_ORU-Titan-CE1.idtoken GLIDEIN_CPUS=16 GLIDEIN_ClusterId=6363343 GLIDEIN_Entry_Name=OSG_US_ORU-Titan-CE1 GLIDEIN_Entry_Signature=891248730926b011d1acca4ac3941b081e3d182d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=oru-titan-ce1.svc.opensciencegrid.org oru-titan-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgsubmit_O0YDsx GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=ORU-Titan-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=ORU-Titan GLIDEIN_Tmp_Dir=/scratch/osgsubmit/glide_Hb60y5/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity version 3.7.1-5.1.ohpc.2.1 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/opt/ohpc/pub/libs/singularity/3.7.1/bin/singularity GWMS_CONTAINERSW_VERSION=3.7.1-5.1.ohpc.2.1 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/opt/ohpc/pub/libs/singularity/3.7.1/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.7.1-5.1.ohpc.2.1 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=oru-titan-ce1.svc.opensciencegrid.org OSG_SITE_NAME=ORU-Titan-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_14@glidein_1855373_12593074@c071.orca.oru.edu$ "; unset PROMPT_COMMAND PS1=slot1_14@glidein_1855373_12593074@c071.orca.oru.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/scratch/osgsubmit/glide_Hb60y5/execute/dir_129685:/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/scratch/osgsubmit/glide_Hb60y5/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity version 3.7.1-5.1.ohpc.2.1 SINGULARITY_IMAGES_DICT=default:/scratch/osgsubmit/glide_Hb60y5/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/opt/ohpc/pub/libs/singularity/3.7.1/bin/singularity SINGULARITY_TMPDIR=/scratch/osgsubmit/glide_Hb60y5/execute/dir_129685/scratch SINGULARITY_VERSION=3.7.1-5.1.ohpc.2.1 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osgsubmit/glide_Hb60y5/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osgsubmit/glide_Hb60y5/execute/dir_129685/scratch _CONDOR_SLOT=slot1_14 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
469238.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on beegfs_nodev 777314461696 285281981440 492032480256 37% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 45.068ms asymm 2 1: _gateway 45.110ms asymm 2 2: 156.110.41.145 45.294ms 3: 156.110.31.225 46.070ms 4: 198.71.46.45 45.847ms asymm 5 5: fourhundredge-0-0-0-9.302.core1.tuls.net.internet2.edu 4.235ms 6: fourhundredge-0-0-0-2.4079.core1.kans.net.internet2.edu 64.232ms asymm 12 7: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 62.013ms asymm 11 8: fourhundredge-0-0-0-23.4079.core1.chic.net.internet2.edu 18.147ms asymm 10 9: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 17.380ms 10: 144.92.233.112 57.937ms 11: perfsonar02.hep.wisc.edu 64.459ms reached Resume: pmtu 1500 hops 11 back 12
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
OU-Sooner-CE1 9 hours ago 1
100.0%
100.0%
100.0%
100.0%
all
OU-Sooner-CE1 — probe output
warning all
Job failed with exit code 0; no results transferred
PDX-Coeus-CE1 2 hours ago 6
33.3%
33.3%
0.0%
33.3%
kernel env df firewall mtu nrp_images http_proxy gpu
PDX-Coeus-CE1 — probe output
pass kernel
5.14.0-611.55.1.el9_7.0.3.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_R47PUT/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_R47PUT/execute/dir_1828646/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1828646 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/tmp/glide_R47PUT/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_R47PUT/cred.d/idtokens/credential_OSPool.main_OSG_US_PDX-Coeus-CE1_compute.idtoken GLIDEIN_CPUS=20 GLIDEIN_ClusterId=6271179 GLIDEIN_Entry_Name=OSG_US_PDX-Coeus-CE1_compute GLIDEIN_Entry_Signature=30a3635e90843bff10d0bdbc0c38d909d09696aa GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=pdx-coeus-ce1.svc.osg-htc.org pdx-coeus-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_el6S0C GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=coeus.cluster:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=PDX-Coeus-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=PDX-Coeus GLIDEIN_Tmp_Dir=/tmp/glide_R47PUT/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/tmp/glide_R47PUT/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/tmp/glide_R47PUT/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=pdx-coeus-ce1.svc.osg-htc.org OSG_SITE_NAME=PDX-Coeus-CE1 OSG_SQUID_LOCATION=coeus.cluster:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_6@glidein_128305_51363925@compute088$ "; unset PROMPT_COMMAND PS1=slot1_6@glidein_128305_51363925@compute088$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_R47PUT/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/tmp/glide_R47PUT/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/tmp/glide_R47PUT/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_R47PUT/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_R47PUT/execute/dir_1828646/scratch _CONDOR_SLOT=slot1_6 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
167.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 195264552 19708428 175556124 11% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: coeus.cluster 0.386ms 1: coeus.cluster 0.488ms 2: 131.252.42.10 1.328ms asymm 3 3: 131.252.42.10 0.892ms asymm 4 4: 131.252.5.19 0.687ms 5: no reply 6: 131.252.5.213 2.850ms 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy coeus.cluster:3128
pass gpu
Tue Jun 2 06:50:09 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX A5000 On | 00000000:01:00.0 Off | Off | | 37% 65C P2 222W / 230W | 670MiB / 24564MiB | 91% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA RTX A5000 On | 00000000:41:00.0 Off | Off | | 30% 24C P8 16W / 230W | 5MiB / 24564MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA RTX A5000 On | 00000000:81:00.0 Off | Off | | 30% 25C P8 18W / 230W | 5MiB / 24564MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA RTX A5000 On | 00000000:C1:00.0 Off | Off | | 30% 25C P8 22W / 230W | 5MiB / 24564MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
PDX-Orca-CE1 2 hours ago 5
80.0%
80.0%
0.0%
80.0%
kernel env df firewall mtu nrp_images http_proxy
PDX-Orca-CE1 — probe output
pass kernel
5.14.0-611.55.1.el9_7.0.3.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_58IkCf/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_58IkCf/execute/dir_1516073/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1516073 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/tmp/glide_58IkCf/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_58IkCf/cred.d/idtokens/credential_OSPool.main_OSG_US_PDX-Orca-CE1.idtoken GLIDEIN_CPUS=32 GLIDEIN_ClusterId=5794201 GLIDEIN_Entry_Name=OSG_US_PDX-Orca-CE1 GLIDEIN_Entry_Signature=1099bb65b4060388f8c245128339d383968a629c GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=pdx-orca-ce1.svc.osg-htc.org pdx-orca-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_oeVKbH GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=PDX-Orca-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=PDX-Orca GLIDEIN_Tmp_Dir=/tmp/glide_58IkCf/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/tmp/glide_58IkCf/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/tmp/glide_58IkCf/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=pdx-orca-ce1.svc.osg-htc.org OSG_SITE_NAME=PDX-Orca-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_815516_193600900@orcaga20$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_815516_193600900@orcaga20$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_58IkCf/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/tmp/glide_58IkCf/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/tmp/glide_58IkCf/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_58IkCf/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_58IkCf/execute/dir_1516073/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
396.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sdb1 460367788 20971552 415937324 5% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 9000 1: _gateway 0.648ms asymm 2 1: _gateway 0.550ms asymm 2 2: _gateway 0.604ms asymm 3 3: 10.252.100.1 0.156ms pmtu 1500 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
PSU-LIGO 2 hours ago 112
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
PSU-LIGO — probe output
pass kernel
4.18.0-553.124.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/localscratch/condor/execute/dir_700522/scratch/glide_7HrZih/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/localscratch/condor/execute/dir_700522/scratch/glide_7HrZih/execute/dir_873007/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=873007 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/localscratch/condor/execute/dir_700522/scratch/glide_7HrZih/cred.d/idtokens/credential_OSPool.main_OSG_US_PSU-LIGO.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=5796311 GLIDEIN_Entry_Name=OSG_US_PSU-LIGO GLIDEIN_Entry_Signature=8abf07a4226a500dbd13f2e929631fe87f71ce8f GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=psu-ligo-ce1.svc.opensciencegrid.org psu-ligo-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_44lOAC GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=ligo-squid-01.gwave.ics.psu.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=PSU-LIGO GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=fc3215af03f5a606519dd42bc9fed8d4e3456f69 GLIDEIN_Site=PSU-LIGO GLIDEIN_Tmp_Dir=/localscratch/condor/execute/dir_700522/scratch/glide_7HrZih/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=psu-ligo-ce1.svc.opensciencegrid.org OSG_SITE_NAME=PSU-LIGO OSG_SQUID_LOCATION=ligo-squid-01.gwave.ics.psu.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_700538_532181383@comp-cc-0477$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_700538_532181383@comp-cc-0477$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/localscratch/condor/execute/dir_700522/scratch/glide_7HrZih/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/localscratch/condor/execute/dir_700522/scratch/glide_7HrZih/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/localscratch/condor/execute/dir_700522/scratch/glide_7HrZih/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/localscratch/condor/execute/dir_700522/scratch/glide_7HrZih/execute/dir_873007/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1028.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p3 1366833076 288409172 1078423904 22% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 1.924ms 1: _gateway 18.928ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy ligo-squid-01.gwave.ics.psu.edu:3128
Purdue-Geddes 2 hours ago 10
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
Purdue-Geddes — probe output
pass kernel
4.18.0-553.22.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-RCuGJR/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-RCuGJR/execute/dir_3513358 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3513358 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Purdue Geddes OSG_SQUID_LOCATION=http://squid1.rcac.purdue.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_2@Purdue-Geddes.osg-geddes-backfill-a-0$ "; unset PROMPT_COMMAND PS1=slot1_2@Purdue-Geddes.osg-geddes-backfill-a-0$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-RCuGJR/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-RCuGJR/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-RCuGJR/execute/dir_3513358 _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=http://squid1.rcac.purdue.edu:3128
pass df
104.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/rbd2 123268672 13592232 109660056 12% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=8950 1?: [LOCALHOST] pmtu 8950 1: 172-21-160-10.pushprox-kube-proxy-client.cattle-monitoring-system.svc.cluster.local 0.167ms 1: 172-21-160-10.csi-rbdplugin-metrics.rook-ceph.svc.cluster.local 0.227ms 2: math-g109-dz9264f-01-vlan1160.tcom.purdue.edu 0.390ms 3: math-g190-c7710-01-eth-3-15.tcom.purdue.edu 0.198ms 4: math-g190-ar7280dr3a-01-vl2500.tcom.purdue.edu 0.317ms 5: math-g190-ar7280dr3a-02-et54-1.tcom.purdue.edu 0.344ms asymm 4 6: research-gigapop-ll-lhcone-400g.tcom.purdue.edu 2.164ms asymm 5 7: lo-0.2.rtr2.chic.indiana.gigapop.net 5.756ms asymm 6 8: no reply 9: uwmadison_se-203--chic-cr6.es.net 23.260ms asymm 8 10: perfsonar02.hep.wisc.edu 22.918ms reached Resume: pmtu 8950 hops 10 back 9
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://squid1.rcac.purdue.edu:3128
Rhodes-HPC-CE1 2 hours ago 8
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
Rhodes-HPC-CE1 — probe output
pass kernel
5.14.0-611.49.1.el9_7.x86_64
pass env
APPTAINER_CACHEDIR=/srv/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1170313 CONTAINERSW_FULL_VERSION=singularity-ce version 4.2.2-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/singularity CONTAINERSW_VERSION=4.2.2-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_MTt48W/cred.d/idtokens/credential_OSPool.main_OSG_US_Rhodes-HPC-CE1.idtoken GLIDEIN_CPUS=48 GLIDEIN_ClusterId=486455 GLIDEIN_Entry_Name=OSG_US_Rhodes-HPC-CE1 GLIDEIN_Entry_Signature=52fc23e495f6cc3904c4c835b0e623d3f26c26af GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=rhodes-hpc-ce1.svc.osg-htc.org rhodes-hpc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_V2enmS GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=10.0.0.1:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Rhodes-HPC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=Rhodes-HPC GLIDEIN_Tmp_Dir=/scratch/glide_MTt48W/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity-ce version 4.2.2-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/singularity GWMS_CONTAINERSW_VERSION=4.2.2-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=4.2.2-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=rhodes-hpc-ce1.svc.osg-htc.org OSG_SITE_NAME=Rhodes-HPC-CE1 OSG_SQUID_LOCATION=10.0.0.1:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_15@glidein_541580_123223814@compute01$ "; unset PROMPT_COMMAND PS1=slot1_15@glidein_541580_123223814@compute01$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/scratch/glide_MTt48W/execute/dir_1170313:/srv,/etc/hosts,/etc/localtime,/cvmfs:/cvmfs:ro SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/scratch/glide_MTt48W/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity-ce version 4.2.2-1.el9 SINGULARITY_IMAGES_DICT=default:/scratch/glide_MTt48W/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/singularity SINGULARITY_TMPDIR=/scratch/glide_MTt48W/execute/dir_1170313/scratch SINGULARITY_VERSION=4.2.2-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_MTt48W/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_MTt48W/execute/dir_1170313/scratch _CONDOR_SLOT=slot1_15 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
126.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 499862048 367592868 132269180 74% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: lotus 0.153ms 1: lotus 0.129ms 2: 10.70.0.1 0.156ms 3: xg-1-1-35-3909-soag12.hornlake.tn.malt.comcast.net 1.240ms 4: no reply 5: no reply 6: be-288-ar01.hornlake.tn.malt.comcast.net 1.572ms 7: 162.151.119.229 0.928ms 8: be-34922-cs02.dallas.tx.ibone.comcast.net 10.159ms 9: be-3111-pe11.1950stemmons.tx.ibone.comcast.net 10.340ms 10: dls-b7-link.ip.twelve99.net 10.219ms 11: dls-b23-link.ip.twelve99.net 10.516ms 12: kanc-bb2-link.ip.twelve99.net 26.204ms 13: chi-bb2-link.ip.twelve99.net 26.186ms asymm 10 14: chi-b24-link.ip.twelve99.net 26.199ms asymm 11 15: wiscnet-ic-308898.ip.twelve99-cust.net 38.244ms asymm 13 16: r-uwmilwaukee-ems-ae1-3473.uwsys.net 29.197ms asymm 14 17: r-uwmadison-cssc-ae4-3455.uwsys.net 31.124ms asymm 15 18: 143.235.41.129 31.151ms asymm 16 19: perfsonar02.hep.wisc.edu 31.211ms reached Resume: pmtu 1500 hops 19 back 18
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 10.0.0.1:3128
SDSC-PRP-OSPool-Provisioner 2 hours ago 777
100.0%
99.5%
100.0%
99.5%
kernel env df firewall mtu nrp_images http_proxy
SDSC-PRP-OSPool-Provisioner — probe output
pass kernel
6.8.0-117-generic
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-EbTqtB/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-EbTqtB/execute/dir_90256 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=90256 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=SDSC-PRP OSG_SQUID_LOCATION=10.110.90.167:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1@SDSC-PRP-OSPool-Provisioner.osg-direct-6a1136f0-007ad1-7477k$ "; unset PROMPT_COMMAND PS1=slot1@SDSC-PRP-OSPool-Provisioner.osg-direct-6a1136f0-007ad1-7477k$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-EbTqtB/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-EbTqtB/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el8 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-EbTqtB/execute/dir_90256 _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=10.110.90.167:3128
pass df
5860.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 7811942400 1666832968 6145109432 22% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=8930 1?: [LOCALHOST] pmtu 8930 1: 192.69.103.144 0.163ms 1: 192.69.103.144 0.180ms 2: 192.69.103.190 0.505ms 3: mghpcc-re-nox-mghpcc-gw1.nox.org 0.985ms 4: nox-mghpcc-gw1-i2-re-chic.nox.org 3.206ms 5: fourhundredge-0-0-0-2.4079.core2.clev.net.internet2.edu 25.380ms asymm 9 6: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 18.719ms asymm 8 7: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 18.823ms 8: 144.92.233.112 20.695ms 9: perfsonar02.hep.wisc.edu 20.647ms reached Resume: pmtu 8930 hops 9 back 10
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 10.110.90.167:3128
SIUE-Kyna-CE1 7 hours ago 2
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
SIUE-Kyna-CE1 — probe output
pass kernel
6.8.0-111-generic
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg/glide_ueJNYl/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg/glide_ueJNYl/execute/dir_2741555/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2741555 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/osg/glide_ueJNYl/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg/glide_ueJNYl/cred.d/idtokens/credential_OSPool.main_OSG_US_SIUE-Kyna-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=479074 GLIDEIN_Entry_Name=OSG_US_SIUE-Kyna-CE1 GLIDEIN_Entry_Signature=9e62c07116621468766296f6922f3615f5e25277 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=siue-kyna-ce1.svc.osg-htc.org siue-kyna-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_LCNp4h GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=SIUE-Kyna-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=SIUE-Kyna GLIDEIN_Tmp_Dir=/scratch/osg/glide_ueJNYl/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/osg/glide_ueJNYl/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/osg/glide_ueJNYl/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv HTTPS_PROXY=http://10.10.44.230:8080 HTTP_PROXY=http://10.10.44.230:8080 JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NO_PROXY=::1,127.0.0.1,localhost,10.0.0.0/8,*.hpc.siue.edu NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=siue-kyna-ce1.svc.osg-htc.org OSG_SITE_NAME=SIUE-Kyna-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2597999_404441228@cn11.hpc.siue.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2597999_404441228@cn11.hpc.siue.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg/glide_ueJNYl/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/scratch/osg/glide_ueJNYl/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/scratch/osg/glide_ueJNYl/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg/glide_ueJNYl/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg/glide_ueJNYl/execute/dir_2741555/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://10.10.44.230:8080 https_proxy=http://10.10.44.230:8080 no_proxy=::1,127.0.0.1,localhost,10.0.0.0/8,*.hpc.siue.edu
pass df
1638.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vgroot-lvroot 1843643996 31864392 1718053908 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.520ms 1: _gateway 0.371ms 2: 146.163.142.254 0.564ms 3: vdom-root120.net.siue.edu 0.628ms 4: dh0122-r1.net.siue.edu 1.857ms 5: mren-router.net.siue.edu 11.446ms 6: hundredge-0-0-0-28.1136.core1.star.net.internet2.edu 10.663ms 7: 144.92.233.112 11.557ms 8: perfsonar02.hep.wisc.edu 11.139ms reached Resume: pmtu 1500 hops 8 back 9
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://10.10.44.230:8080
SLU-Libra-CE1 2 hours ago 8
100.0%
75.0%
0.0%
75.0%
kernel env df firewall mtu nrp_images http_proxy
SLU-Libra-CE1 — probe output
pass kernel
5.14.0-611.38.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg01/glide_4LHNnl/images/htc__almalinux__10.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__almalinux__10.sif APPTAINER_TMPDIR=/scratch/osg01/glide_4LHNnl/execute/dir_1563034/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1563034 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/osg01/glide_4LHNnl/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01/glide_4LHNnl/cred.d/idtokens/credential_OSPool.main_OSG_US_SLU-Libra-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6277932 GLIDEIN_Entry_Name=OSG_US_SLU-Libra-CE1 GLIDEIN_Entry_Signature=e673b0768c20defd270f002db3abc5f93e88a652 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=slu-libra-ce1.svc.osg-htc.org slu-libra-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_v9hL2S GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=SLU-Libra-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=SLU-Libra GLIDEIN_Tmp_Dir=/scratch/osg01/glide_4LHNnl/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/osg01/glide_4LHNnl/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/osg01/glide_4LHNnl/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=slu-libra-ce1.svc.osg-htc.org OSG_SITE_NAME=SLU-Libra-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_1540576_367391825@comp03$ "; unset PROMPT_COMMAND PS1=slot1@glidein_1540576_367391825@comp03$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg01/glide_4LHNnl/images/htc__almalinux__10.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/scratch/osg01/glide_4LHNnl/images/htc__almalinux__10.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__almalinux__10.sif SINGULARITY_PATH=/scratch/osg01/glide_4LHNnl/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01/glide_4LHNnl/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01/glide_4LHNnl/execute/dir_1563034/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
119733.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 10.149.6.1:/data01/scratch/osg01/glide_4LHNnl/execute/dir_1563034 144335981568 18786349056 125549632512 14% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: head01.cm.cluster 0.183ms 1: head01.cm.cluster 0.181ms 2: 10.108.121.254 17.057ms 3: 10.248.100.81 16.863ms 4: 165.134.248.113 1.352ms 5: 10.248.101.30 17.097ms 6: 10.248.100.20 24.947ms 7: 165.134.223.27 24.755ms 8: fourhundredge-0-0-0-10.23.core2.kans.net.internet2.edu 34.302ms 9: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 48.074ms asymm 11 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 46.901ms 11: 144.92.233.112 46.892ms 12: perfsonar02.hep.wisc.edu 46.442ms reached Resume: pmtu 1500 hops 12 back 13
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
SU-ITS-CE2 2 hours ago 108
98.1%
0.0%
100.0%
23.1%
kernel env df firewall mtu nrp_images http_proxy
SU-ITS-CE2 — probe output
warning kernel
3.10.0-1160.66.1.el7.x86_64
pass env
APPTAINER_CACHEDIR=/srv BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=singularity version 3.6.0-rc.4 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/bin/singularity CONTAINERSW_VERSION=3.6.0-rc.4 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_10262/glide_UlljQZ/cred.d/idtokens/credential_OSPool.main_Glow_US_Syracuse_condor-ce2_whole.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=5892581 GLIDEIN_Country=US GLIDEIN_ESTIMATED_CPUS=40 GLIDEIN_Entry_Name=Glow_US_Syracuse_condor-ce2_whole GLIDEIN_Entry_Signature=e7d69650198605501087c90e35ab7eb582780085 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=its-condor-ce2.syr.edu its-condor-ce2.syr.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_nobody_qTEaXF GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=10.5.0.41:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=SU-ITS-CE2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=SU-ITS GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_10262/glide_UlljQZ/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity version 3.6.0-rc.4 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/bin/singularity GWMS_CONTAINERSW_VERSION=3.6.0-rc.4 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.6.0-rc.4 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=its-condor-ce2.syr.edu OSG_SITE_NAME=SU-ITS-CE2 OSG_SQUID_LOCATION=10.5.0.41:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_28@glidein_10264_103013875@CRUSH-OSG-C7-10-5-234-133$ "; unset PROMPT_COMMAND PS1=slot1_28@glidein_10264_103013875@CRUSH-OSG-C7-10-5-234-133$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_CACHEDIR=/srv SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity version 3.6.0-rc.4 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_MODE=privileged SINGULARITY_NAME=rocky:9 SINGULARITY_PATH=/bin/singularity SINGULARITY_TMPDIR=/var/lib/condor/execute/dir_10262/glide_UlljQZ/execute/dir_29503 SINGULARITY_VERSION=3.6.0-rc.4 SLOTS_LAYOUT=partitionable TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_10262/glide_UlljQZ/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_10262/glide_UlljQZ/execute/dir_29503 _CONDOR_SLOT=slot1_28 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass df
156.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/centos-root 206542340 42267360 164274980 21% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.5.255.253 0.713ms 1: 10.5.255.253 0.522ms 2: 10.30.29.35 0.367ms 3: 10.30.29.44 0.781ms asymm 2 4: 128.230.61.33 1.699ms asymm 3 5: syr-9208-su.nysernet.net 0.852ms asymm 4 6: buf-9208-syr-9208.nysernet.net 4.416ms asymm 5 7: i2-clev-buf-9208.nysernet.net 11.019ms asymm 6 8: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 14.595ms asymm 9 9: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 15.257ms asymm 8 10: 144.92.233.112 18.191ms asymm 9 11: perfsonar02.hep.wisc.edu 17.807ms reached Resume: pmtu 1500 hops 11 back 11
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 10.5.0.41:3128
SU-ITS-CE3 2 hours ago 152
92.8%
0.0%
100.0%
74.3%
kernel env df firewall mtu nrp_images http_proxy
SU-ITS-CE3 — probe output
warning kernel
3.10.0-1160.66.1.el7.x86_64
pass env
APPTAINER_CACHEDIR=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027 BATCH_SYSTEM=HTCondor CONDORCE_COLLECTOR_HOST=its-condor-ce3.syr.edu:9619 CONDOR_CONFIG=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/condor_config CONDOR_PARENT_ID=CRUSH-OSG-C7-10-5-189-243:83027:1780408209 CONDOR_PROCD_ADDRESS=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/log/procd_address CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/cred.d/idtokens/credential_OSPool.main_Glow_US_Syracuse_condor-ce3_whole.idtoken GLIDEIN_CPUS=40 GLIDEIN_ClusterId=6277348 GLIDEIN_Country=US GLIDEIN_ESTIMATED_CPUS=40 GLIDEIN_Entry_Name=Glow_US_Syracuse_condor-ce3_whole GLIDEIN_Entry_Signature=08c9bc4822616aa9c6fab335e3c2732ddc70f634 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=its-condor-ce3.syr.edu its-condor-ce3.syr.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_nobody_6U1oac GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=10.5.0.41:3128 GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=SU-ITS-CE3 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=SU-ITS GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/tmp GLIDEIN_UUID=b5e51249-5f50-4b6e-b4c5-03c0d0d5ae37 GLOBUS_LOCATION=/usr GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_PATH=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/.gwms.d/bin: GWMS_SINGULARITY_STATUS=PREFERRED HAS_SINGULARITY=False HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HOLD_GRACE_TIME=0 HOME=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027 JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_Glow_US_Syracuse_condor-ce3_whole/tokens.tgz JULIA_NUM_THREADS=1 LANG=C MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=/osg/app_dir OSG_DATA=/osg/data_dir OSG_DEFAULT_SE= OSG_GRID=/etc/osg/wn-client/ OSG_HOSTNAME=its-condor-ce3.syr.edu OSG_SITE_NAME=SU-ITS-CE3 OSG_SITE_READ= OSG_SITE_WRITE= OSG_SQUID_LOCATION=10.5.0.41:3128 OSG_STORAGE_ELEMENT=False OSG_WN_TMP=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027/.local-tmp PATH=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/.gwms.d/bin::/bin:/usr/bin:/sbin:/usr/sbin PREEMPT_GRACE_TIME=10000000 PWD=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027 PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SHLVL=2 SINGULARITY_CACHEDIR=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027 SLOTS_LAYOUT=partitionable TEMP=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027 TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027 TMPDIR=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_13019=13023:1780388797:496240805 _CONDOR_ANCESTOR_13023=83027:1780408203:607334359 _CONDOR_ANCESTOR_83027=83399:1780408210:1023669700 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/main/condor/bin _CONDOR_CHIRP_CONFIG=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027/.chirp.config _CONDOR_CREDS=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027/.condor_creds _CONDOR_EXECUTE=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute _CONDOR_JOB_AD=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027/.job.ad _CONDOR_JOB_IWD=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027 _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027/.machine.ad _CONDOR_SCRATCH_DIR=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027 _CONDOR_SLOT=slot1_9 _CONDOR_WRAPPER_ERROR_FILE=/var/lib/condor/execute/dir_120825/glide_4Ou3J6/execute/dir_83027/.job_wrapper_failure
pass df
165.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/centos-root 206542340 32567320 173975020 16% /
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.5.255.254 0.829ms 1: 10.5.255.254 0.653ms 2: 10.30.29.35 0.515ms 3: 10.30.29.46 0.791ms asymm 2 4: 128.230.61.33 1.587ms asymm 3 5: syr-9208-su.nysernet.net 70.728ms asymm 4 6: buf-9208-syr-9208.nysernet.net 4.506ms asymm 5 7: i2-clev-buf-9208.nysernet.net 9.733ms asymm 6 8: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 16.248ms asymm 9 9: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 15.631ms asymm 8 10: 144.92.233.112 17.995ms asymm 9 11: perfsonar02.hep.wisc.edu 17.727ms reached Resume: pmtu 1500 hops 11 back 11
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 10.5.0.41:3128
SYRA-PATH-EP 2 hours ago 1
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
SYRA-PATH-EP — probe output
pass kernel
4.18.0-517.el8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-vP6Etc/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-vP6Etc/execute/dir_1207643/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1207643 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=SYRA-PATH PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_27@SYRA-PATH-EP.osgvo-docker-pilot-ospool-7c8b4d4476-zb8zf$ "; unset PROMPT_COMMAND PS1=slot1_27@SYRA-PATH-EP.osgvo-docker-pilot-ospool-7c8b4d4476-zb8zf$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/pilot/osgvo-pilot-vP6Etc/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-vP6Etc/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el9 TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-vP6Etc/execute/dir_1207643/scratch _CONDOR_SLOT=slot1_27 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
372.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/openebs-pvc--c18ea35d--d569--4e76--b24a--f06f1c76ea96 442688776 52404876 390267516 12% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1480 1?: [LOCALHOST] pmtu 1480 1: 128-230-87-133.node-exporter.monitoring.svc.cluster.local 0.072ms 1: 128-230-87-133.node-exporter.monitoring.svc.cluster.local 0.056ms 2: 128.230.87.189 0.537ms 3: 128.230.61.33 1.266ms asymm 5 4: syr-9208-su.nysernet.net 0.500ms asymm 6 5: buf-9208-syr-9208.nysernet.net 3.834ms asymm 7 6: i2-clev-buf-9208.nysernet.net 10.665ms asymm 8 7: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 16.883ms asymm 11 8: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 15.046ms asymm 10 9: 144.92.233.112 17.498ms asymm 11 10: perfsonar02.hep.wisc.edu 17.440ms reached Resume: pmtu 1480 hops 10 back 13
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Swan-CE1 1 hour ago 2
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
Swan-CE1 — probe output
pass kernel
4.18.0-553.123.2.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_prBh4J/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_prBh4J/execute/dir_874433/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=874433 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T2_US_Nebraska GLIDEIN_CONDOR_TOKEN=/scratch/glide_prBh4J/cred.d/idtokens/credential_OSPool.main_HCC_US_Omaha_swan.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=473319 GLIDEIN_Country=US GLIDEIN_DUNESite=US_Swan GLIDEIN_Entry_Name=HCC_US_Omaha_swan GLIDEIN_Entry_Signature=d5b86d13543d7e5528df2693b0931dba85dd99cc GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=swan-gw1.unl.edu swan-gw1.unl.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_aePC6V GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Swan-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=Swan GLIDEIN_Tmp_Dir=/scratch/glide_prBh4J/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=swan-gw1.unl.edu OSG_SITE_NAME=Swan OSG_SQUID_LOCATION=red-squid.unl.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_491947_46768010@c0708.swan.hcc.unl.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_491947_46768010@c0708.swan.hcc.unl.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_prBh4J/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/scratch/glide_prBh4J/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TERM=vt220 TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_prBh4J/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_prBh4J/execute/dir_874433/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
2933.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md3 3123834588 48236504 3075598084 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.155ms 1: c3838.pki.hcc.unl.edu 7.355ms 1: c3838.pki.hcc.unl.edu 0.201ms 2: c3838.pki.hcc.unl.edu 0.122ms asymm 1 3: hcc-pki-mlxe.unl.edu 0.354ms asymm 2 4: 129.93.241.41 0.644ms asymm 3 5: 129.93.6.29 2.470ms asymm 4 6: 10.161.1.14 3.889ms asymm 5 7: fourhundredge-0-0-0-10.1410.core2.kans.net.internet2.edu 29.124ms asymm 6 8: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 19.001ms asymm 9 9: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 19.933ms asymm 8 10: 144.92.233.112 20.520ms asymm 9 11: perfsonar02.hep.wisc.edu 17.686ms reached Resume: pmtu 1500 hops 11 back 11
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy red-squid.unl.edu:3128
Swarthmore-Firebird-CE1 2 hours ago 45
100.0%
84.4%
0.0%
84.4%
kernel env df firewall mtu nrp_images http_proxy
Swarthmore-Firebird-CE1 — probe output
pass kernel
5.14.0-611.55.1.el9_7.0.3.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg01/glide_b3ed6E/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg01/glide_b3ed6E/execute/dir_2000546/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2000546 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01/glide_b3ed6E/cred.d/idtokens/credential_OSPool.main_OSG_US_Swarthmore-firebird-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=6477435 GLIDEIN_Entry_Name=OSG_US_Swarthmore-firebird-CE1 GLIDEIN_Entry_Signature=ce6c93fba87178af8ff102fa86fbe08352989662 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=swarthmore-firebird-ce1.svc.osg-htc.org swarthmore-firebird-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_5APmfE GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Swarthmore-Firebird-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=Swarthmore-Firebird GLIDEIN_Tmp_Dir=/scratch/osg01/glide_b3ed6E/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=swarthmore-firebird-ce1.svc.osg-htc.org OSG_SITE_NAME=Swarthmore-Firebird-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_387368_74044351@himem05.cluster$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_387368_74044351@himem05.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/osg01/glide_b3ed6E/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/scratch/osg01/glide_b3ed6E/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01/glide_b3ed6E/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01/glide_b3ed6E/execute/dir_2000546/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1648.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/disk0-scratch 1814182408 86091912 1728090496 5% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: firebird.cluster 1.353ms 1: firebird.cluster 1.510ms 2: fw-comsvc-colo.lafayette.edu 2.231ms 3: 139.147.1.113 2.311ms asymm 4 4: 172.31.0.33 2.151ms 5: 172.31.0.3 2.410ms 6: gw0-g0-3-ibgp.prd.lafayette.edu 1.646ms asymm 5 7: 164.153.140.160 5.083ms asymm 6 8: hundredge-0-0-0-28.78.core1.phil.net.internet2.edu 6.118ms asymm 7 9: fourhundredge-0-0-0-2.4079.core2.ashb.net.internet2.edu 23.904ms asymm 14 10: fourhundredge-0-0-0-1.4079.core2.clev.net.internet2.edu 23.381ms asymm 13 11: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 22.982ms asymm 12 12: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 24.894ms asymm 11 13: 144.92.233.112 24.986ms asymm 12 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
TCNJ-ELSA-CE2 2 hours ago 166
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy gpu
TCNJ-ELSA-CE2 — probe output
pass kernel
5.14.0-611.55.1.el9_7.0.3.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/scratch/glide_07seMZ/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/local/scratch/glide_07seMZ/execute/dir_3591602/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3591602 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/scratch/glide_07seMZ/cred.d/idtokens/credential_OSPool.main_OSG_US_TCNJ-ELSA-CE2.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=516772 GLIDEIN_Entry_Name=OSG_US_TCNJ-ELSA-CE2 GLIDEIN_Entry_Signature=59d2eb8b6d9f6fd89a02ed3db4d4fa3c2b370615 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=tcnj-elsa-ce2.svc.opensciencegrid.org tcnj-elsa-ce2.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_SJ4V5k GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=8 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=TCNJ-ELSA-CE2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=4a553cb760eb0536cf01a0e964943d71a876fc26 GLIDEIN_Site=ELSA GLIDEIN_Tmp_Dir=/local/scratch/glide_07seMZ/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=tcnj-elsa-ce2.svc.opensciencegrid.org OSG_SITE_NAME=TCNJ-ELSA-CE2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3021770_75753272@node037$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3021770_75753272@node037$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/local/scratch/glide_07seMZ/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/local/scratch/glide_07seMZ/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/scratch/glide_07seMZ/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/scratch/glide_07seMZ/execute/dir_3591602/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
96.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda4 143897004 43136296 100760708 30% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: ohpc-sms.localdomain 0.377ms 1: ohpc-sms.localdomain 0.256ms 2: 10.18.1.251 1.079ms 3: 10.18.0.1 0.453ms 4: 159.91.51.251 0.776ms 5: 159.91.58.177 0.663ms 6: 159.91.58.18 0.921ms 7: 159.91.58.142 1.001ms 8: 130.156.252.77 3.672ms asymm 14 9: 130.156.35.186 6.760ms asymm 15 10: 130.156.35.161 4.156ms asymm 14 11: bundle-ether1.104.core1.phil.net.internet2.edu 6.642ms asymm 16 12: fourhundredge-0-0-0-2.4079.core2.ashb.net.internet2.edu 21.640ms asymm 22 13: fourhundredge-0-0-0-1.4079.core2.clev.net.internet2.edu 23.271ms asymm 21 14: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 23.305ms asymm 20 15: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 23.005ms asymm 20 16: 144.92.233.112 25.564ms asymm 21 17: perfsonar02.hep.wisc.edu 24.475ms reached Resume: pmtu 1500 hops 17 back 22
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass gpu
Tue Jun 2 09:50:08 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.159.04 Driver Version: 580.159.04 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce GTX 1080 Ti On | 00000000:1A:00.0 Off | N/A | | 23% 17C P8 8W / 250W | 5MiB / 11264MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
UA-LR-ITS-EP 2 hours ago 31
100.0%
100.0%
96.8%
90.3%
kernel env df firewall mtu nrp_images http_proxy gpu
UA-LR-ITS-EP — probe output
pass kernel
4.18.0-553.6.1.el8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-ciQFMF/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-ciQFMF/execute/dir_2751194 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2751194 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=UA Little Rock ITS OSG_SQUID_LOCATION=144.167.99.254:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_1@UA-LR-ITS-EP.f5d083451d99$ "; unset PROMPT_COMMAND PS1=slot1_1@UA-LR-ITS-EP.f5d083451d99$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-ciQFMF/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-ciQFMF/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-ciQFMF/execute/dir_2751194 _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=144.167.99.254:3128
pass df
75.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 192080248 112789420 79290828 59% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.081ms 1: _gateway 0.105ms 2: 144.167.99.3 0.167ms 3: FH213-Firewall-Campus.net.ualr.edu 8.028ms 4: rtr-fh213-intf2-0.mgt.ualr.edu 1.005ms 5: 208.90.110.230 8.446ms 6: 208.90.110.230 8.342ms asymm 5 7: no reply 8: no reply 9: no reply 10: 208.90.110.99 20.549ms asymm 11 11: 208.90.110.99 21.645ms 12: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 33.025ms asymm 17 13: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 35.137ms asymm 17 14: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 34.108ms asymm 15 15: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 35.011ms 16: perfsonar02.hep.wisc.edu 36.238ms reached Resume: pmtu 1500 hops 16 back 18
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 144.167.99.254:3128
pass gpu
Tue Jun 2 13:51:37 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3060 Off | 00000000:01:00.0 Off | N/A | | 0% 48C P2 46W / 170W | 165MiB / 12288MiB | 89% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA GeForce RTX 3060 Off | 00000000:06:00.0 Off | N/A | | 0% 43C P8 18W / 170W | 4MiB / 12288MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA GeForce RTX 3060 Off | 00000000:07:00.0 Off | N/A | | 0% 37C P8 19W / 170W | 4MiB / 12288MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA GeForce RTX 3060 Off | 00000000:08:00.0 Off | N/A | | 0% 37C P8 17W / 170W | 4MiB / 12288MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
UAH-Voyager-CE1 2 hours ago 30
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
UAH-Voyager-CE1 — probe output
pass kernel
5.14.0-503.40.1.el9_5.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/data001/OSG/glide_aGqoU4/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/data001/OSG/glide_aGqoU4/execute/dir_3208630/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3208630 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/data001/OSG/glide_aGqoU4/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/data001/OSG/glide_aGqoU4/cred.d/idtokens/credential_OSPool.main_OSG_US_UAH-Voyager-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=5796934 GLIDEIN_Entry_Name=OSG_US_UAH-Voyager-CE1 GLIDEIN_Entry_Signature=c21da4ccf2990d9b1fbb3db1ed1b514fae608f64 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=uah-voyager-ce1.svc.osg-htc.org uah-voyager-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_Aj4FHF GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UAH-Voyager-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=fc3215af03f5a606519dd42bc9fed8d4e3456f69 GLIDEIN_Site=UAH-Voyager GLIDEIN_Tmp_Dir=/data001/OSG/glide_aGqoU4/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/data001/OSG/glide_aGqoU4/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/data001/OSG/glide_aGqoU4/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=uah-voyager-ce1.svc.osg-htc.org OSG_SITE_NAME=UAH-Voyager-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3075744_410267388@node019$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3075744_410267388@node019$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/data001/OSG/glide_aGqoU4/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/data001/OSG/glide_aGqoU4/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/data001/OSG/glide_aGqoU4/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/data001/OSG/glide_aGqoU4/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/data001/OSG/glide_aGqoU4/execute/dir_3208630/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
376764.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 172.17.170.1:/zpool/raid001/data001/OSG/glide_aGqoU4/execute/dir_3208630 395289731072 223907840 395065823232 1% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: stor001 0.363ms 1: stor001 0.321ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UCMerced-Pinnacles-CE1 2 hours ago 64
68.8%
60.9%
0.0%
90.6%
kernel env df firewall mtu nrp_images http_proxy
UCMerced-Pinnacles-CE1 — probe output
pass kernel
5.14.0-611.54.1.el9_7.x86_64
pass env
APPTAINER_CACHEDIR=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch BASH_ENV=/usr/share/lmod/lmod/init/bash BASH_FUNC_ml%%=() { eval "$($LMOD_DIR/ml_cmd "$@")" } BASH_FUNC_module%%=() { if [ -z "${LMOD_SH_DBG_ON+x}" ]; then case "$-" in *v*x*) __lmod_sh_dbg='vx' ;; *v*) __lmod_sh_dbg='v' ;; *x*) __lmod_sh_dbg='x' ;; esac; fi; if [ -n "${__lmod_sh_dbg:-}" ]; then set +$__lmod_sh_dbg; echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output" 1>&2; fi; eval "$($LMOD_CMD shell "$@")" && eval "$(${LMOD_SETTARG_CMD:-:} -s sh)"; __lmod_my_status=$?; if [ -n "${__lmod_sh_dbg:-}" ]; then echo "Shell debugging restarted" 1>&2; set -$__lmod_sh_dbg; fi; unset __lmod_sh_dbg; return $__lmod_my_status } BASH_FUNC_scl%%=() { if [ "$1" = "load" -o "$1" = "unload" ]; then eval "module $@"; else /usr/bin/scl "$@"; fi } BASH_FUNC_which%%=() { ( alias; eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@ } BATCH_SYSTEM=HTCondor BLAHPD_LOCATION=/home/osg01/bosco/glite CONDORCE_COLLECTOR_HOST=ucmerced-pinnacles-ce1.svc.opensciencegrid.org:9619 CONDOR_CONFIG=/localscratch/osg01/glide_P2T17X/condor_config CONDOR_PARENT_ID=node080:1327242:1780408204 CONDOR_PROCD_ADDRESS=/localscratch/osg01/glide_P2T17X/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/localscratch/osg01/glide_P2T17X/log/procd_address CONDOR_STARTER_PID=1327242 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/localscratch/osg01/glide_P2T17X/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/100567/bus DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima: ENVIRONMENT=BATCH FPATH=/usr/share/lmod/lmod/init/ksh_funcs GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/localscratch/osg01/glide_P2T17X/cred.d/idtokens/credential_OSPool.main_OSG_US_UCMerced-Pinnacles-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=506730 GLIDEIN_Entry_Name=OSG_US_UCMerced-Pinnacles-CE1 GLIDEIN_Entry_Signature=07cdf6864bf6f7345d6ea4408501628a06aabda0 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=ucmerced-pinnacles-ce1.svc.opensciencegrid.org ucmerced-pinnacles-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_Uh7ogB GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UCMerced-Pinnacles-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=UCMerced-Pinnacles GLIDEIN_Tmp_Dir=/localscratch/osg01/glide_P2T17X/tmp GLIDEIN_UUID=7865765f-ac40-4580-a78f-9af33291f47d GLITE_LOCATION=/home/osg01/bosco/glite GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/localscratch/osg01/glide_P2T17X/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_PATH=/localscratch/osg01/glide_P2T17X/.gwms.d/bin: GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/localscratch/osg01/glide_P2T17X/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HISTCONTROL=ignoredups HISTSIZE=1000 HOLD_GRACE_TIME=0 HOME=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch HOSTNAME=node080 HYDRA_BOOTSTRAP=slurm HYDRA_LAUNCHER_EXTRA_ARGS=--external-launcher I_MPI_HYDRA_BOOTSTRAP=slurm I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_OSG_US_UCMerced-Pinnacles-CE1/tokens.tgz JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LESSOPEN=||/usr/bin/lesspipe.sh %s LMOD_CMD=/usr/share/lmod/lmod/libexec/lmod LMOD_DIR=/usr/share/lmod/lmod/libexec LMOD_PKG=/usr/share/lmod/lmod LMOD_ROOT=/usr/share/lmod LMOD_SETTARG_FULL_SUPPORT=no LMOD_VERSION=8.7.65 LMOD_sys=Linux LOGNAME=osg01 MAIL=/var/spool/mail/osg01 MANPATH=/usr/share/lmod/lmod/share/man: MKL_NUM_THREADS=1 MODULEPATH=/etc/scl/modulefiles:/etc/scl/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/opt/modulefiles:/opt/modules MODULEPATH_ROOT=/usr/share/modulefiles MODULESHOME=/usr/share/lmod/lmod MOTD_SHOWN=pam NODE_COUNT=4 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OLDPWD=/localscratch/osg01/glide_P2T17X/logs/7865765f-ac40-4580-a78f-9af33291f47d OMPI_MCA_plm_slurm_args=--external-launcher OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=UNAVAILABLE OSG_DATA=UNAVAILABLE OSG_GRID=/root OSG_HOSTNAME=ucmerced-pinnacles-ce1.svc.opensciencegrid.org OSG_SITE_NAME=UCMerced-Pinnacles-CE1 OSG_WN_TMP=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch/.local-tmp PATH=/localscratch/osg01/glide_P2T17X/.gwms.d/bin::/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/osg01/.local/bin:/home/osg01/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin PREEMPT_GRACE_TIME=10000000 PRTE_MCA_plm_slurm_args=--external-launcher PWD=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SHELL=/bin/bash SHLVL=0 SINGULARITY_CACHEDIR=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_MODE=unprivileged SINGULARITY_PATH=/localscratch/osg01/glide_P2T17X/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable SLURMD_NODENAME=node080 SLURM_CLUSTER_NAME=pinnacles SLURM_CONF=/opt/slurm/slurm.conf SLURM_CPUS_ON_NODE=4 SLURM_GTIDS=0 SLURM_JOBID=74873 SLURM_JOB_ACCOUNT=osg SLURM_JOB_CPUS_PER_NODE=4 SLURM_JOB_END_TIME=1780471487 SLURM_JOB_GID=100127 SLURM_JOB_ID=74873 SLURM_JOB_NAME=bl_zOGnlI SLURM_JOB_NODELIST=node080 SLURM_JOB_NUM_NODES=1 SLURM_JOB_PARTITION=cenvalarc.osg SLURM_JOB_QOS=normal SLURM_JOB_START_TIME=1780212287 SLURM_JOB_UID=100567 SLURM_JOB_USER=osg01 SLURM_LOCALID=0 SLURM_MEM_PER_NODE=16000 SLURM_NNODES=1 SLURM_NODEID=0 SLURM_NODELIST=node080 SLURM_NPROCS=4 SLURM_NTASKS=4 SLURM_NTASKS_PER_NODE=4 SLURM_OOM_KILL_STEP=0 SLURM_PRIO_PROCESS=0 SLURM_PROCID=0 SLURM_SUBMIT_DIR=/home/osg01/bosco/sandbox/88b7/88b7bee4/ucmerced-pinnacles-ce1.svc.opensciencegrid.org_9619_ucmerced-pinnacles-ce1.svc.opensciencegrid.org_1249.0_1780212272 SLURM_SUBMIT_HOST=rclogin03.cluster SLURM_TASKS_PER_NODE=4 SLURM_TASK_PID=3237753 SLURM_TOPOLOGY_ADDR=ibspine01.ibsw03.node080 SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node SSH_CLIENT=128.105.82.98 61769 22 SSH_CONNECTION=128.105.82.98 61769 169.236.75.113 22 S_COLORS=auto TEMP=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch TMPDIR=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime USER=osg01 XDG_DATA_DIRS=/home/osg01/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share XDG_RUNTIME_DIR=/run/user/100567 XDG_SESSION_CLASS=user XDG_SESSION_ID=58 XDG_SESSION_TYPE=tty _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_1327242=1328780:1780408207:1466269659 _CONDOR_ANCESTOR_3351516=3351824:1780212468:1556436658 _CONDOR_ANCESTOR_3351824=1327242:1780408203:2681782776 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/localscratch/osg01/glide_P2T17X/main/condor/bin _CONDOR_CHIRP_CONFIG=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/user/.chirp.config _CONDOR_CREDS=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch/.condor_creds _CONDOR_EXECUTE=/localscratch/osg01/glide_P2T17X/execute _CONDOR_JOB_AD=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch/.job.ad _CONDOR_JOB_IWD=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/localscratch/osg01/glide_P2T17X/execute/dir_1327242/scratch/.job_wrapper_failure which_declare=declare -f
pass df
817.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme1n1p1 937233280 79862672 857370608 9% /localscratch
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: pinnacles.cluster 0.293ms 1: pinnacles.cluster 0.324ms 2: 169.236.74.1 0.675ms 3: 10.7.1.61 0.986ms 4: pa-core-0-1.ucmerced.edu 0.732ms 5: edge-1-3.ucmerced.edu 2.042ms 6: frg-agg4--ucmerced--2nd-10g.cenic.net 3.417ms 7: ferg1-agg-02--ferg1-agg-01--200g--01.cenic.net 2.683ms 8: sacr2-agg-01--ferg1-agg-01--400g--01.cenic.net 6.722ms asymm 13 9: emvl1-agg-01--sacr2-agg-01--800g--01.cenic.net 7.085ms asymm 12 10: svl-agg10--emvl1-agg-01--400g--01.cenic.net 6.892ms 11: svl-agg10-hpr--svl-agg10-dc.cenic.net 6.962ms 12: hpr-internet2r&e--svl-agg10-100ge.cenic.net 9.651ms 13: fourhundredge-0-0-0-22.4079.core2.sunn.net.internet2.edu 58.453ms asymm 24 14: fourhundredge-0-0-0-1.4079.core2.sacr.net.internet2.edu 59.188ms asymm 23 15: fourhundredge-0-0-0-1.4079.core2.salt.net.internet2.edu 58.978ms asymm 22 16: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 59.710ms asymm 21 17: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 58.569ms asymm 20 18: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 60.624ms asymm 19 19: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 60.186ms asymm 18 20: 144.92.233.112 60.861ms asymm 19 21: perfsonar02.hep.wisc.edu 60.690ms reached Resume: pmtu 1500 hops 21 back 21
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UCR-HPCC-CE1 2 hours ago 2
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
UCR-HPCC-CE1 — probe output
pass kernel
4.18.0-553.123.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_UJgqAC/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/scratch/glide_UJgqAC/execute/dir_1526905/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1526905 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/glide_UJgqAC/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_UJgqAC/cred.d/idtokens/credential_OSPool.main_OSG_US_UCR-HPCC-CE1.idtoken GLIDEIN_CPUS=10 GLIDEIN_ClusterId=6480387 GLIDEIN_Entry_Name=OSG_US_UCR-HPCC-CE1 GLIDEIN_Entry_Signature=94154951740ca1ce3426c9b12a59c3fc9c3fbf15 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=ucr-hpcc-ce1.svc.osg-htc.org ucr-hpcc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_QHvQ50 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UCR-HPCC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=fc3215af03f5a606519dd42bc9fed8d4e3456f69 GLIDEIN_Site=UCR-HPCC GLIDEIN_Tmp_Dir=/scratch/glide_UJgqAC/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/glide_UJgqAC/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/glide_UJgqAC/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ucr-hpcc-ce1.svc.osg-htc.org OSG_SITE_NAME=UCR-HPCC-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_1482170_331894560@c21.ib.hpcc.ucr.edu$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_1482170_331894560@c21.ib.hpcc.ucr.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_UJgqAC/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/scratch/glide_UJgqAC/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/scratch/glide_UJgqAC/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_UJgqAC/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_UJgqAC/execute/dir_1526905/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
439.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda6 465561308 5125844 460435464 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: nat-clust.clust.hpcc.ucr.edu 0.155ms 1: nat-clust.clust.hpcc.ucr.edu 0.133ms 2: d-genom-1120d-1-v31.ucr.edu 0.639ms 3: 172.23.254.250 0.607ms 4: 172.30.254.9 0.510ms 5: 172.30.254.8 0.467ms 6: b-som-bn-1-v2020.ucr.edu 1.538ms asymm 7 7: 137.164.26.128 13.253ms asymm 13 8: hpr-i2--lax-agg10.cenic.net 14.917ms asymm 14 9: fourhundredge-0-0-0-2.4079.core2.salt.net.internet2.edu 60.762ms asymm 23 10: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 58.591ms asymm 22 11: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 58.634ms asymm 21 12: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 59.517ms asymm 20 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 60.029ms asymm 19 14: 144.92.233.112 61.419ms asymm 20 15: perfsonar02.hep.wisc.edu 61.362ms reached Resume: pmtu 1500 hops 15 back 22
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UChicago_Pile_Backfill 2 hours ago 21
100.0%
95.2%
71.4%
95.2%
kernel env df firewall mtu nrp_images http_proxy
UChicago_Pile_Backfill — probe output
pass kernel
4.18.0-553.117.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-Zoo0Y4/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-Zoo0Y4/execute/dir_824771/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=824771 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=UChicago_Pile OSG_SQUID_LOCATION=head04.pile.uchicago.edu:30244 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_12@UChicago-Pile-Backfill.c021.pile.uchicago.edu$ "; unset PROMPT_COMMAND PS1=slot1_12@UChicago-Pile-Backfill.c021.pile.uchicago.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/pilot/osgvo-pilot-Zoo0Y4/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-Zoo0Y4/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-Zoo0Y4/execute/dir_824771/scratch _CONDOR_SLOT=slot1_12 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=head04.pile.uchicago.edu:30244
pass df
3359.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 3841429688 123824804 3522396232 4% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: no reply 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 9000 Resume: pmtu 9000
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy head04.pile.uchicago.edu:30244
UConn-OSG 2 hours ago 43
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
UConn-OSG — probe output
pass kernel
4.18.0-553.125.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/execute/dir_438605/glide_Cv7buK/images/htc__ubuntu__24.04.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__ubuntu__24.04.sif APPTAINER_TMPDIR=/local/execute/dir_438605/glide_Cv7buK/execute/dir_3968458/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3968458 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/execute/dir_438605/glide_Cv7buK/cred.d/idtokens/credential_OSPool.main_OSG_US_UConn_gluskap_op.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=5793821 GLIDEIN_Country=US GLIDEIN_DUNESite=US_UConn GLIDEIN_Entry_Name=OSG_US_UConn_gluskap_op GLIDEIN_Entry_Signature=795abd5eb3138d2e2a27cee8632c2bf6d4315724 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=gluskap.phys.uconn.edu gluskap.phys.uconn.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_prod01_eChALH GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=gryphn.phys.uconn.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UConn-OSG GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=UConn GLIDEIN_Tmp_Dir=/local/execute/dir_438605/glide_Cv7buK/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=gluskap.phys.uconn.edu OSG_SITE_NAME=UConn-OSG_CE OSG_SQUID_LOCATION=gryphn.phys.uconn.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_7@glidein_438607_473221296@nod30.phys.uconn.edu$ "; unset PROMPT_COMMAND PS1=slot1_7@glidein_438607_473221296@nod30.phys.uconn.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/local/execute/dir_438605/glide_Cv7buK/images/htc__ubuntu__24.04.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/local/execute/dir_438605/glide_Cv7buK/images/htc__ubuntu__24.04.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__ubuntu__24.04.sif SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/execute/dir_438605/glide_Cv7buK/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/execute/dir_438605/glide_Cv7buK/execute/dir_3968458/scratch _CONDOR_SLOT=slot1_7 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
863.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sdb1 937231232 31938456 905292776 4% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 137.99.79.129 0.502ms 1: 137.99.79.129 0.360ms 2: 137.99.254.89 0.414ms 3: mx-hbl-core-po38.net.uconn.edu 0.388ms 4: vss-core-border-inside-po51-638.net.uconn.edu 0.618ms 5: fw-hbl-i-lag1-662.net.uconn.edu 8.527ms 6: vss-core-border-outside-po20-660.net.uconn.edu 9.281ms 7: mx-hbl-border-outside-ae51-628.net.uconn.edu 9.199ms 8: enrt064hhh-mx-ae6-634.net.cen.ct.gov 9.794ms 9: 10.172.96.168 10.111ms 10: bundle-ether2.101.core1.hart2.net.internet2.edu 11.618ms 11: fourhundredge-0-0-0-0.4079.core1.alba.net.internet2.edu 34.591ms asymm 17 12: fourhundredge-0-0-0-2.4079.core2.clev.net.internet2.edu 34.213ms asymm 16 13: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 34.207ms asymm 15 14: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 33.707ms 15: 144.92.233.112 34.782ms 16: perfsonar02.hep.wisc.edu 34.871ms reached Resume: pmtu 1500 hops 16 back 17
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy gryphn.phys.uconn.edu:3128
UIUC-TGI-RAILS-CE1 2 hours ago 20
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy gpu
UIUC-TGI-RAILS-CE1 — probe output
pass kernel
4.18.0-553.105.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/projects/bcxl/svcosguser/glide_RUxl5R/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/projects/bcxl/svcosguser/glide_RUxl5R/execute/dir_841897/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=841897 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/projects/bcxl/svcosguser/glide_RUxl5R/cred.d/idtokens/credential_OSPool.main_OSG_US_UIUC-TGI-RAILS-CE1_cpu-small.idtoken GLIDEIN_CPUS=24 GLIDEIN_ClusterId=6476759 GLIDEIN_Entry_Name=OSG_US_UIUC-TGI-RAILS-CE1_cpu-small GLIDEIN_Entry_Signature=9b4c6dceacd26c92f4fb6d0fd170a12440a4008f GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=uiuc-tgi-rails-ce1.svc.osg-htc.org uiuc-tgi-rails-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_svcosguser_thCtVc GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UIUC-TGI-RAILS-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=UIUC-TGI-RAILS GLIDEIN_Tmp_Dir=/projects/bcxl/svcosguser/glide_RUxl5R/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=uiuc-tgi-rails-ce1.svc.osg-htc.org OSG_SITE_NAME=UIUC-TGI-RAILS-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_14@glidein_1585996_369910424@rails01$ "; unset PROMPT_COMMAND PS1=slot1_14@glidein_1585996_369910424@rails01$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/projects/bcxl/svcosguser/glide_RUxl5R/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-2.el8 SINGULARITY_IMAGES_DICT=default:/projects/bcxl/svcosguser/glide_RUxl5R/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-2.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/projects/bcxl/svcosguser/glide_RUxl5R/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/projects/bcxl/svcosguser/glide_RUxl5R/execute/dir_841897/scratch _CONDOR_SLOT=slot1_14 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
76102.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on pool1.railsvast.internal.ncsa.edu:/projects/bcxl/svcosguser/glide_RUxl5R/execute/dir_841897 80530274304 730954752 79799319552 1% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: tgi-rails-pub-gw.gw.ncsa.edu 0.285ms 1: tgi-rails-pub-gw.gw.ncsa.edu 0.224ms 2: npcf-core-east-tgi-rails-data.gw.ncsa.edu 0.132ms 3: npcf-exit-east-npcf-core-east.gw.ncsa.edu 0.620ms 4: 198.17.196.86 0.550ms 5: 72.36.80.50 20.332ms 6: perfsonar02.hep.wisc.edu 20.293ms reached Resume: pmtu 9000 hops 6 back 6
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass gpu
Tue Jun 2 08:50:10 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 590.48.01 Driver Version: 590.48.01 CUDA Version: 13.1 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA H100 80GB HBM3 On | 00000000:9B:00.0 Off | 0 | | N/A 27C P0 76W / 700W | 4MiB / 81559MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
UND-Talon-CE1 2 hours ago 36
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
UND-Talon-CE1 — probe output
pass kernel
5.14.0-427.42.1.el9_4.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_tL1l48/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_tL1l48/execute/dir_2489983/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2489983 CONTAINERSW_FULL_VERSION=apptainer version 1.1.9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/cm/local/apps/apptainer/current/bin/apptainer CONTAINERSW_VERSION=1.1.9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_tL1l48/cred.d/idtokens/credential_OSPool.main_OSG_US_UND-Talon-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=480425 GLIDEIN_Entry_Name=OSG_US_UND-Talon-CE1 GLIDEIN_Entry_Signature=f9130a815daa3cfa15de0e28dccfd31e0e394960 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=und-talon-ce1.svc.opensciencegrid.org und-talon-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_c36s4V GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UND-Talon-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=4a553cb760eb0536cf01a0e964943d71a876fc26 GLIDEIN_Site=UND-Talon GLIDEIN_Tmp_Dir=/tmp/glide_tL1l48/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.1.9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/cm/local/apps/apptainer/current/bin/apptainer GWMS_CONTAINERSW_VERSION=1.1.9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/cm/local/apps/apptainer/current/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.1.9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=und-talon-ce1.svc.opensciencegrid.org OSG_SITE_NAME=UND-Talon-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2234168_151041033@talon05.cm.cluster$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2234168_151041033@talon05.cm.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_tL1l48/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.1.9 SINGULARITY_IMAGES_DICT=default:/tmp/glide_tL1l48/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/cm/local/apps/apptainer/current/bin/apptainer SINGULARITY_VERSION=1.1.9 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_tL1l48/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_tL1l48/execute/dir_2489983/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
716.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 920325060 168992120 751332940 19% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.128.128.1 7.820ms 1: 10.128.128.1 8.046ms 2: 172.29.10.26 8.291ms 3: 172.29.10.17 8.196ms asymm 4 4: w.av25.3/22/1.ndus.edu 1.020ms asymm 5 5: 165.234.8.181 15.890ms 6: 134.129.0.37 16.171ms 7: 165.234.21.1 26.733ms asymm 8 8: telecomb-gr-01-hu0-5-0-3.northernlights.gigapop.net 34.617ms asymm 9 9: r-minneapolis-511-ae0-110.uwsys.net 34.055ms asymm 11 10: r-uweauclaire-centennial-ae0-3470.uwsys.net 35.237ms asymm 17 11: r-uwmadison-animal-ae3-3507.uwsys.net 37.946ms asymm 13 12: r-uwmadison-cssc-ae3-3439.uwsys.net 35.264ms asymm 14 13: 143.235.41.129 40.098ms asymm 15 14: perfsonar02.hep.wisc.edu 37.701ms reached Resume: pmtu 1500 hops 14 back 15
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UNL-PATH-EP 2 hours ago 24
91.7%
87.5%
100.0%
87.5%
kernel env df firewall mtu nrp_images http_proxy
UNL-PATH-EP — probe output
pass kernel
4.18.0-489.el8.x86_64
pass env
APPTAINER_CACHEDIR=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3092841 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main-container GLIDEIN_ResourceName=UNL-PATH-EP GLIDEIN_Site=UNL-PATH GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_DIR=/pilot/osgvo-pilot-PM3YXm GWMS_SINGULARITY_STATUS=PREFERRED HAS_SINGULARITY=False HOME=/home/osg JULIA_NUM_THREADS=1 LC_CTYPE=C.UTF-8 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=UNL-PATH-EP OSG_SQUID_LOCATION= OSG_WN_TMP=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch/.local-tmp PATH=/usr/local/bin:/usr/bin PWD=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SHLVL=0 SINGULARITY_CACHEDIR=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch TEMP=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch TMPDIR=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch/.local-tmp TZ=UTC _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_2819=2826:1779910416:2281818815 _CONDOR_ANCESTOR_2826=3092841:1780408206:68661544 _CONDOR_ANCESTOR_3092841=3093206:1780408206:1050666569 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/user/.chirp.config _CONDOR_CREDS=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch/.condor_creds _CONDOR_JOB_AD=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch/.job.ad _CONDOR_JOB_IWD=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch _CONDOR_SLOT=slot1_5 _CONDOR_WRAPPER_ERROR_FILE=/pilot/osgvo-pilot-PM3YXm/execute/dir_3092841/scratch/.job_wrapper_failure
pass df
388.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/openebs-pvc--93db991c--1f46--4d14--9f38--4485459dfe69 442688776 35031380 407641012 8% /pilot
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1480 1?: [LOCALHOST] pmtu 1480 1: 129-93-179-160.kubelet.kube-system.svc.cluster.local 0.063ms 1: 129-93-179-160.kubelet.kube-system.svc.cluster.local 0.031ms 2: 129.93.179.190 0.260ms 3: hcc-shor-m1436-vl48.unl.edu 8.162ms 4: no reply 5: chic-cr6--uwmadison_se-203.es.net 15.878ms 6: uwmadison_se-203--chic-cr6.es.net 33.264ms 7: perfsonar02.hep.wisc.edu 33.210ms reached Resume: pmtu 1480 hops 7 back 7
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
USC-CARC-Artemis-Backfill 2 hours ago 7
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
USC-CARC-Artemis-Backfill — probe output
pass kernel
4.18.0-372.16.1.el8_6.0.1.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-Uzg3I8/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-Uzg3I8/execute/dir_63732 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=63732 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Center for Advanced Research Computing OSG_SQUID_LOCATION=68.181.11.8:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_5@USC-CARC-Artemis-Backfill.900425c6a87f$ "; unset PROMPT_COMMAND PS1=slot1_5@USC-CARC-Artemis-Backfill.900425c6a87f$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-Uzg3I8/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-Uzg3I8/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-Uzg3I8/execute/dir_63732 _CONDOR_SLOT=slot1_5 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure http_proxy=68.181.11.8:3128
pass df
12.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on tmpfs 65980400 52579180 13401220 80% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: host.containers.internal 0.064ms 1: host.containers.internal 0.106ms 2: artemis-osg 0.229ms 3: 68.181.11.126 0.299ms 4: 68.181.25.18 0.520ms 5: 10.250.9.113 8.585ms 6: 207.151.53.252 8.946ms 7: 207.151.53.144 8.558ms asymm 6 8: hpr--lax-agg10--csuf--10g--01.cenic.net 8.624ms asymm 7 9: hpr-i2--lax-agg10.cenic.net 11.664ms asymm 8 10: fourhundredge-0-0-0-2.4079.core2.salt.net.internet2.edu 54.126ms asymm 17 11: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 57.128ms asymm 16 12: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 56.150ms asymm 15 13: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 54.678ms asymm 14 14: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 55.834ms asymm 13 15: 144.92.233.112 57.600ms asymm 14 16: perfsonar02.hep.wisc.edu 57.432ms reached Resume: pmtu 1500 hops 16 back 16
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 68.181.11.8:3128
USCMS-FNAL-WC1 1 hour ago 1
0.0%
0.0%
100.0%
0.0%
kernel env df firewall mtu nrp_images http_proxy
USCMS-FNAL-WC1 — probe output
pass kernel
5.14.0-611.55.1.el9_7.x86_64
pass env
APPTAINER_CACHEDIR=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch BATCH_SYSTEM=HTCondor CONDORCE_COLLECTOR_HOST=cmsosgce-opp1.fnal.gov:9619 CONDOR_CONFIG=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/condor_config CONDOR_PARENT_ID=osg-57893-0-cmswn2358:2425259:1780411531 CONDOR_PROCD_ADDRESS=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/log/procd_address CONDOR_STARTER_PID=2425259 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 FERMIHTC_APPTAINER_ARGS=--pid --ipc --contain FERMIHTC_APPTAINER_BIN=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer FERMIHTC_APPTAINER_BIND=/cvmfs /etc/hosts /etc/grid-security FERMIHTC_APPTAINER_CMD=exec FERMIHTC_APPTAINER_PWD=/srv GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=c5959eaac451247e3cdef4b46781022a68b907ed GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=4a2af60a10c29f832c546449bb9517fcc535f6a7 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/cred.d/idtokens/credential_OSPool.main_CMSHTPC_T1_US_FNAL_condce_opp1_whole.idtoken GLIDEIN_CPUS=48 GLIDEIN_ClusterId=6351199 GLIDEIN_Country=US GLIDEIN_DUNESite=US_FNAL-T1 GLIDEIN_ESTIMATED_CPUS=64 GLIDEIN_Entry_Name=CMSHTPC_T1_US_FNAL_condce_opp1_whole GLIDEIN_Entry_Signature=582c5e111d33ea8ac2b0dc0dd1df795ec2b22c5b GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=cmsosgce-opp1.fnal.gov cmsosgce-opp1.fnal.gov:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_KXLDUW GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_ProcId=7 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=USCMS-FNAL-WC1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=69145a89c1960c034a06440ce60acece6d007000 GLIDEIN_Site=FNAL GLIDEIN_Tmp_Dir=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/tmp GLIDEIN_UUID=986e431c-1e87-45e7-aac2-ce6909f6708a GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_PATH=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/.gwms.d/bin: GWMS_SINGULARITY_STATUS=PREFERRED HAS_SINGULARITY=False HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch HOSTNAME=osg-57893-0-cmswn2358.fnal.gov JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_CMSHTPC_T1_US_FNAL_condce_opp1_whole/tokens.tgz JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OLDPWD=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/logs/986e431c-1e87-45e7-aac2-ce6909f6708a OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=/cvmfs/oasis.opensciencegrid.org OSG_DATA=UNAVAILABLE OSG_GRID=/etc/osg/wn-client/ OSG_HOSTNAME=cmsosgce-opp1.fnal.gov OSG_SITE_NAME=USCMS-FNAL-WC1-OPP1 OSG_SITE_READ= OSG_SITE_WRITE= OSG_SQUID_LOCATION=squid.fnal.gov:3128 OSG_WN_TMP=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch/.local-tmp PATH=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/.gwms.d/bin::/bin:/usr/bin:/sbin:/usr/sbin PREEMPT_GRACE_TIME=10000000 PWD=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SHLVL=0 SINGULARITY_CACHEDIR=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch SLOTS_LAYOUT=partitionable TEMP=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch TMPDIR=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_20611=20614:1780296737:4137529901 _CONDOR_ANCESTOR_20614=2425259:1780411531:1825166825 _CONDOR_ANCESTOR_2425259=2425712:1780411532:3318822428 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/main/condor/bin _CONDOR_CHIRP_CONFIG=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/user/.chirp.config _CONDOR_CREDS=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch/.condor_creds _CONDOR_EXECUTE=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute _CONDOR_JOB_AD=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch/.job.ad _CONDOR_JOB_IWD=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/storage/local/data1/condor/execute/dir_2738771/glide_g6s8Js/execute/dir_2425259/scratch/.job_wrapper_failure _JAVA_OPTIONS=-Xmx1024m
pass df
3484.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda8 3769907704 115920276 3653987428 4% /storage/local/data1/condor/execute/dir_2738771
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] 0.005ms pmtu 9000 1: r-cms-fcc2-2-vlan191.fnal.gov 1.333ms 1: r-cms-fcc2-2-vlan191.fnal.gov 1.237ms 2: r-border-fcc2-1-2513-1.fnal.gov 0.278ms 3: no reply 4: fnalgcc-cr6--chic-bb-e.igp.es.net 1.971ms asymm 6 5: no reply 6: uwmadison_se-203--chic-cr6.es.net 19.455ms 7: perfsonar02.hep.wisc.edu 19.292ms reached Resume: pmtu 9000 hops 7 back 7
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy squid.fnal.gov:3128
USD-Lawrence-CE1 2 hours ago 110
100.0%
0.0%
0.0%
0.0%
kernel env df firewall mtu nrp_images http_proxy
USD-Lawrence-CE1 — probe output
pass kernel
4.18.0-372.26.1.el8_6.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/job_1204622/glide_OlKFRB/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/job_1204622/glide_OlKFRB/execute/dir_463533/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=463533 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/job_1204622/glide_OlKFRB/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/job_1204622/glide_OlKFRB/cred.d/idtokens/credential_OSPool.main_OSG_US_USD-Lawrence-CE1.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=5791702 GLIDEIN_Entry_Name=OSG_US_USD-Lawrence-CE1 GLIDEIN_Entry_Signature=ce459c2892b0129b75645de0aaab72eb0083bcd6 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=usd-lawrence-ce1.svc.opensciencegrid.org usd-lawrence-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01@usd.local_eRouVb GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=USD-Lawrence-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=USD-Lawrence GLIDEIN_Tmp_Dir=/scratch/job_1204622/glide_OlKFRB/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/job_1204622/glide_OlKFRB/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/job_1204622/glide_OlKFRB/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=usd-lawrence-ce1.svc.opensciencegrid.org OSG_SITE_NAME=USD-Lawrence-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_799959_232164231@node060.lawrence$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_799959_232164231@node060.lawrence$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/job_1204622/glide_OlKFRB/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/scratch/job_1204622/glide_OlKFRB/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/scratch/job_1204622/glide_OlKFRB/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/job_1204622/glide_OlKFRB/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/job_1204622/glide_OlKFRB/execute/dir_463533/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
187.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda6 207528956 10488364 197040592 6% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: login01.lawrence 0.391ms 1: login01.lawrence 0.277ms 2: 206.209.0.250 0.599ms 3: 206.176.0.181 3.723ms 4: 206.71.32.5 1.337ms 5: 206.71.32.241 2.189ms 6: 206.71.32.162 14.873ms 7: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 23.928ms asymm 9 8: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 24.996ms 9: 144.92.233.112 26.501ms 10: perfsonar02.hep.wisc.edu 25.776ms reached Resume: pmtu 1500 hops 10 back 13
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UTC-Epyc 2 hours ago 6
100.0%
66.7%
0.0%
66.7%
kernel env df firewall mtu nrp_images http_proxy
UTC-Epyc — probe output
pass kernel
5.14.0-611.26.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/mnt/internal/osg01/glide_31ZJMA/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/mnt/internal/osg01/glide_31ZJMA/execute/dir_896620/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=896620 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/mnt/internal/osg01/glide_31ZJMA/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=f29942d112f8b61bda4863d2278e2e0af66f8785 GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/mnt/internal/osg01/glide_31ZJMA/cred.d/idtokens/credential_OSPool.gpu_OSG_US_UTC-Epyc.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=5893708 GLIDEIN_Entry_Name=OSG_US_UTC-Epyc GLIDEIN_Entry_Signature=e037d612ddebdab2bab20613685784ca4b5eff1d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=utc-epyc-ce1.svc.opensciencegrid.org utc-epyc-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_XLRrLN GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UTC-Epyc GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=UTC-Epyc GLIDEIN_Tmp_Dir=/mnt/internal/osg01/glide_31ZJMA/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/mnt/internal/osg01/glide_31ZJMA/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/mnt/internal/osg01/glide_31ZJMA/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=utc-epyc-ce1.svc.opensciencegrid.org OSG_SITE_NAME=UTC-Epyc PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_21@glidein_2822615_27306666@epyc15$ "; unset PROMPT_COMMAND PS1=slot1_21@glidein_2822615_27306666@epyc15$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/mnt/internal/osg01/glide_31ZJMA/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/mnt/internal/osg01/glide_31ZJMA/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SINGULARITY_PATH=/mnt/internal/osg01/glide_31ZJMA/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/mnt/internal/osg01/glide_31ZJMA/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/mnt/internal/osg01/glide_31ZJMA/execute/dir_896620/scratch _CONDOR_SLOT=slot1_21 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
895.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p1 976283900 37014496 939269404 4% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.797ms 1: _gateway 2.974ms 2: 10.249.200.157 7.906ms 3: 192.30.208.30 8.022ms 4: no reply 5: no reply 6: no reply 7: perfsonar02.hep.wisc.edu 37.495ms reached Resume: pmtu 1500 hops 7 back 17
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UW-IT-CE1 2 hours ago 26
100.0%
100.0%
23.1%
100.0%
kernel env df firewall mtu nrp_images http_proxy
UW-IT-CE1 — probe output
pass kernel
4.18.0-553.123.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_gP0t9A/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_gP0t9A/execute/dir_60296/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=60296 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_gP0t9A/cred.d/idtokens/credential_OSPool.main_OSG_US_UW-IT-CE1.idtoken GLIDEIN_CPUS=40 GLIDEIN_ClusterId=516773 GLIDEIN_Entry_Name=OSG_US_UW-IT-CE1 GLIDEIN_Entry_Signature=ec8ccace0d57cc30ad409552e2a46da085187063 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=uw-it-ce1.svc.osg-htc.org uw-it-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_oscigrid_KN10C6 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UW-IT-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=4a553cb760eb0536cf01a0e964943d71a876fc26 GLIDEIN_Site=UW-IT GLIDEIN_Tmp_Dir=/tmp/glide_gP0t9A/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=uw-it-ce1.svc.osg-htc.org OSG_SITE_NAME=UW-IT-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_20756_20778192@n3246.hyak.local$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_20756_20778192@n3246.hyak.local$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_gP0t9A/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/tmp/glide_gP0t9A/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_gP0t9A/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_gP0t9A/execute/dir_60296/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://klone-dip1:3128 no_proxy=localhost,hyak.local,hyakm.washington.edu
pass df
353.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 374896792 4382628 370514164 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: klone-dip1.hyak.local 0.296ms 1: klone-dip1.hyak.local 0.296ms 2: ae3--20.uwar-uwtc-1.infra.washington.edu 0.586ms 3: lo0--20.uwcr-ads-2.infra.washington.edu 0.911ms 4: ae4--308.uwcr-ads-2.infra.washington.edu 0.602ms 5: no reply 6: ae16--500.icar-sttl2-2.infra.pnw-gigapop.net 0.874ms 7: lo0--210.icar-sttl1-1.infra.pnw-gigapop.net 1.096ms 8: 209.124.181.167 4.328ms 9: fourhundredge-0-0-0-17.4079.core1.seat.net.internet2.edu 52.296ms asymm 19 10: fourhundredge-0-0-0-0.4079.core1.salt.net.internet2.edu 51.584ms asymm 18 11: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 51.986ms asymm 17 12: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 53.585ms asymm 16 13: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 50.545ms asymm 15 14: fourhundredge-0-0-0-22.4079.core1.chic.net.internet2.edu 50.941ms 15: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 51.523ms asymm 13 16: 144.92.233.112 53.689ms asymm 14 17: perfsonar02.hep.wisc.edu 53.517ms reached Resume: pmtu 1500 hops 17 back 16
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://klone-dip1:3128
UWEC-BOSE-CE1 2 hours ago 4
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
UWEC-BOSE-CE1 — probe output
pass kernel
4.18.0-348.el8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/scratch/job_372302/glide_eOpZyn/images/htc__almalinux__10.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__almalinux__10.sif APPTAINER_TMPDIR=/local/scratch/job_372302/glide_eOpZyn/execute/dir_1921575/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1921575 CONTAINERSW_FULL_VERSION=apptainer version 1.2.5 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/data/software/default/apptainer/1.2.5/bin/apptainer CONTAINERSW_VERSION=1.2.5 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/scratch/job_372302/glide_eOpZyn/cred.d/idtokens/credential_OSPool.main_OSG_US_UWEC-BOSE-CE1.idtoken GLIDEIN_CONTAINER_ENV=keepall GLIDEIN_CPUS=16 GLIDEIN_ClusterId=396527 GLIDEIN_Entry_Name=OSG_US_UWEC-BOSE-CE1 GLIDEIN_Entry_Signature=3ba5cbfd39b77389947bb79b07e1ad1b1d65c658 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=uwec-bose-ce1.svc.osg-htc.org uwec-bose-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_wgwW0o GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UWEC-BOSE-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=UWEC-BOSE GLIDEIN_Tmp_Dir=/local/scratch/job_372302/glide_eOpZyn/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.2.5 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/data/software/default/apptainer/1.2.5/bin/apptainer GWMS_CONTAINERSW_VERSION=1.2.5 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/data/software/default/apptainer/1.2.5/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.2.5 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=uwec-bose-ce1.svc.osg-htc.org OSG_SITE_NAME=UWEC-BOSE-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_2417356_827437270@cn34.head.bose.hpc.cluster$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_2417356_827437270@cn34.head.bose.hpc.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIN=/data/software/default/apptainer/1.2.5/bin SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/local/scratch/job_372302/glide_eOpZyn/images/htc__almalinux__10.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.2.5 SINGULARITY_IMAGES_DICT=default:/local/scratch/job_372302/glide_eOpZyn/images/htc__almalinux__10.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__almalinux__10.sif SINGULARITY_PATH=/data/software/default/apptainer/1.2.5/bin/apptainer SINGULARITY_VERSION=1.2.5 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/scratch/job_372302/glide_eOpZyn/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/scratch/job_372302/glide_eOpZyn/execute/dir_1921575/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
723.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda8 837052400 78437060 758615340 10% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 10.826ms 1: _gateway 7.890ms 2: r-uweauclaire-cvtc-ae2-0.uwsys.net 7.892ms 3: r-uweauclaire-centennial-et-0-1-9-3503.uwsys.net 12.586ms asymm 8 4: r-uwmadison-animal-ae3-3507.uwsys.net 4.689ms 5: r-uwmadison-cssc-ae3-3439.uwsys.net 16.442ms 6: 143.235.41.129 11.996ms 7: perfsonar02.hep.wisc.edu 11.887ms reached Resume: pmtu 1500 hops 7 back 6
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UWL-Test-EP 2 hours ago 14
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
UWL-Test-EP — probe output
pass kernel
5.14.0-611.54.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-pGcv5X/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-pGcv5X/execute/dir_22690 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=22690 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=UW-La Crosse CSCPE PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_1@UWL-Test-EP.3f50e5f4e568$ "; unset PROMPT_COMMAND PS1=slot1_1@UWL-Test-EP.3f50e5f4e568$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-pGcv5X/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-pGcv5X/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-pGcv5X/execute/dir_22690 _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass df
133.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 235337452 95407628 139929824 41% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.050ms 1: _gateway 0.043ms 2: 138.49.30.1 0.736ms asymm 3 3: 10.159.96.1 0.511ms 4: 138.49.0.2 0.923ms asymm 5 5: r-uwlacrosse-wing-ae3-182.uwsys.net 0.997ms 6: r-uwmadison-animal-et-0-1-4-3504.uwsys.net 3.843ms 7: r-uwmadison-cssc-ae3-3439.uwsys.net 3.717ms 8: 143.235.41.129 3.640ms 9: perfsonar02.hep.wisc.edu 3.467ms reached Resume: pmtu 1500 hops 9 back 8
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
UWM-Mortimer-CE1 2 hours ago 170
100.0%
99.4%
100.0%
99.4%
kernel env df firewall mtu nrp_images http_proxy
UWM-Mortimer-CE1 — probe output
pass kernel
4.18.0-553.123.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_GURbGz/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_GURbGz/execute/dir_1453367/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1453367 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_GURbGz/cred.d/idtokens/credential_OSPool.main_OSG_US_UWM-Mortimer-CE1.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=470417 GLIDEIN_Entry_Name=OSG_US_UWM-Mortimer-CE1 GLIDEIN_Entry_Signature=7d3a1198aa6a858dcf1ee7d3a297390a19a7cbc9 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=uwm-mortimer-ce1.svc.osg-htc.org uwm-mortimer-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_JkYWzr GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=uwm-proxy.uwm.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UWM-Mortimer-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=UWM-Mortimer GLIDEIN_Tmp_Dir=/tmp/glide_GURbGz/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=uwm-mortimer-ce1.svc.osg-htc.org OSG_SITE_NAME=UWM-Mortimer-CE1 OSG_SQUID_LOCATION=uwm-proxy.uwm.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_1609201_568960967@execute-140.mortimer.hpc.uwm.edu$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_1609201_568960967@execute-140.mortimer.hpc.uwm.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/tmp/glide_GURbGz/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.4.5-3.el8 SINGULARITY_IMAGES_DICT=default:/tmp/glide_GURbGz/images/htc__rocky__9.sif SINGULARITY_MODE=privileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.4.5-3.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_GURbGz/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_GURbGz/execute/dir_1453367/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
314.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/VolGroup-tmp 366991360 36773784 330217576 11% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: _gateway 44.055ms 1: _gateway 44.006ms 2: no reply 3: r-uwmilwaukee-ems-ae9-160.uwsys.net 45.923ms 4: r-uwmadison-cssc-ae4-3455.uwsys.net 5.624ms 5: 143.235.41.129 48.036ms 6: perfsonar02.hep.wisc.edu 48.420ms reached Resume: pmtu 9000 hops 6 back 7
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy uwm-proxy.uwm.edu:3128
Utah-Granite-CE1 2 hours ago 68
100.0%
16.2%
0.0%
16.2%
kernel env df firewall mtu nrp_images http_proxy
Utah-Granite-CE1 — probe output
pass kernel
4.18.0-553.89.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/local/osguserg/1379260/glide_8eG4Z5/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/local/osguserg/1379260/glide_8eG4Z5/execute/dir_3187057/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3187057 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/local/osguserg/1379260/glide_8eG4Z5/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/local/osguserg/1379260/glide_8eG4Z5/cred.d/idtokens/credential_OSPool.main_OSG_US_Utah-Granite-CE1.idtoken GLIDEIN_CPUS=16 GLIDEIN_ClusterId=5794990 GLIDEIN_Entry_Name=OSG_US_Utah-Granite-CE1 GLIDEIN_Entry_Signature=6c45d997483c193262f03c8920cb27c46af500d9 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=utah-granite-ce1.svc.opensciencegrid.org utah-granite-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osguserg_xddLDN GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Utah-Granite-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=765966d611cba83c37d46bbeacfce9747f2208a1 GLIDEIN_Site=Utah-Granite GLIDEIN_Tmp_Dir=/scratch/local/osguserg/1379260/glide_8eG4Z5/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/local/osguserg/1379260/glide_8eG4Z5/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/local/osguserg/1379260/glide_8eG4Z5/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=utah-granite-ce1.svc.opensciencegrid.org OSG_SITE_NAME=Utah-Granite-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_1145227_306780615@grn040.int.chpc.utah.edu$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_1145227_306780615@grn040.int.chpc.utah.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/local/osguserg/1379260/glide_8eG4Z5/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/scratch/local/osguserg/1379260/glide_8eG4Z5/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/scratch/local/osguserg/1379260/glide_8eG4Z5/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/local/osguserg/1379260/glide_8eG4Z5/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/local/osguserg/1379260/glide_8eG4Z5/execute/dir_3187057/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
5792.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_peaks_crypt-lv_scratch 6148450304 75053836 6073396468 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.242.160.2 0.116ms 1: 10.242.160.2 0.059ms 2: rt-inscc-ddc-core-101.chpc.utah.edu 0.740ms asymm 3 3: 10.2.0.1 0.432ms asymm 4 4: 155.99.130.11 0.530ms asymm 5 5: 155.99.130.64 0.601ms asymm 6 6: 199.104.93.34 1.175ms 7: 199.104.93.21 1.532ms 8: 140.197.253.138 2.615ms 9: 140.197.248.40 2.150ms asymm 11 10: lv3-beibr-b-182-int.uen.net 2.553ms 11: fourhundredge-0-0-0-8.1013.core1.salt.net.internet2.edu 2.822ms 12: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 37.110ms asymm 20 13: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 37.688ms asymm 19 14: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 37.662ms asymm 18 15: fourhundredge-0-0-0-22.4079.core1.chic.net.internet2.edu 36.575ms asymm 17 16: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 37.887ms 17: 144.92.233.112 38.683ms 18: perfsonar02.hep.wisc.edu 38.671ms reached Resume: pmtu 1500 hops 18 back 19
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
Utah-Notchpeak-CE1 2 hours ago 100
98.0%
47.0%
0.0%
47.0%
kernel env df firewall mtu nrp_images http_proxy
Utah-Notchpeak-CE1 — probe output
pass kernel
4.18.0-553.89.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/local/osgusern/13237118/glide_HFNS2c/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/local/osgusern/13237118/glide_HFNS2c/execute/dir_992777/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=992777 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/local/osgusern/13237118/glide_HFNS2c/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/local/osgusern/13237118/glide_HFNS2c/cred.d/idtokens/credential_OSPool.main_OSG_US_Utah-Notchpeak-CE1.idtoken GLIDEIN_CPUS=16 GLIDEIN_ClusterId=478048 GLIDEIN_Entry_Name=OSG_US_Utah-Notchpeak-CE1 GLIDEIN_Entry_Signature=66ce3eff281023136811a1c72a0c142640335746 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=utah-notchpeak-ce1.svc.opensciencegrid.org utah-notchpeak-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgusern_kA1Abg GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Utah-Notchpeak-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=Utah-Notchpeak GLIDEIN_Tmp_Dir=/scratch/local/osgusern/13237118/glide_HFNS2c/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/local/osgusern/13237118/glide_HFNS2c/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/local/osgusern/13237118/glide_HFNS2c/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=utah-notchpeak-ce1.svc.opensciencegrid.org OSG_SITE_NAME=Utah-Notchpeak-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_12@glidein_64174_341692200@notch098.ipoib.int.chpc.utah.edu$ "; unset PROMPT_COMMAND PS1=slot1_12@glidein_64174_341692200@notch098.ipoib.int.chpc.utah.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/local/osgusern/13237118/glide_HFNS2c/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/scratch/local/osgusern/13237118/glide_HFNS2c/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/scratch/local/osgusern/13237118/glide_HFNS2c/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/local/osgusern/13237118/glide_HFNS2c/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/local/osgusern/13237118/glide_HFNS2c/execute/dir_992777/scratch _CONDOR_SLOT=slot1_12 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1788.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_peaks_crypt-lv_scratch 1902232068 27347712 1874884356 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.242.154.2 0.100ms 1: 10.242.154.2 0.096ms 2: rt-inscc-ddc-core-101.chpc.utah.edu 0.554ms asymm 3 3: 10.2.0.1 0.468ms asymm 4 4: 155.99.130.11 0.522ms asymm 5 5: 155.99.130.64 0.666ms asymm 6 6: 199.104.93.34 1.322ms 7: no reply 8: 140.197.253.138 2.573ms 9: 140.197.248.40 2.363ms asymm 11 10: lv3-beibr-b-182-int.uen.net 2.561ms 11: fourhundredge-0-0-0-8.1013.core1.salt.net.internet2.edu 3.820ms 12: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 37.534ms asymm 20 13: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 36.976ms asymm 19 14: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 36.411ms asymm 18 15: fourhundredge-0-0-0-22.4079.core1.chic.net.internet2.edu 36.051ms asymm 17 16: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 37.011ms 17: 144.92.233.112 38.380ms 18: perfsonar02.hep.wisc.edu 38.490ms reached Resume: pmtu 1500 hops 18 back 19
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
VU-AUGIE-CE1 2 hours ago 2
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
VU-AUGIE-CE1 — probe output
pass kernel
5.15.0-57-generic
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_QrXTMh/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_QrXTMh/execute/dir_3405193/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3405193 CONTAINERSW_FULL_VERSION=apptainer version 1.1.8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.1.8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_QrXTMh/cred.d/idtokens/credential_OSPool.main_VU-AUGIE-CE1.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=481180 GLIDEIN_Entry_Name=VU-AUGIE-CE1 GLIDEIN_Entry_Signature=e21a224724e42d85c81d0ccc105807ed67ea08b1 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=vu-augie-ce1.svc.opensciencegrid.org vu-augie-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_23R3E5 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=augie.villanova.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=VU-AUGIE-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=VU-AUGIE GLIDEIN_Tmp_Dir=/scratch/glide_QrXTMh/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.1.8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.1.8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.1.8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=vu-augie-ce1.svc.opensciencegrid.org OSG_SITE_NAME=VU-AUGIE-CE1 OSG_SQUID_LOCATION=augie.villanova.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_24@glidein_4034091_163566832@n006.cluster.com$ "; unset PROMPT_COMMAND PS1=slot1_24@glidein_4034091_163566832@n006.cluster.com$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/scratch/glide_QrXTMh/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.1.8 SINGULARITY_IMAGES_DICT=default:/scratch/glide_QrXTMh/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.1.8 SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_QrXTMh/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_QrXTMh/execute/dir_3405193/scratch _CONDOR_SLOT=slot1_24 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
685.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p4 760688044 41777960 718910084 6% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: clusterhn.cluster.pssclabs.com 0.192ms 1: clusterhn.cluster.pssclabs.com 0.207ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy augie.villanova.edu:3128
WISC-PATH-EP 2 hours ago 10
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
WISC-PATH-EP — probe output
pass kernel
6.12.0-124.38.1.el10_1.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-ZZKAyT/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-ZZKAyT/execute/dir_1738472/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1738472 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=WISC-PATH PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_8@WISC-PATH-EP.osgvo-docker-pilot-ospool-665dff69c8-n4vrk$ "; unset PROMPT_COMMAND PS1=slot1_8@WISC-PATH-EP.osgvo-docker-pilot-ospool-665dff69c8-n4vrk$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/pilot/osgvo-pilot-ZZKAyT/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-ZZKAyT/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.0-1.el9 TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-ZZKAyT/execute/dir_1738472/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
350.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/openebs-pvc--d9d79a9d--83e3--4f91--82f0--68a8ec4730d7 442688776 75088216 367584176 17% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1440 1?: [LOCALHOST] 0.009ms pmtu 1440 1: wisc-cpu-13.facility.path-cc.io 0.102ms 1: wisc-cpu-13.facility.path-cc.io 0.097ms 2: 2607:f388:2200:c0::2 0.388ms 3: perfsonar02.hep.wisc.edu 0.378ms reached Resume: pmtu 1440 hops 3 back 3
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
WSU-GRID-CE1 2 hours ago 31
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
WSU-GRID-CE1 — probe output
pass kernel
5.15.80-200.el7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/wsu/tmp/glide_U5IMss/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/wsu/tmp/glide_U5IMss/execute/dir_17687 BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=apptainer version 1.3.1-1.el7 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/wsu/tmp/glide_U5IMss/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.3.1-1.el7 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/wsu/tmp/glide_U5IMss/cred.d/idtokens/credential_OSPool.main_OSG_US_WSU-GRID_himem.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=401722 GLIDEIN_Entry_Name=OSG_US_WSU-GRID_himem GLIDEIN_Entry_Signature=0bb60d14c81867a2e3fb8b86c2bfe71b197417ec GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=wsu-grid-ce1.svc.opensciencegrid.org wsu-grid-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_ZMunTD GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=vsquid1.grid.wayne.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=WSU-GRID-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=c226af78d5610a8f317517d6c384dc441cee4ee4 GLIDEIN_Site=GRID_ce2 GLIDEIN_Tmp_Dir=/wsu/tmp/glide_U5IMss/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.3.1-1.el7 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/wsu/tmp/glide_U5IMss/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.3.1-1.el7 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/wsu/tmp/glide_U5IMss/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.3.1-1.el7 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=wsu-grid-ce1.svc.opensciencegrid.org OSG_SITE_NAME=WSU-GRID-CE1 OSG_SQUID_LOCATION=vsquid1.grid.wayne.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_559_558051480@mem1$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_559_558051480@mem1$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/wsu/tmp/glide_U5IMss/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.3.1-1.el7 SINGULARITY_IMAGES_DICT=default:/wsu/tmp/glide_U5IMss/images/htc__rocky__8.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__8.sif SINGULARITY_PATH=/wsu/tmp/glide_U5IMss/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.3.1-1.el7 SLOTS_LAYOUT=partitionable TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/wsu/tmp/glide_U5IMss/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/wsu/tmp/glide_U5IMss/execute/dir_17687 _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass df
46.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on panfs://10.3.140.1/wsu 48828112 64736 48763376 1% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: gwha1.nas.grid.wayne.edu 0.474ms 1: gwha1.nas.grid.wayne.edu 0.259ms 2: 141.217.136.1 0.588ms 3: 193-cr-c6509e-1-r515-fo-8-1-v98.core.infra 1.153ms 4: 193-br-dcs-7280sr3k-1-po-1.core.infra 0.975ms 5: ae1x160.dtrt-wsudc-c1.mich.net 8.610ms 6: et-3-1-0x3.gdrp-eberhard-c1.mich.net 12.849ms 7: no reply 8: ae102x0.chcg-lumen-600w-c1.mich.net 15.320ms asymm 9 9: 192.122.182.34 18.301ms asymm 10 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 17.484ms 11: 144.92.233.112 20.308ms 12: perfsonar02.hep.wisc.edu 19.432ms reached Resume: pmtu 1500 hops 12 back 13
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy vsquid1.grid.wayne.edu:3128
WY-MedicineBow-CE1 2 hours ago 48
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy
WY-MedicineBow-CE1 — probe output
pass kernel
5.14.0-611.55.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/srv/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/project/openscigrid/glide_AaKK63/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/project/openscigrid/glide_AaKK63/execute/dir_2926602/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2926602 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/project/openscigrid/glide_AaKK63/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.0-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=06c0210ed57939fdeb2d97058de94d494c14e51f GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=7f215391a0fa3a915d48d5b0e4e04b8186b17e67 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/project/openscigrid/glide_AaKK63/cred.d/idtokens/credential_OSPool.main_OSG_US_WY-MedicineBow-CE1.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6480388 GLIDEIN_Entry_Name=OSG_US_WY-MedicineBow-CE1 GLIDEIN_Entry_Signature=d0b6e5faf651b8cf97d5be7949f4938bb28174b9 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=wy-medicinebow-ce1.svc.opensciencegrid.org wy-medicinebow-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_qANblm GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=WY-MedicineBow-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=fc3215af03f5a606519dd42bc9fed8d4e3456f69 GLIDEIN_Site=WY-MedicineBow GLIDEIN_Tmp_Dir=/project/openscigrid/glide_AaKK63/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/project/openscigrid/glide_AaKK63/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/project/openscigrid/glide_AaKK63/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=wy-medicinebow-ce1.svc.opensciencegrid.org OSG_SITE_NAME=WY-MedicineBow-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2617513_655295928@mbcpu-001.cluster$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2617513_655295928@mbcpu-001.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/srv/scratch SINGULARITY_CONTAINER=/project/openscigrid/glide_AaKK63/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.0-1.el8 SINGULARITY_IMAGES_DICT=default:/project/openscigrid/glide_AaKK63/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/project/openscigrid/glide_AaKK63/main/condor/usr/libexec/apptainer SINGULARITY_VERSION=1.5.0-1.el8 SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/project/openscigrid/glide_AaKK63/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/project/openscigrid/glide_AaKK63/execute/dir_2926602/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
5023.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on vast-eth:/cluster/medbow/project/openscigrid/glide_AaKK63/execute/dir_2926602 5368709120 100728832 5267980288 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: no reply 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260526T053001Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set