OSPool Resource Status

Refreshes every 5 minutes
75
Resources
67
All Passing
4
Failing
4
Warnings
0
Not Available
Resource Last Updated Glideins Singularity SIF CVMFS Pelican Probes
AGLT2 1 hour ago 83
100.0%
100.0%
98.8%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
AGLT2 — probe output
pass kernel
5.14.0-611.49.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/var/lib/condor/execute/dir_1114894/glide_TX7U8S/execute/dir_1282825/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_1114894/glide_TX7U8S/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_1114894/glide_TX7U8S/execute/dir_1282825/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1282825 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_1114894/glide_TX7U8S/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.HCC_US_Michigan_gate02.idtoken.idtoken GLIDEIN_ClusterId=6562077 GLIDEIN_Country=US GLIDEIN_DUNESite=US_Michigan GLIDEIN_Entry_Name=HCC_US_Michigan_gate02 GLIDEIN_Entry_Signature=b242348519e3dc39536df5e3a9f6505e69036f2d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=gate02.aglt2.org gate02.aglt2.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_NsodqL GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_Nickname=AGLT2 GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=5 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=AGLT2 GLIDEIN_SEs=head01.aglt2.org GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=Michigan GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_1114894/glide_TX7U8S/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Michigan PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_1114967_427637118@c7-2-38.aglt2.org$ "; unset PROMPT_COMMAND PS1=slot1@glidein_1114967_427637118@c7-2-38.aglt2.org$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_1114894/glide_TX7U8S/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__8.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_1114894/glide_TX7U8S/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_1114894/glide_TX7U8S/execute/dir_1282825/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
2106.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_sys-lv_condor 2558026032 219718004 2208292796 10% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] 0.012ms pmtu 9000 1: 2001:48a8:68f7:1:7ead:4fff:fe6b:b017 0.849ms 1: 2001:48a8:68f7:1:fa6b:d9ff:fe2d:b1cb 0.744ms 2: 2001:48a8:68f7:ff02:: 0.793ms 3: l3-pereseb1-perencas1.pe-re-seb-1.umnet.umich.edu 1.564ms asymm 2 4: chic-cr6--uwmadison_se-203.es.net 5.567ms 5: chic-cr6--uwmadison_se-203.es.net 5.721ms asymm 4 6: perfsonar02.hep.wisc.edu 16.568ms reached Resume: pmtu 9000 hops 6 back 6
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg (uid 789087)
ASU-Sol-CE1 1 hour ago 3
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
ASU-Sol-CE1 — probe output
pass kernel
4.18.0-553.124.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/scratch/osg/osg01/glide_ENBuCd/execute/dir_3524748/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg/osg01/glide_ENBuCd/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg/osg01/glide_ENBuCd/execute/dir_3524748/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3524748 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg/osg01/glide_ENBuCd/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_ASU-Sol-CE1.idtoken.idtoken GLIDEIN_CPUS=24 GLIDEIN_ClusterId=7089260 GLIDEIN_Entry_Name=OSG_US_ASU-Sol-CE1 GLIDEIN_Entry_Signature=fa1ac3dc0447c59fe78f45bbe7e38f0ac39788d3 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=asu-sol-ce1.svc.opensciencegrid.org asu-sol-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_YZHjGO GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=ASU-Sol-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=76d5c6db6877413d123d8ddff4f1753932659ba9 GLIDEIN_Site=ASU-Sol GLIDEIN_Tmp_Dir=/scratch/osg/osg01/glide_ENBuCd/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=asu-sol-ce1.svc.opensciencegrid.org OSG_SITE_NAME=ASU-Sol-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_1565953_49996848@sg234$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_1565953_49996848@sg234$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/scratch/osg/osg01/glide_ENBuCd/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg/osg01/glide_ENBuCd/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg/osg01/glide_ENBuCd/execute/dir_3524748/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
476547.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on beegfs_nodev 3996579659776 3496883466240 499696193536 88% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.082ms 1: _gateway 0.083ms 2: 206.206.198.2 0.335ms 3: im-sl-gw.netmgmt.asu.edu 10.849ms 4: 172.29.21.17 10.951ms 5: 172.29.7.33 6.898ms 6: vprsenglish.asurite.ad.asu.edu 1.220ms 7: 206.207.226.103 1.231ms 8: fourhundredge-0-0-0-23.1010.core1.phoe.net.internet2.edu 4.025ms 9: fourhundredge-0-0-0-0.4079.core1.tucs.net.internet2.edu 44.421ms asymm 22 10: fourhundredge-0-0-0-0.4079.core1.elpa.net.internet2.edu 44.974ms asymm 21 11: fourhundredge-0-0-0-22.4079.core2.elpa.net.internet2.edu 45.996ms asymm 20 12: fourhundredge-0-0-0-2.4079.core2.denv.net.internet2.edu 44.873ms asymm 19 13: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 45.849ms asymm 18 14: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 43.814ms asymm 17 15: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 46.437ms asymm 16 16: 144.92.233.112 47.150ms asymm 17 17: perfsonar02.hep.wisc.edu 47.052ms reached Resume: pmtu 1500 hops 17 back 19
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 77777769)
Alabama-CHPC-CE1 1 hour ago 44
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Alabama-CHPC-CE1 — probe output
pass kernel
4.18.0-553.150.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/tmp/glide_olOdAM/execute/dir_3663157/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_olOdAM/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_olOdAM/execute/dir_3663157/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3663157 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/tmp/glide_olOdAM/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_olOdAM/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Alabama-CHPC-CE1.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=804078 GLIDEIN_Entry_Name=OSG_US_Alabama-CHPC-CE1 GLIDEIN_Entry_Signature=1c5c244d16194d61b53e4c8b89b5578286b2d828 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=ua-hpc-ce1.svc.osg-htc.org ua-hpc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_yFymyZ GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Alabama-CHPC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=04973478840ac6f9501c3524ab345e903645979b GLIDEIN_Site=Alabama-CHPC GLIDEIN_Tmp_Dir=/tmp/glide_olOdAM/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/tmp/glide_olOdAM/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/tmp/glide_olOdAM/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ua-hpc-ce1.svc.osg-htc.org OSG_SITE_NAME=Alabama-CHPC-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2994841_203801956@chpc-compute003.cm.cluster$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2994841_203801956@chpc-compute003.cm.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/tmp/glide_olOdAM/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_olOdAM/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_olOdAM/execute/dir_3663157/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
14277.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda7 15418689536 447996992 14970692544 3% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: chpc-head001.cm.cluster 0.107ms 1: chpc-head001.cm.cluster 0.079ms 2: 10.8.96.2 1.862ms 3: no reply 4: 192.168.9.17 2.809ms asymm 5 5: no reply 6: EDGE-ASR9K-PE-9.oit.ua.edu 2.904ms 7: 192.168.9.14 3.963ms 8: 192.168.13.252 4.003ms 9: TCL-ASRK9-R1-4001.oit.ua.edu 4.516ms 10: 130.160.15.233 10.985ms 11: core1-to-core2-re.56mar.sox.net 11.575ms 12: sidco-to-365dc-re.sox.net 13.958ms asymm 10 13: i2-to-sox-re.sidco.sox.net 17.231ms asymm 11 14: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 25.493ms asymm 18 15: fourhundredge-0-0-0-21.4079.core2.cinc.net.internet2.edu 26.034ms asymm 17 16: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 25.749ms 17: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 26.285ms asymm 15 18: 144.92.233.112 27.294ms asymm 16 19: perfsonar02.hep.wisc.edu 28.521ms reached Resume: pmtu 1500 hops 19 back 18
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 700)
BNL-SDCC-SPOOLCE01 1 hour ago 2
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
BNL-SDCC-SPOOLCE01 — probe output
pass kernel
5.14.0-687.15.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/home/condor/local/execute/dir_2813050/scratch/glide_fTxslH/execute/dir_1746138/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/home/condor/local/execute/dir_2813050/scratch/glide_fTxslH/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/home/condor/local/execute/dir_2813050/scratch/glide_fTxslH/execute/dir_1746138/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1746138 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/home/condor/local/execute/dir_2813050/scratch/glide_fTxslH/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.DUNE_US_BNL_spoolce01.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=7041829 GLIDEIN_Country=US GLIDEIN_DUNESite=US_BNL GLIDEIN_Entry_Name=DUNE_US_BNL_spoolce01 GLIDEIN_Entry_Signature=8d1844b424fb04b5223cf1455777f4c7f330f404 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=spoolce01.sdcc.bnl.gov spoolce01.sdcc.bnl.gov:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_0Y9DqM GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=BNL-SDCC-SPOOLCE01 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=BNL-SDCC GLIDEIN_Tmp_Dir=/home/condor/local/execute/dir_2813050/scratch/glide_fTxslH/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=BNL-SDCC PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_2813073_395034102@spool0242.sdcc.bnl.gov$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_2813073_395034102@spool0242.sdcc.bnl.gov$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/home/condor/local/execute/dir_2813050/scratch/glide_fTxslH/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/home/condor/local/execute/dir_2813050/scratch/glide_fTxslH/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/home/condor/local/execute/dir_2813050/scratch/glide_fTxslH/execute/dir_1746138/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
13148.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/sysvg-scratch 15172947224 621403108 13786796956 5% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.012ms pmtu 1500 1: ar0725-p1-l9.s149-96.sdcc.bnl.gov 1.098ms 1: ar0725-p1-l9.s149-96.sdcc.bnl.gov 0.244ms 2: 2620:0:210:e013::16 0.219ms 3: 2620:0:210:e013::229 0.829ms 4: fddd:7d33:c9b4:ff::51 2.383ms asymm 5 5: fddd:7d33:c9b4:ff::42 2.465ms 6: no reply 7: no reply 8: newy32aoa-cr6--wash-bb-e.igp.es.net 21.408ms asymm 16 9: wash-cr6--eqxdc4-bb-d.igp.es.net 21.346ms asymm 15 10: eqxdc4-cr6--eqxch2-bb-e.igp.es.net 21.786ms asymm 14 11: eqxch2-cr6--chic-bb-c.igp.es.net 21.333ms asymm 13 12: no reply 13: uwmadison_se-203--chic-cr6.es.net 33.671ms 14: perfsonar02.hep.wisc.edu 32.518ms reached Resume: pmtu 1500 hops 14 back 14
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg (uid 9878)
BNL-SDCC-SPOOLCE02 43 mins ago 3
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
BNL-SDCC-SPOOLCE02 — probe output
pass kernel
5.14.0-687.15.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/home/condor/local/execute/dir_1490433/scratch/glide_ksMK2X/execute/dir_2296718/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/home/condor/local/execute/dir_1490433/scratch/glide_ksMK2X/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/home/condor/local/execute/dir_1490433/scratch/glide_ksMK2X/execute/dir_2296718/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2296718 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/home/condor/local/execute/dir_1490433/scratch/glide_ksMK2X/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.DUNE_US_BNL_spoolce02.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=6351940 GLIDEIN_Country=US GLIDEIN_DUNESite=US_BNL GLIDEIN_Entry_Name=DUNE_US_BNL_spoolce02 GLIDEIN_Entry_Signature=1f30725a8c8bd948d69f4c16fd01b27145f95b42 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=spoolce02.sdcc.bnl.gov spoolce02.sdcc.bnl.gov:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_DWydc8 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=BNL-SDCC-SPOOLCE02 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=BNL-SDCC GLIDEIN_Tmp_Dir=/home/condor/local/execute/dir_1490433/scratch/glide_ksMK2X/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=BNL-SDCC PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_1490446_409045947@spool0132.sdcc.bnl.gov$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_1490446_409045947@spool0132.sdcc.bnl.gov$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/home/condor/local/execute/dir_1490433/scratch/glide_ksMK2X/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/home/condor/local/execute/dir_1490433/scratch/glide_ksMK2X/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/home/condor/local/execute/dir_1490433/scratch/glide_ksMK2X/execute/dir_2296718/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
5744.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/sysvg-scratch 7048416776 669441916 6023679628 11% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.014ms pmtu 1500 1: ar0725-p1-l20.s201-192.sdcc.bnl.gov 0.193ms 1: ar0725-p1-l20.s201-192.sdcc.bnl.gov 0.145ms 2: 2620:0:210:e023::38 0.194ms 3: 2620:0:210:e013::229 0.270ms 4: fddd:7d33:c9b4:ff::51 0.310ms asymm 5 5: fddd:7d33:c9b4:ff::43 0.433ms 6: no reply 7: no reply 8: newy32aoa-cr6--wash-bb-e.igp.es.net 21.236ms asymm 16 9: wash-cr6--eqxdc4-bb-d.igp.es.net 21.540ms asymm 15 10: eqxdc4-cr6--eqxch2-bb-e.igp.es.net 21.427ms asymm 14 11: eqxch2-cr6--chic-bb-c.igp.es.net 21.233ms asymm 13 12: no reply 13: uwmadison_se-203--chic-cr6.es.net 32.977ms 14: perfsonar02.hep.wisc.edu 32.407ms reached Resume: pmtu 1500 hops 14 back 14
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg (uid 9878)
Boise-State-Borah-CE1 1 hour ago 15
0.0%
0.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Boise-State-Borah-CE1 — probe output
warning kernel
3.10.0-1160.119.1.el7.x86_64
pass env
APPTAINER_CACHEDIR=/local/osg01/glide_bjiWAx/execute/dir_48967 BASH_ENV=/cm/local/apps/environment-modules/4.4.0//init/bash BASH_FUNC_module%%=() { unset _mlshdbg; if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then case "$-" in *v*x*) set +vx; _mlshdbg='vx' ;; *v*) set +v; _mlshdbg='v' ;; *x*) set +x; _mlshdbg='x' ;; *) _mlshdbg='' ;; esac; fi; unset _mlre _mlIFS; if [ -n "${IFS+x}" ]; then _mlIFS=$IFS; fi; IFS=' '; for _mlv in ${MODULES_RUN_QUARANTINE:-}; do if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' "; fi; _mlrv="MODULES_RUNENV_${_mlv}"; _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' "; fi; done; if [ -n "${_mlre:-}" ]; then eval `eval ${_mlre}/usr/bin/tclsh /cm/local/apps/environment-modules/4.4.0/libexec/modulecmd.tcl bash '"$@"'`; else eval `/usr/bin/tclsh /cm/local/apps/environment-modules/4.4.0/libexec/modulecmd.tcl bash "$@"`; fi; _mlstatus=$?; if [ -n "${_mlIFS+x}" ]; then IFS=$_mlIFS; else unset IFS; fi; unset _mlre _mlv _mlrv _mlIFS; if [ -n "${_mlshdbg:-}" ]; then set -$_mlshdbg; fi; unset _mlshdbg; return $_mlstatus } BASH_FUNC_module()=() { unset _mlshdbg; if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then case "$-" in *v*x*) set +vx; _mlshdbg='vx' ;; *v*) set +v; _mlshdbg='v' ;; *x*) set +x; _mlshdbg='x' ;; *) _mlshdbg='' ;; esac; fi; unset _mlre _mlIFS; if [ -n "${IFS+x}" ]; then _mlIFS=$IFS; fi; IFS=' '; for _mlv in ${MODULES_RUN_QUARANTINE:-}; do if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' "; fi; _mlrv="MODULES_RUNENV_${_mlv}"; _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' "; fi; done; if [ -n "${_mlre:-}" ]; then eval `eval ${_mlre}/usr/bin/tclsh /cm/local/apps/environment-modules/4.4.0/libexec/modulecmd.tcl bash '"$@"'`; else eval `/usr/bin/tclsh /cm/local/apps/environment-modules/4.4.0/libexec/modulecmd.tcl bash "$@"`; fi; _mlstatus=$?; if [ -n "${_mlIFS+x}" ]; then IFS=$_mlIFS; else unset IFS; fi; unset _mlre _mlv _mlrv _mlIFS; if [ -n "${_mlshdbg:-}" ]; then set -$_mlshdbg; fi; unset _mlshdbg; return $_mlstatus } BASH_FUNC_switchml%%=() { typeset swfound=1; if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then typeset swname='main'; if [ -e /cm/local/apps/environment-modules/4.4.0//libexec/modulecmd.tcl ]; then typeset swfound=0; unset MODULES_USE_COMPAT_VERSION; fi; else typeset swname='compatibility'; if [ -e /cm/local/apps/environment-modules/4.4.0//libexec/modulecmd-compat ]; then typeset swfound=0; MODULES_USE_COMPAT_VERSION=1; export MODULES_USE_COMPAT_VERSION; fi; fi; if [ $swfound -eq 0 ]; then echo "Switching to Modules $swname version"; source /cm/local/apps/environment-modules/4.4.0//init/bash; else echo "Cannot switch to Modules $swname version, command not found"; return 1; fi } BASH_FUNC_switchml()=() { typeset swfound=1; if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then typeset swname='main'; if [ -e /cm/local/apps/environment-modules/4.4.0//libexec/modulecmd.tcl ]; then typeset swfound=0; unset MODULES_USE_COMPAT_VERSION; fi; else typeset swname='compatibility'; if [ -e /cm/local/apps/environment-modules/4.4.0//libexec/modulecmd-compat ]; then typeset swfound=0; MODULES_USE_COMPAT_VERSION=1; export MODULES_USE_COMPAT_VERSION; fi; fi; if [ $swfound -eq 0 ]; then echo "Switching to Modules $swname version"; source /cm/local/apps/environment-modules/4.4.0//init/bash; else echo "Cannot switch to Modules $swname version, command not found"; return 1; fi } BASH_FUNC_which%%=() { ( alias; eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@ } BATCH_SYSTEM=HTCondor BLAHPD_LOCATION=/bsuhome/osg01/bosco/glite CMD_WLM_CLUSTER_NAME=slurm CONDORCE_COLLECTOR_HOST=boise-state-borah-ce1.svc.osg-htc.org:9619 CONDOR_CONFIG=/local/osg01/glide_bjiWAx/condor_config CONDOR_PARENT_ID=gpu111:48967:1790772824 CONDOR_PROCD_ADDRESS=/local/osg01/glide_bjiWAx/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/local/osg01/glide_bjiWAx/log/procd_address CONTAINERSW_FULL_VERSION=singularity version 3.7.3 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/cm/shared/apps/singularity/3.7.3/bin/singularity CONTAINERSW_VERSION=3.7.3 CPATH=/cm/shared/apps/slurm/current/include CPATH_modshare=/cm/shared/apps/slurm/current/include:1 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1022/bus DEBUGINFOD_URLS=https://debuginfod.centos.org/ ENABLE_LMOD=0 ENV=/cm/local/apps/environment-modules/4.4.0//init/profile.sh ENVIRONMENT=BATCH GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/osg01/glide_bjiWAx/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Boise-State-Borah-CE1.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=7096134 GLIDEIN_DEBUG_OPTIONS= GLIDEIN_Entry_Name=OSG_US_Boise-State-Borah-CE1 GLIDEIN_Entry_Signature=3b6274020185e096cbb87255e59d56b7486ab8e9 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=boise-state-borah-ce1.svc.osg-htc.org boise-state-borah-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_G5EOzF GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=3 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Boise-State-Borah-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=Boise-State-Borah GLIDEIN_Tmp_Dir=/local/osg01/glide_bjiWAx/tmp GLIDEIN_UUID=1184de38-1c92-4a24-b20a-2784860bb93b GLITE_LOCATION=/bsuhome/osg01/bosco/glite GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_CONTAINERSW_FULL_VERSION=singularity version 3.7.3 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/cm/shared/apps/singularity/3.7.3/bin/singularity GWMS_CONTAINERSW_VERSION=3.7.3 GWMS_PATH=/local/osg01/glide_bjiWAx/.gwms.d/bin: GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/cm/shared/apps/singularity/3.7.3/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.7.3 HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HISTCONTROL=ignoredups HISTSIZE=1000 HOLD_GRACE_TIME=0 HOME=/tmp/osg01/glide_bjiWAx/execute/dir_48967 HOSTNAME=gpu111 JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_OSG_US_Boise-State-Borah-CE1/tokens.tgz JULIA_NUM_THREADS=1 LANG=C LD_LIBRARY_PATH=/cm/shared/apps/python/3.9.7/lib:/cm/shared/apps/slurm/current/lib64/slurm:/cm/shared/apps/slurm/current/lib64:/cm/local/apps/gcc/9.2.0/lib:/cm/local/apps/gcc/9.2.0/lib64 LD_LIBRARY_PATH_modshare=/cm/shared/apps/slurm/current/lib64:1:/cm/local/apps/gcc/9.2.0/lib:1:/cm/shared/apps/python/3.9.7/lib:1:/cm/shared/apps/slurm/current/lib64/slurm:1:/cm/local/apps/gcc/9.2.0/lib64:1 LESSOPEN=||/usr/bin/lesspipe.sh %s LIBRARY_PATH=/cm/shared/apps/slurm/current/lib64/slurm:/cm/shared/apps/slurm/current/lib64 LIBRARY_PATH_modshare=/cm/shared/apps/slurm/current/lib64:1:/cm/shared/apps/slurm/current/lib64/slurm:1 LOADEDMODULES=gcc/9.2.0:slurm/slurm/23.02.7:python/3.9.7 LOADEDMODULES_modshare=python/3.9.7:1:slurm/slurm/23.02.7:1:gcc/9.2.0:1 LOGNAME=osg01 MAIL=/var/spool/mail/osg01 MANPATH=/cm/shared/apps/slurm/current/man:: MANPATH_modshare=:1:/cm/shared/apps/slurm/current/man:1 MKL_NUM_THREADS=1 MODULEPATH=/cm/local/modulefiles:/cm/shared/modulefiles MODULESHOME=/cm/local/apps/environment-modules/4.4.0/ MODULES_CMD=/cm/local/apps/environment-modules/4.4.0/libexec/modulecmd.tcl NODE_COUNT=4 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=UNAVAILABLE OSG_DATA=UNAVAILABLE OSG_GRID=/root OSG_HOSTNAME=boise-state-borah-ce1.svc.osg-htc.org OSG_SITE_NAME=Boise-State-Borah-CE1 OSG_WN_TMP=/local/osg01/glide_bjiWAx/execute/dir_48967/.local-tmp PATH=/local/osg01/glide_bjiWAx/.gwms.d/bin::/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/cm/shared/apps/python/3.9.7/bin:/cm/shared/apps/slurm/current/sbin:/cm/shared/apps/slurm/current/bin:/cm/local/apps/gcc/9.2.0/bin:/cm/local/apps/environment-modules/4.4.0//bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/sbin:/sbin:/usr/sbin:/cm/local/apps/environment-modules/4.4.0/bin:/opt/dell/srvadmin/sbin:/bsuhome/osg01/.local/bin:/bsuhome/osg01/bin PATH_modshare=/cm/local/apps/environment-modules/4.4.0//bin:1:/opt/dell/srvadmin/sbin:1:/usr/bin:1:/usr/local/bin:1:/cm/shared/apps/slurm/current/bin:1:/cm/shared/apps/python/3.9.7/bin:1:/cm/shared/apps/slurm/current/sbin:1:/cm/local/apps/environment-modules/4.4.0/bin:1:/bsuhome/osg01/bin:1:/sbin:1:/bsuhome/osg01/.local/bin:1:/usr/sbin:1:/usr/local/sbin:1:/cm/local/apps/gcc/9.2.0/bin:1 PREEMPT_GRACE_TIME=10000000 PWD=/tmp/osg01/glide_bjiWAx/execute/dir_48967 PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SELINUX_LEVEL_REQUESTED= SELINUX_ROLE_REQUESTED= SELINUX_USE_CURRENT_RANGE= SHELL=/bin/bash SHLVL=6 SINGULARITY_CACHEDIR=/local/osg01/glide_bjiWAx/execute/dir_48967 SINGULARITY_FULL_VERSION=singularity version 3.7.3 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_MODE=privileged SINGULARITY_PATH=/cm/shared/apps/singularity/3.7.3/bin/singularity SINGULARITY_VERSION=3.7.3 SLOTS_LAYOUT=partitionable SLURMD_NODENAME=gpu111 SLURM_CLUSTER_NAME=slurm SLURM_CONF=/etc/slurm/slurm.conf SLURM_CPUS_ON_NODE=4 SLURM_GTIDS=0 SLURM_JOBID=3290193 SLURM_JOB_ACCOUNT=osg01 SLURM_JOB_CPUS_PER_NODE=4 SLURM_JOB_END_TIME=1791000592 SLURM_JOB_GID=1022 SLURM_JOB_ID=3290193 SLURM_JOB_NAME=bl_xlQGFA SLURM_JOB_NODELIST=gpu111 SLURM_JOB_NUM_NODES=1 SLURM_JOB_PARTITION=osg SLURM_JOB_QOS=normal SLURM_JOB_START_TIME=1790741392 SLURM_JOB_UID=1022 SLURM_JOB_USER=osg01 SLURM_LOCALID=0 SLURM_MEM_PER_NODE=32768 SLURM_NNODES=1 SLURM_NODEID=0 SLURM_NODELIST=gpu111 SLURM_NODE_ALIASES=(null) SLURM_NPROCS=4 SLURM_NTASKS=4 SLURM_NTASKS_PER_NODE=4 SLURM_PRIO_PROCESS=0 SLURM_PROCID=0 SLURM_SUBMIT_DIR=/bsuhome/osg01/bosco/sandbox/edcd/edcdd433/boise-state-borah-ce1.svc.osg-htc.org_9619_boise-state-borah-ce1.svc.osg-htc.org_45155.0_1790741083 SLURM_SUBMIT_HOST=borah-login SLURM_TASKS_PER_NODE=4 SLURM_TASK_PID=25764 SLURM_TOPOLOGY_ADDR=gpu111 SLURM_TOPOLOGY_ADDR_PATTERN=node SLURM_WORKING_CLUSTER=slurm:borah-hn001:6817:9984:109 SQUEUE_SORT=t SSH_CLIENT=128.105.82.103 56260 22 SSH_CONNECTION=128.105.82.103 56260 132.178.231.141 22 TEMP=/local/osg01/glide_bjiWAx/execute/dir_48967 TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/local/osg01/glide_bjiWAx/execute/dir_48967 TMPDIR=/local/osg01/glide_bjiWAx/execute/dir_48967/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime USER=osg01 XDG_RUNTIME_DIR=/run/user/1022 XDG_SESSION_ID=11 _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_14716=16099:1790741676:4208922578 _CONDOR_ANCESTOR_16099=48967:1790772822:2861014675 _CONDOR_ANCESTOR_48967=50114:1790772837:1437696824 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/osg01/glide_bjiWAx/main/condor/bin _CONDOR_CHIRP_CONFIG=/local/osg01/glide_bjiWAx/execute/dir_48967/.chirp.config _CONDOR_CREDS=/local/osg01/glide_bjiWAx/execute/dir_48967/.condor_creds _CONDOR_EXECUTE=/local/osg01/glide_bjiWAx/execute _CONDOR_JOB_AD=/local/osg01/glide_bjiWAx/execute/dir_48967/.job.ad _CONDOR_JOB_IWD=/local/osg01/glide_bjiWAx/execute/dir_48967 _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/local/osg01/glide_bjiWAx/execute/dir_48967/.machine.ad _CONDOR_SCRATCH_DIR=/local/osg01/glide_bjiWAx/execute/dir_48967 _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/local/osg01/glide_bjiWAx/execute/dir_48967/.job_wrapper_failure _LMFILES_=/cm/local/modulefiles/gcc/9.2.0:/cm/local/modulefiles/slurm/slurm/23.02.7:/cm/shared/modulefiles/python/3.9.7 _LMFILES__modshare=/cm/shared/modulefiles/python/3.9.7:1:/cm/local/modulefiles/slurm/slurm/23.02.7:1:/cm/local/modulefiles/gcc/9.2.0:1 which_declare=declare -f
pass df
270.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p4 382609888 98626684 283983204 26% /tmp
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: borah-hn001.cm.cluster 0.429ms 1: borah-hn001.cm.cluster 10.425ms 2: l3-c3-nexus-01-10-32-4-2.boisestate.edu 10.933ms 3: no reply 4: fw-5450-ae1-616.boisestate.edu 8.362ms 5: l3-101mec-204-sci-dmz-01-po53-500.boisestate.edu 19.957ms 6: 74.118.19.33 19.960ms 7: 74.118.19.130 19.960ms 8: hundredge-0-0-0-24.4007.core1.salt.net.internet2.edu 28.822ms 9: fourhundredge-0-0-0-0.4079.core1.denv.net.internet2.edu 55.663ms asymm 17 10: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 52.243ms asymm 16 11: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 55.108ms asymm 15 12: fourhundredge-0-0-0-22.4079.core1.chic.net.internet2.edu 51.575ms asymm 14 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 61.956ms 14: 144.92.233.112 53.552ms 15: perfsonar02.hep.wisc.edu 53.671ms reached Resume: pmtu 1500 hops 15 back 16
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 1022)
CHTC 1 hour ago 437
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
CHTC — probe output
pass kernel
5.14.0-620.el9.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/cvmfs APPTAINER_CACHEDIR=/var/lib/condor/execute/slot1/dir_1863934/scratch/glide_PQ6Aov/execute/dir_2725669/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/slot1/dir_1863934/scratch/glide_PQ6Aov/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/slot1/dir_1863934/scratch/glide_PQ6Aov/execute/dir_2725669/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2725669 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 FTP_PROXY=http://squid-cs-2360.chtc.wisc.edu:3128 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/slot1/dir_1863934/scratch/glide_PQ6Aov/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.HCCHTPC_US_Wisconsin_osg01_rhel7.idtoken.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=874494 GLIDEIN_Country=US GLIDEIN_DUNESite=US_Wisconsin GLIDEIN_Entry_Name=HCCHTPC_US_Wisconsin_osg01_rhel7 GLIDEIN_Entry_Signature=f3519cb6a51a3aeffbedd1143e10fc3d302acee8 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=osggrid01.hep.wisc.edu osggrid01.hep.wisc.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_slot1_2_etA4Jp GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=5 GLIDEIN_Proxy_URL=osgsquid.hep.wisc.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=CHTC GLIDEIN_SE_VONAME_LOWERCASE=srm://cmssrm.hep.wisc.edu:8443/srm/v2/server?SFN=/hdfs/osg/vo/ GLIDEIN_SEs=cmssrm.hep.wisc.edu GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=CHTC GLIDEIN_Tmp_Dir=/var/lib/condor/execute/slot1/dir_1863934/scratch/glide_PQ6Aov/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=osggrid01.hep.wisc.edu OSG_SITE_NAME=GLOW-OSG OSG_SQUID_LOCATION=osgsquid.hep.wisc.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_8@glidein_1863936_287858340@e2610.chtc.wisc.edu$ "; unset PROMPT_COMMAND PS1=slot1_8@glidein_1863936_287858340@e2610.chtc.wisc.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/cvmfs SINGULARITY_CONTAINER=/var/lib/condor/execute/slot1/dir_1863934/scratch/glide_PQ6Aov/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/slot1/dir_1863934/scratch/glide_PQ6Aov/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/slot1/dir_1863934/scratch/glide_PQ6Aov/execute/dir_2725669/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://squid-cs-2360.chtc.wisc.edu:3128
pass df
737.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p1 1537155984 685535700 773463280 47% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.009ms pmtu 1500 1: 2607:f388:2200:100::3 0.571ms 1: 2607:f388:2200:100::3 0.440ms 2: 2607:f388:0:33f::2 0.478ms 3: perfsonar02.hep.wisc.edu 0.283ms reached Resume: pmtu 1500 hops 3 back 3
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://squid-cs-2360.chtc.wisc.edu:3128
pass user
slot1_2 (uid 5301)
CHTC-DEV-PROD-EP-mwestphall 1 hour ago 1
100.0%
0.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
CHTC-DEV-PROD-EP-mwestphall — probe output
pass kernel
5.14.0-725.el9.aarch64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/pilot/osgvo-pilot-ifi0NS/execute/dir_4544/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-ifi0NS/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-ifi0NS/execute/dir_4544/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=4544 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=CHTC PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_1@CHTC-DEV-PROD-EP-mwestphall.8313dff9e758$ "; unset PROMPT_COMMAND PS1=slot1_1@CHTC-DEV-PROD-EP-mwestphall.8313dff9e758$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/pilot/osgvo-pilot-ifi0NS/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-ifi0NS/execute/dir_4544/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
99.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 263112772 145028616 104646000 59% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.087ms 1: _gateway 0.110ms 2: 128.104.100.2 0.244ms 3: 146.151.164.1 0.367ms 4: perfsonar02.hep.wisc.edu 0.298ms reached Resume: pmtu 1500 hops 4 back 4
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg (uid 1000)
CHTC-Spark-CE1 1 hour ago 30
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
CHTC-Spark-CE1 — probe output
pass kernel
5.14.0-511.el9.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/var/lib/condor/execute/osg01/glide_bqhHn5/execute/dir_1763944/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/osg01/glide_bqhHn5/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/var/lib/condor/execute/osg01/glide_bqhHn5/execute/dir_1763944/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1763944 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=282e7263f1755563ecd8b0c3a7dfabafef58aaed GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/osg01/glide_bqhHn5/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_CHTC-Spark-CE1-gpu.idtoken.idtoken GLIDEIN_CPUS=16 GLIDEIN_ClusterId=6562625 GLIDEIN_Entry_Name=OSG_US_CHTC-Spark-CE1-gpu GLIDEIN_Entry_Signature=67977bd80b18ed481d8dbfaffc6e159d77a113f6 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=chtc-spark-ce1.svc.osg-htc.org chtc-spark-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_fn22m2 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=CHTC-Spark-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=CHTC-Spark GLIDEIN_Tmp_Dir=/var/lib/condor/execute/osg01/glide_bqhHn5/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=chtc-spark-ce1.svc.osg-htc.org OSG_SITE_NAME=CHTC-Spark-CE1 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_13@glidein_1380060_439445689@spark-agpu225.chtc.wisc.edu$ "; unset PROMPT_COMMAND PS1=slot1_13@glidein_1380060_439445689@spark-agpu225.chtc.wisc.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/var/lib/condor/execute/osg01/glide_bqhHn5/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/osg01/glide_bqhHn5/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/osg01/glide_bqhHn5/execute/dir_1763944/scratch _CONDOR_SLOT=slot1_13 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
2614.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p2 3075412536 177785756 2741329720 7% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 4092 1: _gateway 0.153ms 1: _gateway 0.111ms 2: _gateway 0.075ms pmtu 1500 2: 128.104.100.3 0.295ms 3: 146.151.164.9 0.465ms 4: perfsonar02.hep.wisc.edu 0.224ms reached Resume: pmtu 1500 hops 4 back 4
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 4977)
CHTC-ce2000 1 hour ago 9
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
CHTC-ce2000 — probe output
pass kernel
6.12.0-211.28.2.el10_2.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/cvmfs APPTAINER_CACHEDIR=/var/lib/condor/execute/slot1/dir_3523562/scratch/glide_PNttfx/execute/dir_3860653/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/slot1/dir_3523562/scratch/glide_PNttfx/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/var/lib/condor/execute/slot1/dir_3523562/scratch/glide_PNttfx/execute/dir_3860653/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3860653 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el10_2 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el10_2 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 FTP_PROXY=http://squid-cs-2360.chtc.wisc.edu:3128 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/slot1/dir_3523562/scratch/glide_PNttfx/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_CHTC-ce2000.idtoken.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=959075 GLIDEIN_Entry_Name=OSG_US_CHTC-ce2000 GLIDEIN_Entry_Signature=d6cad8461f18c8ae54590f1e4b266d5579b12fb4 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=ce2000.chtc.wisc.edu ce2000.chtc.wisc.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgvopilot_sZDzpo GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=squid-cs-2360.chtc.wisc.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=CHTC-ce2000 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=CHTC GLIDEIN_Tmp_Dir=/var/lib/condor/execute/slot1/dir_3523562/scratch/glide_PNttfx/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el10_2 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el10_2 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el10_2 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ce2000.chtc.wisc.edu OSG_SITE_NAME=CHTC-ce2000 OSG_SQUID_LOCATION=squid-cs-2360.chtc.wisc.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_3523580_149892999@htcss2002.chtc.wisc.edu$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_3523580_149892999@htcss2002.chtc.wisc.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/cvmfs SINGULARITY_CONTAINER=/var/lib/condor/execute/slot1/dir_3523562/scratch/glide_PNttfx/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__8.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/slot1/dir_3523562/scratch/glide_PNttfx/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/slot1/dir_3523562/scratch/glide_PNttfx/execute/dir_3860653/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://squid-cs-2360.chtc.wisc.edu:3128
pass df
60.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/condor_vg-backfill1_1+2162940_2413 66002476 2271004 63715088 4% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.005ms pmtu 1500 1: 2607:f388:2200:100::3 0.602ms 1: 2607:f388:2200:100::3 0.541ms 2: 2607:f388:0:33f::2 0.268ms 3: perfsonar02.hep.wisc.edu 0.204ms reached Resume: pmtu 1500 hops 3 back 3
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://squid-cs-2360.chtc.wisc.edu:3128
pass user
osgvopilot (uid 4984)
CHTC-ce2001 1 hour ago 10
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
CHTC-ce2001 — probe output
pass kernel
5.14.0-620.el9.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/cvmfs APPTAINER_CACHEDIR=/var/lib/condor/execute/slot1/dir_2204214/scratch/glide_s9Wbyq/execute/dir_3103394/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/slot1/dir_2204214/scratch/glide_s9Wbyq/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/var/lib/condor/execute/slot1/dir_2204214/scratch/glide_s9Wbyq/execute/dir_3103394/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3103394 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 FTP_PROXY=http://squid-cs-2360.chtc.wisc.edu:3128 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/slot1/dir_2204214/scratch/glide_s9Wbyq/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_CHTC-ce2001.idtoken.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=688680 GLIDEIN_Entry_Name=OSG_US_CHTC-ce2001 GLIDEIN_Entry_Signature=8f6579a7e947084b55d481664ef32211c8cd34b1 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=ce2001.chtc.wisc.edu ce2001.chtc.wisc.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgvopilot_JtIx7G GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=squid-cs-2360.chtc.wisc.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=CHTC-ce2001 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=CHTC GLIDEIN_Tmp_Dir=/var/lib/condor/execute/slot1/dir_2204214/scratch/glide_s9Wbyq/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ce2000.chtc.wisc.edu OSG_SITE_NAME=CHTC-ce2000 OSG_SQUID_LOCATION=squid-cs-2360.chtc.wisc.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_7@glidein_2204216_68077440@e2605.chtc.wisc.edu$ "; unset PROMPT_COMMAND PS1=slot1_7@glidein_2204216_68077440@e2605.chtc.wisc.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/cvmfs SINGULARITY_CONTAINER=/var/lib/condor/execute/slot1/dir_2204214/scratch/glide_s9Wbyq/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__8.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/slot1/dir_2204214/scratch/glide_s9Wbyq/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/slot1/dir_2204214/scratch/glide_s9Wbyq/execute/dir_3103394/scratch _CONDOR_SLOT=slot1_7 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://squid-cs-2360.chtc.wisc.edu:3128
pass df
738.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p1 1537155984 684688120 774310860 47% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.006ms pmtu 1500 1: 2607:f388:2200:100::3 0.573ms 1: 2607:f388:2200:100::3 0.530ms 2: 2607:f388:0:33f::2 0.404ms 3: perfsonar02.hep.wisc.edu 0.315ms reached Resume: pmtu 1500 hops 3 back 3
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://squid-cs-2360.chtc.wisc.edu:3128
pass user
osgvopilot (uid 4984)
Cincinnati-ARC-OSG-CE1 1 hour ago 8
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Cincinnati-ARC-OSG-CE1 — probe output
pass kernel
4.18.0-553.30.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/scratch-local/glide_e2SYhJ/execute/dir_451030/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch-local/glide_e2SYhJ/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch-local/glide_e2SYhJ/execute/dir_451030/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=451030 CONTAINERSW_FULL_VERSION=apptainer version 1.3.6-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.3.6-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch-local/glide_e2SYhJ/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Cincinnati-ARC-OSG-CE1.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=6550663 GLIDEIN_Entry_Name=OSG_US_Cincinnati-ARC-OSG-CE1 GLIDEIN_Entry_Signature=5a53d17611c99331907a9d86772213f49c94f1d5 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=cincinnati-arc-osg-ce1.svc.opensciencegrid.org cincinnati-arc-osg-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_Z33n6c GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Cincinnati-ARC-OSG-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=76ba0eb3c7154bcebd6c7df094e963d26ecf4544 GLIDEIN_Site=Cincinnati-ARC-OSG GLIDEIN_Tmp_Dir=/scratch-local/glide_e2SYhJ/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.3.6-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.3.6-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.3.6-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=cincinnati-arc-osg-ce1.svc.opensciencegrid.org OSG_SITE_NAME=Cincinnati-ARC-OSG-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_1219766_135339384@compute-4.localdomain$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_1219766_135339384@compute-4.localdomain$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/scratch-local/glide_e2SYhJ/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch-local/glide_e2SYhJ/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch-local/glide_e2SYhJ/execute/dir_451030/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
360.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/localscratch-lv_local_scratch 460365864 59261916 377645132 14% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: arc-osg 0.221ms 1: arc-osg 0.242ms 2: msbb-00g95-hpc.manage.uc.edu 0.458ms asymm 3 3: 10.0.27.17 1.041ms 4: 10.0.27.253 0.664ms 5: 10.0.0.2 1.169ms 6: 129.137.1.3 1.739ms 7: 129.137.1.5 2.707ms 8: cncne-r4-et-0-2-0s812.core.oar.net 2.243ms asymm 9 9: cncno-r5-et-4-3-0s735.core.oar.net 2.440ms asymm 8 10: hundredge-0-0-0-25.1260.core1.cinc.net.internet2.edu 3.949ms asymm 9 11: fourhundredge-0-0-0-23.4079.core2.cinc.net.internet2.edu 9.292ms asymm 14 12: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 9.291ms asymm 13 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 10.863ms asymm 12 14: 144.92.233.112 11.796ms asymm 13 15: perfsonar02.hep.wisc.edu 11.725ms reached Resume: pmtu 1500 hops 15 back 15
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 1001)
Clemson-Palmetto-CE1 1 hour ago 253
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Clemson-Palmetto-CE1 — probe output
pass kernel
4.18.0-553.150.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/tmp/glide_uolpgB/execute/dir_2715530/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_uolpgB/images/htc__rocky__8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__8.sif APPTAINER_TMPDIR=/tmp/glide_uolpgB/execute/dir_2715530/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2715530 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_uolpgB/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Clemson-Palmetto-CE1.idtoken.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=6344501 GLIDEIN_Entry_Name=OSG_US_Clemson-Palmetto-CE1 GLIDEIN_Entry_Signature=4430872225ca10dfe37fc6f0ef13362dae5250ae GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=clemson-palmetto-ce1.svc.opensciencegrid.org clemson-palmetto-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg001_6CWGdL GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Clemson-Palmetto-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=76d5c6db6877413d123d8ddff4f1753932659ba9 GLIDEIN_Site=Clemson-Palmetto GLIDEIN_Tmp_Dir=/tmp/glide_uolpgB/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=clemson-palmetto-ce1.svc.opensciencegrid.org OSG_SITE_NAME=Clemson-Palmetto-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_6@glidein_1169396_171314520@node2048.palmetto.clemson.edu$ "; unset PROMPT_COMMAND PS1=slot1_6@glidein_1169396_171314520@node2048.palmetto.clemson.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/tmp/glide_uolpgB/images/htc__rocky__8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__8.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_uolpgB/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_uolpgB/execute/dir_2715530/scratch _CONDOR_SLOT=slot1_6 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
2695.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sdb1 2927240708 101272620 2825968088 4% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: _gateway 18.336ms 1: _gateway 6.341ms 2: 205.186.62.81 0.622ms 3: xe-1-1-1-816-t01-cle.culr.net 1.260ms 4: xe-1-1-1-816-t01-sox.culr.net 11.691ms asymm 9 5: sidco-to-56mar-al2s.sox.net 17.827ms asymm 12 6: i2-to-sox-re.sidco.sox.net 18.212ms asymm 9 7: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 26.275ms asymm 12 8: fourhundredge-0-0-0-23.4079.core2.cinc.net.internet2.edu 26.325ms asymm 11 9: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 25.068ms asymm 10 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 26.350ms asymm 9 11: 144.92.233.112 27.775ms asymm 10 12: perfsonar02.hep.wisc.edu 27.794ms reached Resume: pmtu 9000 hops 12 back 12
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg001 (uid 527198)
Colgate-CCARE-EP 7 hours ago 2
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Colgate-CCARE-EP — probe output
pass kernel
6.8.0-124-generic
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/pilot/osgvo-pilot-BUdLDj/execute/dir_398925/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-BUdLDj/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-BUdLDj/execute/dir_398925/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=398925 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Colgate ITS OSG_SQUID_LOCATION=http://149.43.3.11:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_7@Colgate-CCARE-EP.452ac55443fd$ "; unset PROMPT_COMMAND PS1=slot1_7@Colgate-CCARE-EP.452ac55443fd$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/pilot/osgvo-pilot-BUdLDj/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-BUdLDj/execute/dir_398925/scratch _CONDOR_SLOT=slot1_7 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://149.43.3.11:3128
pass df
950.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/ccare--g04--vg-ccare--g04--lv 1837860872 747887992 996540868 43% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.095ms 1: _gateway 0.109ms 2: 149.43.3.1 0.637ms 3: edge-router-ocon-bgp.colgate.edu 1.298ms 4: 172.16.2.2 0.803ms asymm 3 5: alb-9204-colgate.nysernet.net 3.930ms asymm 4 6: syr-57c3-alb-9204.nysernet.net 7.026ms asymm 5 7: buf-55a1-syr-57c3.nysernet.net 10.052ms asymm 6 8: buf-9208-buf-55a1.nysernet.net 12.795ms asymm 7 9: i2-clev-buf-9208.nysernet.net 15.618ms asymm 8 10: fourhundredge-0-0-0-21.4079.core1.clev.net.internet2.edu 26.797ms asymm 17 11: fourhundredge-0-0-0-3.4079.core1.cinc.net.internet2.edu 26.028ms asymm 16 12: fourhundredge-0-0-0-23.4079.core2.cinc.net.internet2.edu 25.243ms asymm 15 13: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 24.486ms asymm 14 14: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 26.082ms asymm 13 15: 144.92.233.112 27.037ms asymm 14 16: perfsonar02.hep.wisc.edu 26.985ms reached Resume: pmtu 1500 hops 16 back 16
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://149.43.3.11:3128
pass user
osg (uid 1000)
Duke-NCShare-CE1 1 hour ago 44
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Duke-NCShare-CE1 — probe output
pass kernel
6.8.0-111-generic
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/scratch/glide_fRWIih/execute/dir_3040016/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_fRWIih/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_fRWIih/execute/dir_3040016/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3040016 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_fRWIih/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Duke-NCShare-CE1.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=861419 GLIDEIN_Entry_Name=OSG_US_Duke-NCShare-CE1 GLIDEIN_Entry_Signature=57efe65f90b82da4998e4860038edeba1ea6a7a6 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=duke-ncshare-ce1.svc.osg-htc.org duke-ncshare-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_nosg1_6ODMk7 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Duke-NCShare-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=Duke-NCShare GLIDEIN_Tmp_Dir=/scratch/glide_fRWIih/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=duke-ncshare-ce1.svc.osg-htc.org OSG_SITE_NAME=Duke-NCShare-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2934812_162251982@compute-10$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2934812_162251982@compute-10$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/scratch/glide_fRWIih/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_fRWIih/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_fRWIih/execute/dir_3040016/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
696.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/ubuntu--vg-lstor--lv 824572824 52141272 730472128 7% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.385ms asymm 2 1: _gateway 0.381ms asymm 2 2: rtp-gw-to-scidmz-gw-1.ncren.net 0.674ms 3: bundle-ether250.613.core1.rale.net.internet2.edu 3.981ms 4: fourhundredge-0-0-0-6.4079.core1.ashb.net.internet2.edu 21.182ms asymm 10 5: fourhundredge-0-0-0-1.4079.core1.clev.net.internet2.edu 21.043ms asymm 9 6: fourhundredge-0-0-0-2.4079.core1.eqch.net.internet2.edu 22.036ms asymm 8 7: fourhundredge-0-0-0-6.4079.core1.star.net.internet2.edu 23.627ms 8: 144.92.233.112 23.947ms 9: perfsonar02.hep.wisc.edu 23.894ms reached Resume: pmtu 1500 hops 9 back 10
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
nosg1 (uid 3000012)
FANDM-ITS-CE1 1 hour ago 32
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
FANDM-ITS-CE1 — probe output
pass kernel
4.18.0-553.50.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/scratch/glide_u34unJ/execute/dir_330200/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_u34unJ/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_u34unJ/execute/dir_330200/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=330200 CONTAINERSW_FULL_VERSION=apptainer version 1.5.2-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.2-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_u34unJ/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_FANDM-ITS-CE1.idtoken.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=6764592 GLIDEIN_Entry_Name=OSG_US_FANDM-ITS-CE1 GLIDEIN_Entry_Signature=09d31ab425a304f84e787e731631c545527d9d51 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=fandm-its-ce1.svc.opensciencegrid.org fandm-its-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_uJfGro GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=FANDM-ITS-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins5@gfactory-1.osg-htc.org GLIDEIN_Signature=e977401dd4ee29c9252d463c389cf2702ca41418 GLIDEIN_Site=FANDM-ITS GLIDEIN_Tmp_Dir=/scratch/glide_u34unJ/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.2-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.2-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.2-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=fandm-its-ce1.svc.opensciencegrid.org OSG_SITE_NAME=FANDM-ITS-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_8@glidein_926143_141538800@n01.cluster$ "; unset PROMPT_COMMAND PS1=slot1_8@glidein_926143_141538800@n01.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/scratch/glide_u34unJ/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_u34unJ/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_u34unJ/execute/dir_330200/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
992.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p5 1092553500 52295808 1040257692 5% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: cv01.cluster 0.270ms 1: cv01.cluster 0.323ms 2: 10.68.128.1 8.650ms 3: 10.68.0.9 5.051ms 4: 10.68.0.14 0.621ms 5: 10.255.255.25 1.199ms 6: 10.255.255.34 1.089ms asymm 5 7: 164.153.140.194 2.900ms asymm 6 8: 172.24.1.20 9.840ms asymm 7 9: hundredge-0-0-0-25.68.core1.pitt.net.internet2.edu 12.964ms asymm 8 10: fourhundredge-0-0-0-3.4079.core1.clev.net.internet2.edu 19.856ms asymm 13 11: fourhundredge-0-0-0-2.4079.core1.eqch.net.internet2.edu 22.740ms asymm 12 12: fourhundredge-0-0-0-6.4079.core1.star.net.internet2.edu 20.628ms asymm 11 13: 144.92.233.112 22.636ms asymm 12 14: perfsonar02.hep.wisc.edu 22.867ms reached Resume: pmtu 1500 hops 14 back 14
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 1004)
FIU-IRCC-CE1 1 hour ago 61
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
FIU-IRCC-CE1 — probe output
pass kernel
5.14.0-611.54.3.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/scratch/osg/glide_ywXJ1D/execute/dir_3338804/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg/glide_ywXJ1D/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg/glide_ywXJ1D/execute/dir_3338804/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3338804 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/osg/glide_ywXJ1D/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg/glide_ywXJ1D/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_FIU-IRCC.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=861439 GLIDEIN_Entry_Name=OSG_US_FIU-IRCC GLIDEIN_Entry_Signature=579764b33bde8e2fa74d75ed653784bccd599768 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=fiu-ircc-ce1.svc.osg-htc.org fiu-ircc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_8UxBqR GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=FIU-IRCC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=f90b138f62b42316e2088dea04273e4626194e65 GLIDEIN_Site=FIU-IRCC GLIDEIN_Tmp_Dir=/scratch/osg/glide_ywXJ1D/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/osg/glide_ywXJ1D/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/osg/glide_ywXJ1D/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=fiu-ircc-ce1.svc.osg-htc.org OSG_SITE_NAME=FIU-IRCC-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3481764_143140340@osg007.roary.net$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3481764_143140340@osg007.roary.net$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/scratch/osg/glide_ywXJ1D/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg/glide_ywXJ1D/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg/glide_ywXJ1D/execute/dir_3338804/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
39605.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 192.168.232.21@tcp:192.168.232.22@tcp:192.168.232.23@tcp:192.168.232.24@tcp:/scratch 50398964480 8358038204 41528974468 17% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: osgce.roary.net 0.262ms 1: osgce.roary.net 0.145ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg (uid 84159)
FSU-HNPGRID 1 hour ago 68
100.0%
100.0%
98.5%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
FSU-HNPGRID — probe output
pass kernel
5.14.0-687.49.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/scratch/glide_3k93Y8/execute/dir_2842389/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_3k93Y8/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_3k93Y8/execute/dir_2842389/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2842389 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_3k93Y8/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_FSU-HNPGRID.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=6558045 GLIDEIN_Entry_Name=OSG_US_FSU-HNPGRID GLIDEIN_Entry_Signature=6e6cf7a1b6b10c90a073b8877250bcb0357d4ac2 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=fsu-hnpgrid-ce1.svc.osg-htc.org fsu-hnpgrid-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_Gzneah GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=FSU-HNPGRID GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=e977401dd4ee29c9252d463c389cf2702ca41418 GLIDEIN_Site=OSG_US_FSU_HNPGRID GLIDEIN_Tmp_Dir=/scratch/glide_3k93Y8/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=fsu-hnpgrid-ce1.svc.osg-htc.org OSG_SITE_NAME=FSU-HNPGRID PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_1072146_67167660@scigrid8.physics.fsu.edu$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_1072146_67167660@scigrid8.physics.fsu.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/scratch/glide_3k93Y8/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_3k93Y8/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_3k93Y8/execute/dir_2842389/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1354.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda4 3899867640 2480088408 1419779232 64% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: bvi100411551.dsl-pe.coas-physics.net.fsu.edu 0.649ms 1: vlan99.6506.ken.fsu.edu 0.721ms 2: no reply 3: no reply 4: no reply 5: illegally-used-at.fsu 1.642ms 6: 128.186.253.197 1.776ms 7: ae22.1901.rsb-br.net.fsu.edu 2.017ms 8: tlh-flrcore-asr9010-1-te0203-1902.net.flrnet.org 4.590ms 9: tlh-bb-rtr-2-fh0001-1.rtr.net.flrnet.org 7.708ms asymm 11 10: jax-bb-rtr-1-fh0001-1.rtr.net.flrnet.org 8.346ms 11: prov-i2-atla-renet-1070.net.flrnet.org 7.715ms 12: fourhundredge-0-0-0-6.4079.core1.atla.net.internet2.edu 30.163ms asymm 24 13: fourhundredge-0-0-0-22.4079.core2.atla.net.internet2.edu 30.151ms asymm 23 14: fourhundredge-0-0-0-4.4079.core2.nash.net.internet2.edu 29.611ms asymm 22 15: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 29.495ms asymm 21 16: fourhundredge-0-0-0-21.4079.core2.cinc.net.internet2.edu 31.429ms asymm 20 17: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 31.210ms asymm 19 18: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 31.057ms 19: 144.92.233.112 32.411ms 20: perfsonar02.hep.wisc.edu 32.421ms reached Resume: pmtu 1500 hops 20 back 21
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 9004)
GSU-ACIDS 1 hour ago 13
15.4%
15.4%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
GSU-ACIDS — probe output
pass kernel
6.8.0-111-generic
pass env
APPTAINER_CACHEDIR=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch BASH_ENV=/usr/share/lmod/lmod/init/bash BATCH_SYSTEM=HTCondor BLAHPD_LOCATION=/home/users/svc-osgaccnt/bosco_ce1/glite CONDORCE_COLLECTOR_HOST=gsu-acids-ce1.svc.opensciencegrid.org:9619 CONDOR_CONFIG=/scratch/users/svc-osgaccnt/glide_lYRqYa/condor_config CONDOR_PARENT_ID=acidsgn005:849985:1790772824 CONDOR_PROCD_ADDRESS=/scratch/users/svc-osgaccnt/glide_lYRqYa/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/scratch/users/svc-osgaccnt/glide_lYRqYa/log/procd_address CONDOR_STARTER_PID=849985 CONTAINERSW_FULL_VERSION=apptainer version 1.4.1 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/sysapps/ubuntu-applications/apptainer/1.4.1/install/bin/apptainer CONTAINERSW_VERSION=1.4.1 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 CVMFSEXEC_REPOS=oasis.opensciencegrid.org,singularity.opensciencegrid.org CVMFSEXEC_TARBALL=http://gfactory-1.osg-htc.org/cvmfsexec-osg-v4.38.tar.gz DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/330104/bus ENVIRONMENT=BATCH GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/users/svc-osgaccnt/glide_lYRqYa/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_GSU-ACIDS.idtoken.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=6563058 GLIDEIN_DEBUG_OPTIONS= GLIDEIN_Entry_Name=OSG_US_GSU-ACIDS GLIDEIN_Entry_Signature=0f8ed206c193e12e85708aca9669000552e81494 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=gsu-acids-ce1.svc.opensciencegrid.org gsu-acids-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_svc-osgaccnt_gwkMT3 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=arcosg04.rs.gsu.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=GSU-ACIDS GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=GSU-ACIDS GLIDEIN_Tmp_Dir=/scratch/users/svc-osgaccnt/glide_lYRqYa/tmp GLIDEIN_UUID=b6d7346a-a092-47ca-a82d-e795905aa4f3 GLITE_LOCATION=/home/users/svc-osgaccnt/bosco_ce1/glite GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.1 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/sysapps/ubuntu-applications/apptainer/1.4.1/install/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.1 GWMS_PATH=/scratch/users/svc-osgaccnt/glide_lYRqYa/.gwms.d/bin: GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/sysapps/ubuntu-applications/apptainer/1.4.1/install/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.1 HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch HOSTNAME=acidsgn005 HYDRA_BOOTSTRAP=slurm HYDRA_LAUNCHER_EXTRA_ARGS=--external-launcher I_MPI_HYDRA_BOOTSTRAP=slurm I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_OSG_US_GSU-ACIDS/tokens.tgz JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/sysapps/ubuntu-applications/apptainer/1.4.1/install/libexec LMOD_CMD=/usr/share/lmod/lmod/libexec/lmod LMOD_DIR=/usr/share/lmod/lmod/libexec LMOD_PKG=/usr/share/lmod/lmod LMOD_ROOT=/usr/share/lmod LMOD_SETTARG_FULL_SUPPORT=no LMOD_VERSION=8.6.19 LMOD_sys=Linux LOADEDMODULES=apptainer/1.4.1 LOGNAME=svc-osgaccnt MANPATH=/usr/share/lmod/lmod/share/man: MKL_NUM_THREADS=1 MODULEPATH=/application/ubuntumodules/localmodules::/etc/lmod/modules:/usr/share/lmod/lmod/modulefiles MODULEPATH_ROOT=/usr/share/modulefiles MODULESHOME=/usr/share/lmod/lmod NODE_COUNT=64 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OLDPWD=/scratch/users/svc-osgaccnt/glide_lYRqYa/logs/b6d7346a-a092-47ca-a82d-e795905aa4f3 OMPI_MCA_plm_slurm_args=--external-launcher OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=UNAVAILABLE OSG_DATA=UNAVAILABLE OSG_GRID=/home/users/svc-osgaccnt/bosco-osg-wn-client OSG_HOSTNAME=gsu-acids-ce1.svc.opensciencegrid.org OSG_SITE_NAME=GSU-ACIDS OSG_SQUID_LOCATION=arcosg04.rs.gsu.edu:3128 OSG_WN_TMP=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch/.local-tmp PATH=/scratch/users/svc-osgaccnt/glide_lYRqYa/.gwms.d/bin::/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sysapps/ubuntu-applications/apptainer/1.4.1/install/bin:/sysapps/ubuntu-applications/apptainer/1.4.1/install/libexec/apptainer/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/users/svc-osgaccnt/.local/bin:/home/users/svc-osgaccnt/bin PREEMPT_GRACE_TIME=10000000 PRTE_MCA_plm_slurm_args=--external-launcher PWD=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SCRATCH=/scratch/users/svc-osgaccnt/4516563 SHELL=/bin/bash SHLVL=2 SINGULARITY_CACHEDIR=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch SINGULARITY_FULL_VERSION=apptainer version 1.4.1 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_MODE=privileged SINGULARITY_PATH=/sysapps/ubuntu-applications/apptainer/1.4.1/install/bin/apptainer SINGULARITY_VERSION=1.4.1 SLOTS_LAYOUT=partitionable SLURMD_NODENAME=acidsgn005 SLURM_CLUSTER_NAME=acids SLURM_CONF=/var/spool/slurmd/conf-cache/slurm.conf SLURM_CPUS_ON_NODE=64 SLURM_GTIDS=0 SLURM_JOBID=4516563 SLURM_JOB_ACCOUNT=opensciencegrid SLURM_JOB_CPUS_PER_NODE=64 SLURM_JOB_END_TIME=1790939486 SLURM_JOB_GID=100513 SLURM_JOB_ID=4516563 SLURM_JOB_NAME=bl_ZTaeof SLURM_JOB_NODELIST=acidsgn005 SLURM_JOB_NUM_NODES=1 SLURM_JOB_PARTITION=qOSG SLURM_JOB_QOS=normal SLURM_JOB_START_TIME=1790766686 SLURM_JOB_UID=330104 SLURM_JOB_USER=svc-osgaccnt SLURM_LOCALID=0 SLURM_MEM_PER_NODE=250000 SLURM_NNODES=1 SLURM_NODEID=0 SLURM_NODELIST=acidsgn005 SLURM_NPROCS=64 SLURM_NTASKS=64 SLURM_NTASKS_PER_NODE=64 SLURM_OOM_KILL_STEP=0 SLURM_PRIO_PROCESS=0 SLURM_PROCID=0 SLURM_SCRIPT_CONTEXT=prolog_task SLURM_SUBMIT_DIR=/home/users/svc-osgaccnt/bosco_ce1/sandbox/99c2/99c26d20/gsu-acids-ce1.svc.opensciencegrid.org_9619_gsu-acids-ce1.svc.opensciencegrid.org_45227.0_1790766156 SLURM_SUBMIT_HOST=arclogin02 SLURM_TASKS_PER_NODE=64 SLURM_TASK_PID=786236 SLURM_TOPOLOGY_ADDR=acidsgn005 SLURM_TOPOLOGY_ADDR_PATTERN=node SSH_CLIENT=10.245.14.22 44234 22 SSH_CONNECTION=10.245.14.22 44234 10.245.18.86 22 TEMP=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch TMPDIR=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime USER=svc-osgaccnt XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop XDG_RUNTIME_DIR=/run/user/330104 XDG_SESSION_CLASS=user XDG_SESSION_ID=4294967295 XDG_SESSION_TYPE=tty _=/scratch/users/svc-osgaccnt/glide_lYRqYa/main/condor/sbin/condor_master _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_812276=812283:1790767209:4035984618 _CONDOR_ANCESTOR_812283=849985:1790772822:2919944808 _CONDOR_ANCESTOR_849985=850440:1790772848:3951539570 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/users/svc-osgaccnt/glide_lYRqYa/main/condor/bin _CONDOR_CHIRP_CONFIG=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/user/.chirp.config _CONDOR_CREDS=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch/.condor_creds _CONDOR_EXECUTE=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute _CONDOR_JOB_AD=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch/.job.ad _CONDOR_JOB_IWD=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch _CONDOR_SLOT=slot1_42 _CONDOR_WRAPPER_ERROR_FILE=/scratch/users/svc-osgaccnt/glide_lYRqYa/execute/dir_849985/scratch/.job_wrapper_failure _LMFILES_=/application/ubuntumodules/localmodules/apptainer/1.4.1 _ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0ge30sCm1UID0gewphcHB0YWluZXIgPSB7CmZuID0gIi9hcHBsaWNhdGlvbi91YnVudHVtb2R1bGVzL2xvY2FsbW9kdWxlcy9hcHB0YWluZXIvMS40LjEiLApmdWxsTmFtZSA9ICJhcHB0YWluZXIvMS40LjEiLApsb2FkT3JkZXIgPSAxLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImFwcHRhaW5lci8xLjQuMSIsCndWID0gIjAwMDAwMDAwMS4wMDAwMDAwMDQuMDAwMDAwMDAxLip6ZmluYWwiLAp9LAp9LAptcGF0aEEgPSB7CiIvYXBwbGljYXRp _ModuleTable002_=b24vdWJ1bnR1bW9kdWxlcy9sb2NhbG1vZHVsZXMiLCAiICIsICIvZXRjL2xtb2QvbW9kdWxlcyIsICIvdXNyL3NoYXJlL2xtb2QvbG1vZC9tb2R1bGVmaWxlcyIsCn0sCnN5c3RlbUJhc2VNUEFUSCA9ICIvYXBwbGljYXRpb24vdWJ1bnR1bW9kdWxlcy9sb2NhbG1vZHVsZXM6IDovZXRjL2xtb2QvbW9kdWxlczovdXNyL3NoYXJlL2xtb2QvbG1vZC9tb2R1bGVmaWxlcyIsCn0K _ModuleTable_Sz_=2 __LMOD_REF_COUNT_LD_LIBRARY_PATH=/sysapps/ubuntu-applications/apptainer/1.4.1/install/libexec:1 __LMOD_REF_COUNT_MODULEPATH=/application/ubuntumodules/localmodules:1;/etc/lmod/modules:1;/usr/share/lmod/lmod/modulefiles:1 __LMOD_REF_COUNT_PATH=/sysapps/ubuntu-applications/apptainer/1.4.1/install/bin:1;/sysapps/ubuntu-applications/apptainer/1.4.1/install/libexec/apptainer/bin:1;/usr/local/sbin:1;/usr/local/bin:1;/usr/sbin:1;/usr/bin:1;/sbin:1;/bin:1;/usr/games:1;/usr/local/games:1;/snap/bin:1;/home/users/svc-osgaccnt/.local/bin:1;/home/users/svc-osgaccnt/bin:1
pass df
159733.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on beegfs_nodev 270018342912 102525657088 167492685824 38% /scratch
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 9000 1: 10.245.10.3 0.452ms 1: 10.245.10.2 0.451ms 2: rtcdfars-rtlsdc01rs-gw.gsu.edu 0.503ms 3: fwcdfxrs-rtcdfars-rtcdfdrs-gw.gsu.edu 0.153ms 4: fwcdfrs-mgmt.gsunet 0.379ms pmtu 1500 4: fwcdfrs-rtcdfars.gsunet 0.706ms 5: rtcdfaedge-rtcdfars-gw.gsu.edu 0.418ms 6: SOX-10G-Primary-TO-GSU.gsu.edu 2.629ms 7: sidco-to-56mar-al2s.sox.net 9.284ms asymm 9 8: i2-to-sox-re.sidco.sox.net 9.686ms asymm 10 9: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 20.153ms asymm 17 10: fourhundredge-0-0-0-23.4079.core2.cinc.net.internet2.edu 20.171ms asymm 16 11: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 28.118ms asymm 15 12: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 20.959ms asymm 14 13: 144.92.233.112 22.172ms asymm 15 14: perfsonar02.hep.wisc.edu 22.098ms reached Resume: pmtu 1500 hops 14 back 17
fail nrp_images
Failed to download https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt: <urlopen error _ssl.c:983: The handshake operation timed out>
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy arcosg04.rs.gsu.edu:3128
pass user
svc-osgaccnt (uid 330104)
GSU-Adonis-EP 50 mins ago 1
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
GSU-Adonis-EP — probe output
pass kernel
6.8.0-136-lowlatency
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/pilot/osgvo-pilot-svRps3/execute/dir_326171/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-svRps3/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-svRps3/execute/dir_326171/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=326171 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=GSU-Adonis OSG_SQUID_LOCATION=http://arcosg04.rs.gsu.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_14@GSU-Adonis-EP.osg-endpoint-5fb6854cd7-8c4fb$ "; unset PROMPT_COMMAND PS1=slot1_14@GSU-Adonis-EP.osg-endpoint-5fb6854cd7-8c4fb$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/pilot/osgvo-pilot-svRps3/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-svRps3/execute/dir_326171/scratch _CONDOR_SLOT=slot1_14 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://arcosg04.rs.gsu.edu:3128
pass df
33.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on tmpfs 67108864 32150816 34958048 48% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1430 1?: [LOCALHOST] pmtu 1430 1: 10-245-19-162.ceph-csi-cephfs-nodeplugin-http-metrics.ceph-csi.svc.cluster.local 0.034ms 1: 10-245-19-162.ceph-csi-cephfs-nodeplugin-http-metrics.ceph-csi.svc.cluster.local 0.048ms 2: 10.245.18.3 0.389ms 3: rtcdfars-rtlsdc01rs-gw.gsu.edu 0.452ms 4: fwcdfxrs-rtcdfars-rtcdfdrs-gw.gsu.edu 0.156ms 5: fwcdfrs-rtcdfdrs.gsunet 0.776ms 6: rtcdfaedge-rtcdfars-gw.gsu.edu 0.478ms 7: SOX-10G-Primary-TO-GSU.gsu.edu 2.389ms 8: no reply 9: sidco-to-365dc-re.sox.net 8.728ms asymm 10 10: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 20.285ms asymm 18 11: fourhundredge-0-0-0-22.4079.core2.cinc.net.internet2.edu 20.996ms asymm 17 12: fourhundredge-0-0-0-21.4079.core2.cinc.net.internet2.edu 19.946ms asymm 17 13: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 20.517ms asymm 16 14: 144.92.233.112 22.266ms asymm 16 15: 144.92.233.112 21.888ms asymm 16 16: perfsonar02.hep.wisc.edu 21.716ms reached Resume: pmtu 1430 hops 16 back 18
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://arcosg04.rs.gsu.edu:3128
pass user
osg (uid 1000)
GWU-Viper-CE1 1 hour ago 42
100.0%
14.3%
0.0%
14.3%
kernel env df firewall mtu nrp_images http_proxy user
GWU-Viper-CE1 — probe output
pass kernel
5.14.0-570.52.1.el9_6.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/scratch/osg/dir_1344372/glide_XDFpfs/execute/dir_687071/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg/dir_1344372/glide_XDFpfs/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg/dir_1344372/glide_XDFpfs/execute/dir_687071/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=687071 CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg/dir_1344372/glide_XDFpfs/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_GWU-Viper-CE1.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=805868 GLIDEIN_Entry_Name=OSG_US_GWU-Viper-CE1 GLIDEIN_Entry_Signature=39236ebe7f0b3171c0c611d6137b787fb960d112 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=gwu-viper-ce1.svc.osg-htc.org gwu-viper-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_1FSyht GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=GWU-Viper-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=ad419b1b3db36bf5c7633f8e9c4c175eb881dd81 GLIDEIN_Site=GWU-Viper GLIDEIN_Tmp_Dir=/scratch/osg/dir_1344372/glide_XDFpfs/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=gwu-viper-ce1.svc.osg-htc.org OSG_SITE_NAME=GWU-Viper-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_1344375_577077780@node007$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_1344375_577077780@node007$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/scratch/osg/dir_1344372/glide_XDFpfs/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg/dir_1344372/glide_XDFpfs/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg/dir_1344372/glide_XDFpfs/execute/dir_687071/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
815.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 960302824 56574260 854874104 7% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.008ms pmtu 1500 1: 2620:118:500f:2b8a::1 2.422ms 1: 2620:118:500f:2b8a::1 3.671ms 2: 2001:468:9:208:: 1.860ms 3: no reply 4: no reply 5: 2001:468:0:1::13f 17.528ms asymm 6 6: 2607:f388:0:2204:: 17.872ms 7: perfsonar02.hep.wisc.edu 17.866ms reached Resume: pmtu 1500 hops 7 back 8
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 1002)
Harrisburg-HT24A01-EP 1 hour ago 1
100.0%
0.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Harrisburg-HT24A01-EP — probe output
pass kernel
5.14.0-611.54.1.el9_7.aarch64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/pilot/osgvo-pilot-wENtzE/execute/dir_18176/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-wENtzE/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-wENtzE/execute/dir_18176/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=18176 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Harrisburg - CARC PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_1@Harrisburg-HT24A01-EP.65b84c542c91$ "; unset PROMPT_COMMAND PS1=slot1_1@Harrisburg-HT24A01-EP.65b84c542c91$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/pilot/osgvo-pilot-wENtzE/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-wENtzE/execute/dir_18176/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
6987.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme1n1 7499379032 172988432 7326390600 3% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.159ms 1: _gateway 0.175ms 2: 172.16.107.5 0.472ms 3: 172.30.191.251 0.892ms 4: 50.173.121.1 0.980ms 5: te-0-3-0-1-3989-sur03.lowerpaxton.pa.pitt.comcast.net 2.033ms 6: be-5-sur02.lowerpaxton.pa.pitt.comcast.net 1.936ms 7: be-84-rar01.lancaster.pa.pitt.comcast.net 3.717ms 8: be-34-ar01.mckeesport.pa.pitt.comcast.net 11.603ms 9: be-31631-cs03.pittsburgh.pa.ibone.comcast.net 11.657ms 10: be-4411-pe11.ashburn.va.ibone.comcast.net 20.491ms asymm 11 11: bundle-ether5.agg1.ashb.net.internet2.edu 20.606ms asymm 12 12: no reply 13: no reply 14: no reply 15: no reply 16: 198.71.47.15 30.985ms asymm 18 17: perfsonar02.hep.wisc.edu 30.973ms reached Resume: pmtu 1500 hops 17 back 20
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg (uid 1000)
Hawaii-Koa-CE1 1 hour ago 75
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Hawaii-Koa-CE1 — probe output
pass kernel
5.14.0-687.42.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/mnt/nfs/scratch/osg01/glide_1zSVio/execute/dir_4115876/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/mnt/nfs/scratch/osg01/glide_1zSVio/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/mnt/nfs/scratch/osg01/glide_1zSVio/execute/dir_4115876/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=4115876 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/mnt/nfs/scratch/osg01/glide_1zSVio/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Hawaii-Koa-CE1.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=874428 GLIDEIN_Entry_Name=OSG_US_Hawaii-Koa-CE1 GLIDEIN_Entry_Signature=689aeee4c610847fbb0a46e9ece9098cc5ca32d9 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=hawaii-koa-ce1.svc.osg-htc.org hawaii-koa-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_6UvVZh GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Hawaii-Koa-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=Hawaii-Koa GLIDEIN_Tmp_Dir=/mnt/nfs/scratch/osg01/glide_1zSVio/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=hawaii-koa-ce1.svc.osg-htc.org OSG_SITE_NAME=Hawaii-Koa-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3305342_472524768@cn-03-23-08.mana.ci.its.hawaii.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3305342_472524768@cn-03-23-08.mana.ci.its.hawaii.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/mnt/nfs/scratch/osg01/glide_1zSVio/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/mnt/nfs/scratch/osg01/glide_1zSVio/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/mnt/nfs/scratch/osg01/glide_1zSVio/execute/dir_4115876/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
9420.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 10.0.0.99:/datastore/scratch/osg01/glide_1zSVio/execute/dir_4115876 10585395200 707674112 9877721088 7% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: _gateway 0.242ms 1: _gateway 0.240ms 2: et-1-2-0-2500-scidmz-re0.uhnet.net 0.327ms 3: et-0-2-0-35-ohelo-re0.uhnet.net 0.308ms 4: internet2-uhm-losa.uhnet.net 49.143ms 5: fourhundredge-0-0-0-2.4079.core2.salt.net.internet2.edu 94.584ms asymm 13 6: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 94.166ms asymm 12 7: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 94.625ms asymm 11 8: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 126.462ms asymm 10 9: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 94.722ms 10: 144.92.233.112 96.345ms 11: perfsonar02.hep.wisc.edu 96.314ms reached Resume: pmtu 9000 hops 11 back 12
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 40002)
IRISHEP-SSL-RIVER-DEV-BACKFILL 1 hour ago 65
100.0%
98.5%
100.0%
98.5%
kernel env df firewall mtu nrp_images http_proxy user
IRISHEP-SSL-RIVER-DEV-BACKFILL — probe output
pass kernel
5.14.0-687.51.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/pilot/osgvo-pilot-rEwKIu/execute/dir_540691/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-rEwKIu/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-rEwKIu/execute/dir_540691/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=540691 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=IRISHEP-SSL-UCHICAGO OSG_SQUID_LOCATION=squid.grid.uchicago.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_28@IRISHEP-SSL-RIVER-DEV-BACKFILL.river-c002.ssl-hep.org$ "; unset PROMPT_COMMAND PS1=slot1_28@IRISHEP-SSL-RIVER-DEV-BACKFILL.river-c002.ssl-hep.org$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/pilot/osgvo-pilot-rEwKIu/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-rEwKIu/execute/dir_540691/scratch _CONDOR_SLOT=slot1_28 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=squid.grid.uchicago.edu:3128
pass df
258.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sdb3 765870024 456246356 270646184 63% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.012ms pmtu 1500 1: 2605:9a00:10:200c::2 0.272ms 1: 2605:9a00:10:200c::2 0.360ms 2: hinds-a-to-hinds-b.sci.uchicago.edu 0.224ms asymm 1 3: a08-s004-u3-s251b-to-h01-391-s500a.p2p.uchicago.net 0.372ms asymm 2 4: b65-ll129-s500-to-h01-391-l21-s500.p2p.uchicago.net 0.290ms asymm 2 5: perfsonar02.hep.wisc.edu 11.881ms reached Resume: pmtu 1500 hops 5 back 5
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy squid.grid.uchicago.edu:3128
pass user
nobody (uid 65534)
ISI 1 hour ago 1
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
ISI — probe output
pass kernel
4.18.0-553.168.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/scitech/nas/scratch-90-days/alphafold3,/cvmfs APPTAINER_CACHEDIR=/var/lib/condor/execute/dir_2867171/scratch/glide_lI9kHV/execute/dir_2990795/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_2867171/scratch/glide_lI9kHV/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_2867171/scratch/glide_lI9kHV/execute/dir_2990795/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2990795 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_2867171/scratch/glide_lI9kHV/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_ISI_osg.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=683367 GLIDEIN_Country=US GLIDEIN_Entry_Name=OSG_US_ISI_osg GLIDEIN_Entry_Signature=729b6e78a408301d9352388e8a9be59dcf34985d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=osg.isi.edu osg.isi.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_Dy19OH GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=workflow.isi.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=ISI GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=f90b138f62b42316e2088dea04273e4626194e65 GLIDEIN_Site=ISI GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_2867171/scratch/glide_lI9kHV/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=osg.isi.edu OSG_SITE_NAME=ISI OSG_SQUID_LOCATION=workflow.isi.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_2867174_283279961@compute-2.isi.edu$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_2867174_283279961@compute-2.isi.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/scitech/nas/scratch-90-days/alphafold3,/cvmfs SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_2867171/scratch/glide_lI9kHV/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_2867171/scratch/glide_lI9kHV/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_2867171/scratch/glide_lI9kHV/execute/dir_2990795/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=workflow.isi.edu:3128
pass df
72.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/rl-root 101697536 25956300 75741236 26% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] 0.003ms pmtu 1500 1: 2606:d5c0:0:1::1 0.805ms 1: 2606:d5c0:0:1::1 0.923ms 2: 2606:d5c0::1 1.117ms 3: 2001:1878:15::1 1.710ms 4: lax-agg10--losa4-agg-01--400g--01.cenic.net 1.591ms 5: lax-agg10-hpr--lax-agg10-dc.cenic.net 1.808ms 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: 2607:f388:0:2204:: 56.746ms asymm 13 13: perfsonar02.hep.wisc.edu 54.952ms reached Resume: pmtu 1500 hops 13 back 15
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy workflow.isi.edu:3128
pass user
osg (uid 61082)
IU-Jetstream2-Backfill 1 hour ago 99
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
IU-Jetstream2-Backfill — probe output
pass kernel
5.15.0-179-generic
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/ospool-public-data APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-tufCYy/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-tufCYy/execute/dir_8152 BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=8152 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Pervasive Technology Institute PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_1@IU-Jetstream2-Backfill.blue-8866699c5-qg52r$ "; unset PROMPT_COMMAND PS1=slot1_1@IU-Jetstream2-Backfill.blue-8866699c5-qg52r$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/ospool-public-data SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/pilot/osgvo-pilot-tufCYy/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.3-1.el9 SINGULARITY_IMAGES_DICT=default:/pilot/osgvo-pilot-tufCYy/images/htc__rocky__9.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.3-1.el9 TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-tufCYy/execute/dir_8152 _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass df
123.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 152243760 22359628 129867748 15% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=8850 1?: [LOCALHOST] pmtu 8850 1: 10-10-0-105.ceph-csi-cephfs-nodeplugin-http-metrics.ceph-csi.svc.cluster.local 0.126ms 1: 10-10-0-105.ceph-csi-cephfs-nodeplugin-http-metrics.ceph-csi.svc.cluster.local 0.045ms 2: host-10-10-0-1.js2local 0.654ms 3: host-10-10-3-17.js2local 3.928ms asymm 2 4: 149.165.159.1 5.993ms asymm 3 5: 172.16.11.141 4.447ms asymm 4 6: e-12-1.leafc1.bldc.net.rt.iu.edu 1.963ms asymm 5 7: 10.9.213.9 4.645ms asymm 6 8: 10.9.213.9 2.391ms asymm 6 9: et-4-2-3.2012.rtr.ictc.indiana.gigapop.net 14.179ms asymm 7 10: ae-9.0.rtr.ll.indiana.gigapop.net 9.375ms asymm 8 11: ae-7.1.rtr2.chic.indiana.gigapop.net 10.530ms asymm 9 12: 144.92.233.118 31.252ms asymm 10 13: perfsonar02.hep.wisc.edu 22.179ms reached Resume: pmtu 8850 hops 13 back 11
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg (uid 1000)
JSU-Tigers-CE1 46 mins ago 2
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
JSU-Tigers-CE1 — probe output
pass kernel
5.14.0-701.el9.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/var/lib/condor/execute/dir_222679/glide_Ed6xO7/execute/dir_563250/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/lib/condor/execute/dir_222679/glide_Ed6xO7/images/htc__almalinux__10.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__almalinux__10.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_222679/glide_Ed6xO7/execute/dir_563250/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=563250 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_222679/glide_Ed6xO7/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_JSU-Tigers-CE1.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=808020 GLIDEIN_Entry_Name=OSG_US_JSU-Tigers-CE1 GLIDEIN_Entry_Signature=b812de2d552f19651ca082ff54f40b346c4d3af1 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=jsu-tigers-ce1.svc.opensciencegrid.org jsu-tigers-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_nAXAyQ GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=JSU-Tigers-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=JSU-Tigers GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_222679/glide_Ed6xO7/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=jsu-tigers-ce1.svc.opensciencegrid.org OSG_SITE_NAME=JSU-Tigers-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_222683_14742060@chtc.ep0.jsums.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_222683_14742060@chtc.ep0.jsums.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/var/lib/condor/execute/dir_222679/glide_Ed6xO7/images/htc__almalinux__10.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__almalinux__10.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_222679/glide_Ed6xO7/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_222679/glide_Ed6xO7/execute/dir_563250/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1674.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/condor_vg-condor_lv 1844816532 3455332 1755593400 1% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.539ms 1: _gateway 0.377ms 2: 173.235.100.112 5.356ms 3: 173.235.100.7 9.524ms asymm 2 4: 173.235.100.17 19.075ms asymm 3 5: fourhundredge-0-0-0-22.4079.core2.atla.net.internet2.edu 36.259ms asymm 14 6: fourhundredge-0-0-0-4.4079.core2.nash.net.internet2.edu 36.655ms asymm 13 7: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 33.078ms asymm 12 8: fourhundredge-0-0-0-23.4079.core2.cinc.net.internet2.edu 33.495ms asymm 11 9: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 35.712ms asymm 10 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 33.386ms asymm 9 11: 144.92.233.112 34.562ms asymm 10 12: perfsonar02.hep.wisc.edu 34.439ms reached Resume: pmtu 1500 hops 12 back 12
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 4977)
KENT-STATE-RESEARCH-EP 1 hour ago 4
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
KENT-STATE-RESEARCH-EP — probe output
pass kernel
5.14.0-710.el9.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/pilot/osgvo-pilot-iAeUya/execute/dir_339493/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-iAeUya/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-iAeUya/execute/dir_339493/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=339493 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Kent_State_Research PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_31@KENT-STATE-RESEARCH-EP.c618e805555b$ "; unset PROMPT_COMMAND PS1=slot1_31@KENT-STATE-RESEARCH-EP.c618e805555b$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/pilot/osgvo-pilot-iAeUya/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-iAeUya/execute/dir_339493/scratch _CONDOR_SLOT=slot1_31 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
41248.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg--main-home 45986104216 425907804 43251635108 1% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.094ms 1: _gateway 0.103ms 2: gw2.research.cs.kent.edu 10.777ms 3: 10.255.255.1 0.610ms 4: akrnq-r4-et-0-2-1s3112.core.oar.net 0.750ms 5: yntww-r5-et-1-0-0s100.core.oar.net 2.420ms 6: clevb-r5-et-5-1-0s100.core.oar.net 4.324ms 7: cncne-r4-et-0-0-4s100.core.oar.net 8.541ms asymm 9 8: cncno-r5-et-4-3-0s735.core.oar.net 8.864ms asymm 10 9: hundredge-0-0-0-25.1260.core1.cinc.net.internet2.edu 9.935ms asymm 10 10: cncno-r5-et-4-3-0s735.core.oar.net 10.699ms 11: hundredge-0-0-0-25.1260.core1.cinc.net.internet2.edu 10.919ms asymm 10 12: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 14.637ms asymm 10 13: 144.92.233.112 15.369ms asymm 11 14: perfsonar02.hep.wisc.edu 15.513ms reached Resume: pmtu 1500 hops 14 back 13
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg (uid 1000)
KSU-Beocat-CE1 1 hour ago 395
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
KSU-Beocat-CE1 — probe output
pass kernel
5.14.0-687.39.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/tmp/glide_IADAgu/execute/dir_1696019/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_IADAgu/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_IADAgu/execute/dir_1696019/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1696019 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_IADAgu/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_KSU-Beocat-CE1.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=6349927 GLIDEIN_Entry_Name=OSG_US_KSU-Beocat-CE1 GLIDEIN_Entry_Signature=5bc97af60bac6347ea61225769e37b50c753d141 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=ksu-beocat-ce1.svc.opensciencegrid.org ksu-beocat-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgsubmit_yYlnSj GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=squid.beocat.ksu.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=KSU-Beocat-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=BEOCAT-SLATE GLIDEIN_Tmp_Dir=/tmp/glide_IADAgu/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ksu-beocat-ce1.svc.opensciencegrid.org OSG_SITE_NAME=KSU-Beocat-CE1 OSG_SQUID_LOCATION=squid.beocat.ksu.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_2287415_525010629@dwarf33.beocat.ksu.edu$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_2287415_525010629@dwarf33.beocat.ksu.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/tmp/glide_IADAgu/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_IADAgu/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_IADAgu/execute/dir_1696019/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
811.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg00-root 968898324 118154284 850744040 13% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: no reply 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 9000 Resume: pmtu 9000
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy squid.beocat.ksu.edu:3128
pass user
osgsubmit (uid 3838)
LSUHSC-Tigerfish-CE2 1 hour ago 6
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
LSUHSC-Tigerfish-CE2 — probe output
pass kernel
4.18.0-240.15.1.el8_3.x86_64
pass env
BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3910184 CONTAINERSW_FULL_VERSION=singularity version 3.7.4-1.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/singularity CONTAINERSW_VERSION=3.7.4-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/mnt/beegfs/scratch/glide_GZJPrm/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_LSUHSC-Tigerfish-CE2.idtoken.idtoken GLIDEIN_CPUS=40 GLIDEIN_ClusterId=6562850 GLIDEIN_Entry_Name=OSG_US_LSUHSC-Tigerfish-CE2 GLIDEIN_Entry_Signature=1db449a206570d631b635eaf499272f8bda7514d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=lsuhsc-tf-ce2.svc.opensciencegrid.org lsuhsc-tf-ce2.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_TZ8yhS GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=LSUHSC-Tigerfish-CE2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=LSUHSC-Tigerfish GLIDEIN_Tmp_Dir=/mnt/beegfs/scratch/glide_GZJPrm/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity version 3.7.4-1.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/singularity GWMS_CONTAINERSW_VERSION=3.7.4-1.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.7.4-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=lsuhsc-tf-ce2.svc.opensciencegrid.org OSG_SITE_NAME=LSUHSC-Tigerfish-CE2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_12@glidein_3805195_258457976@node34.cluster$ "; unset PROMPT_COMMAND PS1=slot1_12@glidein_3805195_258457976@node34.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/mnt/beegfs/scratch/glide_GZJPrm/execute/dir_3910184:/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/mnt/beegfs/scratch/glide_GZJPrm/execute/dir_3910184/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/mnt/beegfs/scratch/glide_GZJPrm/images/htc__ubuntu__24.04.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__ubuntu__24.04.sif SINGULARITY_TMPDIR=/mnt/beegfs/scratch/glide_GZJPrm/execute/dir_3910184/scratch SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/mnt/beegfs/scratch/glide_GZJPrm/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/mnt/beegfs/scratch/glide_GZJPrm/execute/dir_3910184/scratch _CONDOR_SLOT=slot1_12 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
348855.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on beegfs_nodev 596209287168 230408054272 365801232896 39% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: tigerfish.cluster 0.200ms 1: tigerfish.cluster 0.240ms 2: 155.58.254.1 0.734ms 3: rcb-bdr-r1.lsuhsc.edu 0.858ms 4: 208.100.102.142 1.073ms 5: 10.250.25.153 7.740ms asymm 13 6: 10.250.24.121 8.338ms asymm 12 7: 10.250.25.69 7.288ms asymm 11 8: 10.250.24.117 7.097ms asymm 10 9: 10.250.27.69 7.565ms 10: 10.250.27.70 7.360ms asymm 9 11: 10.250.27.102 7.259ms asymm 8 12: 208.100.66.98 10.419ms asymm 9 13: fourhundredge-0-0-0-2.4079.core1.houh.net.internet2.edu 35.464ms asymm 21 14: fourhundredge-0-0-0-2.4079.core1.hous.net.internet2.edu 35.275ms asymm 20 15: fourhundredge-0-0-0-0.4079.core2.dall.net.internet2.edu 35.115ms asymm 19 16: fourhundredge-0-0-0-4.4079.core2.dall3.net.internet2.edu 36.309ms asymm 18 17: fourhundredge-0-0-0-2.4079.core2.kans.net.internet2.edu 34.458ms 18: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 35.856ms asymm 16 19: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 35.546ms asymm 15 20: 144.92.233.112 37.330ms asymm 16 21: perfsonar02.hep.wisc.edu 37.220ms reached Resume: pmtu 1500 hops 21 back 18
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 100017)
Lafayette-Firebird-CE1 1 hour ago 29
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Lafayette-Firebird-CE1 — probe output
pass kernel
5.14.0-687.39.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/scratch/osg01/glide_QeWjw7/execute/dir_2586720/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg01/glide_QeWjw7/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg01/glide_QeWjw7/execute/dir_2586720/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2586720 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01/glide_QeWjw7/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Lafayette-firebird-CE1.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=861391 GLIDEIN_Entry_Name=OSG_US_Lafayette-firebird-CE1 GLIDEIN_Entry_Signature=fece894c99d46d5fe5efcfa91bf77152cd04a238 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=lafayette-firebird-ce1.svc.osg-htc.org lafayette-firebird-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_9Zy7Rj GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Lafayette-Firebird-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=Lafayette-Firebird GLIDEIN_Tmp_Dir=/scratch/osg01/glide_QeWjw7/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=lafayette-firebird-ce1.svc.osg-htc.org OSG_SITE_NAME=Lafayette-Firebird-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_2534561_98139440@node06.cluster$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_2534561_98139440@node06.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/scratch/osg01/glide_QeWjw7/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01/glide_QeWjw7/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01/glide_QeWjw7/execute/dir_2586720/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
799.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/disk0-scratch 877647156 39458064 838189092 5% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: firebird.cluster 1.576ms 1: firebird.cluster 1.645ms 2: fw-comsvc-colo.lafayette.edu 2.299ms 3: 139.147.1.113 2.341ms asymm 4 4: 172.31.0.33 2.134ms 5: 172.31.0.3 2.903ms 6: gw0-g0-3-ibgp.prd.lafayette.edu 2.955ms asymm 5 7: 164.153.140.160 6.196ms asymm 6 8: no reply 9: hundredge-0-0-0-25.68.core1.pitt.net.internet2.edu 16.694ms asymm 8 10: fourhundredge-0-0-0-3.4079.core1.clev.net.internet2.edu 23.354ms asymm 13 11: fourhundredge-0-0-0-2.4079.core1.eqch.net.internet2.edu 24.525ms asymm 12 12: fourhundredge-0-0-0-6.4079.core1.star.net.internet2.edu 26.173ms asymm 11 13: 144.92.233.112 26.248ms asymm 12 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 200079)
Langston-Lion-CE1 1 hour ago 11
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Langston-Lion-CE1 — probe output
pass kernel
5.14.0-570.49.1.el9_6.x86_64
pass env
BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=707322 CONTAINERSW_FULL_VERSION=singularity-ce version 4.3.3-3.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/singularity CONTAINERSW_VERSION=4.3.3-3.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osgccre/glide_dMnI1f/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Langston-Lion-CE1.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=7086943 GLIDEIN_Entry_Name=OSG_US_Langston-Lion-CE1 GLIDEIN_Entry_Signature=296e29304fde9496aed79e299e1df728c53308b0 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=langston-lion-ce1.svc.osg-htc.org langston-lion-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osgccre_uECY86 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Langston-Lion-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=88d76433adc669450d89762935ce2ce292bb77fe GLIDEIN_Site=Langston-Lion GLIDEIN_Tmp_Dir=/scratch/osgccre/glide_dMnI1f/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity-ce version 4.3.3-3.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/singularity GWMS_CONTAINERSW_VERSION=4.3.3-3.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=4.3.3-3.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=langston-lion-ce1.svc.osg-htc.org OSG_SITE_NAME=Langston-Lion-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_1756007_249777792@c000.ccre.langston.edu$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_1756007_249777792@c000.ccre.langston.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/scratch/osgccre/glide_dMnI1f/execute/dir_707322:/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/scratch/osgccre/glide_dMnI1f/execute/dir_707322/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/scratch/osgccre/glide_dMnI1f/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_TMPDIR=/scratch/osgccre/glide_dMnI1f/execute/dir_707322/scratch SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osgccre/glide_dMnI1f/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osgccre/glide_dMnI1f/execute/dir_707322/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
798.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 936705024 99851384 836853640 11% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: lionvmhost.ccre.langston.edu 0.157ms 1: lionvmhost.ccre.langston.edu 0.156ms 2: 164.58.29.113 0.582ms 3: 198.71.46.45 2.278ms asymm 4 4: fourhundredge-0-0-0-9.302.core1.tuls.net.internet2.edu 5.124ms 5: fourhundredge-0-0-0-2.4079.core1.kans.net.internet2.edu 18.312ms asymm 11 6: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 18.495ms asymm 10 7: fourhundredge-0-0-0-23.4079.core1.chic.net.internet2.edu 18.860ms asymm 9 8: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 17.689ms 9: 144.92.233.112 20.490ms 10: perfsonar02.hep.wisc.edu 20.536ms reached Resume: pmtu 1500 hops 10 back 11
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osgccre (uid 20006)
Lehigh_-_Hawk 1 hour ago 442
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Lehigh_-_Hawk — probe output
pass kernel
5.14.0-687.48.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/scratch/osg/14365638/glide_0bHGFL/execute/dir_3677047/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg/14365638/glide_0bHGFL/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg/14365638/glide_0bHGFL/execute/dir_3677047/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3677047 CONTAINERSW_FULL_VERSION=apptainer version 1.5.4-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.4-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg/14365638/glide_0bHGFL/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Lehigh-Hawk_osg-hawk-ce.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=804418 GLIDEIN_Country=US GLIDEIN_Entry_Name=OSG_US_Lehigh-Hawk_osg-hawk-ce GLIDEIN_Entry_Signature=8db42494da6fff3cd7495d1c937b2a7111d57b0b GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=osg-hawk-ce.cc.lehigh.edu osg-hawk-ce.cc.lehigh.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_RAZ2d1 GLIDEIN_MaxMemMBs=7000 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=osg-squid.cc.lehigh.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Lehigh - Hawk GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=04973478840ac6f9501c3524ab345e903645979b GLIDEIN_Site=Lehigh - Hawk GLIDEIN_Tmp_Dir=/scratch/osg/14365638/glide_0bHGFL/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.4-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.4-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.4-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv HTTPS_PROXY=osg-squid.cc.lehigh.edu:3128 JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=osg-hawk-ce.cc.lehigh.edu OSG_SITE_NAME=LEHIGH-HAWK-CE OSG_SQUID_LOCATION=osg-squid.cc.lehigh.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_852796_172618172@hawk-a702.cc.lehigh.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_852796_172618172@hawk-a702.cc.lehigh.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/scratch/osg/14365638/glide_0bHGFL/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TERM=linux TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg/14365638/glide_0bHGFL/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg/14365638/glide_0bHGFL/execute/dir_3677047/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=osg-squid.cc.lehigh.edu:3128
pass df
514.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/almalinux-scratch 874028092 290587092 538969092 36% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 192.168.7.251 0.477ms 1: 192.168.7.251 0.615ms 2: borderfw-e.cc.lehigh.edu 0.076ms 3: pack34-out.cc.lehigh.edu 0.169ms 4: 164.153.140.133 2.525ms 5: 172.24.1.20 9.613ms 6: hundredge-0-0-0-25.68.core1.pitt.net.internet2.edu 11.277ms 7: fourhundredge-0-0-0-3.4079.core1.clev.net.internet2.edu 20.723ms asymm 11 8: fourhundredge-0-0-0-2.4079.core1.eqch.net.internet2.edu 44.771ms asymm 10 9: fourhundredge-0-0-0-6.4079.core1.star.net.internet2.edu 19.217ms 10: 144.92.233.112 22.347ms 11: perfsonar02.hep.wisc.edu 22.407ms reached Resume: pmtu 1500 hops 11 back 12
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy osg-squid.cc.lehigh.edu:3128
pass user
osg (uid 257299)
MI-HORUS-CE2 1 hour ago 1322
98.4%
98.4%
28.1%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
MI-HORUS-CE2 — probe output
pass kernel
5.14.0-427.42.1.el9_4.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/scratch/glide_CwYPUD/execute/dir_322232/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_CwYPUD/images/htc__ubuntu__24.04.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__ubuntu__24.04.sif APPTAINER_TMPDIR=/scratch/glide_CwYPUD/execute/dir_322232/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=322232 CONTAINERSW_FULL_VERSION=apptainer version 1.3.5-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.3.5-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_CwYPUD/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_MI-HORUS-CE2.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6560791 GLIDEIN_Entry_Name=OSG_US_MI-HORUS-CE2 GLIDEIN_Entry_Signature=f54e7c3e68218c973d5ca198627dcec7ac77c752 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=mi-horus-ce2.svc.osg-htc.org mi-horus-ce2.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_TQwLnc GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=MI-HORUS-CE2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=MI-HORUS GLIDEIN_Tmp_Dir=/scratch/glide_CwYPUD/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.3.5-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.3.5-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.3.5-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=mi-horus-ce2.svc.osg-htc.org OSG_SITE_NAME=MI-HORUS-CE2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_2138234_120041875@um-lc02.osris.org$ "; unset PROMPT_COMMAND PS1=slot1@glidein_2138234_120041875@um-lc02.osris.org$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/scratch/glide_CwYPUD/images/htc__ubuntu__24.04.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__ubuntu__24.04.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_CwYPUD/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_CwYPUD/execute/dir_322232/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
2662.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_scratch-lv_scratch 3124098648 332611140 2791487508 11% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 141.211.169.2 0.252ms 1: 141.211.169.2 0.954ms 2: l3-maccc-osiris1.r-macc-c.umnet.umich.edu 0.417ms 3: l3-legmacc1-maccc-primaryvrf.leg-macc-1.umnet.umich.edu 0.387ms 4: no reply 5: no reply 6: l3-binncas1-pelegseb1.bin-ncas-1.umnet.umich.edu 0.767ms 7: l3-binncas1-binseb1.bin-seb-1.umnet.umich.edu 0.788ms asymm 6 8: 144.92.233.112 9.474ms 9: 144.92.233.112 9.516ms asymm 8 10: perfsonar02.hep.wisc.edu 9.381ms reached Resume: pmtu 1500 hops 10 back 10
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 100194)
MTState-Tempest-CE1 1 hour ago 521
96.5%
96.5%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user gpu
MTState-Tempest-CE1 — probe output
pass kernel
6.12.0-211.53.1.el10_2.x86_64
pass env
APPTAINER_CACHEDIR=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch BATCH_SYSTEM=HTCondor BLAHPD_LOCATION=/home/osg01/bosco/glite CONDORCE_COLLECTOR_HOST=mtstate-tempest-ce1.svc.opensciencegrid.org:9619 CONDOR_CONFIG=/home/group/osg/jobtemps/358099/glide_Mery9b/condor_config CONDOR_PARENT_ID=gpu023:2060971:1790772826 CONDOR_PROCD_ADDRESS=/home/group/osg/jobtemps/358099/glide_Mery9b/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/home/group/osg/jobtemps/358099/glide_Mery9b/log/procd_address CONDOR_STARTER_PID=2060971 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el10_2 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el10_2 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/200494/bus DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima: ENVIRONMENT=BATCH GFAL_CONFIG_DIR=/home/osg01/bosco-osg-wn-client/etc/gfal2.d/ GFAL_PLUGIN_DIR=/home/osg01/bosco-osg-wn-client/usr/lib64/gfal2-plugins/ GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=282e7263f1755563ecd8b0c3a7dfabafef58aaed GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/home/group/osg/jobtemps/358099/glide_Mery9b/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_MTState-Tempest-CE1_gpu.idtoken.idtoken GLIDEIN_CPUS=16 GLIDEIN_ClusterId=874454 GLIDEIN_DEBUG_OPTIONS= GLIDEIN_Entry_Name=OSG_US_MTState-Tempest-CE1_gpu GLIDEIN_Entry_Signature=4a0ec26f86f035dbea750fcf90e04a0b7524a8f1 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=mtstate-tempest-ce1.svc.opensciencegrid.org mtstate-tempest-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_4MjteA GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=MTState-Tempest-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=MTState-Tempest GLIDEIN_Tmp_Dir=/home/group/osg/jobtemps/358099/glide_Mery9b/tmp GLIDEIN_UUID=4fe99b6f-8787-4252-9905-6bdf50e35efd GLITE_LOCATION=/home/osg01/bosco/glite GLOBUS_LOCATION=/home/osg01/bosco-osg-wn-client/usr GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el10_2 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el10_2 GWMS_PATH=/home/group/osg/jobtemps/358099/glide_Mery9b/.gwms.d/bin: GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el10_2 HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HISTCONTROL=ignoredups HISTSIZE=1000 HOLD_GRACE_TIME=0 HOME=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch HOSTNAME=gpu023 HYDRA_BOOTSTRAP=slurm HYDRA_LAUNCHER_EXTRA_ARGS=--external-launcher I_MPI_HYDRA_BOOTSTRAP=slurm I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_OSG_US_MTState-Tempest-CE1_gpu/tokens.tgz JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/home/osg01/bosco-osg-wn-client/lib64:/home/osg01/bosco-osg-wn-client/lib:/home/osg01/bosco-osg-wn-client/usr/lib64:/home/osg01/bosco-osg-wn-client/usr/lib:/home/osg01/bosco-osg-wn-client/usr/lib64/dcap:/home/osg01/bosco-osg-wn-client/usr/lib64/lcgdm LESSOPEN=||/usr/bin/lesspipe.sh %s LOGNAME=osg01 MAIL=/var/spool/mail/osg01 MANPATH=/home/osg01/bosco-osg-wn-client/usr/share/man MEM_SCRATCH=/dev/shm/jobs/358099 MKL_NUM_THREADS=1 MOTD_SHOWN=pam NODE_COUNT=16 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OLDPWD=/home/group/osg/jobtemps/358099/glide_Mery9b/logs/4fe99b6f-8787-4252-9905-6bdf50e35efd OMPI_MCA_plm_slurm_args=--external-launcher OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=UNAVAILABLE OSG_DATA=UNAVAILABLE OSG_GRID=/home/osg01/bosco-osg-wn-client OSG_HOSTNAME=mtstate-tempest-ce1.svc.opensciencegrid.org OSG_LOCATION=/home/osg01/bosco-osg-wn-client OSG_SITE_NAME=MTState-Tempest-CE1 OSG_WN_TMP=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch/.local-tmp PATH=/home/group/osg/jobtemps/358099/glide_Mery9b/.gwms.d/bin::/home/osg01/bosco-osg-wn-client/usr/bin:/home/osg01/bosco-osg-wn-client/usr/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/slurm/current/bin:/opt/slurm/current/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin PERL5LIB=/home/osg01/bosco-osg-wn-client/usr/share/perl5/vendor_perl:/home/osg01/bosco-osg-wn-client/usr/share/perl5:/home/osg01/bosco-osg-wn-client/usr/lib64/perl5/vendor_perl:/home/osg01/bosco-osg-wn-client/usr/lib64/perl5 PREEMPT_GRACE_TIME=10000000 PRTE_MCA_plm_slurm_args=--external-launcher PWD=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch PYTHONPATH=/home/osg01/bosco-osg-wn-client/usr/lib/python3.12/site-packages:/home/osg01/bosco-osg-wn-client/usr/lib64/python3.12/site-packages PYTHON_CPU_COUNT=1 ROCR_VISIBLE_DEVICES=0 ROOT_MAX_THREADS=1 SELINUX_LEVEL_REQUESTED= SELINUX_ROLE_REQUESTED= SELINUX_USE_CURRENT_RANGE= SHELL=/bin/bash SHLVL=0 SINGULARITY_CACHEDIR=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch SINGULARITY_FULL_VERSION=apptainer version 1.5.3-1.el10_2 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9-cuda-12.6.0 SINGULARITY_MODE=privileged SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.5.3-1.el10_2 SLOTS_LAYOUT=partitionable SLURMD_NODENAME=gpu023 SLURM_CLUSTER_NAME=tempest SLURM_CONF=/etc/slurm/slurm.conf SLURM_CPUS_ON_NODE=16 SLURM_GPUS_ON_NODE=1 SLURM_GTIDS=0 SLURM_JOBID=358099 SLURM_JOB_ACCOUNT=group-osg SLURM_JOB_CPUS_PER_NODE=16 SLURM_JOB_END_TIME=1790985252 SLURM_JOB_EXCLUSIVE=NO SLURM_JOB_GID=200000 SLURM_JOB_GPUS=1 SLURM_JOB_ID=358099 SLURM_JOB_NAME=bl_ckUuiT SLURM_JOB_NODELIST=gpu023 SLURM_JOB_NUM_NODES=1 SLURM_JOB_OVERSUBSCRIBE=NO SLURM_JOB_PARTITION=osg-gpu SLURM_JOB_QOS=normal SLURM_JOB_SLUID=s0Q9M3QSE6VR00 SLURM_JOB_START_TIME=1790726052 SLURM_JOB_UID=200494 SLURM_JOB_USER=osg01 SLURM_LOCALID=0 SLURM_MEM_PER_NODE=65536 SLURM_NNODES=1 SLURM_NODEID=0 SLURM_NODELIST=gpu023 SLURM_NPROCS=16 SLURM_NTASKS=16 SLURM_NTASKS_PER_NODE=16 SLURM_OOM_KILL_STEP=0 SLURM_PRIO_PROCESS=0 SLURM_PROCID=0 SLURM_SCRIPT_CONTEXT=prolog_task SLURM_STEPID=-5 SLURM_STEP_ID=-5 SLURM_SUBMIT_DIR=/home/osg01/bosco/sandbox/305c/305c3127/mtstate-tempest-ce1.svc.opensciencegrid.org_9619_mtstate-tempest-ce1.svc.opensciencegrid.org_153604.0_1790726038 SLURM_SUBMIT_HOST=tempest-osg-submit SLURM_TASKS_PER_NODE=16 SLURM_TASK_PID=762268 SLURM_TOPOLOGY_ADDR=gpu023 SLURM_TOPOLOGY_ADDR_PATTERN=node SSH_CLIENT=128.105.82.103 46906 22 SSH_CONNECTION=128.105.82.103 46906 10.226.104.10 22 TEMP=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch TMPDIR=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch/.local-tmp TMP_SCRATCH=/home/group/osg/jobtemps/358099 TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime USER=osg01 VOMS_USERCONF=/home/osg01/bosco-osg-wn-client/etc/vomses X509_CERT_DIR=/home/osg01/bosco-osg-wn-client/etc/grid-security/certificates X509_VOMS_DIR=/home/osg01/bosco-osg-wn-client/etc/grid-security/vomsdir XDG_RUNTIME_DIR=/run/user/200494 XDG_SESSION_CLASS=user XDG_SESSION_ID=461 XDG_SESSION_TYPE=tty ZE_AFFINITY_MASK=0 _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_2060971=2061721:1790772827:406505244 _CONDOR_ANCESTOR_840396=840609:1790727336:198256866 _CONDOR_ANCESTOR_840609=2060971:1790772823:693405577 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/home/group/osg/jobtemps/358099/glide_Mery9b/main/condor/bin _CONDOR_CHIRP_CONFIG=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/user/.chirp.config _CONDOR_CREDS=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch/.condor_creds _CONDOR_EXECUTE=/home/group/osg/jobtemps/358099/glide_Mery9b/execute _CONDOR_JOB_AD=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch/.job.ad _CONDOR_JOB_IWD=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch _CONDOR_SLOT=slot1_13 _CONDOR_WRAPPER_ERROR_FILE=/home/group/osg/jobtemps/358099/glide_Mery9b/execute/dir_2060971/scratch/.job_wrapper_failure mem_scratch=/dev/shm/jobs/358099 tmp_scratch=/home/group/osg/jobtemps/358099
pass df
8280.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 10.226.112.1:/home/group/osg 10737418240 2054722560 8682695680 20% /home/group/osg
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: 10.226.113.252 0.075ms 1: 10.226.113.252 0.064ms 2: 10.226.119.253 0.680ms 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 9000 Resume: pmtu 9000
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 200494)
pass gpu
Wed Sep 30 06:53:47 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 Tesla T4 On | 00000000:18:00.0 Off | 0 | | N/A 36C P8 15W / 70W | 3MiB / 15360MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
MWT2 1 hour ago 268
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
MWT2 — probe output
pass kernel
5.14.0-687.25.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/scratch.local/condor/execute/dir_2960660/glide_7AfDMT/execute/dir_464196/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch.local/condor/execute/dir_2960660/glide_7AfDMT/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch.local/condor/execute/dir_2960660/glide_7AfDMT/execute/dir_464196/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=464196 CONTAINERSW_FULL_VERSION=apptainer version 1.5.4-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.5.4-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/scratch.local/condor/execute/dir_2960660/glide_7AfDMT/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.Engage_US_MWT2_iut2_gk02_condce_mcore.idtoken.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=873368 GLIDEIN_Country=US GLIDEIN_DUNESite=US_UChicago GLIDEIN_Entry_Name=Engage_US_MWT2_iut2_gk02_condce_mcore GLIDEIN_Entry_Signature=7d6b0eea34ab9b61c0bd23b180179a5baecac49f GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=iut2-gk02.mwt2.org iut2-gk02.mwt2.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_ZDzYCS GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=MWT2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=ad419b1b3db36bf5c7633f8e9c4c175eb881dd81 GLIDEIN_Site=UChicago GLIDEIN_Tmp_Dir=/scratch.local/condor/execute/dir_2960660/glide_7AfDMT/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.4-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.4-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.4-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=iut2-gk02.mwt2.org OSG_SITE_NAME=MWT2_CE_IU2 OSG_SQUID_LOCATION=iut2-squid.mwt2.org:32200 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_6@glidein_2960662_16345971@mwt2-c213.campuscluster.illinois.edu$ "; unset PROMPT_COMMAND PS1=slot1_6@glidein_2960662_16345971@mwt2-c213.campuscluster.illinois.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/scratch.local/condor/execute/dir_2960660/glide_7AfDMT/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch.local/condor/execute/dir_2960660/glide_7AfDMT/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch.local/condor/execute/dir_2960660/glide_7AfDMT/execute/dir_464196/scratch _CONDOR_SLOT=slot1_6 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
3440.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda 7440848848 3458709572 3607112796 49% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] 0.011ms pmtu 9000 1: vl-1675-cc-core.gw.ncsa.edu 0.171ms 1: vl-1675-cc-core.gw.ncsa.edu 0.224ms 2: 2620:0:c80:1::216 0.194ms 3: npcf-exit-west-npcf-core-west.gw.ncsa.edu 0.437ms 4: 2620:0:c80:106::11 0.864ms 5: no reply 6: uwmadison_se-203--chic-cr6.es.net 14.056ms 7: perfsonar02.hep.wisc.edu 14.047ms reached Resume: pmtu 9000 hops 7 back 7
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy iut2-squid.mwt2.org:32200
pass user
osg (uid 21039)
Maine-Penobscot-CE1 1 hour ago 20
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Maine-Penobscot-CE1 — probe output
pass kernel
5.14.0-611.54.1.el9_7.x86_64
pass env
BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1262891 CONTAINERSW_FULL_VERSION=singularity-ce version 3.11.5-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/singularity CONTAINERSW_VERSION=3.11.5-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01p/glide_TvD8gY/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Maine-Penobscot-CE1_bw.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=7035751 GLIDEIN_Entry_Name=OSG_US_Maine-Penobscot-CE1_bw GLIDEIN_Entry_Signature=761e215d5afd02c934d77a91e85dcce6569d7051 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=maine-penobscot-ce1.svc.opensciencegrid.org maine-penobscot-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01p_B6TAxi GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Maine-Penobscot-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=e977401dd4ee29c9252d463c389cf2702ca41418 GLIDEIN_Site=Maine-Penobscot GLIDEIN_Tmp_Dir=/scratch/osg01p/glide_TvD8gY/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity-ce version 3.11.5-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/singularity GWMS_CONTAINERSW_VERSION=3.11.5-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.11.5-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=maine-penobscot-ce1.svc.opensciencegrid.org OSG_SITE_NAME=Maine-Penobscot-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_426918_345383648@node-81.localdomain$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_426918_345383648@node-81.localdomain$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/scratch/osg01p/glide_TvD8gY/execute/dir_1262891:/srv,/etc/hosts,/etc/localtime SINGULARITY_CACHEDIR=/scratch/osg01p/glide_TvD8gY/execute/dir_1262891/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/scratch/osg01p/glide_TvD8gY/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_TMPDIR=/scratch/osg01p/glide_TvD8gY/execute/dir_1262891/scratch SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01p/glide_TvD8gY/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01p/glide_TvD8gY/execute/dir_1262891/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
775.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 961230208 99290696 813101388 11% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: login1.localdomain 0.217ms 1: login1.localdomain 0.211ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01p (uid 1011)
Missouri-Hellbender-CE1 1 hour ago 375
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Missouri-Hellbender-CE1 — probe output
pass kernel
4.18.0-553.159.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/local/scratch/glide_jlgmOv/execute/dir_57684/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/scratch/glide_jlgmOv/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/local/scratch/glide_jlgmOv/execute/dir_57684/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=57684 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/scratch/glide_jlgmOv/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Missouri-Hellbender-CE1.idtoken.idtoken GLIDEIN_CPUS=2 GLIDEIN_ClusterId=7097293 GLIDEIN_Entry_Name=OSG_US_Missouri-Hellbender-CE1 GLIDEIN_Entry_Signature=42f1686b0a02a8b12d33b2f928986e423999286b GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=missouri-hellbender-ce1.svc.opensciencegrid.org missouri-hellbender-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_s2c_osg_01_pLIscY GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Missouri-Hellbender-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=Missouri-Hellbender GLIDEIN_Tmp_Dir=/local/scratch/glide_jlgmOv/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=missouri-hellbender-ce1.svc.opensciencegrid.org OSG_SITE_NAME=Missouri-Hellbender-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_1354850_1091179053@c218.mgmt.hellbender$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_1354850_1091179053@c218.mgmt.hellbender$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/local/scratch/glide_jlgmOv/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/scratch/glide_jlgmOv/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/scratch/glide_jlgmOv/execute/dir_57684/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
899.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme1n1p1 1562049324 618740776 943308548 40% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: rcore1-hellbender-mgmt-v348.rnet.missouri.edu 10.838ms 1: rcore1-hellbender-mgmt-v348.rnet.missouri.edu 10.939ms 2: tcore-agg-rnet.bb.missouri.edu 10.955ms 3: ncore-tcore-rnet.bb.missouri.edu 10.956ms asymm 2 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
s2c_osg_01 (uid 100753443)
NCSU-OSG-CE1 15 mins ago 4
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user gpu
NCSU-OSG-CE1 — probe output
pass kernel
5.14.0-570.23.1.el9_6.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/scratch/glide_ulvNvK/execute/dir_3346300/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_ulvNvK/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/scratch/glide_ulvNvK/execute/dir_3346300/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3346300 CONTAINERSW_FULL_VERSION=apptainer version 1.4.1-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.1-1.el9 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=a9c5840f9c0c9c69a1ccc8df7a49c87f7b1ed2ce GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_ulvNvK/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_NCSU-OSG-CE1.idtoken.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=954532 GLIDEIN_Entry_Name=OSG_US_NCSU-OSG-CE1 GLIDEIN_Entry_Signature=a50abf29bd913d1c4de2fe7014ffd785edac3df6 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=ncsu-osg-ce1.svc.osg-htc.org ncsu-osg-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_OSG01_UrIOS5 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=NCSU-OSG-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=b7ad1a6a40bd6a6100fb8503c2b0a86ab4628f8c GLIDEIN_Site=NCSU-OSG GLIDEIN_Tmp_Dir=/scratch/glide_ulvNvK/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.1-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.1-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.1-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ncsu-osg-ce1.svc.osg-htc.org OSG_SITE_NAME=NCSU-OSG-CE1 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_10@glidein_1042682_260844848@vcledch0201.hpc.ncsu.edu$ "; unset PROMPT_COMMAND PS1=slot1_10@glidein_1042682_260844848@vcledch0201.hpc.ncsu.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/scratch/glide_ulvNvK/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_ulvNvK/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_ulvNvK/execute/dir_3346300/scratch _CONDOR_SLOT=slot1_10 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
3191.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p7 3671331348 324380184 3346951164 9% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: vl2140-dc2-6509-1-fwsm-2.ncstate.net 9.231ms 1: vl2140-dc2-6509-1-fwsm-2.ncstate.net 9.678ms 2: 10.250.142.110 10.123ms asymm 3 3: 10.250.133.5 9.520ms 4: 128.109.1.245 7.165ms 5: rtp-gw-to-ws-gw.ncren.net 18.254ms 6: bundle-ether250.613.core1.rale.net.internet2.edu 20.457ms 7: fourhundredge-0-0-0-6.4079.core1.ashb.net.internet2.edu 39.467ms asymm 13 8: fourhundredge-0-0-0-1.4079.core1.clev.net.internet2.edu 40.140ms asymm 12 9: fourhundredge-0-0-0-2.4079.core1.eqch.net.internet2.edu 39.200ms asymm 11 10: fourhundredge-0-0-0-6.4079.core1.star.net.internet2.edu 39.670ms 11: 144.92.233.112 41.618ms 12: perfsonar02.hep.wisc.edu 41.281ms reached Resume: pmtu 1500 hops 12 back 13
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
OSG01 (uid 1002)
pass gpu
Wed Sep 30 10:06:52 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.158.01 Driver Version: 570.158.01 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA L40 Off | 00000000:17:00.0 Off | 0 | | N/A 57C P0 193W / 300W | 2181MiB / 46068MiB | 51% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA L40 Off | 00000000:31:00.0 Off | 0 | | N/A 36C P8 35W / 300W | 3MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| +-----------------------------------------------------------------------------------------+
ND-CAML-GPU-CE1 1 hour ago 29
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user gpu
ND-CAML-GPU-CE1 — probe output
pass kernel
5.14.0-611.55.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/var/condor/execute/dir_2192570/glide_C55oYL/execute/dir_2930983/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/var/condor/execute/dir_2192570/glide_C55oYL/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/var/condor/execute/dir_2192570/glide_C55oYL/execute/dir_2930983/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2930983 CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.0-1.el9 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=282e7263f1755563ecd8b0c3a7dfabafef58aaed GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_NotreDame GLIDEIN_CONDOR_TOKEN=/var/condor/execute/dir_2192570/glide_C55oYL/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.CMS_US_ND-CAML_gpu.idtoken.idtoken GLIDEIN_CPUS=12 GLIDEIN_ClusterId=958350 GLIDEIN_Entry_Name=CMS_US_ND-CAML_gpu GLIDEIN_Entry_Signature=a8682bb50d94a41424d24c6d7b83d55b90db5518 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=nd-caml-gpu-ce1.svc.osg-htc.org nd-caml-gpu-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_r2rpiw GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=ND-CAML-GPU-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=ND_CAMLGPU GLIDEIN_Tmp_Dir=/var/condor/execute/dir_2192570/glide_C55oYL/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.0-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.0-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.0-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=nd-caml-gpu-ce1.svc.osg-htc.org OSG_SITE_NAME=ND-CAML-GPU-CE1 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_6@glidein_2192572_223747272@qa-rtx6k-032.crc.nd.edu$ "; unset PROMPT_COMMAND PS1=slot1_6@glidein_2192572_223747272@qa-rtx6k-032.crc.nd.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/var/condor/execute/dir_2192570/glide_C55oYL/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/condor/execute/dir_2192570/glide_C55oYL/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/condor/execute/dir_2192570/glide_C55oYL/execute/dir_2930983/scratch _CONDOR_SLOT=slot1_6 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
375.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda2 467572672 73820828 393751844 16% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: r64cisco9508-85-secondary.crc.nd.edu 0.521ms 1: r64cisco9508-85-secondary.crc.nd.edu 0.501ms 2: crc-def2compute-3.gw.nd.edu 0.536ms 3: 172.21.0.29 0.605ms asymm 4 4: border-pa.fw.nd.edu 0.487ms 5: no reply 6: 172.21.0.148 214.840ms asymm 5 7: et-4-0-0.401.rtr.ictc.indiana.gigapop.net 218.608ms asymm 6 8: ae-9.0.rtr.ll.indiana.gigapop.net 217.399ms asymm 7 9: ae-7.1.rtr2.chic.indiana.gigapop.net 218.115ms asymm 8 10: 144.92.233.118 231.224ms asymm 9 11: perfsonar02.hep.wisc.edu 232.256ms reached Resume: pmtu 1500 hops 11 back 10
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 1001)
pass gpu
Wed Sep 30 08:53:45 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 Quadro RTX 6000 Off | 00000000:86:00.0 Off | 0 | | N/A 20C P8 12W / 250W | 3MiB / 23040MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 Quadro RTX 6000 Off | 00000000:D8:00.0 Off | 0 | | N/A 73C P0 246W / 250W | 371MiB / 23040MiB | 89% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
NMSU-Discovery-CE1 1 hour ago 27
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
NMSU-Discovery-CE1 — probe output
pass kernel
4.18.0-553.92.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/fs1/osg/osg01/glide_jbIvFq/execute/dir_3300157/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/fs1/osg/osg01/glide_jbIvFq/images/htc__debian__13.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__debian__13.sif APPTAINER_TMPDIR=/fs1/osg/osg01/glide_jbIvFq/execute/dir_3300157/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3300157 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/fs1/osg/osg01/glide_jbIvFq/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_NMSU-Discovery-CE1.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=861235 GLIDEIN_Entry_Name=OSG_US_NMSU-Discovery-CE1 GLIDEIN_Entry_Signature=6ae2114cbf21bab3ff7fffdf8944f3b14a77f9ac GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=nmsu-discovery-ce1.svc.osg-htc.org nmsu-discovery-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_0Dchzs GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=192.168.211.16:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=NMSU-Discovery-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=NMSU_DISCOVERY GLIDEIN_Tmp_Dir=/fs1/osg/osg01/glide_jbIvFq/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=nmsu-discovery-ce1.svc.osg-htc.org OSG_SITE_NAME=NMSU-Discovery-CE1 OSG_SQUID_LOCATION=192.168.211.16:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_3212054_565620440@discovery-g13.cluster.local$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_3212054_565620440@discovery-g13.cluster.local$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/fs1/osg/osg01/glide_jbIvFq/images/htc__debian__13.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__debian__13.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/fs1/osg/osg01/glide_jbIvFq/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/fs1/osg/osg01/glide_jbIvFq/execute/dir_3300157/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
35439.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on fs1 37580963840 420380672 37160583168 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.88.37.254 6.823ms 1: 10.88.37.254 3.984ms 2: dc-core-vlan-88-dc.nmsu.edu 6.991ms asymm 3 3: 10.123.128.157 10.969ms asymm 4 4: 10.123.128.49 10.978ms asymm 6 5: 10.123.128.45 5.979ms asymm 7 6: ??? 10.987ms 7: abqg-rgon.checs.net 11.973ms 8: 208.77.77.233 10.976ms asymm 7 9: fourhundredge-0-0-0-0.4079.core1.kans.net.internet2.edu 41.953ms asymm 15 10: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 44.965ms asymm 14 11: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 44.977ms asymm 14 12: fourhundredge-0-0-0-22.4079.core1.chic.net.internet2.edu 44.973ms asymm 13 13: 144.92.233.112 45.978ms 14: perfsonar02.hep.wisc.edu 45.935ms reached Resume: pmtu 1500 hops 14 back 15
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 192.168.211.16:3128
pass user
osg01 (uid 3001)
Nebraska 1 hour ago 5
80.0%
80.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Nebraska — probe output
pass kernel
5.14.0-687.25.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/var/lib/condor/execute/dir_751020,/cvmfs,/etc/cvmfs,/etc/grid-security,/var/lib/cvmfs,/var/lib/sss/pipes,/var/lib/sss/mc,/etc/nsswitch.conf,/dev/fuse,/dev/log APPTAINER_CACHEDIR=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/mnt/t2ceph/user/jthiltge/cmssif/el8.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=el8.sif APPTAINER_TMPDIR=/var/lib/condor/execute/dir_751020 BATCH_SYSTEM=HTCondor CONDORCE_COLLECTOR_HOST=red.unl.edu:9619 CONDOR_CONFIG=/var/lib/condor/execute/dir_751020/glide_f3jP0I/condor_config CONDOR_PARENT_ID=red-c7224:1205393:1790772894 CONDOR_PROCD_ADDRESS=/var/lib/condor/execute/dir_751020/glide_f3jP0I/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/var/lib/condor/execute/dir_751020/glide_f3jP0I/log/procd_address CONDOR_STARTER_PID=1205393 CONTAINERSW_FULL_VERSION=apptainer version 1.5.4-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer CONTAINERSW_VERSION=1.5.4-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T2_US_Nebraska GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_751020/glide_f3jP0I/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.CMS_T2_US_Nebraska_Red_test_apptainer.idtoken.idtoken GLIDEIN_CPUS=48 GLIDEIN_ClusterId=943139 GLIDEIN_Country=US GLIDEIN_DEBUG_OPTIONS= GLIDEIN_ESTIMATED_CPUS=36 GLIDEIN_Entry_Name=CMS_T2_US_Nebraska_Red_test_apptainer GLIDEIN_Entry_Signature=072b8cf75cdfc94dfbab899faf393d21208a04d4 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=red.unl.edu red.unl.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_guCzIS GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=red-squid.unl.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Nebraska GLIDEIN_SE_VONAME_LOWERCASE=srm://srm.unl.edu:8443/srm/v2/server?SFN=/mnt/hadoop/user/ GLIDEIN_SEs=srm.unl.edu GLIDEIN_SINGULARITY_BINARY_OVERRIDE=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=63af73ae490192bb0dd515c1346f3fabca411a9b GLIDEIN_Site=Nebraska GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_751020/glide_f3jP0I/tmp GLIDEIN_UUID=2000bf95-a345-43a0-a6b9-265d7e00f55f GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.4-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.4-1.el8 GWMS_PATH=/var/lib/condor/execute/dir_751020/glide_f3jP0I/.gwms.d/bin: GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.4-1.el8 HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_CMS_T2_US_Nebraska_Red_test_apptainer/tokens.tgz JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OLDPWD=/var/lib/condor/execute/dir_751020/glide_f3jP0I/logs/2000bf95-a345-43a0-a6b9-265d7e00f55f OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=UNAVAILABLE OSG_DATA=UNAVAILABLE OSG_GRID=/etc/osg/wn-client OSG_HOSTNAME=red.unl.edu OSG_SITE_NAME=Red OSG_SQUID_LOCATION=red-squid.unl.edu:3128 OSG_WN_TMP=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch/.local-tmp PATH=/var/lib/condor/execute/dir_751020/glide_f3jP0I/.gwms.d/bin::/bin:/usr/bin:/sbin:/usr/sbin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@red-c7224.unl.edu$ "; unset PROMPT_COMMAND PWD=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SHLVL=2 SINGULARITY_BIND=/var/lib/condor/execute/dir_751020,/cvmfs,/etc/cvmfs,/etc/grid-security,/var/lib/cvmfs,/var/lib/sss/pipes,/var/lib/sss/mc,/etc/nsswitch.conf,/dev/fuse,/dev/log SINGULARITY_CACHEDIR=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch SINGULARITY_CONTAINER=/mnt/t2ceph/user/jthiltge/cmssif/el8.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.5.4-1.el8 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/debian:13 SINGULARITY_MODE=unprivileged SINGULARITY_NAME=el8.sif SINGULARITY_PATH=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/testing/bin/apptainer SINGULARITY_VERSION=1.5.4-1.el8 SLOTS_LAYOUT=partitionable TEMP=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch TMPDIR=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_1205393=1205760:1790772894:747502530 _CONDOR_ANCESTOR_29396=29399:1790618339:2107691338 _CONDOR_ANCESTOR_29399=1205393:1790772893:3205694687 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_751020/glide_f3jP0I/main/condor/bin _CONDOR_CHIRP_CONFIG=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/user/.chirp.config _CONDOR_CREDS=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch/.condor_creds _CONDOR_EXECUTE=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute _CONDOR_JOB_AD=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch/.job.ad _CONDOR_JOB_IWD=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/var/lib/condor/execute/dir_751020/glide_f3jP0I/execute/dir_1205393/scratch/.job_wrapper_failure http_proxy=red-squid.unl.edu:3128
pass df
794.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg1-lv_var 909723648 76274580 833449068 9% /var/lib/condor/execute/dir_751020
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] 0.007ms pmtu 9000 1: 2600:900:6:1102::32 0.433ms 1: 2600:900:6:1102::32 0.420ms 2: 2600:900:6:1048::1 0.557ms 3: no reply 4: chic-cr6--uwmadison_se-203.es.net 15.786ms 5: uwmadison_se-203--chic-cr6.es.net 26.767ms 6: perfsonar02.hep.wisc.edu 26.732ms reached Resume: pmtu 9000 hops 6 back 6
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy red-squid.unl.edu:3128
pass user
osg (uid 32016)
OU-Sooner-CE1 1 hour ago 2
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
OU-Sooner-CE1 — probe output
pass kernel
5.14.0-687.39.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/lscratch/34662862/glide_Oszyjn/execute/dir_948023/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/lscratch/34662862/glide_Oszyjn/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/lscratch/34662862/glide_Oszyjn/execute/dir_948023/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=948023 CONTAINERSW_FULL_VERSION=apptainer version 1.5.1-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.1-1.el9 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=a9c5840f9c0c9c69a1ccc8df7a49c87f7b1ed2ce GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/lscratch/34662862/glide_Oszyjn/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_OU-Sooner-CE1.idtoken.idtoken GLIDEIN_CPUS=32 GLIDEIN_ClusterId=870025 GLIDEIN_Entry_Name=OSG_US_OU-Sooner-CE1 GLIDEIN_Entry_Signature=c478ee834ce275fd6437f287e3e291573450adf6 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=ou-sooner-ce1.svc.osg-htc.org ou-sooner-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_GlhSs9 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=OU-Sooner-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=04973478840ac6f9501c3524ab345e903645979b GLIDEIN_Site=OU-Sooner GLIDEIN_Tmp_Dir=/lscratch/34662862/glide_Oszyjn/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.1-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.1-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.1-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ou-sooner-ce1.svc.osg-htc.org OSG_SITE_NAME=OU-Sooner-CE1 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_2566047_173018916@c1041.oscer.ou.edu$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_2566047_173018916@c1041.oscer.ou.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/lscratch/34662862/glide_Oszyjn/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/lscratch/34662862/glide_Oszyjn/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/lscratch/34662862/glide_Oszyjn/execute/dir_948023/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
329.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vg_lscratch-lscratch_34662862 379726592 34297616 345428976 10% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: 10.251.224.20 0.819ms 1: 10.251.224.20 1.228ms 2: 10.0.102.12 0.786ms 3: 156.110.31.129 0.876ms 4: site--ou-se1547-v4.lhcone.es.net 3.866ms asymm 7 5: no reply 6: chic-cr6--uwmadison_se-203.es.net 18.973ms 7: uwmadison_se-203--chic-cr6.es.net 29.934ms 8: perfsonar02.hep.wisc.edu 29.945ms reached Resume: pmtu 9000 hops 8 back 8
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 22858)
PDX-Coeus-CE1 1 hour ago 10
90.0%
90.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user gpu
PDX-Coeus-CE1 — probe output
pass kernel
5.14.0-687.42.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/tmp/glide_PpbfM1/execute/dir_2575600/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_PpbfM1/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/tmp/glide_PpbfM1/execute/dir_2575600/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2575600 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/tmp/glide_PpbfM1/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=282e7263f1755563ecd8b0c3a7dfabafef58aaed GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_PpbfM1/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_PDX-Coeus-CE1.idtoken.idtoken GLIDEIN_CPUS=32 GLIDEIN_ClusterId=687627 GLIDEIN_Entry_Name=OSG_US_PDX-Coeus-CE1 GLIDEIN_Entry_Signature=f6a1acfb5e169135c6b831784426e408e9a6d9e7 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=pdx-coeus-ce1.svc.osg-htc.org pdx-coeus-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_OHoSBi GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=coeus.cluster:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=PDX-Coeus-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=ad419b1b3db36bf5c7633f8e9c4c175eb881dd81 GLIDEIN_Site=PDX-Coeus GLIDEIN_Tmp_Dir=/tmp/glide_PpbfM1/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/tmp/glide_PpbfM1/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/tmp/glide_PpbfM1/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=pdx-coeus-ce1.svc.osg-htc.org OSG_SITE_NAME=PDX-Coeus-CE1 OSG_SQUID_LOCATION=coeus.cluster:3128 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_5@glidein_1337573_178148655@gpu07$ "; unset PROMPT_COMMAND PS1=slot1_5@glidein_1337573_178148655@gpu07$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/tmp/glide_PpbfM1/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_PpbfM1/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_PpbfM1/execute/dir_2575600/scratch _CONDOR_SLOT=slot1_5 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1621.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p1 1844812308 50580016 1700447256 3% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 9000 1: _gateway 0.968ms asymm 2 1: _gateway 1.024ms asymm 2 2: _gateway 0.801ms asymm 3 3: 10.252.100.1 0.185ms pmtu 1500 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy coeus.cluster:3128
pass user
osg01 (uid 10262)
pass gpu
Wed Sep 30 05:54:56 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 610.57.04 KMD Version: 610.57.04 CUDA UMD Version: 13.3 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX A5000 Off | 00000000:01:00.0 Off | Off | | 40% 68C P2 223W / 230W | 13835MiB / 24564MiB | 100% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA RTX A5000 Off | 00000000:41:00.0 Off | Off | | 32% 63C P2 198W / 230W | 9111MiB / 24564MiB | 73% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA RTX A5000 Off | 00000000:81:00.0 Off | Off | | 44% 71C P2 227W / 230W | 13835MiB / 24564MiB | 100% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA RTX A5000 Off | 00000000:C1:00.0 Off | Off | | 30% 26C P8 20W / 230W | 5MiB / 24564MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
PDX-Orca-CE1 1 hour ago 6
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
PDX-Orca-CE1 — probe output
pass kernel
5.14.0-687.42.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/tmp/glide_nRnS8J/execute/dir_2120002/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_nRnS8J/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_nRnS8J/execute/dir_2120002/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2120002 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/tmp/glide_nRnS8J/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_nRnS8J/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_PDX-Orca-CE1.idtoken.idtoken GLIDEIN_CPUS=32 GLIDEIN_ClusterId=870492 GLIDEIN_Entry_Name=OSG_US_PDX-Orca-CE1 GLIDEIN_Entry_Signature=c7389b6bcba6668e137dd1798ba173fa36655a73 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=pdx-orca-ce1.svc.osg-htc.org pdx-orca-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_mz37EK GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=PDX-Orca-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=b7ad1a6a40bd6a6100fb8503c2b0a86ab4628f8c GLIDEIN_Site=PDX-Orca GLIDEIN_Tmp_Dir=/tmp/glide_nRnS8J/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/tmp/glide_nRnS8J/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/tmp/glide_nRnS8J/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=pdx-orca-ce1.svc.osg-htc.org OSG_SITE_NAME=PDX-Orca-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_15@glidein_3550749_103109016@orcaga20$ "; unset PROMPT_COMMAND PS1=slot1_15@glidein_3550749_103109016@orcaga20$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/tmp/glide_nRnS8J/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_nRnS8J/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_nRnS8J/execute/dir_2120002/scratch _CONDOR_SLOT=slot1_15 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
305.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 460367788 116778896 320129980 27% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 9000 1: _gateway 0.841ms asymm 2 1: _gateway 0.716ms asymm 2 2: _gateway 0.635ms asymm 3 3: 10.252.100.1 0.136ms pmtu 1500 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 10262)
PSU-LIGO 1 hour ago 128
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
PSU-LIGO — probe output
pass kernel
4.18.0-553.147.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/localscratch/condor/execute/dir_161502/scratch/glide_Ri7ioH/execute/dir_1342179/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/localscratch/condor/execute/dir_161502/scratch/glide_Ri7ioH/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/localscratch/condor/execute/dir_161502/scratch/glide_Ri7ioH/execute/dir_1342179/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1342179 CONTAINERSW_FULL_VERSION=apptainer version 1.5.2-1.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.2-1.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/localscratch/condor/execute/dir_161502/scratch/glide_Ri7ioH/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_PSU-LIGO.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=959091 GLIDEIN_Entry_Name=OSG_US_PSU-LIGO GLIDEIN_Entry_Signature=b14f94e287e6bb9cc0e1c9abd64f73e9e1ba5411 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=psu-ligo-ce1.svc.opensciencegrid.org psu-ligo-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_VSzysR GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=ligo-squid-01.gwave.ics.psu.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=PSU-LIGO GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=PSU-LIGO GLIDEIN_Tmp_Dir=/localscratch/condor/execute/dir_161502/scratch/glide_Ri7ioH/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.2-1.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.2-1.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.2-1.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=psu-ligo-ce1.svc.opensciencegrid.org OSG_SITE_NAME=PSU-LIGO OSG_SQUID_LOCATION=ligo-squid-01.gwave.ics.psu.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_4@glidein_161518_611103750@comp-ex-0202$ "; unset PROMPT_COMMAND PS1=slot1_4@glidein_161518_611103750@comp-ex-0202$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/localscratch/condor/execute/dir_161502/scratch/glide_Ri7ioH/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/localscratch/condor/execute/dir_161502/scratch/glide_Ri7ioH/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/localscratch/condor/execute/dir_161502/scratch/glide_Ri7ioH/execute/dir_1342179/scratch _CONDOR_SLOT=slot1_4 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
1134.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p3 1366833076 177756052 1189077024 14% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 21.760ms 1: _gateway 5.974ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy ligo-squid-01.gwave.ics.psu.edu:3128
pass user
osg01 (uid 40131)
Purdue-Anvil-CE1 1 hour ago 14
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Purdue-Anvil-CE1 — probe output
pass kernel
4.18.0-553.162.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/tmp/glide_13tM0P/execute/dir_3123065/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_13tM0P/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_13tM0P/execute/dir_3123065/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3123065 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_13tM0P/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Purdue-Anvil-CE1.idtoken.idtoken GLIDEIN_CPUS=128 GLIDEIN_ClusterId=854003 GLIDEIN_Entry_Name=OSG_US_Purdue-Anvil-CE1 GLIDEIN_Entry_Signature=d2db0058c8948632c4cddca1f00f6c3e00f5df23 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=purdue-anvil-ce1.svc.osg-htc.org purdue-anvil-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_x-ob_CqdD78 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=squid1.rcac.purdue.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Purdue-Anvil-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=f90b138f62b42316e2088dea04273e4626194e65 GLIDEIN_Site=Purdue-Anvil GLIDEIN_Tmp_Dir=/tmp/glide_13tM0P/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=purdue-anvil-ce1.svc.osg-htc.org OSG_SITE_NAME=Purdue-Anvil-CE1 OSG_SQUID_LOCATION=squid1.rcac.purdue.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_37@glidein_247932_545992700@a163.anvil.rcac.purdue.edu$ "; unset PROMPT_COMMAND PS1=slot1_37@glidein_247932_545992700@a163.anvil.rcac.purdue.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/tmp/glide_13tM0P/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_13tM0P/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_13tM0P/execute/dir_3123065/scratch _CONDOR_SLOT=slot1_37 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
184.9 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda2 374917264 181063868 193853396 49% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=4092 1?: [LOCALHOST] pmtu 4092 1: 172.17.3.179 0.436ms 1: 172.17.3.179 0.464ms 2: math-g190-ar7280dr3a-01-et7-1.tcom.purdue.edu 0.818ms 3: math-g190-ar7280dr3a-01-vl2500.tcom.purdue.edu 0.339ms 4: math-g190-ar7280dr3a-02-et54-1.tcom.purdue.edu 0.299ms asymm 3 5: research-gigapop-ll-lhcone-400g.tcom.purdue.edu 2.179ms asymm 4 6: lo-0.2.rtr2.chic.indiana.gigapop.net 6.042ms asymm 5 7: no reply 8: uwmadison_se-203--chic-cr6.es.net 16.581ms asymm 7 9: perfsonar02.hep.wisc.edu 16.538ms reached Resume: pmtu 4092 hops 9 back 8
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy squid1.rcac.purdue.edu:3128
pass user
x-ob (uid 7941029)
Purdue-Geddes 1 hour ago 10
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Purdue-Geddes — probe output
pass kernel
4.18.0-553.22.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/pilot/osgvo-pilot-lUwc0h/execute/dir_3274234/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-lUwc0h/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-lUwc0h/execute/dir_3274234/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=3274234 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Purdue Geddes OSG_SQUID_LOCATION=http://squid1.rcac.purdue.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_1@Purdue-Geddes.osg-geddes-backfill-a-3$ "; unset PROMPT_COMMAND PS1=slot1_1@Purdue-Geddes.osg-geddes-backfill-a-3$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/pilot/osgvo-pilot-lUwc0h/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-lUwc0h/execute/dir_3274234/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://squid1.rcac.purdue.edu:3128
pass df
100.1 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/rbd1 123268672 18248104 105004184 15% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=8950 1?: [LOCALHOST] pmtu 8950 1: 172-21-160-10.rancher-monitoring-kubelet.kube-system.svc.cluster.local 0.145ms 1: 172-21-160-10.rancher-monitoring-kubelet.kube-system.svc.cluster.local 0.168ms 2: math-g109-dz9264f-01-vlan1160.tcom.purdue.edu 0.359ms 3: math-g109-dz9264f-01-eth-1-1-65.tcom.purdue.edu 0.215ms 4: math-g190-ar7280dr3a-01-vl2500.tcom.purdue.edu 0.313ms 5: math-g190-ar7280dr3a-02-et54-1.tcom.purdue.edu 0.322ms asymm 4 6: research-gigapop-ll-lhcone-400g.tcom.purdue.edu 2.176ms asymm 5 7: lo-0.2.rtr2.chic.indiana.gigapop.net 5.538ms asymm 6 8: no reply 9: uwmadison_se-203--chic-cr6.es.net 16.826ms asymm 8 10: perfsonar02.hep.wisc.edu 16.445ms reached Resume: pmtu 8950 hops 10 back 9
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://squid1.rcac.purdue.edu:3128
pass user
osg (uid 1000)
Rhodes-HPC-CE1 1 hour ago 9
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Rhodes-HPC-CE1 — probe output
pass kernel
5.14.0-687.39.1.el9_8.x86_64
pass env
BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2891462 CONTAINERSW_FULL_VERSION=singularity-ce version 4.4.1-1.el9 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/singularity CONTAINERSW_VERSION=4.4.1-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_AxtWqZ/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Rhodes-HPC-CE1.idtoken.idtoken GLIDEIN_CPUS=48 GLIDEIN_ClusterId=870598 GLIDEIN_Entry_Name=OSG_US_Rhodes-HPC-CE1 GLIDEIN_Entry_Signature=2d0eebd64dd538b35175189acd7aa1ad6785fb5d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=rhodes-hpc-ce1.svc.osg-htc.org rhodes-hpc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_yYGjAt GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=10.0.0.1:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Rhodes-HPC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=b7ad1a6a40bd6a6100fb8503c2b0a86ab4628f8c GLIDEIN_Site=Rhodes-HPC GLIDEIN_Tmp_Dir=/scratch/glide_AxtWqZ/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity-ce version 4.4.1-1.el9 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/singularity GWMS_CONTAINERSW_VERSION=4.4.1-1.el9 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=4.4.1-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=rhodes-hpc-ce1.svc.osg-htc.org OSG_SITE_NAME=Rhodes-HPC-CE1 OSG_SQUID_LOCATION=10.0.0.1:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_27@glidein_1024159_146184312@compute09$ "; unset PROMPT_COMMAND PS1=slot1_27@glidein_1024159_146184312@compute09$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/scratch/glide_AxtWqZ/execute/dir_2891462:/srv,/etc/hosts,/etc/localtime,/cvmfs:/cvmfs:ro SINGULARITY_CACHEDIR=/scratch/glide_AxtWqZ/execute/dir_2891462/scratch SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/scratch/glide_AxtWqZ/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SINGULARITY_TMPDIR=/scratch/glide_AxtWqZ/execute/dir_2891462/scratch SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_AxtWqZ/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_AxtWqZ/execute/dir_2891462/scratch _CONDOR_SLOT=slot1_27 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
66.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 499862048 430010532 69851516 87% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: lotus 0.314ms 1: lotus 0.122ms 2: 10.70.0.1 0.212ms 3: xg-1-1-35-3909-soag12.hornlake.tn.malt.comcast.net 1.610ms 4: no reply 5: no reply 6: be-288-ar01.hornlake.tn.malt.comcast.net 1.703ms 7: 162.151.119.229 0.945ms 8: be-34922-cs02.dallas.tx.ibone.comcast.net 10.184ms 9: be-3101-pe01.1950stemmons.tx.ibone.comcast.net 10.931ms 10: bundle-ether4.agg1.dall3.net.internet2.edu 10.513ms 11: no reply 12: no reply 13: no reply 14: no reply 15: 198.71.47.15 30.033ms asymm 16 16: perfsonar02.hep.wisc.edu 29.852ms reached Resume: pmtu 1500 hops 16 back 18
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 10.0.0.1:3128
pass user
osg01 (uid 10000)
SDSC-PRP-OSPool-Provisioner 1 hour ago 307
0.0%
0.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
SDSC-PRP-OSPool-Provisioner — probe output
pass kernel
6.8.0-136-generic
pass env
APPTAINER_CACHEDIR=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=254044 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main-container GLIDEIN_ResourceName=SDSC-PRP-OSPool-Provisioner GLIDEIN_Site=SDSC-PRP GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_DIR=/pilot/osgvo-pilot-PuoxN2 GWMS_SINGULARITY_STATUS=PREFERRED HAS_SINGULARITY=False HOME=/home/osg JULIA_NUM_THREADS=1 LC_CTYPE=C.UTF-8 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=SDSC-PRP-OSPool-Provisioner OSG_SQUID_LOCATION=10.110.90.167:3128 OSG_WN_TMP=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch/.local-tmp PATH=/usr/local/bin:/usr/bin PWD=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SHLVL=0 SINGULARITY_CACHEDIR=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch TEMP=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch TMPDIR=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch/.local-tmp TZ=UTC _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_254044=254410:1790772823:610352036 _CONDOR_ANCESTOR_3031=3035:1790751650:1459176771 _CONDOR_ANCESTOR_3035=254044:1790772822:476708158 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/user/.chirp.config _CONDOR_CREDS=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch/.condor_creds _CONDOR_JOB_AD=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch/.job.ad _CONDOR_JOB_IWD=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/pilot/osgvo-pilot-PuoxN2/execute/dir_254044/scratch/.job_wrapper_failure http_proxy=10.110.90.167:3128
pass df
465.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 733644800 245540600 488104200 34% /
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=8930 1?: [LOCALHOST] pmtu 8930 1: pas.bois.net.internet2.edu 0.197ms 1: pas.bois.net.internet2.edu 0.143ms 2: hundredge-0-0-0-33.9.core1.bois.net.internet2.edu 3.651ms 3: fourhundredge-0-0-0-0.4079.core2.salt.net.internet2.edu 40.511ms asymm 11 4: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 41.932ms asymm 10 5: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 41.462ms asymm 9 6: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 41.536ms asymm 8 7: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 41.243ms 8: 144.92.233.112 43.449ms 9: perfsonar02.hep.wisc.edu 43.519ms reached Resume: pmtu 8930 hops 9 back 10
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 10.110.90.167:3128
pass user
osg (uid 1000)
SIUE-Kyna-CE1 6 hours ago 2
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
SIUE-Kyna-CE1 — probe output
pass kernel
6.8.0-111-generic
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/scratch/osg/glide_CEgO0P/execute/dir_846329/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg/glide_CEgO0P/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg/glide_CEgO0P/execute/dir_846329/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=846329 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/osg/glide_CEgO0P/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg/glide_CEgO0P/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_SIUE-Kyna-CE1.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6562301 GLIDEIN_Entry_Name=OSG_US_SIUE-Kyna-CE1 GLIDEIN_Entry_Signature=2e7ed7c3ac588bd551f87500dba7c4fdf5a0ae34 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=siue-kyna-ce1.svc.osg-htc.org siue-kyna-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_kvmWlt GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=SIUE-Kyna-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=SIUE-Kyna GLIDEIN_Tmp_Dir=/scratch/osg/glide_CEgO0P/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/osg/glide_CEgO0P/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/osg/glide_CEgO0P/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv HTTPS_PROXY=http://10.10.44.230:8080 HTTP_PROXY=http://10.10.44.230:8080 JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NO_PROXY=::1,127.0.0.1,localhost,10.0.0.0/8,*.hpc.siue.edu NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=siue-kyna-ce1.svc.osg-htc.org OSG_SITE_NAME=SIUE-Kyna-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_814265_208287706@cn11.hpc.siue.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_814265_208287706@cn11.hpc.siue.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/scratch/osg/glide_CEgO0P/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg/glide_CEgO0P/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg/glide_CEgO0P/execute/dir_846329/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=http://10.10.44.230:8080 https_proxy=http://10.10.44.230:8080 no_proxy=::1,127.0.0.1,localhost,10.0.0.0/8,*.hpc.siue.edu
pass df
1620.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vgroot-lvroot 1843643996 50729120 1699189180 3% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.374ms 1: _gateway 0.333ms 2: 146.163.142.254 0.355ms 3: vdom-root120.net.siue.edu 0.413ms 4: dh0122-r1.net.siue.edu 1.462ms 5: mren-router.net.siue.edu 14.433ms 6: hundredge-0-0-0-28.1136.core1.star.net.internet2.edu 9.528ms 7: 144.92.233.112 11.871ms 8: perfsonar02.hep.wisc.edu 11.950ms reached Resume: pmtu 1500 hops 8 back 9
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy http://10.10.44.230:8080
pass user
osg (uid 64034)
SLU-Libra-CE1 1 hour ago 5
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
SLU-Libra-CE1 — probe output
pass kernel
5.14.0-611.55.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/scratch/osg01/glide_9EgvkT/execute/dir_715952/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg01/glide_9EgvkT/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/scratch/osg01/glide_9EgvkT/execute/dir_715952/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=715952 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/osg01/glide_9EgvkT/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=a9c5840f9c0c9c69a1ccc8df7a49c87f7b1ed2ce GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01/glide_9EgvkT/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_SLU-Libra-CE1_gpu.idtoken.idtoken GLIDEIN_CPUS=32 GLIDEIN_ClusterId=954116 GLIDEIN_Entry_Name=OSG_US_SLU-Libra-CE1_gpu GLIDEIN_Entry_Signature=ffdc63a4ca2d1a5a9988d583fc4bd5f8102b5594 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=slu-libra-ce1.svc.osg-htc.org slu-libra-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_3JiCr7 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=SLU-Libra-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-2.osg-htc.org GLIDEIN_Signature=b7ad1a6a40bd6a6100fb8503c2b0a86ab4628f8c GLIDEIN_Site=SLU-Libra GLIDEIN_Tmp_Dir=/scratch/osg01/glide_9EgvkT/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/osg01/glide_9EgvkT/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/osg01/glide_9EgvkT/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=slu-libra-ce1.svc.osg-htc.org OSG_SITE_NAME=SLU-Libra-CE1 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_7@glidein_3310518_114561090@gpu02.cm.cluster$ "; unset PROMPT_COMMAND PS1=slot1_7@glidein_3310518_114561090@gpu02.cm.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/scratch/osg01/glide_9EgvkT/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01/glide_9EgvkT/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01/glide_9EgvkT/execute/dir_715952/scratch _CONDOR_SLOT=slot1_7 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
79352.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 10.149.6.1:/data01/scratch/osg01/glide_9EgvkT/execute/dir_715952 110909175808 27702017024 83207158784 25% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: head01.cm.cluster 0.172ms 1: head01.cm.cluster 0.247ms 2: 10.108.121.254 16.497ms 3: no reply 4: no reply 5: 10.248.101.14 16.692ms 6: 10.248.100.20 24.689ms 7: 165.134.223.27 24.731ms 8: fourhundredge-0-0-0-10.23.core2.kans.net.internet2.edu 34.310ms 9: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 45.732ms asymm 11 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 47.431ms 11: 144.92.233.112 44.445ms 12: perfsonar02.hep.wisc.edu 46.484ms reached Resume: pmtu 1500 hops 12 back 13
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 1001)
SU-ITS-CE2 1 hour ago 352
97.7%
0.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
SU-ITS-CE2 — probe output
warning kernel
3.10.0-1160.66.1.el7.x86_64
pass env
APPTAINER_CACHEDIR=/srv BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=singularity version 3.6.0-rc.4 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/bin/singularity CONTAINERSW_VERSION=3.6.0-rc.4 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_36995/glide_wuFgSM/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.Glow_US_Syracuse_condor-ce2_whole.idtoken.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=687982 GLIDEIN_Country=US GLIDEIN_ESTIMATED_CPUS=40 GLIDEIN_Entry_Name=Glow_US_Syracuse_condor-ce2_whole GLIDEIN_Entry_Signature=53f6ab91126cceb964b8a9e93c9e3bcbc0945b8f GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=its-condor-ce2.syr.edu its-condor-ce2.syr.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_nobody_z8XxM4 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=4 GLIDEIN_Proxy_URL=10.5.0.41:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=SU-ITS-CE2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=SU-ITS GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_36995/glide_wuFgSM/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity version 3.6.0-rc.4 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/bin/singularity GWMS_CONTAINERSW_VERSION=3.6.0-rc.4 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.6.0-rc.4 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=its-condor-ce2.syr.edu OSG_SITE_NAME=SU-ITS-CE2 OSG_SQUID_LOCATION=10.5.0.41:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_57@glidein_36997_210828464@CRUSH-OSG-C7-10-5-221-5$ "; unset PROMPT_COMMAND PS1=slot1_57@glidein_36997_210828464@CRUSH-OSG-C7-10-5-221-5$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_CACHEDIR=/srv SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:8 SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity version 3.6.0-rc.4 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:8 SINGULARITY_MODE=privileged SINGULARITY_NAME=rocky:8 SINGULARITY_PATH=/bin/singularity SINGULARITY_TMPDIR=/var/lib/condor/execute/dir_36995/glide_wuFgSM/execute/dir_43856 SINGULARITY_VERSION=3.6.0-rc.4 SLOTS_LAYOUT=partitionable TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_36995/glide_wuFgSM/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_36995/glide_wuFgSM/execute/dir_43856 _CONDOR_SLOT=slot1_57 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass df
167.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/centos-root 206542340 30606696 175935644 15% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.5.255.254 0.484ms 1: 10.5.255.254 0.617ms 2: 10.30.29.35 0.224ms 3: 10.30.29.46 0.678ms asymm 2 4: 128.230.61.33 1.451ms asymm 3 5: syr-9208-su.nysernet.net 0.687ms asymm 4 6: buf-9208-syr-9208.nysernet.net 5.409ms asymm 5 7: i2-clev-buf-9208.nysernet.net 9.195ms asymm 6 8: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 16.870ms asymm 9 9: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 14.906ms asymm 8 10: 144.92.233.112 17.792ms asymm 9 11: perfsonar02.hep.wisc.edu 17.769ms reached Resume: pmtu 1500 hops 11 back 11
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 10.5.0.41:3128
pass user
nobody (uid 99)
SU-ITS-CE3 1 hour ago 155
91.0%
0.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
SU-ITS-CE3 — probe output
warning kernel
3.10.0-1160.66.1.el7.x86_64
pass env
APPTAINER_CACHEDIR=/srv BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=singularity version 3.6.0-rc.4 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/bin/singularity CONTAINERSW_VERSION=3.6.0-rc.4 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_OSG GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_23762/glide_4XPetG/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.Glow_US_Syracuse_condor-ce3_whole.idtoken.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=871551 GLIDEIN_Country=US GLIDEIN_ESTIMATED_CPUS=40 GLIDEIN_Entry_Name=Glow_US_Syracuse_condor-ce3_whole GLIDEIN_Entry_Signature=a864bbc84606161bd381a2e2c55caa75ea5cc3b0 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=its-condor-ce3.syr.edu its-condor-ce3.syr.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_nobody_nb0Ucb GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=10.5.0.41:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=SU-ITS-CE3 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-2.osg-htc.org GLIDEIN_Signature=ad419b1b3db36bf5c7633f8e9c4c175eb881dd81 GLIDEIN_Site=SU-ITS GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_23762/glide_4XPetG/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=singularity version 3.6.0-rc.4 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/bin/singularity GWMS_CONTAINERSW_VERSION=3.6.0-rc.4 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/bin/singularity GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=3.6.0-rc.4 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=its-condor-ce3.syr.edu OSG_SITE_NAME=SU-ITS-CE3 OSG_SQUID_LOCATION=10.5.0.41:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_30@glidein_23764_382256444@CRUSH-OSG-C7-10-5-171-89$ "; unset PROMPT_COMMAND PS1=slot1_30@glidein_23764_382256444@CRUSH-OSG-C7-10-5-171-89$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_CACHEDIR=/srv SINGULARITY_COMMAND=exec SINGULARITY_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/ubuntu:24.04 SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=singularity version 3.6.0-rc.4 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/ubuntu:24.04 SINGULARITY_MODE=privileged SINGULARITY_NAME=ubuntu:24.04 SINGULARITY_PATH=/bin/singularity SINGULARITY_TMPDIR=/var/lib/condor/execute/dir_23762/glide_4XPetG/execute/dir_6807 SINGULARITY_VERSION=3.6.0-rc.4 SLOTS_LAYOUT=partitionable TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_23762/glide_4XPetG/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_23762/glide_4XPetG/execute/dir_6807 _CONDOR_SLOT=slot1_30 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass df
168.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/centos-root 206542340 29761292 176781048 15% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.5.255.254 0.507ms 1: 10.5.255.254 0.403ms 2: 10.30.29.35 0.249ms 3: 10.30.29.46 0.695ms asymm 2 4: 128.230.61.33 1.473ms asymm 3 5: syr-9208-su.nysernet.net 1.029ms asymm 4 6: buf-9208-syr-9208.nysernet.net 4.592ms asymm 5 7: i2-clev-buf-9208.nysernet.net 9.050ms asymm 6 8: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 16.260ms asymm 9 9: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 16.257ms asymm 8 10: 144.92.233.112 18.004ms asymm 9 11: perfsonar02.hep.wisc.edu 17.820ms reached Resume: pmtu 1500 hops 11 back 11
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 10.5.0.41:3128
pass user
nobody (uid 99)
SYRA-PATH-EP 1 hour ago 2
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
SYRA-PATH-EP — probe output
pass kernel
4.18.0-517.el8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/pilot/osgvo-pilot-gpwfjZ/execute/dir_1003799/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-gpwfjZ/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-gpwfjZ/execute/dir_1003799/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1003799 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=SYRA-PATH PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_105@SYRA-PATH-EP.osgvo-docker-pilot-ospool-7c8b4d4476-zb8zf$ "; unset PROMPT_COMMAND PS1=slot1_105@SYRA-PATH-EP.osgvo-docker-pilot-ospool-7c8b4d4476-zb8zf$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/pilot/osgvo-pilot-gpwfjZ/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-gpwfjZ/execute/dir_1003799/scratch _CONDOR_SLOT=slot1_105 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
249.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/openebs-pvc--c18ea35d--d569--4e76--b24a--f06f1c76ea96 442688776 181383772 261288620 41% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1480 1?: [LOCALHOST] pmtu 1480 1: 128-230-87-133.kubelet.kube-system.svc.cluster.local 0.076ms 1: 128-230-87-133.kubelet.kube-system.svc.cluster.local 0.142ms 2: 128.230.87.189 0.351ms 3: 128.230.61.33 1.627ms asymm 5 4: syr-9208-su.nysernet.net 0.739ms asymm 6 5: buf-9208-syr-9208.nysernet.net 9.805ms asymm 7 6: i2-clev-buf-9208.nysernet.net 9.873ms asymm 8 7: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 15.351ms asymm 11 8: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 14.942ms asymm 10 9: 144.92.233.112 17.551ms asymm 11 10: perfsonar02.hep.wisc.edu 17.543ms reached Resume: pmtu 1480 hops 10 back 13
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg (uid 1000)
Swan-CE1 1 hour ago 300
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Swan-CE1 — probe output
pass kernel
4.18.0-553.157.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/scratch/glide_7iXBHg/execute/dir_1929734/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_7iXBHg/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_7iXBHg/execute/dir_1929734/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1929734 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T2_US_Nebraska GLIDEIN_CONDOR_TOKEN=/scratch/glide_7iXBHg/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.HCC_US_Omaha_swan.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=7020055 GLIDEIN_Country=US GLIDEIN_DUNESite=US_Swan GLIDEIN_Entry_Name=HCC_US_Omaha_swan GLIDEIN_Entry_Signature=e81ea0d7dea4ae1d1a322cd68894b3304901f9a0 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=swan-gw1.unl.edu swan-gw1.unl.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_XYYuPk GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Swan-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=76ba0eb3c7154bcebd6c7df094e963d26ecf4544 GLIDEIN_Site=Swan GLIDEIN_Tmp_Dir=/scratch/glide_7iXBHg/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=unavailable HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=swan-gw1.unl.edu OSG_SITE_NAME=Swan OSG_SQUID_LOCATION=red-squid.unl.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_1590070_142877589@c1909.swan.hcc.unl.edu$ "; unset PROMPT_COMMAND PS1=slot1@glidein_1590070_142877589@c1909.swan.hcc.unl.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/scratch/glide_7iXBHg/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TERM=vt220 TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_7iXBHg/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_7iXBHg/execute/dir_1929734/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
3516.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/md3 3748646536 61560576 3687085960 2% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.254ms 1: c3838.pki.hcc.unl.edu 5.884ms 1: c3838.pki.hcc.unl.edu 0.107ms 2: c3838.pki.hcc.unl.edu 0.136ms asymm 1 3: hcc-pki-mlxe.unl.edu 0.456ms asymm 2 4: 129.93.241.41 0.685ms asymm 3 5: 129.93.6.29 2.419ms asymm 4 6: 10.161.1.14 3.958ms asymm 5 7: fourhundredge-0-0-0-10.1410.core2.kans.net.internet2.edu 8.629ms asymm 6 8: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 17.948ms asymm 9 9: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 18.607ms asymm 8 10: 144.92.233.112 20.278ms asymm 9 11: perfsonar02.hep.wisc.edu 17.448ms reached Resume: pmtu 1500 hops 11 back 10
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy red-squid.unl.edu:3128
pass user
osg (uid 32016)
Swarthmore-Firebird-CE1 1 hour ago 28
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
Swarthmore-Firebird-CE1 — probe output
pass kernel
5.14.0-687.39.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/scratch/osg01/glide_weny7E/execute/dir_2831725/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/osg01/glide_weny7E/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/osg01/glide_weny7E/execute/dir_2831725/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2831725 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/osg01/glide_weny7E/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_Swarthmore-firebird-CE1.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=7097864 GLIDEIN_Entry_Name=OSG_US_Swarthmore-firebird-CE1 GLIDEIN_Entry_Signature=1c501b0ced07b87e42423be3bdc6843909ed7e5d GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=swarthmore-firebird-ce1.svc.osg-htc.org swarthmore-firebird-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_F9ezeX GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=Swarthmore-Firebird-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=Swarthmore-Firebird GLIDEIN_Tmp_Dir=/scratch/osg01/glide_weny7E/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=swarthmore-firebird-ce1.svc.osg-htc.org OSG_SITE_NAME=Swarthmore-Firebird-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_2697182_310318020@node02.cluster$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_2697182_310318020@node02.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/scratch/osg01/glide_weny7E/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/osg01/glide_weny7E/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/osg01/glide_weny7E/execute/dir_2831725/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
7029.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/disk0-scratch 7439005696 68176500 7370829196 1% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: firebird.cluster 0.279ms 1: firebird.cluster 0.261ms 2: fw-comsvc-colo.lafayette.edu 0.972ms 3: 139.147.1.113 1.032ms asymm 4 4: 172.31.0.33 0.909ms 5: 172.31.0.3 1.574ms 6: gw0-g0-3-ibgp.prd.lafayette.edu 1.650ms asymm 5 7: 164.153.140.160 5.008ms asymm 6 8: no reply 9: hundredge-0-0-0-25.68.core1.pitt.net.internet2.edu 16.349ms asymm 8 10: fourhundredge-0-0-0-3.4079.core1.clev.net.internet2.edu 23.582ms asymm 13 11: fourhundredge-0-0-0-2.4079.core1.eqch.net.internet2.edu 23.606ms asymm 12 12: fourhundredge-0-0-0-6.4079.core1.star.net.internet2.edu 23.783ms asymm 11 13: 144.92.233.112 25.977ms asymm 12 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 200079)
TCNJ-ELSA-CE2 1 hour ago 166
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user gpu
TCNJ-ELSA-CE2 — probe output
pass kernel
5.14.0-611.55.1.el9_7.0.3.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/local/scratch/glide_L7ONu0/execute/dir_2523458/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/scratch/glide_L7ONu0/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/local/scratch/glide_L7ONu0/execute/dir_2523458/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2523458 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/scratch/glide_L7ONu0/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_TCNJ-ELSA-CE2.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=861412 GLIDEIN_Entry_Name=OSG_US_TCNJ-ELSA-CE2 GLIDEIN_Entry_Signature=2e89201bc658fbfc1d38e1e7687717d53cc19081 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=tcnj-elsa-ce2.svc.opensciencegrid.org tcnj-elsa-ce2.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_i9mD3D GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=TCNJ-ELSA-CE2 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=ELSA GLIDEIN_Tmp_Dir=/local/scratch/glide_L7ONu0/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=tcnj-elsa-ce2.svc.opensciencegrid.org OSG_SITE_NAME=TCNJ-ELSA-CE2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_1944720_10337150@node028$ "; unset PROMPT_COMMAND PS1=slot1@glidein_1944720_10337150@node028$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/local/scratch/glide_L7ONu0/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/scratch/glide_L7ONu0/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/scratch/glide_L7ONu0/execute/dir_2523458/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
86.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda4 143897004 53421212 90475792 38% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: ohpc-sms.localdomain 0.241ms 1: ohpc-sms.localdomain 0.207ms 2: 10.18.1.251 1.007ms 3: 10.18.0.1 0.461ms 4: 159.91.51.251 0.609ms 5: 159.91.58.161 0.996ms 6: 172.16.4.3 0.990ms 7: 159.91.58.142 1.148ms 8: 130.156.252.77 3.437ms asymm 14 9: 130.156.35.186 6.896ms asymm 15 10: 130.156.35.161 3.746ms asymm 14 11: bundle-ether1.104.core1.phil.net.internet2.edu 7.695ms asymm 16 12: fourhundredge-0-0-0-2.4079.core2.ashb.net.internet2.edu 22.358ms asymm 22 13: fourhundredge-0-0-0-1.4079.core2.clev.net.internet2.edu 22.258ms asymm 21 14: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 22.769ms asymm 20 15: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 23.728ms asymm 20 16: 144.92.233.112 25.905ms asymm 21 17: perfsonar02.hep.wisc.edu 24.366ms reached Resume: pmtu 1500 hops 17 back 22
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 64101)
pass gpu
Wed Sep 30 08:53:47 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.178.04 Driver Version: 580.178.04 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce GTX 1080 Ti On | 00000000:1A:00.0 Off | N/A | | 23% 17C P8 8W / 250W | 5MiB / 11264MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
UA-LR-ITS-EP 1 hour ago 30
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
UA-LR-ITS-EP — probe output
pass kernel
4.18.0-553.6.1.el8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/pilot/osgvo-pilot-saGOmy/execute/dir_243794/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-saGOmy/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-saGOmy/execute/dir_243794/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=243794 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=UA Little Rock ITS OSG_SQUID_LOCATION=144.167.99.254:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_8@UA-LR-ITS-EP.19fd01f468e5$ "; unset PROMPT_COMMAND PS1=slot1_8@UA-LR-ITS-EP.19fd01f468e5$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/pilot/osgvo-pilot-saGOmy/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-saGOmy/execute/dir_243794/scratch _CONDOR_SLOT=slot1_8 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=144.167.99.254:3128
pass df
108.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on overlay 132653920 11895772 113946912 10% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.051ms 1: _gateway 0.063ms 2: 144.167.99.3 0.177ms 3: FH213-Firewall-Campus.net.ualr.edu 7.718ms 4: rtr-fh213-intf1-0.mgt.ualr.edu 1.168ms 5: 100.64.0.5 8.268ms asymm 4 6: 208.90.110.230 8.418ms asymm 5 7: no reply 8: no reply 9: no reply 10: 208.90.110.99 22.001ms 11: 208.90.110.99 20.023ms asymm 10 12: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 34.859ms asymm 16 13: fourhundredge-0-0-0-3.4079.core2.chic.net.internet2.edu 32.710ms asymm 16 14: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 33.752ms 15: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 33.549ms asymm 14 16: 144.92.233.112 36.964ms asymm 15 17: perfsonar02.hep.wisc.edu 36.842ms reached Resume: pmtu 1500 hops 17 back 17
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 144.167.99.254:3128
fail user
uid 1000 does not map to a user
UAH-Voyager-CE1 1 hour ago 31
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
UAH-Voyager-CE1 — probe output
pass kernel
5.14.0-503.40.1.el9_5.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/data001/OSG/glide_zRIRTx/execute/dir_2076862/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/data001/OSG/glide_zRIRTx/images/htc__debian__13.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__debian__13.sif APPTAINER_TMPDIR=/data001/OSG/glide_zRIRTx/execute/dir_2076862/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2076862 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/data001/OSG/glide_zRIRTx/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/data001/OSG/glide_zRIRTx/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_UAH-Voyager-CE1.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=6350076 GLIDEIN_Entry_Name=OSG_US_UAH-Voyager-CE1 GLIDEIN_Entry_Signature=dada6c1dc3486a2e40ad9d36d0fc30ce1c1c43f5 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=uah-voyager-ce1.svc.osg-htc.org uah-voyager-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_yertms GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UAH-Voyager-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=UAH-Voyager GLIDEIN_Tmp_Dir=/data001/OSG/glide_zRIRTx/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/data001/OSG/glide_zRIRTx/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/data001/OSG/glide_zRIRTx/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=uah-voyager-ce1.svc.osg-htc.org OSG_SITE_NAME=UAH-Voyager-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_3187675_18429026@node024$ "; unset PROMPT_COMMAND PS1=slot1@glidein_3187675_18429026@node024$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/data001/OSG/glide_zRIRTx/images/htc__debian__13.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__debian__13.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/data001/OSG/glide_zRIRTx/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/data001/OSG/glide_zRIRTx/execute/dir_2076862/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
366654.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on 172.17.170.1:/zpool/raid001/data001/OSG/glide_zRIRTx/execute/dir_2076862 384864538624 399659008 384464879616 1% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: stor001 0.260ms 1: stor001 0.257ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 50004)
UCMerced-Pinnacles-CE1 1 hour ago 64
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
UCMerced-Pinnacles-CE1 — probe output
pass kernel
5.14.0-687.24.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/localscratch/osg01/glide_n2nLrX/execute/dir_2150447/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/localscratch/osg01/glide_n2nLrX/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/localscratch/osg01/glide_n2nLrX/execute/dir_2150447/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2150447 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/localscratch/osg01/glide_n2nLrX/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/localscratch/osg01/glide_n2nLrX/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_UCMerced-Pinnacles-CE1.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=7086363 GLIDEIN_Entry_Name=OSG_US_UCMerced-Pinnacles-CE1 GLIDEIN_Entry_Signature=c04095ea1e76951261c12b488f2c6f8b765950f7 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=ucmerced-pinnacles-ce1.svc.opensciencegrid.org ucmerced-pinnacles-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_eP47C9 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UCMerced-Pinnacles-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=76ba0eb3c7154bcebd6c7df094e963d26ecf4544 GLIDEIN_Site=UCMerced-Pinnacles GLIDEIN_Tmp_Dir=/localscratch/osg01/glide_n2nLrX/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/localscratch/osg01/glide_n2nLrX/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/localscratch/osg01/glide_n2nLrX/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ucmerced-pinnacles-ce1.svc.opensciencegrid.org OSG_SITE_NAME=UCMerced-Pinnacles-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_3426532_233350182@node078.cluster$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_3426532_233350182@node078.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/localscratch/osg01/glide_n2nLrX/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/localscratch/osg01/glide_n2nLrX/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/localscratch/osg01/glide_n2nLrX/execute/dir_2150447/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
731.2 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme1n1p1 937233280 170463536 766769744 19% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: pinnacles.cluster 0.507ms 1: pinnacles.cluster 0.370ms 2: 169.236.74.1 0.926ms 3: 10.7.1.69 1.198ms 4: pa-core-0-1.ucmerced.edu 1.162ms 5: 10.253.1.0 1.856ms 6: 10.253.1.5 2.219ms asymm 5 7: sacr2-agg-01--ucm--10g--01.cenic.net 7.070ms asymm 11 8: hpr-emvl1-agg-01--sacr2-agg-01--100g.cenic.net 6.496ms asymm 10 9: hpr-svl-agg10--emvl1-agg-01--100g.cenic.net 6.835ms 10: 137.164.26.127 9.540ms 11: fourhundredge-0-0-0-23.4079.core2.sunn.net.internet2.edu 60.894ms asymm 22 12: fourhundredge-0-0-0-1.4079.core2.sacr.net.internet2.edu 58.869ms asymm 21 13: fourhundredge-0-0-0-1.4079.core2.salt.net.internet2.edu 59.236ms asymm 20 14: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 58.299ms asymm 19 15: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 59.493ms asymm 18 16: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 59.779ms asymm 17 17: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 58.415ms asymm 16 18: 144.92.233.112 61.085ms asymm 17 19: perfsonar02.hep.wisc.edu 60.904ms reached Resume: pmtu 1500 hops 19 back 19
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 100567)
UCR-HPCC-CE1 53 mins ago 2
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
UCR-HPCC-CE1 — probe output
pass kernel
4.18.0-553.144.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/scratch/glide_yZkGwn/execute/dir_776216/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_yZkGwn/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_yZkGwn/execute/dir_776216/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=776216 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/scratch/glide_yZkGwn/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_yZkGwn/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_UCR-HPCC-CE1.idtoken.idtoken GLIDEIN_CPUS=10 GLIDEIN_ClusterId=688865 GLIDEIN_Entry_Name=OSG_US_UCR-HPCC-CE1 GLIDEIN_Entry_Signature=c2a98f39d418af12a185b4afb7f0c8591872f727 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=ucr-hpcc-ce1.svc.osg-htc.org ucr-hpcc-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_39W8ds GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UCR-HPCC-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=UCR-HPCC GLIDEIN_Tmp_Dir=/scratch/glide_yZkGwn/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/scratch/glide_yZkGwn/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/scratch/glide_yZkGwn/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=ucr-hpcc-ce1.svc.osg-htc.org OSG_SITE_NAME=UCR-HPCC-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_1@glidein_660899_567130518@c21.ib.hpcc.ucr.edu$ "; unset PROMPT_COMMAND PS1=slot1_1@glidein_660899_567130518@c21.ib.hpcc.ucr.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/scratch/glide_yZkGwn/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_yZkGwn/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_yZkGwn/execute/dir_776216/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
435.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda6 465561308 9444888 456116420 3% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: nat-clust.clust.hpcc.ucr.edu 0.168ms 1: nat-clust.clust.hpcc.ucr.edu 0.139ms 2: d-genom-1120d-1-v31.ucr.edu 0.934ms 3: 172.23.254.250 0.690ms 4: 172.30.254.9 0.491ms 5: 172.30.254.8 0.413ms 6: b-som-bn-1-v2020.ucr.edu 1.074ms asymm 7 7: 137.164.26.128 12.508ms asymm 13 8: hpr-i2--lax-agg10.cenic.net 15.433ms asymm 14 9: fourhundredge-0-0-0-2.4079.core2.salt.net.internet2.edu 59.268ms asymm 23 10: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 59.538ms asymm 22 11: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 59.699ms asymm 21 12: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 60.347ms asymm 20 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 59.682ms asymm 19 14: 144.92.233.112 61.521ms asymm 20 15: perfsonar02.hep.wisc.edu 61.460ms reached Resume: pmtu 1500 hops 15 back 22
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 4542)
UConn-OSG 1 hour ago 27
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
UConn-OSG — probe output
pass kernel
4.18.0-553.158.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/local/execute/dir_1334159/glide_CPBZCq/execute/dir_1444385/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/execute/dir_1334159/glide_CPBZCq/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/local/execute/dir_1334159/glide_CPBZCq/execute/dir_1444385/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1444385 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/execute/dir_1334159/glide_CPBZCq/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_UConn_gluskap_op.idtoken.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=6338554 GLIDEIN_Country=US GLIDEIN_DUNESite=US_UConn GLIDEIN_Entry_Name=OSG_US_UConn_gluskap_op GLIDEIN_Entry_Signature=433332afeaa97d68ffbf93e023ce6aa979144756 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=gluskap.phys.uconn.edu gluskap.phys.uconn.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_prod02_pICjOv GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=1 GLIDEIN_Proxy_URL=gryphn.phys.uconn.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UConn-OSG GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=76ba0eb3c7154bcebd6c7df094e963d26ecf4544 GLIDEIN_Site=UConn GLIDEIN_Tmp_Dir=/local/execute/dir_1334159/glide_CPBZCq/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=gluskap.phys.uconn.edu OSG_SITE_NAME=UConn-OSG_CE OSG_SQUID_LOCATION=gryphn.phys.uconn.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_1334164_608995900@nod45.phys.uconn.edu$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_1334164_608995900@nod45.phys.uconn.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/local/execute/dir_1334159/glide_CPBZCq/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/execute/dir_1334159/glide_CPBZCq/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/execute/dir_1334159/glide_CPBZCq/execute/dir_1444385/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
848.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda1 937231232 47439648 889791584 6% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.499ms 1: _gateway 0.544ms 2: vss-sf-voip-svrs-vl201.net.uconn.edu 0.369ms 3: mx-hbl-core-po38.net.uconn.edu 0.313ms 4: vss-core-border-inside-po51-638.net.uconn.edu 0.598ms 5: fw-msb-i-lag1.666.net.uconn.edu 8.811ms 6: vrf-border-outside-po21-664.net.uconn.edu 9.357ms 7: mx-hbl-border-outside-ae51-628.net.uconn.edu 8.769ms 8: rt1-064hhh-57b16d-be-706-634.net.cen.ct.gov 9.939ms 9: 10.172.96.168 10.211ms 10: bundle-ether2.101.core1.hart2.net.internet2.edu 13.245ms 11: fourhundredge-0-0-0-0.4079.core1.alba.net.internet2.edu 34.148ms asymm 17 12: fourhundredge-0-0-0-2.4079.core2.clev.net.internet2.edu 34.041ms asymm 16 13: fourhundredge-0-0-0-2.4079.core2.eqch.net.internet2.edu 34.617ms asymm 15 14: fourhundredge-0-0-0-10.4079.core1.star.net.internet2.edu 33.068ms 15: 144.92.233.112 34.893ms 16: perfsonar02.hep.wisc.edu 34.907ms reached Resume: pmtu 1500 hops 16 back 17
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy gryphn.phys.uconn.edu:3128
pass user
prod02 (uid 7902)
UIUC-TGI-RAILS-CE1 1 hour ago 19
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
UIUC-TGI-RAILS-CE1 — probe output
pass kernel
4.18.0-553.105.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/projects/bcxl/svcosguser/glide_YTda1d/execute/dir_2741578/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/projects/bcxl/svcosguser/glide_YTda1d/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/projects/bcxl/svcosguser/glide_YTda1d/execute/dir_2741578/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2741578 CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.4.5-2.el8 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=282e7263f1755563ecd8b0c3a7dfabafef58aaed GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/projects/bcxl/svcosguser/glide_YTda1d/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_UIUC-TGI-RAILS-CE1_gpu.idtoken.idtoken GLIDEIN_CPUS=24 GLIDEIN_ClusterId=6563332 GLIDEIN_Entry_Name=OSG_US_UIUC-TGI-RAILS-CE1_gpu GLIDEIN_Entry_Signature=996010a28cea1133ac7912d95a0edd15237ef5df GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=uiuc-tgi-rails-ce1.svc.osg-htc.org uiuc-tgi-rails-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_svcosguser_pePfDR GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UIUC-TGI-RAILS-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins3@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=UIUC-TGI-RAILS GLIDEIN_Tmp_Dir=/projects/bcxl/svcosguser/glide_YTda1d/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.4.5-2.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.4.5-2.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.4.5-2.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=uiuc-tgi-rails-ce1.svc.osg-htc.org OSG_SITE_NAME=UIUC-TGI-RAILS-CE1 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_10@glidein_2604625_48514908@rails06$ "; unset PROMPT_COMMAND PS1=slot1_10@glidein_2604625_48514908@rails06$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/projects/bcxl/svcosguser/glide_YTda1d/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/projects/bcxl/svcosguser/glide_YTda1d/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/projects/bcxl/svcosguser/glide_YTda1d/execute/dir_2741578/scratch _CONDOR_SLOT=slot1_10 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
74303.8 GB available Filesystem 1024-blocks Used Available Capacity Mounted on pool1.railsvast.internal.ncsa.edu:/projects/bcxl/svcosguser/glide_YTda1d/execute/dir_2741578 80530274304 2617088000 77913186304 4% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: tgi-rails-pub-gw.gw.ncsa.edu 0.246ms 1: tgi-rails-pub-gw.gw.ncsa.edu 0.258ms 2: npcf-core-east-tgi-rails-data.gw.ncsa.edu 0.207ms 3: npcf-exit-west-npcf-core-east.gw.ncsa.edu 0.446ms 4: 198.17.196.90 0.838ms 5: 72.36.80.50 15.913ms 6: perfsonar02.hep.wisc.edu 17.805ms reached Resume: pmtu 9000 hops 6 back 6
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
svcosguser (uid 72387)
UND-Talon-CE1 1 hour ago 33
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
UND-Talon-CE1 — probe output
pass kernel
5.14.0-427.42.1.el9_4.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/tmp/glide_yyvb5h/execute/dir_258093/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_yyvb5h/images/htc__almalinux__10.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__almalinux__10.sif APPTAINER_TMPDIR=/tmp/glide_yyvb5h/execute/dir_258093/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=258093 CONTAINERSW_FULL_VERSION=apptainer version 1.1.9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/cm/local/apps/apptainer/current/bin/apptainer CONTAINERSW_VERSION=1.1.9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_yyvb5h/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_UND-Talon-CE1.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=808780 GLIDEIN_Entry_Name=OSG_US_UND-Talon-CE1 GLIDEIN_Entry_Signature=f8da08217a35668291f2ac0d657c3613daac2519 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=und-talon-ce1.svc.opensciencegrid.org und-talon-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_vgTS7Z GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UND-Talon-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=UND-Talon GLIDEIN_Tmp_Dir=/tmp/glide_yyvb5h/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.1.9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/cm/local/apps/apptainer/current/bin/apptainer GWMS_CONTAINERSW_VERSION=1.1.9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/cm/local/apps/apptainer/current/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.1.9 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=und-talon-ce1.svc.opensciencegrid.org OSG_SITE_NAME=UND-Talon-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_155236_393663788@talon23.cm.cluster$ "; unset PROMPT_COMMAND PS1=slot1@glidein_155236_393663788@talon23.cm.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/tmp/glide_yyvb5h/images/htc__almalinux__10.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__almalinux__10.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_yyvb5h/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_yyvb5h/execute/dir_258093/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
764.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 920325060 118645288 801679772 13% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 10.128.128.1 7.784ms 1: 10.128.128.1 7.785ms 2: 172.29.10.26 8.398ms 3: 172.29.10.17 8.069ms asymm 4 4: 10.228.6.253 0.846ms asymm 5 5: 134.129.131.33 7.907ms asymm 6 6: 165.234.8.181 8.180ms 7: 134.129.0.37 8.750ms 8: 165.234.21.1 19.278ms asymm 9 9: telecomb-gr-01-hu0-5-0-3.northernlights.gigapop.net 27.525ms asymm 10 10: r-minneapolis-511-ae0-110.uwsys.net 27.087ms asymm 11 11: r-uweauclaire-centennial-ae0-3470.uwsys.net 28.159ms asymm 17 12: r-uwmadison-animal-ae3-3507.uwsys.net 30.935ms asymm 13 13: r-uwmadison-cssc-ae3-3439.uwsys.net 27.528ms asymm 14 14: 143.235.41.129 32.468ms asymm 15 15: perfsonar02.hep.wisc.edu 30.412ms reached Resume: pmtu 1500 hops 15 back 15
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 1454600096)
USC-CARC-Artemis-Backfill 1 hour ago 5
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
USC-CARC-Artemis-Backfill — probe output
pass kernel
4.18.0-372.16.1.el8_6.0.1.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/pilot/osgvo-pilot-vwDDVd/execute/dir_72700/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-vwDDVd/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-vwDDVd/execute/dir_72700/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=72700 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=Center for Advanced Research Computing OSG_SQUID_LOCATION=68.181.11.8:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_13@USC-CARC-Artemis-Backfill.79148798e6bf$ "; unset PROMPT_COMMAND PS1=slot1_13@USC-CARC-Artemis-Backfill.79148798e6bf$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/pilot/osgvo-pilot-vwDDVd/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-vwDDVd/execute/dir_72700/scratch _CONDOR_SLOT=slot1_13 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure http_proxy=68.181.11.8:3128
pass df
22.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on tmpfs 65980400 42558704 23421696 65% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: host.containers.internal 0.072ms 1: host.containers.internal 0.044ms 2: artemis-osg 0.205ms 3: 68.181.11.126 0.361ms 4: 68.181.25.18 0.398ms 5: 10.250.9.113 8.230ms 6: 207.151.53.251 8.546ms 7: hpr--lax-agg10--csuf--10g--01.cenic.net 8.800ms 8: hpr-i2--lax-agg10.cenic.net 10.921ms 9: fourhundredge-0-0-0-2.4079.core2.salt.net.internet2.edu 56.685ms asymm 17 10: fourhundredge-0-0-0-0.4079.core2.denv.net.internet2.edu 57.566ms asymm 16 11: fourhundredge-0-0-0-0.4079.core2.kans.net.internet2.edu 55.265ms asymm 15 12: fourhundredge-0-0-0-0.4079.core1.chic.net.internet2.edu 54.865ms asymm 14 13: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 56.340ms 14: 144.92.233.112 57.530ms 15: perfsonar02.hep.wisc.edu 57.532ms reached Resume: pmtu 1500 hops 15 back 16
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy 68.181.11.8:3128
fail user
uid 1003 does not map to a user
USCMS-FNAL-WC1 1 hour ago 1
0.0%
0.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
USCMS-FNAL-WC1 — probe output
pass kernel
5.14.0-687.46.1.el9_8.x86_64
pass env
APPTAINER_CACHEDIR=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch BATCH_SYSTEM=HTCondor CONDORCE_COLLECTOR_HOST=cmsosgce-opp1.fnal.gov:9619 CONDOR_CONFIG=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/condor_config CONDOR_PARENT_ID=osg-61088-0-cmswn6031:745803:1790772979 CONDOR_PROCD_ADDRESS=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/log/procd_address CONDOR_PROCD_ADDRESS_BASE=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/log/procd_address CONDOR_STARTER_PID=745803 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 FERMIHTC_APPTAINER_ARGS=--pid --ipc --contain FERMIHTC_APPTAINER_BIN=/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer FERMIHTC_APPTAINER_BIND=/cvmfs /etc/hosts /etc/grid-security FERMIHTC_APPTAINER_CMD=exec FERMIHTC_APPTAINER_PWD=/srv GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=24c135ab93f5200c32fa6d30d8360c2519aaf34c GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=935df4bf8aab44533e1f8cd09ec5c765ce400f87 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.CMSHTPC_T1_US_FNAL_condce_opp1_whole.idtoken.idtoken GLIDEIN_CPUS=128 GLIDEIN_ClusterId=7011146 GLIDEIN_Country=US GLIDEIN_DEBUG_OPTIONS= GLIDEIN_DUNESite=US_FNAL-T1 GLIDEIN_ESTIMATED_CPUS=64 GLIDEIN_Entry_Name=CMSHTPC_T1_US_FNAL_condce_opp1_whole GLIDEIN_Entry_Signature=6625622e89df893047bec8a28dcad008248346d4 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=cmsosgce-opp1.fnal.gov cmsosgce-opp1.fnal.gov:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_3lE61X GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_ProcId=2 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=USCMS-FNAL-WC1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=80cf435b7691ad316cb69740b3fef0c9d0f7c843 GLIDEIN_Site=FNAL GLIDEIN_Tmp_Dir=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/tmp GLIDEIN_UUID=ba434f15-588a-48b6-ac15-d5cb7a003c69 GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GSI_AUTHZ_CONF=/dev/null GWMS_PATH=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/.gwms.d/bin: GWMS_SINGULARITY_STATUS=PREFERRED HAS_SINGULARITY=False HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch HOSTNAME=osg-61088-0-cmswn6031.fnal.gov JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_CMSHTPC_T1_US_FNAL_condce_opp1_whole/tokens.tgz JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OLDPWD=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/logs/ba434f15-588a-48b6-ac15-d5cb7a003c69 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_APP=/cvmfs/oasis.opensciencegrid.org OSG_DATA=UNAVAILABLE OSG_GRID=/etc/osg/wn-client/ OSG_HOSTNAME=cmsosgce-opp1.fnal.gov OSG_SITE_NAME=USCMS-FNAL-WC1-OPP1 OSG_SITE_READ= OSG_SITE_WRITE= OSG_SQUID_LOCATION=squid.fnal.gov:3128 OSG_WN_TMP=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch/.local-tmp PATH=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/.gwms.d/bin::/bin:/usr/bin:/sbin:/usr/sbin PREEMPT_GRACE_TIME=10000000 PWD=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SHLVL=0 SINGULARITY_CACHEDIR=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch SLOTS_LAYOUT=partitionable TEMP=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch TMPDIR=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch/.local-tmp TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM TZ=:/etc/localtime _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_ANCESTOR_29186=29189:1790738552:1061082272 _CONDOR_ANCESTOR_29189=745803:1790772979:1256210643 _CONDOR_ANCESTOR_745803=746172:1790772985:142682556 _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/main/condor/bin _CONDOR_CHIRP_CONFIG=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/user/.chirp.config _CONDOR_CREDS=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch/.condor_creds _CONDOR_EXECUTE=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute _CONDOR_JOB_AD=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch/.job.ad _CONDOR_JOB_IWD=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/storage/local/data1/condor/execute/dir_3466062/glide_4WmNV3/execute/dir_745803/scratch/.job_wrapper_failure _JAVA_OPTIONS=-Xmx1024m
pass df
6907.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sdb1 7499377664 256320100 7243057564 4% /storage/local/data1/condor/execute/dir_3466062
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] 0.006ms pmtu 9000 1: r-cms-fcc2-2-vlan191.fnal.gov 1.652ms 1: r-cms-fcc2-2-vlan191.fnal.gov 1.226ms 2: r-border-fcc2-1-2513-1.fnal.gov 0.253ms 3: no reply 4: fnalgcc-cr6--chic-bb-e.igp.es.net 1.982ms asymm 6 5: no reply 6: uwmadison_se-203--chic-cr6.es.net 12.973ms 7: perfsonar02.hep.wisc.edu 12.905ms reached Resume: pmtu 9000 hops 7 back 7
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy squid.fnal.gov:3128
pass user
osg (uid 12740)
UTC-Epyc 1 hour ago 6
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
UTC-Epyc — probe output
pass kernel
5.14.0-687.5.4.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/mnt/internal/osg01/glide_CGEWzi/execute/dir_1989013/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/mnt/internal/osg01/glide_CGEWzi/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/mnt/internal/osg01/glide_CGEWzi/execute/dir_1989013/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1989013 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/mnt/internal/osg01/glide_CGEWzi/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=282e7263f1755563ecd8b0c3a7dfabafef58aaed GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/mnt/internal/osg01/glide_CGEWzi/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_UTC-Epyc.idtoken.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=857379 GLIDEIN_Entry_Name=OSG_US_UTC-Epyc GLIDEIN_Entry_Signature=d1f884b0757947ea3611b68b54fa80013d3aa485 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=utc-epyc-ce1.svc.opensciencegrid.org utc-epyc-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_RKTxRk GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UTC-Epyc GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins5@gfactory-2.osg-htc.org GLIDEIN_Signature=ad419b1b3db36bf5c7633f8e9c4c175eb881dd81 GLIDEIN_Site=UTC-Epyc GLIDEIN_Tmp_Dir=/mnt/internal/osg01/glide_CGEWzi/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/mnt/internal/osg01/glide_CGEWzi/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/mnt/internal/osg01/glide_CGEWzi/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=utc-epyc-ce1.svc.opensciencegrid.org OSG_SITE_NAME=UTC-Epyc PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_29@glidein_653880_555564863@epyc02$ "; unset PROMPT_COMMAND PS1=slot1_29@glidein_653880_555564863@epyc02$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/etc/OpenCL/vendors SINGULARITY_CONTAINER=/mnt/internal/osg01/glide_CGEWzi/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/mnt/internal/osg01/glide_CGEWzi/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/mnt/internal/osg01/glide_CGEWzi/execute/dir_1989013/scratch _CONDOR_SLOT=slot1_29 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
831.7 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p1 976284608 104200696 872083912 11% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.662ms 1: _gateway 0.536ms 2: 10.249.200.157 8.152ms 3: 192.30.208.30 8.257ms 4: no reply 5: no reply 6: no reply 7: perfsonar02.hep.wisc.edu 37.564ms reached Resume: pmtu 1500 hops 7 back 17
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 300001)
UWEC-BOSE-CE1 1 hour ago 4
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
UWEC-BOSE-CE1 — probe output
pass kernel
4.18.0-348.el8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/local/scratch/job_407374/glide_JYEKkZ/execute/dir_1819328/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/local/scratch/job_407374/glide_JYEKkZ/images/htc__debian__13.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__debian__13.sif APPTAINER_TMPDIR=/local/scratch/job_407374/glide_JYEKkZ/execute/dir_1819328/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=1819328 CONTAINERSW_FULL_VERSION=apptainer version 1.2.5 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/data/software/default/apptainer/1.2.5/bin/apptainer CONTAINERSW_VERSION=1.2.5 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/local/scratch/job_407374/glide_JYEKkZ/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_UWEC-BOSE-CE1.idtoken.idtoken GLIDEIN_CONTAINER_ENV=keepall GLIDEIN_CPUS=16 GLIDEIN_ClusterId=6352164 GLIDEIN_Entry_Name=OSG_US_UWEC-BOSE-CE1 GLIDEIN_Entry_Signature=d6f7f1ec03737541bcb45a359c4c51afc493f5bb GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=uwec-bose-ce1.svc.osg-htc.org uwec-bose-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_CYto1P GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UWEC-BOSE-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=UWEC-BOSE GLIDEIN_Tmp_Dir=/local/scratch/job_407374/glide_JYEKkZ/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.2.5 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/data/software/default/apptainer/1.2.5/bin/apptainer GWMS_CONTAINERSW_VERSION=1.2.5 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/data/software/default/apptainer/1.2.5/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.2.5 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=uwec-bose-ce1.svc.osg-htc.org OSG_SITE_NAME=UWEC-BOSE-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_7@glidein_1768573_15609039@cn12.head.bose.hpc.cluster$ "; unset PROMPT_COMMAND PS1=slot1_7@glidein_1768573_15609039@cn12.head.bose.hpc.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/local/scratch/job_407374/glide_JYEKkZ/images/htc__debian__13.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__debian__13.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/local/scratch/job_407374/glide_JYEKkZ/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/local/scratch/job_407374/glide_JYEKkZ/execute/dir_1819328/scratch _CONDOR_SLOT=slot1_7 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
770.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda8 837052400 29615624 807436776 4% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: 172.23.255.254 7.945ms 1: 172.23.255.254 7.750ms 2: r-uweauclaire-cvtc-ae2-0.uwsys.net 26.198ms 3: r-uweauclaire-centennial-et-0-1-9-3503.uwsys.net 26.542ms asymm 8 4: r-uwmadison-animal-ae3-3507.uwsys.net 14.758ms 5: r-uwmadison-cssc-ae3-3439.uwsys.net 31.396ms 6: 143.235.41.129 29.813ms 7: perfsonar02.hep.wisc.edu 29.617ms reached Resume: pmtu 1500 hops 7 back 6
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 2011)
UWL-Test-EP 1 hour ago 18
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
UWL-Test-EP — probe output
pass kernel
5.14.0-687.42.1.el9_8.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/pilot/osgvo-pilot-WkSlzh/execute/dir_119646/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/pilot/osgvo-pilot-WkSlzh/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/pilot/osgvo-pilot-WkSlzh/execute/dir_119646/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=119646 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-1.el9 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_REQUIRED_OS=any GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-1.el9 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-1.el9 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-1.el9 HAS_SINGULARITY=True HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_SITE_NAME=UW-La Crosse CSCPE PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROMPT_COMMAND=PS1="slot1_1@UWL-Test-EP.d5adab584e23$ "; unset PROMPT_COMMAND PS1=slot1_1@UWL-Test-EP.d5adab584e23$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/pilot/osgvo-pilot-WkSlzh/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/usr/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/pilot/osgvo-pilot-WkSlzh/execute/dir_119646/scratch _CONDOR_SLOT=slot1_1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
109.4 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 235337452 120642756 114694696 52% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: _gateway 0.083ms 1: _gateway 0.124ms 2: 138.49.30.1 0.762ms asymm 3 3: 10.159.96.1 1.121ms 4: 138.49.0.2 0.791ms asymm 5 5: 138.49.0.1 0.724ms 6: r-uwmadison-animal-et-0-1-4-3504.uwsys.net 3.599ms 7: r-uwmadison-animal-et-0-1-4-3504.uwsys.net 3.855ms asymm 6 8: r-uwmadison-cssc-ae3-3439.uwsys.net 3.553ms asymm 7 9: 143.235.41.129 3.512ms asymm 8 10: perfsonar02.hep.wisc.edu 3.552ms reached Resume: pmtu 1500 hops 10 back 8
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
fail user
uid 1000 does not map to a user
UWM-Mortimer-CE1 1 hour ago 30
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
UWM-Mortimer-CE1 — probe output
pass kernel
4.18.0-553.143.1.el8_10.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/tmp/glide_ndI0XC/execute/dir_101874/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/tmp/glide_ndI0XC/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/tmp/glide_ndI0XC/execute/dir_101874/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=101874 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=privileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/tmp/glide_ndI0XC/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_UWM-Mortimer-CE1.idtoken.idtoken GLIDEIN_CPUS=4 GLIDEIN_ClusterId=7088208 GLIDEIN_Entry_Name=OSG_US_UWM-Mortimer-CE1 GLIDEIN_Entry_Signature=e0ccd0fa5592eb35a1ec8677815ff5cdc26b6d02 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=uwm-mortimer-ce1.svc.osg-htc.org uwm-mortimer-ce1.svc.osg-htc.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_ehlCEs GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/privileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=uwm-proxy.uwm.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=UWM-Mortimer-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=88d76433adc669450d89762935ce2ce292bb77fe GLIDEIN_Site=UWM-Mortimer GLIDEIN_Tmp_Dir=/tmp/glide_ndI0XC/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=privileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=privileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=uwm-mortimer-ce1.svc.osg-htc.org OSG_SITE_NAME=UWM-Mortimer-CE1 OSG_SQUID_LOCATION=uwm-proxy.uwm.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_3@glidein_3494801_110035203@execute-139.mortimer.hpc.uwm.edu$ "; unset PROMPT_COMMAND PS1=slot1_3@glidein_3494801_110035203@execute-139.mortimer.hpc.uwm.edu$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CONTAINER=/tmp/glide_ndI0XC/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/tmp/glide_ndI0XC/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/tmp/glide_ndI0XC/execute/dir_101874/scratch _CONDOR_SLOT=slot1_3 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
318.6 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/VolGroup-tmp 366991360 32904840 334086520 9% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=9000 1?: [LOCALHOST] pmtu 9000 1: _gateway 44.212ms 1: _gateway 44.300ms 2: no reply 3: r-uwmilwaukee-ems-ae9-160.uwsys.net 51.942ms 4: r-uwmadison-cssc-ae4-3455.uwsys.net 4.919ms 5: 143.235.41.129 47.216ms 6: perfsonar02.hep.wisc.edu 46.900ms reached Resume: pmtu 9000 hops 6 back 7
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy uwm-proxy.uwm.edu:3128
pass user
osg01 (uid 2606)
VU-AUGIE-CE1 1 hour ago 6
83.3%
83.3%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
VU-AUGIE-CE1 — probe output
pass kernel
5.15.0-57-generic
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/scratch/glide_w60vwf/execute/dir_2238390/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/scratch/glide_w60vwf/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/scratch/glide_w60vwf/execute/dir_2238390/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2238390 CONTAINERSW_FULL_VERSION=apptainer version 1.1.8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.1.8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/scratch/glide_w60vwf/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.VU-AUGIE-CE1.idtoken.idtoken GLIDEIN_CPUS=64 GLIDEIN_ClusterId=7037877 GLIDEIN_Entry_Name=VU-AUGIE-CE1 GLIDEIN_Entry_Signature=faff2cc799a6279ea87afcc8e3e28aa40dece7c5 GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_Gatekeeper=vu-augie-ce1.svc.opensciencegrid.org vu-augie-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_sT03R2 GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=augie.villanova.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=VU-AUGIE-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=gfactory-1.osg-htc.org GLIDEIN_Signature=e977401dd4ee29c9252d463c389cf2702ca41418 GLIDEIN_Site=VU-AUGIE GLIDEIN_Tmp_Dir=/scratch/glide_w60vwf/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.1.8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.1.8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.1.8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=vu-augie-ce1.svc.opensciencegrid.org OSG_SITE_NAME=VU-AUGIE-CE1 OSG_SQUID_LOCATION=augie.villanova.edu:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_15@glidein_1467444_10342325@n005.cluster.com$ "; unset PROMPT_COMMAND PS1=slot1_15@glidein_1467444_10342325@n005.cluster.com$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/scratch/glide_w60vwf/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=partitionable TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/scratch/glide_w60vwf/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/scratch/glide_w60vwf/execute/dir_2238390/scratch _CONDOR_SLOT=slot1_15 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
427.0 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/nvme0n1p4 760688044 312969344 447718700 42% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: clusterhn.cluster.pssclabs.com 0.221ms 1: clusterhn.cluster.pssclabs.com 0.275ms 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy augie.villanova.edu:3128
pass user
osg01 (uid 1052)
WSU-GRID-CE1 1 hour ago 10
100.0%
100.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
WSU-GRID-CE1 — probe output
pass kernel
5.15.80-200.el7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/wsu/tmp/glide_waKpdo/images/htc__rocky__9-cuda-12.6.0.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9-cuda-12.6.0.sif APPTAINER_TMPDIR=/wsu/tmp/glide_waKpdo/execute/dir_13554 BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=apptainer version 1.3.2-1.el7 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/usr/bin/apptainer CONTAINERSW_VERSION=1.3.2-1.el7 CUBACORES=1 CUDA_VERSION=12.6.0 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=gpu GLIDECLIENT_Group_Signature=282e7263f1755563ecd8b0c3a7dfabafef58aaed GLIDECLIENT_Name=OSPool.gpu GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/wsu/tmp/glide_waKpdo/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_WSU-GRID_gpu.idtoken.idtoken GLIDEIN_CPUS=8 GLIDEIN_ClusterId=7096290 GLIDEIN_Entry_Name=OSG_US_WSU-GRID_gpu GLIDEIN_Entry_Signature=6433e04b5891f9cb722b985fd14bc2c55c4ac3ba GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=Tiger-Prod GLIDEIN_GPUS=1 GLIDEIN_Gatekeeper=wsu-grid-ce1.svc.opensciencegrid.org wsu-grid-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_k7NNUL GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=vsquid1.grid.wayne.edu:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=WSU-GRID-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins4@gfactory-1.osg-htc.org GLIDEIN_Signature=34a4e608cf5515b002eae52b57e389e227043365 GLIDEIN_Site=GRID_ce2 GLIDEIN_Tmp_Dir=/wsu/tmp/glide_waKpdo/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.3.2-1.el7 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/usr/bin/apptainer GWMS_CONTAINERSW_VERSION=1.3.2-1.el7 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/usr/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.3.2-1.el7 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs LIBRARY_PATH=/usr/local/cuda/lib64/stubs MKL_NUM_THREADS=1 NCCL_VERSION=2.22.3 NUMEXPR_NUM_THREADS=1 NVARCH=x86_64 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 NVIDIA_VISIBLE_DEVICES=none NV_CUDA_CUDART_DEV_VERSION=12.6.37-1 NV_CUDA_CUDART_VERSION=12.6.37-1 NV_CUDA_LIB_VERSION=12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6-12.6.0-1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.0-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.3.0.75-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.3.0.75-1 NV_CUDNN_VERSION=9.3.0.75-1 NV_LIBCUBLAS_DEV_VERSION=12.6.0.22-1 NV_LIBCUBLAS_VERSION=12.6.0.22-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.22.3-1+cuda12.6 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_PACKAGE=libnccl-2.22.3-1+cuda12.6 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.22.3-1 NV_LIBNCCL_VERSION=2.22.3 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-6-12.3.1.23-1 NV_LIBNPP_DEV_VERSION=12.3.1.23-1 NV_LIBNPP_PACKAGE=libnpp-12-6-12.3.1.23-1 NV_LIBNPP_VERSION=12.3.1.23-1 NV_NVML_DEV_VERSION=12.6.37-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6-12.6.37-1 NV_NVPROF_VERSION=12.6.37-1 NV_NVTX_VERSION=12.6.37-1 OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=wsu-grid-ce1.svc.opensciencegrid.org OSG_SITE_NAME=WSU-GRID-CE1 OSG_SQUID_LOCATION=vsquid1.grid.wayne.edu:3128 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1_2@glidein_53190_506179324@alh3$ "; unset PROMPT_COMMAND PS1=slot1_2@glidein_53190_506179324@alh3$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs,/etc/OpenCL/vendors SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/wsu/tmp/glide_waKpdo/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.3.2-1.el7 SINGULARITY_IMAGES_DICT=default:/wsu/tmp/glide_waKpdo/images/htc__rocky__9-cuda-12.6.0.sif SINGULARITY_MODE=unprivileged SINGULARITY_NAME=htc__rocky__9-cuda-12.6.0.sif SINGULARITY_PATH=/usr/bin/apptainer SINGULARITY_VERSION=1.3.2-1.el7 SLOTS_LAYOUT=partitionable TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/wsu/tmp/glide_waKpdo/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/wsu/tmp/glide_waKpdo/execute/dir_13554 _CONDOR_SLOT=slot1_2 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass df
46.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on panfs://10.3.140.2/wsu 48828112 64864 48763248 1% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: gwha1.nas.grid.wayne.edu 0.197ms 1: gwha1.nas.grid.wayne.edu 0.211ms 2: 141.217.136.1 0.408ms 3: 193-cr-c6509e-1-r515-fo-8-1-v98.core.infra 0.529ms 4: 193-br-dcs-7280sr3k-1-po-1.core.infra 1.017ms 5: ae1x160.dtrt-wsudc-c1.mich.net 8.201ms 6: et-3-1-0x3.gdrp-eberhard-c1.mich.net 11.697ms 7: no reply 8: ae102x0.chcg-lumen-600w-c1.mich.net 14.943ms asymm 9 9: 192.122.182.34 15.847ms asymm 10 10: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 17.338ms 11: 144.92.233.112 18.937ms 12: perfsonar02.hep.wisc.edu 18.877ms reached Resume: pmtu 1500 hops 12 back 13
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy vsquid1.grid.wayne.edu:3128
pass user
osg01 (uid 9215)
WY-MedicineBow-CE1 1 hour ago 49
100.0%
100.0%
0.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
WY-MedicineBow-CE1 — probe output
pass kernel
5.14.0-611.55.1.el9_7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime APPTAINER_CACHEDIR=/project/openscigrid/glide_VCEBwE/execute/dir_2074463/scratch APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/project/openscigrid/glide_VCEBwE/images/htc__rocky__9.sif APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=htc__rocky__9.sif APPTAINER_TMPDIR=/project/openscigrid/glide_VCEBwE/execute/dir_2074463/scratch BATCH_SYSTEM=HTCondor CONDOR_STARTER_PID=2074463 CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/project/openscigrid/glide_VCEBwE/main/condor/usr/libexec/apptainer CONTAINERSW_VERSION=1.5.3-3.el8 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=f4b310e1d65321e1e03b0fa2d39388be9f25de98 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=b544a8f15e488de0ed35698ed909d41164ee245e GLIDEIN_BLACKHOLE=False GLIDEIN_CONDOR_TOKEN=/project/openscigrid/glide_VCEBwE/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.OSG_US_WY-MedicineBow-CE1.idtoken.idtoken GLIDEIN_CPUS=1 GLIDEIN_ClusterId=808566 GLIDEIN_Entry_Name=OSG_US_WY-MedicineBow-CE1 GLIDEIN_Entry_Signature=0fc2b03c2c8969e121f1119f2557fd4891c9796b GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=wy-medicinebow-ce1.svc.opensciencegrid.org wy-medicinebow-ce1.svc.opensciencegrid.org:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg01_yFi7Yc GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=None GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=WY-MedicineBow-CE1 GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=schedd_glideins2@gfactory-2.osg-htc.org GLIDEIN_Signature=1bf7b64272f3246f030cf0a613f6225f1afbedb2 GLIDEIN_Site=WY-MedicineBow GLIDEIN_Tmp_Dir=/project/openscigrid/glide_VCEBwE/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.5.3-3.el8 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/project/openscigrid/glide_VCEBwE/main/condor/usr/libexec/apptainer GWMS_CONTAINERSW_VERSION=1.5.3-3.el8 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/project/openscigrid/glide_VCEBwE/main/condor/usr/libexec/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.5.3-3.el8 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=wy-medicinebow-ce1.svc.opensciencegrid.org OSG_SITE_NAME=WY-MedicineBow-CE1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_749001_481961120@mbcpu-001.cluster$ "; unset PROMPT_COMMAND PS1=slot1@glidein_749001_481961120@mbcpu-001.cluster$ PWD=/srv/scratch PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime SINGULARITY_CONTAINER=/project/openscigrid/glide_VCEBwE/images/htc__rocky__9.sif SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_NAME=htc__rocky__9.sif SLOTS_LAYOUT=fixed TEMP=/srv/scratch TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv/scratch TMPDIR=/srv/scratch TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/project/openscigrid/glide_VCEBwE/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/user/.chirp.config _CONDOR_CREDS=/srv/scratch/.condor_creds _CONDOR_JOB_AD=/srv/scratch/.job.ad _CONDOR_JOB_IWD=/srv/scratch _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/scratch/.machine.ad _CONDOR_SCRATCH_DIR=/srv/scratch _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/project/openscigrid/glide_VCEBwE/execute/dir_2074463/scratch _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/scratch/.job_wrapper_failure
pass df
4946.5 GB available Filesystem 1024-blocks Used Available Capacity Mounted on vast-eth:/cluster/medbow/project/openscigrid/glide_VCEBwE/execute/dir_2074463 5368709120 181885952 5186823168 4% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: no reply 2: no reply 3: no reply 4: no reply 5: no reply 6: no reply 7: no reply 8: no reply 9: no reply 10: no reply 11: no reply 12: no reply 13: no reply 14: no reply 15: no reply 16: no reply 17: no reply 18: no reply 19: no reply 20: no reply 21: no reply 22: no reply 23: no reply 24: no reply 25: no reply 26: no reply 27: no reply 28: no reply 29: no reply 30: no reply Too many hops: pmtu 1500 Resume: pmtu 1500
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
not_available http_proxy
Neither http_proxy nor OSG_SQUID_LOCATION is set
pass user
osg01 (uid 5720062)
uprm-cms 6 hours ago 2
100.0%
0.0%
100.0%
100.0%
kernel env df firewall mtu nrp_images http_proxy user
uprm-cms — probe output
warning kernel
3.10.0-1160.105.1.el7.x86_64
pass env
APPTAINER_APPNAME= APPTAINER_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs APPTAINER_CACHEDIR=/srv APPTAINER_COMMAND=exec APPTAINER_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh APPTAINER_NAME=rocky:9 APPTAINER_TMPDIR=/var/lib/condor/execute/dir_13855/glide_1fxjON/execute/dir_31597 BATCH_SYSTEM=HTCondor CONTAINERSW_FULL_VERSION=apptainer version 1.2.5-1.el7 CONTAINERSW_MODE=unprivileged CONTAINERSW_PATH=/bin/apptainer CONTAINERSW_VERSION=1.2.5-1.el7 CUBACORES=1 CUDA_VISIBLE_DEVICES=10000 GLIDECLIENT_Group=main GLIDECLIENT_Group_Signature=a0978184afcb5a34036c1e257052e985513b3323 GLIDECLIENT_Name=OSPool.main GLIDECLIENT_OSG_VO=OSG GLIDECLIENT_Signature=9fe9221a11a0eaafae7519290ef0e01e341a5e42 GLIDEIN_BLACKHOLE=False GLIDEIN_CMSSite=T3_US_PuertoRico GLIDEIN_CONDOR_TOKEN=/var/lib/condor/execute/dir_13855/glide_1fxjON/cred.d/idtokens/credential_callback_IdTokenGenerator_63XUYJG2.CMS_T3_US_PuertoRico_UPRM.idtoken.idtoken GLIDEIN_ClusterId=684628 GLIDEIN_Country=US GLIDEIN_DUNESite=US_PuertoRico GLIDEIN_Entry_Name=CMS_T3_US_PuertoRico_UPRM GLIDEIN_Entry_Signature=1651a8761a89e3965713f7dde9a5937346e930aa GLIDEIN_Expose_Grid_Env=True GLIDEIN_Factory=NRP-Prod GLIDEIN_Gatekeeper=cms-grid0.hep.uprm.edu cms-grid0.hep.uprm.edu:9619 GLIDEIN_Glexec_Use=NEVER GLIDEIN_GridType=condor GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_osg_og6xAG GLIDEIN_Monitoring_Enabled=False GLIDEIN_NODES=1 GLIDEIN_Name=gfactory_instance GLIDEIN_OVERLOAD_ENABLED=False GLIDEIN_PROVIDES=singularity/unprivileged GLIDEIN_ProcId=0 GLIDEIN_Proxy_URL=squid-0-0-0.local:3128 GLIDEIN_REQUIRED_OS=any GLIDEIN_REQUIRE_VOMS=False GLIDEIN_ResourceName=uprm-cms GLIDEIN_SEs=cms-se.hep.uprm.edu GLIDEIN_SINGULARITY_BINDPATH=/alphafold3,/mnt/alphafold,/mnt/alphafold/alphafold3_7a4a2f,/scitech/nas/scratch-90-days/alphafold3 GLIDEIN_Schedd=gfactory-2.osg-htc.org GLIDEIN_Signature=f90b138f62b42316e2088dea04273e4626194e65 GLIDEIN_Site=PuertoRico GLIDEIN_Tmp_Dir=/var/lib/condor/execute/dir_13855/glide_1fxjON/tmp GOMAXPROCS=1 GPU_DEVICE_ORDINAL=10000 GWMS_CONTAINERSW_FULL_VERSION=apptainer version 1.2.5-1.el7 GWMS_CONTAINERSW_MODE=unprivileged GWMS_CONTAINERSW_PATH=/bin/apptainer GWMS_CONTAINERSW_VERSION=1.2.5-1.el7 GWMS_SINGULARITY_MODE=unprivileged GWMS_SINGULARITY_PATH=/bin/apptainer GWMS_SINGULARITY_STATUS=PREFERRED GWMS_SINGULARITY_VERSION=1.2.5-1.el7 HAS_SINGULARITY=True HAS_UNPRIVILEGED_USER_NAMESPACES=enabled HOLD_GRACE_TIME=0 HOME=/srv JULIA_NUM_THREADS=1 LANG=C LC_CTYPE=C.UTF-8 LD_LIBRARY_PATH=/.singularity.d/libs MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 NVIDIA_VISIBLE_DEVICES=none OMP_NUM_THREADS=1 OMP_THREAD_LIMIT=1 OPENBLAS_NUM_THREADS=1 OSG_HOSTNAME=cms-grid0.hep.uprm.edu OSG_SITE_NAME=uprm-cms-ce OSG_SQUID_LOCATION=squid-0-0-0.local:3128 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PREEMPT_GRACE_TIME=10000000 PROMPT_COMMAND=PS1="slot1@glidein_13858_112456470@compute-2-5.local$ "; unset PROMPT_COMMAND PS1=slot1@glidein_13858_112456470@compute-2-5.local$ PWD=/srv PYTHON_CPU_COUNT=1 ROOT_MAX_THREADS=1 SINGULARITY_BIND=/srv,/etc/hosts,/etc/localtime,/cvmfs SINGULARITY_CACHEDIR=/srv SINGULARITY_CONTAINER=/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh SINGULARITY_FULL_VERSION=apptainer version 1.2.5-1.el7 SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/htc/rocky:9 SINGULARITY_MODE=unprivileged SINGULARITY_NAME=rocky:9 SINGULARITY_PATH=/bin/apptainer SINGULARITY_VERSION=1.2.5-1.el7 SLOTS_LAYOUT=fixed TEMP=/srv TF_LOOP_PARALLEL_ITERATIONS=1 TF_NUM_THREADS=1 TMP=/srv TMPDIR=/srv TRUST_DOMAIN=cm-1.ospool.osg-htc.org:9618?sock=collectorRANDOM _CHIRP_DELAYED_UPDATE_PREFIX=Chirp* _CONDOR_AssignedGPUs=10000 _CONDOR_BIN=/var/lib/condor/execute/dir_13855/glide_1fxjON/main/condor/bin _CONDOR_CHIRP_CONFIG=/srv/.chirp.config _CONDOR_CREDS=/srv/.condor_creds _CONDOR_JOB_AD=/srv/.job.ad _CONDOR_JOB_IWD=/srv _CONDOR_JOB_PIDS= _CONDOR_MACHINE_AD=/srv/.machine.ad _CONDOR_SCRATCH_DIR=/srv _CONDOR_SCRATCH_DIR_OUTSIDE_CONTAINER=/var/lib/condor/execute/dir_13855/glide_1fxjON/execute/dir_31597 _CONDOR_SLOT=slot1 _CONDOR_WRAPPER_ERROR_FILE=/srv/.job_wrapper_failure
pass df
157.3 GB available Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sda3 206292968 30805312 164985512 16% /srv
pass firewall
All ports reachable on ospool-network-test.osg.chtc.io http://ospool-network-test.osg.chtc.io:80: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:1094: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:2880: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8000: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:8443: OK (HTTP 200) http://ospool-network-test.osg.chtc.io:27017: OK (HTTP 200)
pass mtu
pmtu=1500 1?: [LOCALHOST] pmtu 1500 1: cms-hn.local 0.291ms 1: cms-hn.local 0.373ms 2: 136.145.77.1 0.590ms 3: 136.145.230.114 0.540ms 4: R8500.uprm.edu 0.608ms 5: hub787-rum-a.upr.edu 5.123ms 6: et-0-0-1-819.rt04.be.ampath.net 35.231ms 7: et-0-0-1-67.rt05.bb.ampath.net 35.376ms 8: hundredge-0-0-0-26.1804.core1.jack.net.internet2.edu 44.319ms 9: fourhundredge-0-0-0-6.4079.core1.atla.net.internet2.edu 67.286ms asymm 21 10: fourhundredge-0-0-0-23.4079.core2.atla.net.internet2.edu 67.733ms asymm 20 11: fourhundredge-0-0-0-4.4079.core2.nash.net.internet2.edu 68.066ms asymm 19 12: fourhundredge-0-0-0-1.4079.core1.cinc.net.internet2.edu 67.814ms asymm 18 13: fourhundredge-0-0-0-22.4079.core2.cinc.net.internet2.edu 67.300ms asymm 17 14: fourhundredge-0-0-0-4.4079.core1.chic.net.internet2.edu 66.012ms asymm 16 15: fourhundredge-0-0-0-4.4079.core1.star.net.internet2.edu 67.063ms 16: 144.92.233.112 69.642ms 17: perfsonar02.hep.wisc.edu 69.415ms reached Resume: pmtu 1500 hops 17 back 18
pass nrp_images
Downloaded https://ospool-images-web.nrp-nautilus.io/v2/x86_64/htc__rocky__9/latest.txt successfully 20260911T164447Z.sif
pass http_proxy
Downloaded http://ospool-network-test.osg.chtc.io/index.html successfully via proxy squid-0-0-0.local:3128
pass user
osg (uid 5458)