==== wrapper started @ Sun Aug 16 18:50:29 EDT 2020 ==== == startup: hostname: qa-v100-001.crc.nd.edu == startup: kernel: Linux qa-v100-001.crc.nd.edu 3.10.0-1062.el7.x86_64 #1 SMP Thu Jul 18 20:25:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux ==== tracing google @ Sun Aug 16 18:50:34 EDT 2020 ==== wrapper.sh: line 20: traceroute: command not found ==== environment at startup @ Sun Aug 16 18:50:39 EDT 2020 ==== == env: VC3_VERSION_SED=4.2 == env: CCTOOLS_RESOURCE_PROCESS_START=1597618239655545 == env: VC3_INSIDE_WRAPPER_SCRIPT=yes == env: VC3_ROOT_GZIP=/usr/ == env: SHELL=/bin/sh == env: TERM=xterm-256color == env: MAKEFLAGS=-j4 == env: TMPDIR=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: VC3_ROOT_WHICH=/disk/vc3-root/x86_64/redhat7.4.1708/which/v2.20.0 == env: LOBSTER_CVMFS_PROXY=http://eddie.crc.nd.edu:3128 == env: LD_PRELOAD=/disk/vc3-root/tmp/worker-230547-7899/t.43/librmonitor_helper.so.3N9QXP == env: LOBSTER_FRONTIER_PROXY=http://eddie.crc.nd.edu:3128 == env: LOBSTER_CMSSW_VERSION=CMSSW_9_3_6 == env: VC3_VERSION_FULL_TAR=1.26.0 == env: VC3_MACHINE_OS=Linux/redhat7.4.1708 == env: VC3_VERSION_FULL_CCTOOLS_WQ_WORKER=999.999.999 == env: USER=kmohrman == env: TEMP=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: VC3_BUILDER_PATH=/disk/vc3-builder == env: LD_LIBRARY_PATH=/.singularity.d/libs == env: VC3_VERSION_VC3_BUILDER_HANDS=1.0 == env: VC3_ROOT_MAKE=/usr/ == env: VC3_DISTFILES=/disk/vc3-distfiles == env: WORKER_TMPDIR=/disk/vc3-root/tmp/worker-230547-7899/cache/tmp == env: VC3_VERSION_TAR=1.26 == env: ENV=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX == env: VC3_VERSION_CCTOOLS_WQ_WORKER=999.999 == env: VC3_VERSION_FULL_PKG_CONFIG=0.27.1 == env: VC3_VERSION_GZIP=1.5 == env: VC3_ENV_TAG_GLOBAL=7645 == env: PATH=/disk/vc3-root/x86_64/redhat7.4.1708/which/v2.20.0/bin:/disk/vc3-root/x86_64/redhat7.4.1708/cctools-wq-worker/v999.999.999/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin == env: VC3_ROOT=/disk/vc3-root == env: VC3_SH_PROFILE_ENV=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX == env: VC3_ROOT_PKG_CONFIG=/usr/ == env: PWD=/disk/vc3-root/tmp/worker-230547-7899/t.43 == env: LOBSTER_OSG_VERSION=3.4 == env: VC3_VERSION_FULL_SED=4.2.2 == env: VC3_VERSION_WHICH=2.20 == env: VC3_MACHINE_GLIBC_VERSION=2.17 == env: MODULEPATH=/afs/crc.nd.edu/x86_64_linux/Modules/modules/development_tools_and_libraries:/afs/crc.nd.edu/x86_64_linux/Modules/modules/general_software:/afs/crc.nd.edu/x86_64_linux/Modules/modules/restricted_software:/afs/crc.nd.edu/x86_64_linux/Modules/modules/system_modules:/afs/crc.nd.edu/x86_64_linux/Modules/modules/deprecated_software:/afs/crc.nd.edu/x86_64_linux/Modules/current/modulefiles == env: WORK_QUEUE_SANDBOX=/disk/vc3-root/tmp/worker-230547-7899 == env: VC3_ROOT_CCTOOLS_WQ_WORKER=/disk/vc3-root/x86_64/redhat7.4.1708/cctools-wq-worker/v999.999.999 == env: VC3_ROOT_SED=/usr/ == env: VC3_MACHINE_TARGET=x86_64/redhat7.4.1708 == env: CXX=g++ == env: SHLVL=3 == env: HOME=/disk/vc3-root/vc3-home == env: VC3_VERSION_MAKE=3.82 == env: VC3_VERSION_FULL_GZIP=1.5.0 == env: DISK=1000 == env: MEMORY=1200 == env: VC3_VERSION_FULL_WHICH=2.20.0 == env: TMP=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: VC3_INSTALL_USER_HOME=/disk/vc3-root/vc3-home == env: VC3_VERSION_FULL_VC3_BUILDER_HANDS=1.0.0 == env: VC3_VERSION_PKG_CONFIG=0.27 == env: VC3_ROOT_TAR=/usr/ == env: VC3_SH_PROFILE_WRAPPER=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX.wrapper == env: CCTOOLS_ND=lobster-171-cd5e3e2c-cvmfs-70dfa0d6 == env: CC=gcc == env: CORES=1 == env: VC3_VERSION_FULL_MAKE=3.82.0 == env: CCTOOLS_RESOURCE_MONITOR_INFO=1033 == env: VC3_ROOT_VC3_BUILDER_HANDS=/disk/vc3-root/x86_64/redhat7.4.1708/vc3-builder-hands/v1.0.0 == env: VC3_MACHINE_ARCH=x86_64 == env: VC3_COMMAND_LINE=/disk/vc3-builder --var CCTOOLS_ND=lobster-171-cd5e3e2c-cvmfs-70dfa0d6 --require cctools-wq-worker -- work_queue_worker -M lobster_kmohrman_EFT_ALL_genOnly.* -t 300 -C catalog.cse.nd.edu,backup-catalog.cse.nd.edu -d all -o worker.log --cores=12 --memory=16000 --disk=17780 == env: _=/usr/bin/env ==== cpu info @ Sun Aug 16 18:50:44 EDT 2020 ==== == cpu: processor : 0 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 2904.418 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 0 == cpu: cpu cores : 8 == cpu: apicid : 0 == cpu: initial apicid : 0 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 1 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 800.061 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 1 == cpu: cpu cores : 8 == cpu: apicid : 2 == cpu: initial apicid : 2 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 2 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 2 == cpu: cpu cores : 8 == cpu: apicid : 4 == cpu: initial apicid : 4 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 3 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 3 == cpu: cpu cores : 8 == cpu: apicid : 6 == cpu: initial apicid : 6 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 4 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 4 == cpu: cpu cores : 8 == cpu: apicid : 8 == cpu: initial apicid : 8 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 5 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 5 == cpu: cpu cores : 8 == cpu: apicid : 10 == cpu: initial apicid : 10 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 6 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 6 == cpu: cpu cores : 8 == cpu: apicid : 12 == cpu: initial apicid : 12 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 7 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 800.061 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 7 == cpu: cpu cores : 8 == cpu: apicid : 14 == cpu: initial apicid : 14 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 8 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 2987.731 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 0 == cpu: cpu cores : 8 == cpu: apicid : 16 == cpu: initial apicid : 16 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 9 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 1 == cpu: cpu cores : 8 == cpu: apicid : 18 == cpu: initial apicid : 18 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 10 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 2 == cpu: cpu cores : 8 == cpu: apicid : 20 == cpu: initial apicid : 20 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 11 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 3 == cpu: cpu cores : 8 == cpu: apicid : 22 == cpu: initial apicid : 22 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 12 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 800.061 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 4 == cpu: cpu cores : 8 == cpu: apicid : 24 == cpu: initial apicid : 24 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 13 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 5 == cpu: cpu cores : 8 == cpu: apicid : 26 == cpu: initial apicid : 26 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 14 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 6 == cpu: cpu cores : 8 == cpu: apicid : 28 == cpu: initial apicid : 28 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 15 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 7 == cpu: cpu cores : 8 == cpu: apicid : 30 == cpu: initial apicid : 30 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: === using CVMFS proxy: http://eddie.crc.nd.edu:3128 @ Sun Aug 16 18:51:09 EDT 2020 === using Frontier proxy: http://eddie.crc.nd.edu:3128 @ Sun Aug 16 18:51:09 EDT 2020 === parrot helper: /disk/vc3-root/tmp/worker-230547-7899/t.43/lib/libparrot_helper.so @ Sun Aug 16 18:51:14 EDT 2020 ==== content of /disk/vc3-root/tmp/worker-230547-7899/cache/tmp @ Sun Aug 16 18:51:15 EDT 2020 ==== == cache: total 0 === OSG certificate location: /cvmfs/oasis.opensciencegrid.org/mis/certificates @ Sun Aug 16 18:51:20 EDT 2020 === testing parrot usage @ Sun Aug 16 18:51:20 EDT 2020 === parrot OK @ Sun Aug 16 18:51:25 EDT 2020 === starting parrot to access CMSSW... @ Sun Aug 16 18:51:25 EDT 2020 ==== wrapper started @ Sun Aug 16 18:51:25 EDT 2020 ==== == startup: hostname: qa-v100-001.crc.nd.edu == startup: kernel: Linux qa-v100-001.crc.nd.edu 3.10.0-1062.el7.x86_64 #1 SMP Thu Jul 18 20:25:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux ==== tracing google @ Sun Aug 16 18:51:25 EDT 2020 ==== wrapper.sh: line 20: traceroute: command not found ==== environment at startup @ Sun Aug 16 18:51:25 EDT 2020 ==== == env: VC3_VERSION_SED=4.2 == env: CCTOOLS_RESOURCE_PROCESS_START=1597618285472205 == env: VC3_INSIDE_WRAPPER_SCRIPT=yes == env: PARROT_HTTP_PROXY=http://eddie.crc.nd.edu:3128 == env: TERM=xterm-256color == env: SHELL=/bin/sh == env: VC3_ROOT_GZIP=/usr/ == env: LOBSTER_GFAL_COPY= == env: MAKEFLAGS=-j4 == env: PARROT_HELPER=/disk/vc3-root/tmp/worker-230547-7899/t.43/lib/libparrot_helper.so == env: TMPDIR=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: VC3_ROOT_WHICH=/disk/vc3-root/x86_64/redhat7.4.1708/which/v2.20.0 == env: LD_PRELOAD=/disk/vc3-root/tmp/worker-230547-7899/t.43/librmonitor_helper.so.3N9QXP == env: LOBSTER_CVMFS_PROXY=http://eddie.crc.nd.edu:3128 == env: PARROT_PATH=./bin == env: LOBSTER_FRONTIER_PROXY=http://eddie.crc.nd.edu:3128 == env: LOBSTER_CMSSW_VERSION=CMSSW_9_3_6 == env: VC3_MACHINE_OS=Linux/redhat7.4.1708 == env: VC3_VERSION_FULL_TAR=1.26.0 == env: USER=kmohrman == env: VC3_VERSION_FULL_CCTOOLS_WQ_WORKER=999.999.999 == env: LD_LIBRARY_PATH=/.singularity.d/libs:lib == env: VC3_BUILDER_PATH=/disk/vc3-builder == env: TEMP=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: VC3_VERSION_VC3_BUILDER_HANDS=1.0 == env: VC3_DISTFILES=/disk/vc3-distfiles == env: VC3_ROOT_MAKE=/usr/ == env: ENV=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX == env: VC3_VERSION_TAR=1.26 == env: WORKER_TMPDIR=/disk/vc3-root/tmp/worker-230547-7899/cache/tmp == env: FRONTIER_PROXY=http://eddie.crc.nd.edu:3128 == env: PARROT_ALLOW_SWITCHING_CVMFS_REPOSITORIES=TRUE == env: VC3_VERSION_CCTOOLS_WQ_WORKER=999.999 == env: VC3_VERSION_FULL_PKG_CONFIG=0.27.1 == env: PARROT_CVMFS_REPO= == env: VC3_VERSION_GZIP=1.5 == env: PATH=/disk/vc3-root/x86_64/redhat7.4.1708/which/v2.20.0/bin:/disk/vc3-root/x86_64/redhat7.4.1708/cctools-wq-worker/v999.999.999/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin == env: VC3_ENV_TAG_GLOBAL=7645 == env: VC3_ROOT=/disk/vc3-root == env: VC3_SH_PROFILE_ENV=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX == env: PWD=/disk/vc3-root/tmp/worker-230547-7899/t.43 == env: VC3_ROOT_PKG_CONFIG=/usr/ == env: PARROT_ENABLED=TRUE == env: VC3_VERSION_FULL_SED=4.2.2 == env: LOBSTER_OSG_VERSION=3.4 == env: VC3_VERSION_WHICH=2.20 == env: MODULEPATH=/afs/crc.nd.edu/x86_64_linux/Modules/modules/development_tools_and_libraries:/afs/crc.nd.edu/x86_64_linux/Modules/modules/general_software:/afs/crc.nd.edu/x86_64_linux/Modules/modules/restricted_software:/afs/crc.nd.edu/x86_64_linux/Modules/modules/system_modules:/afs/crc.nd.edu/x86_64_linux/Modules/modules/deprecated_software:/afs/crc.nd.edu/x86_64_linux/Modules/current/modulefiles == env: VC3_MACHINE_GLIBC_VERSION=2.17 == env: LOBSTER_LCG_CP= == env: WORK_QUEUE_SANDBOX=/disk/vc3-root/tmp/worker-230547-7899 == env: VC3_ROOT_CCTOOLS_WQ_WORKER=/disk/vc3-root/x86_64/redhat7.4.1708/cctools-wq-worker/v999.999.999 == env: VC3_MACHINE_TARGET=x86_64/redhat7.4.1708 == env: VC3_ROOT_SED=/usr/ == env: OASIS_CERTIFICATES=/cvmfs/oasis.opensciencegrid.org/mis/certificates == env: CXX=g++ == env: HOME=/disk/vc3-root/vc3-home == env: SHLVL=3 == env: VC3_VERSION_FULL_GZIP=1.5.0 == env: VC3_VERSION_MAKE=3.82 == env: DISK=1000 == env: PARROT_CACHE=/disk/vc3-root/tmp/worker-230547-7899/cache/tmp == env: PARROT_CVMFS_ALIEN_CACHE=/disk/vc3-root/tmp/worker-230547-7899/cache/tmp/cvmfs == env: VC3_VERSION_FULL_WHICH=2.20.0 == env: MEMORY=1200 == env: HTTP_PROXY=http://eddie.crc.nd.edu:3128 == env: TMP=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: PYTHONPATH=python: == env: VC3_INSTALL_USER_HOME=/disk/vc3-root/vc3-home == env: VC3_VERSION_FULL_VC3_BUILDER_HANDS=1.0.0 == env: VC3_ROOT_TAR=/usr/ == env: VC3_VERSION_PKG_CONFIG=0.27 == env: CCTOOLS_ND=lobster-171-cd5e3e2c-cvmfs-70dfa0d6 == env: VC3_SH_PROFILE_WRAPPER=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX.wrapper == env: CC=gcc == env: CCTOOLS_RESOURCE_MONITOR_INFO=1033 == env: VC3_VERSION_FULL_MAKE=3.82.0 == env: CORES=1 == env: PARROT_DEBUG_FLAGS= == env: VC3_MACHINE_ARCH=x86_64 == env: VC3_ROOT_VC3_BUILDER_HANDS=/disk/vc3-root/x86_64/redhat7.4.1708/vc3-builder-hands/v1.0.0 == env: VC3_COMMAND_LINE=/disk/vc3-builder --var CCTOOLS_ND=lobster-171-cd5e3e2c-cvmfs-70dfa0d6 --require cctools-wq-worker -- work_queue_worker -M lobster_kmohrman_EFT_ALL_genOnly.* -t 300 -C catalog.cse.nd.edu,backup-catalog.cse.nd.edu -d all -o worker.log --cores=12 --memory=16000 --disk=17780 == env: _=/usr/bin/env ==== cpu info @ Sun Aug 16 18:51:25 EDT 2020 ==== == cpu: processor : 0 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 2422.100 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 0 == cpu: cpu cores : 8 == cpu: apicid : 0 == cpu: initial apicid : 0 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 1 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 1 == cpu: cpu cores : 8 == cpu: apicid : 2 == cpu: initial apicid : 2 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 2 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 2 == cpu: cpu cores : 8 == cpu: apicid : 4 == cpu: initial apicid : 4 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 3 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 1888.513 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 3 == cpu: cpu cores : 8 == cpu: apicid : 6 == cpu: initial apicid : 6 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 4 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 801.086 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 4 == cpu: cpu cores : 8 == cpu: apicid : 8 == cpu: initial apicid : 8 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 5 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 1964.904 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 5 == cpu: cpu cores : 8 == cpu: apicid : 10 == cpu: initial apicid : 10 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 6 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 816.467 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 6 == cpu: cpu cores : 8 == cpu: apicid : 12 == cpu: initial apicid : 12 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 7 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 925.799 == cpu: cache size : 11264 KB == cpu: physical id : 0 == cpu: siblings : 8 == cpu: core id : 7 == cpu: cpu cores : 8 == cpu: apicid : 14 == cpu: initial apicid : 14 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4200.00 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 8 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 2415.179 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 0 == cpu: cpu cores : 8 == cpu: apicid : 16 == cpu: initial apicid : 16 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 9 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 1804.046 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 1 == cpu: cpu cores : 8 == cpu: apicid : 18 == cpu: initial apicid : 18 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 10 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 838.385 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 2 == cpu: cpu cores : 8 == cpu: apicid : 20 == cpu: initial apicid : 20 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 11 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 3 == cpu: cpu cores : 8 == cpu: apicid : 22 == cpu: initial apicid : 22 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 12 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 834.411 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 4 == cpu: cpu cores : 8 == cpu: apicid : 24 == cpu: initial apicid : 24 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 13 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 1314.550 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 5 == cpu: cpu cores : 8 == cpu: apicid : 26 == cpu: initial apicid : 26 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 14 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 846.716 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 6 == cpu: cpu cores : 8 == cpu: apicid : 28 == cpu: initial apicid : 28 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: == cpu: processor : 15 == cpu: vendor_id : GenuineIntel == cpu: cpu family : 6 == cpu: model : 85 == cpu: model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz == cpu: stepping : 4 == cpu: microcode : 0x200005e == cpu: cpu MHz : 799.932 == cpu: cache size : 11264 KB == cpu: physical id : 1 == cpu: siblings : 8 == cpu: core id : 7 == cpu: cpu cores : 8 == cpu: apicid : 30 == cpu: initial apicid : 30 == cpu: fpu : yes == cpu: fpu_exception : yes == cpu: cpuid level : 22 == cpu: wp : yes == cpu: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear spec_ctrl intel_stibp flush_l1d == cpu: bogomips : 4205.20 == cpu: clflush size : 64 == cpu: cache_alignment : 64 == cpu: address sizes : 46 bits physical, 48 bits virtual == cpu: power management: == cpu: === using parrot @ Sun Aug 16 18:51:28 EDT 2020 === sourcing CMS setup @ Sun Aug 16 18:51:28 EDT 2020 LibCvmfs version 2.4, revision 25 === sourcing OSG setup @ Sun Aug 16 18:51:31 EDT 2020 ==== environment after sourcing startup scripts @ Sun Aug 16 18:51:31 EDT 2020 ==== == env: VC3_VERSION_SED=4.2 == env: MANPATH=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/share/man:/cvmfs/cms.cern.ch/share/man: == env: CCTOOLS_RESOURCE_PROCESS_START=1597618291273555 == env: VC3_INSIDE_WRAPPER_SCRIPT=yes == env: GFAL_PLUGIN_DIR=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/gfal2-plugins/ == env: PARROT_HTTP_PROXY=http://eddie.crc.nd.edu:3128 == env: TERM=xterm-256color == env: SHELL=/bin/sh == env: VC3_ROOT_GZIP=/usr/ == env: LOBSTER_GFAL_COPY=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/bin/gfal-copy == env: VOMS_USERCONF=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/etc/vomses == env: MAKEFLAGS=-j4 == env: GLOBUS_LOCATION=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr == env: TMPDIR=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: PERL5LIB=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/share/perl5/vendor_perl:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/share/perl5:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/perl5/vendor_perl:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/perl5 == env: VC3_ROOT_WHICH=/disk/vc3-root/x86_64/redhat7.4.1708/which/v2.20.0 == env: CVSROOT=:gserver:cmssw.cvs.cern.ch:/local/reps/CMSSW == env: LD_PRELOAD=/disk/vc3-root/tmp/worker-230547-7899/t.43/librmonitor_helper.so.3N9QXP == env: LOBSTER_CVMFS_PROXY=http://eddie.crc.nd.edu:3128 == env: PARROT_PATH=./bin == env: LOBSTER_FRONTIER_PROXY=http://eddie.crc.nd.edu:3128 == env: X509_CERT_DIR=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/etc/grid-security/certificates == env: SCRAM_ARCH=slc7_amd64_gcc820 == env: LOBSTER_CMSSW_VERSION=CMSSW_9_3_6 == env: VC3_MACHINE_OS=Linux/redhat7.4.1708 == env: VC3_VERSION_FULL_TAR=1.26.0 == env: USER=kmohrman == env: VC3_VERSION_FULL_CCTOOLS_WQ_WORKER=999.999.999 == env: LD_LIBRARY_PATH=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/lib64:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/lib:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/dcap:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/lcgdm:/.singularity.d/libs:lib:lib == env: VC3_BUILDER_PATH=/disk/vc3-builder == env: TEMP=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: VC3_VERSION_VC3_BUILDER_HANDS=1.0 == env: VC3_DISTFILES=/disk/vc3-distfiles == env: VC3_ROOT_MAKE=/usr/ == env: OSG_LOCATION=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64 == env: ENV=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX == env: VC3_VERSION_TAR=1.26 == env: WORKER_TMPDIR=/disk/vc3-root/tmp/worker-230547-7899/cache/tmp == env: FRONTIER_PROXY=http://eddie.crc.nd.edu:3128 == env: PARROT_ALLOW_SWITCHING_CVMFS_REPOSITORIES=TRUE == env: VC3_VERSION_CCTOOLS_WQ_WORKER=999.999 == env: VC3_VERSION_FULL_PKG_CONFIG=0.27.1 == env: PARROT_CVMFS_REPO= == env: VC3_VERSION_GZIP=1.5 == env: PATH=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/bin:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/sbin:/cvmfs/cms.cern.ch/common:/disk/vc3-root/x86_64/redhat7.4.1708/which/v2.20.0/bin:/disk/vc3-root/x86_64/redhat7.4.1708/cctools-wq-worker/v999.999.999/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin == env: VC3_ENV_TAG_GLOBAL=7645 == env: CMS_PATH=/cvmfs/cms.cern.ch == env: VC3_ROOT=/disk/vc3-root == env: VC3_SH_PROFILE_ENV=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX == env: PWD=/disk/vc3-root/tmp/worker-230547-7899/t.43 == env: VC3_ROOT_PKG_CONFIG=/usr/ == env: PARROT_ENABLED=TRUE == env: VC3_VERSION_FULL_SED=4.2.2 == env: LOBSTER_OSG_VERSION=3.4 == env: VC3_VERSION_WHICH=2.20 == env: MODULEPATH=/afs/crc.nd.edu/x86_64_linux/Modules/modules/development_tools_and_libraries:/afs/crc.nd.edu/x86_64_linux/Modules/modules/general_software:/afs/crc.nd.edu/x86_64_linux/Modules/modules/restricted_software:/afs/crc.nd.edu/x86_64_linux/Modules/modules/system_modules:/afs/crc.nd.edu/x86_64_linux/Modules/modules/deprecated_software:/afs/crc.nd.edu/x86_64_linux/Modules/current/modulefiles == env: VC3_MACHINE_GLIBC_VERSION=2.17 == env: X509_VOMS_DIR=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/etc/grid-security/vomsdir == env: LOBSTER_LCG_CP= == env: WORK_QUEUE_SANDBOX=/disk/vc3-root/tmp/worker-230547-7899 == env: VC3_ROOT_CCTOOLS_WQ_WORKER=/disk/vc3-root/x86_64/redhat7.4.1708/cctools-wq-worker/v999.999.999 == env: VC3_MACHINE_TARGET=x86_64/redhat7.4.1708 == env: VC3_ROOT_SED=/usr/ == env: OASIS_CERTIFICATES=/cvmfs/oasis.opensciencegrid.org/mis/certificates == env: CXX=g++ == env: HOME=/disk/vc3-root/vc3-home == env: SHLVL=3 == env: VC3_VERSION_FULL_GZIP=1.5.0 == env: VC3_VERSION_MAKE=3.82 == env: DISK=1000 == env: PARROT_CACHE=/disk/vc3-root/tmp/worker-230547-7899/cache/tmp == env: PARROT_CVMFS_ALIEN_CACHE=/disk/vc3-root/tmp/worker-230547-7899/cache/tmp/cvmfs == env: GFAL_CONFIG_DIR=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/etc/gfal2.d/ == env: VC3_VERSION_FULL_WHICH=2.20.0 == env: MEMORY=1200 == env: HTTP_PROXY=http://eddie.crc.nd.edu:3128 == env: TMP=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: PYTHONPATH=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib/python2.7/site-packages:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/python2.7/site-packages:python:python: == env: VC3_INSTALL_USER_HOME=/disk/vc3-root/vc3-home == env: VC3_VERSION_FULL_VC3_BUILDER_HANDS=1.0.0 == env: VC3_ROOT_TAR=/usr/ == env: VC3_VERSION_PKG_CONFIG=0.27 == env: CCTOOLS_ND=lobster-171-cd5e3e2c-cvmfs-70dfa0d6 == env: VC3_SH_PROFILE_WRAPPER=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX.wrapper == env: CC=gcc == env: CCTOOLS_RESOURCE_MONITOR_INFO=1033 == env: VC3_VERSION_FULL_MAKE=3.82.0 == env: CORES=1 == env: PARROT_DEBUG_FLAGS= == env: VC3_MACHINE_ARCH=x86_64 == env: VC3_ROOT_VC3_BUILDER_HANDS=/disk/vc3-root/x86_64/redhat7.4.1708/vc3-builder-hands/v1.0.0 == env: VC3_COMMAND_LINE=/disk/vc3-builder --var CCTOOLS_ND=lobster-171-cd5e3e2c-cvmfs-70dfa0d6 --require cctools-wq-worker -- work_queue_worker -M lobster_kmohrman_EFT_ALL_genOnly.* -t 300 -C catalog.cse.nd.edu,backup-catalog.cse.nd.edu -d all -o worker.log --cores=12 --memory=16000 --disk=17780 == env: _=/usr/bin/env ==== proxy information @ Sun Aug 16 18:51:31 EDT 2020 ==== == proxy: subject : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=kmohrman/CN=825714/CN=Kelci Ann Mohrman/CN=3523194095 == proxy: issuer : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=kmohrman/CN=825714/CN=Kelci Ann Mohrman == proxy: identity : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=kmohrman/CN=825714/CN=Kelci Ann Mohrman == proxy: type : RFC compliant proxy == proxy: strength : 1024 bits == proxy: path : proxy == proxy: timeleft : 74:35:27 ==== working directory at startup @ Sun Aug 16 18:51:32 EDT 2020 ==== == dir: total 860 == dir: -rw-r--r-- 9 kmohrman campus 8037 Aug 16 18:50 GEN-00000-ttHJets_1_base-0_cfg.py == dir: drwxr-xr-x 2 kmohrman campus 71 Aug 16 18:50 bin == dir: -rwxr-xr-x 11 kmohrman campus 627272 Aug 16 18:50 cctools-monitor == dir: -rw-r--r-- 1 kmohrman campus 0 Aug 16 18:50 cctools-monitor.summary == dir: drwxrwxrwx 2 kmohrman campus 6 Aug 16 18:50 cctools-temp-t.43.YBXcWS == dir: -rw------- 1 kmohrman campus 51 Aug 16 18:51 cvmfschecksum.cms.cern.ch == dir: -rw------- 1 kmohrman campus 51 Aug 16 18:51 cvmfschecksum.oasis.opensciencegrid.org == dir: drwxr-xr-x 2 kmohrman campus 33 Aug 16 18:50 lib == dir: -rwxrwxrwx 1 kmohrman campus 146912 Aug 16 18:50 librmonitor_helper.so.3N9QXP == dir: -rw-r--r-- 2 kmohrman campus 2010 Aug 16 18:50 parameters.json == dir: -rw------- 2 kmohrman campus 9884 Aug 16 18:50 proxy == dir: drwxr-xr-x 3 kmohrman campus 20 Aug 16 18:50 python == dir: -rw-r--r-- 11 kmohrman campus 570 Aug 16 18:50 report.json.in == dir: -rw-r--r-- 11 kmohrman campus 449 Aug 16 18:50 sandbox-CMSSW_9_3_6-slc7_amd64_gcc630.tar.bz2 == dir: drwxr-xr-x 4 kmohrman campus 37 Aug 16 18:50 siteconf == dir: -rw-r--r-- 1 kmohrman campus 11 Aug 16 18:51 t_wrapper_start == dir: -rwxr-xr-x 11 kmohrman campus 47217 Aug 16 18:50 task.py == dir: -rwxr-xr-x 11 kmohrman campus 5986 Aug 16 18:50 wrapper.sh === creating new release CMSSW_9_3_6 for scram arch slc7_amd64_gcc630 @ Sun Aug 16 18:51:32 EDT 2020 WARNING: In non-interactive mode release checks e.g. deprecated releases, production architectures are disabled. === unpacking sandbox-CMSSW_9_3_6-slc7_amd64_gcc630.tar.bz2 @ Sun Aug 16 18:51:48 EDT 2020 ==== machine load @ Sun Aug 16 18:51:50 EDT 2020 ==== top: unknown option 'M' Usage: top -hv | -bcHiOSs -d secs -n max -u|U user -p pid(s) -o field -w [cols] ==== environment before execution @ Sun Aug 16 18:51:50 EDT 2020 ==== == env: G4LEVELGAMMADATA=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4PhotonEvaporation/3.2/data/PhotonEvaporation3.2 == env: UTM_XSD_DIR=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/utm/utm_0.6.5 == env: SRT_G4LEVELGAMMADATA_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4PhotonEvaporation/3.2/data/PhotonEvaporation3.2 == env: VC3_VERSION_SED=4.2 == env: CMSSW_RELEASE_BASE=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6 == env: SCRAMRT_SET=CMSSW:CMSSW_9_3_6:slc7_amd64_gcc630:V2_2_9_pre10:SRT_ == env: MANPATH=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/share/man:/cvmfs/cms.cern.ch/share/man: == env: CCTOOLS_RESOURCE_PROCESS_START=1597618310469658 == env: CMSSW_FWLITE_INCLUDE_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/hepmc/2.06.07/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/clhep/2.3.4.2-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/boost/1.63.0-mlhled2/include == env: COMPILER_RUNTIME_OBJECTS=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0 == env: SRT_MALLOC_CONF_SCRAMRTDEL=lg_chunk:18,lg_dirty_mult:4 == env: VC3_INSIDE_WRAPPER_SCRIPT=yes == env: ROOT_INCLUDE_PATH=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/src:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/src:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/coral/CORAL_2_3_21-njopjo6/include/LCG:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/mctester/1.25.0a-ghjeda4/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/lcg/root/6.10.04-ghjeda/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/QtDesigner:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/tauolapp/1.1.5-ghjeda3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/charybdis/1.003-ghjeda3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/thepeg/1.9.2p1-ghjeda/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/include/SHERPA-MC:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/QtOpenGL:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/QtGui:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pythia8/230-ghjeda/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/herwig/6.521-ghjeda3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/rivet/2.5.2-ghjeda4/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/Qt3Support:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/lwtnn/1.0-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/classlib/3.1.3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/lhapdf/6.2.1-ghjeda/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/cgal/4.2-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/tkonlinesw/4.1.0-1-ghjeda4/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/starlight/r193-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/Qt:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/QtCore:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/QtXml:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/mcdb/1.0.3/interface:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libungif/4.1.4/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libtiff/4.0.3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libpng/1.6.16/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4/10.02.p02-ghjeda3/include/Geant4:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/frontier_client/2.8.20-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pcre/8.37/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/boost/1.63.0-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/xrootd/4.6.1/include/xrootd/private:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/vdt/v0.3.2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/valgrind/3.12.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/utm/utm_0.6.5/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/toprex/4.23/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/tbb/2017_20161004oss/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/tauola/27.121.5/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sigcpp/2.6.2/include/sigc++-2.0:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sqlite/3.15.1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/protobuf/3.2.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pacparser/1.3.5/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/oracle/12.1.0.2.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/meschach/1.2.pCMS1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/madgraph5amcatnlo/2.4.3-ghjeda6:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libhepml/0.2.1/interface:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/ktjet/1.06-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/jimmy/4.2-ghjeda3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/jemalloc/4.5.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/xz/5.2.2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/heppdt/3.03.00/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/hector/1.3.4_patch1-ghjeda4/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gsl/2.2.1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libjpeg-turbo/1.3.1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/giflib/4.2.3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/xerces-c/3.1.3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gdbm/1.10/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/freetype/2.5.3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/fftw3/3.3.2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/fftjet/1.5.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/fastjet/3.1.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/expat/2.1.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/hepmc/2.06.07/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/dpm/1.8.0.1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/dcap/2.47.8/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libxml2/2.9.1/include/libxml2:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/curl/7.51.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/cppunit/1.12.1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/clhep/2.3.4.2-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/openssl/1.0.2d/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pythia6/426/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/photos/215.5/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/zlib-x86_64/1.2.11/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libuuid/2.22.2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/castor/2.1.13.9/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/castor/2.1.13.9/include/shift:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/cascade/2.2.04-ghjeda3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/bz2lib/1.0.6/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/python/2.7.11-mlhled2/include/python2.7:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/ittnotify/16.06.18/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/include/c++/6.3.0:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/include/c++/6.3.0/x86_64-unknown-linux-gnu:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/include/c++/6.3.0/backward:/usr/local/include:/usr/include == env: INTEL_LICENSE_FILE=28518@lxlicen01.cern.ch,28518@lxlicen02.cern.ch,28518@lxlicen03.cern.ch == env: SRT_TEXMFHOME_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/rivet/2.5.2-ghjeda4/share/Rivet/texmf == env: SRT_GEANT4_PARFULLCMS_CONFIG_DIR_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-parfullcms/2014.01.27-ghjeda3/share/ParFullCMS == env: GFAL_PLUGIN_DIR=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/gfal2-plugins/ == env: PARROT_HTTP_PROXY=http://eddie.crc.nd.edu:3128 == env: TERM=xterm-256color == env: SHELL=/bin/sh == env: VC3_ROOT_GZIP=/usr/ == env: LOBSTER_GFAL_COPY=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/bin/gfal-copy == env: PDFPATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/rivet/2.5.2-ghjeda4/share == env: DBSCMD_HOME=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/dbs-client/DBS_2_1_9-mlhled2/lib/DBSAPI == env: SRT_TNS_ADMIN_SCRAMRTDEL=/etc == env: VOMS_USERCONF=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/etc/vomses == env: MAKEFLAGS=-j4 == env: ROOT_GCC_TOOLCHAIN=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0 == env: GLOBUS_LOCATION=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr == env: TMPDIR=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: G4FORCENUMBEROFTHREADS=max == env: OPENBLAS_NUM_THREADS=1 == env: SRT_PDFPATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/rivet/2.5.2-ghjeda4/share == env: PERL5LIB=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/git/1.8.3.1/share/perl5:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/share/perl5/vendor_perl:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/share/perl5:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/perl5/vendor_perl:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/perl5 == env: VC3_ROOT_WHICH=/disk/vc3-root/x86_64/redhat7.4.1708/which/v2.20.0 == env: CMSSW_DATA_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms == env: LHAPDF_DATA_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/lhapdf/6.2.1-ghjeda/share/LHAPDF == env: CVSROOT=:gserver:cmssw.cvs.cern.ch:/local/reps/CMSSW == env: LD_PRELOAD=/disk/vc3-root/tmp/worker-230547-7899/t.43/librmonitor_helper.so.3N9QXP == env: LOBSTER_CVMFS_PROXY=http://eddie.crc.nd.edu:3128 == env: PARROT_PATH=./bin == env: CMSSW_GIT_HASH=CMSSW_9_3_6 == env: SRT_G4ABLADATA_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4ABLA/3.0/data/G4ABLA3.0 == env: QTDIR=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7 == env: SRT_SCONS_LIB_DIR_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/scons/1.2.0-mlhled2/lib == env: OLDPWD=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6 == env: LOBSTER_FRONTIER_PROXY=http://eddie.crc.nd.edu:3128 == env: G4NEUTRONXSDATA=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4NEUTRONXS/1.4/data/G4NEUTRONXS1.4 == env: QTINC=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include == env: SRT_COMPILER_RUNTIME_OBJECTS_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0 == env: X509_CERT_DIR=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/etc/grid-security/certificates == env: SCRAM_ARCH=slc7_amd64_gcc630 == env: GIT_SSL_CAINFO=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/git/1.8.3.1/share/ssl/certs/ca-bundle.crt == env: DBS_CLIENT_CONFIG=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/dbs-client/DBS_2_1_9-mlhled2/lib/DBSAPI/dbs.config == env: SRT_CMS_OPENLOOPS_PREFIX_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/openloops/1.3.1-mlhled2 == env: SRT_GIT_EXEC_PATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/git/1.8.3.1/libexec/git-core == env: LOBSTER_CMSSW_VERSION=CMSSW_9_3_6 == env: ROOT_TTREECACHE_SIZE=0 == env: EVTGENDATA=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/evtgen/1.6.0-ghjeda3/share == env: SRT_INTEL_LICENSE_FILE_SCRAMRTDEL=28518@lxlicen01.cern.ch,28518@lxlicen02.cern.ch,28518@lxlicen03.cern.ch == env: SRT_ROOT_INCLUDE_PATH_SCRAMRTDEL=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/src:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/src:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/coral/CORAL_2_3_21-njopjo6/include/LCG:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/mctester/1.25.0a-ghjeda4/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/lcg/root/6.10.04-ghjeda/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/QtDesigner:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/tauolapp/1.1.5-ghjeda3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/charybdis/1.003-ghjeda3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/thepeg/1.9.2p1-ghjeda/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/include/SHERPA-MC:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/QtOpenGL:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/QtGui:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pythia8/230-ghjeda/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/herwig/6.521-ghjeda3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/rivet/2.5.2-ghjeda4/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/Qt3Support:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/lwtnn/1.0-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/classlib/3.1.3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/lhapdf/6.2.1-ghjeda/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/cgal/4.2-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/tkonlinesw/4.1.0-1-ghjeda4/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/starlight/r193-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/Qt:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/QtCore:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include/QtXml:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/mcdb/1.0.3/interface:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libungif/4.1.4/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libtiff/4.0.3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libpng/1.6.16/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4/10.02.p02-ghjeda3/include/Geant4:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/frontier_client/2.8.20-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pcre/8.37/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/boost/1.63.0-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/xrootd/4.6.1/include/xrootd/private:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/vdt/v0.3.2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/valgrind/3.12.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/utm/utm_0.6.5/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/toprex/4.23/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/tbb/2017_20161004oss/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/tauola/27.121.5/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sigcpp/2.6.2/include/sigc++-2.0:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sqlite/3.15.1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/protobuf/3.2.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pacparser/1.3.5/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/oracle/12.1.0.2.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/meschach/1.2.pCMS1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/madgraph5amcatnlo/2.4.3-ghjeda6:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libhepml/0.2.1/interface:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/ktjet/1.06-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/jimmy/4.2-ghjeda3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/jemalloc/4.5.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/xz/5.2.2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/heppdt/3.03.00/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/hector/1.3.4_patch1-ghjeda4/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gsl/2.2.1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libjpeg-turbo/1.3.1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/giflib/4.2.3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/xerces-c/3.1.3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gdbm/1.10/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/freetype/2.5.3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/fftw3/3.3.2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/fftjet/1.5.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/fastjet/3.1.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/expat/2.1.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/hepmc/2.06.07/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/dpm/1.8.0.1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/dcap/2.47.8/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libxml2/2.9.1/include/libxml2:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/curl/7.51.0/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/cppunit/1.12.1/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/clhep/2.3.4.2-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/openssl/1.0.2d/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pythia6/426/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/photos/215.5/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/zlib-x86_64/1.2.11/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/libuuid/2.22.2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/castor/2.1.13.9/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/castor/2.1.13.9/include/shift:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/cascade/2.2.04-ghjeda3/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/bz2lib/1.0.6/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/python/2.7.11-mlhled2/include/python2.7:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/ittnotify/16.06.18/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/include/c++/6.3.0:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/include/c++/6.3.0/x86_64-unknown-linux-gnu:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/include/c++/6.3.0/backward:/usr/local/include:/usr/include == env: VC3_MACHINE_OS=Linux/redhat7.4.1708 == env: VC3_VERSION_FULL_TAR=1.26.0 == env: G4LEDATA=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4EMLOW/6.48/data/G4EMLOW6.48 == env: SRT_VINCIADATA_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/vincia/1.2.02-ghjeda3/share/Vincia/xmldoc == env: SRT_ROOT_TTREECACHE_PREFILL_SCRAMRTDEL=0 == env: OPAL_PREFIX=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/openmpi/2.1.2rc4 == env: USER=kmohrman == env: VC3_VERSION_FULL_CCTOOLS_WQ_WORKER=999.999.999 == env: ROOT_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/madgraph5amcatnlo/2.4.3-ghjeda6 == env: SHERPA_INCLUDE_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/include/SHERPA-MC == env: SRT_LHAPDF_DATA_PATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/lhapdf/6.2.1-ghjeda/share/LHAPDF == env: LD_LIBRARY_PATH=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/biglib/slc7_amd64_gcc630:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/external/slc7_amd64_gcc630/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/biglib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/external/slc7_amd64_gcc630/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/llvm/4.0.0-njopjo/lib64:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/lib64:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/lib:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/lib64:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/lib:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/dcap:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/lcgdm:/.singularity.d/libs:lib:lib == env: VC3_BUILDER_PATH=/disk/vc3-builder == env: TEMP=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: SRT_CMSSW_RELEASE_BASE_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6 == env: SRT_RIVET_ANALYSIS_PATH_SCRAMRTDEL=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/rivet/2.5.2-ghjeda4/lib == env: VC3_VERSION_VC3_BUILDER_HANDS=1.0 == env: SRT_PERL5LIB_SCRAMRT=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/git/1.8.3.1/share/perl5 == env: SRT_CMSSW_VERSION_SCRAMRTDEL=CMSSW_9_3_6 == env: VC3_DISTFILES=/disk/vc3-distfiles == env: VC3_ROOT_MAKE=/usr/ == env: VALGRIND_LIB=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/valgrind/3.12.0/lib/valgrind == env: SRT_CMSSW_GIT_HASH_SCRAMRTDEL=CMSSW_9_3_6 == env: SRT_ROOTSYS_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/lcg/root/6.10.04-ghjeda/ == env: OSG_LOCATION=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64 == env: ENV=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX == env: VC3_VERSION_TAR=1.26 == env: WORKER_TMPDIR=/disk/vc3-root/tmp/worker-230547-7899/cache/tmp == env: G4NEUTRONHPDATA=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4NDL/4.5/data/G4NDL4.5 == env: GEANT4_PARFULLCMS_CONFIG_DIR=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-parfullcms/2014.01.27-ghjeda3/share/ParFullCMS == env: FRONTIER_PROXY=http://eddie.crc.nd.edu:3128 == env: PARROT_ALLOW_SWITCHING_CVMFS_REPOSITORIES=TRUE == env: CSC_TRACK_FINDER_DATA_DIR=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/CSCTrackFinderEmulation/1.2/data/ == env: SRT_QTINC_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/include == env: VC3_VERSION_CCTOOLS_WQ_WORKER=999.999 == env: G4ENSDFSTATEDATA=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4ENSDFSTATE/1.2.3/data/G4ENSDFSTATE1.2.3 == env: G4RADIOACTIVEDATA=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4RadioactiveDecay/4.3.2/data/RadioactiveDecay4.3.2 == env: ROOFITSYS=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/lcg/root/6.10.04-ghjeda/ == env: SRT_ROOFITSYS_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/lcg/root/6.10.04-ghjeda/ == env: SRT_G4RADIOACTIVEDATA_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4RadioactiveDecay/4.3.2/data/RadioactiveDecay4.3.2 == env: SRT_G4ENSDFSTATEDATA_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4ENSDFSTATE/1.2.3/data/G4ENSDFSTATE1.2.3 == env: SRT_CLHEP_PARAM_PATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/clhep/2.3.4.2-mlhled2 == env: G4ABLADATA=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4ABLA/3.0/data/G4ABLA3.0 == env: VC3_VERSION_FULL_PKG_CONFIG=0.27.1 == env: SRT_SHERPA_INCLUDE_PATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/include/SHERPA-MC == env: SRT_G4FORCENUMBEROFTHREADS_SCRAMRTDEL=max == env: SRT_WORKER_DATA_PATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/blackhat/0.9.9-mlhled2/share/blackhat/datafiles/ == env: PARROT_CVMFS_REPO= == env: VC3_VERSION_GZIP=1.5 == env: TNS_ADMIN=/etc == env: SRT_HEPPDT_PARAM_PATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/heppdt/3.03.00 == env: PATH=/cvmfs/cms.cern.ch/share/overrides/bin:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/bin/slc7_amd64_gcc630:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/external/slc7_amd64_gcc630/bin:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/bin/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/external/slc7_amd64_gcc630/bin:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/llvm/4.0.0-njopjo/bin:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/bin:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/bin:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/sbin:/cvmfs/cms.cern.ch/common:/disk/vc3-root/x86_64/redhat7.4.1708/which/v2.20.0/bin:/disk/vc3-root/x86_64/redhat7.4.1708/cctools-wq-worker/v999.999.999/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin == env: VC3_ENV_TAG_GLOBAL=7645 == env: SRT_DBS_CLIENT_CONFIG_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/dbs-client/DBS_2_1_9-mlhled2/lib/DBSAPI/dbs.config == env: SRT_CMSSW_SEARCH_PATH_SCRAMRTDEL=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/src:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/external/slc7_amd64_gcc630/data:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/src:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/external/slc7_amd64_gcc630/data == env: SRT_GIT_SSL_CAINFO_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/git/1.8.3.1/share/ssl/certs/ca-bundle.crt == env: CMS_PATH=/cvmfs/cms.cern.ch == env: VC3_ROOT=/disk/vc3-root == env: SRT_UTM_XSD_DIR_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/utm/utm_0.6.5 == env: SRT_PATH_SCRAMRT=/cvmfs/cms.cern.ch/share/overrides/bin:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/bin/slc7_amd64_gcc630:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/external/slc7_amd64_gcc630/bin:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/bin/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/external/slc7_amd64_gcc630/bin:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/llvm/4.0.0-njopjo/bin:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/bin == env: VC3_SH_PROFILE_ENV=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX == env: RIVET_ANALYSIS_PATH=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/rivet/2.5.2-ghjeda4/lib == env: SRT_LOCALRT_SCRAMRTDEL=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6 == env: SRT_CMSSW_BASE_SCRAMRTDEL=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6 == env: PWD=/disk/vc3-root/tmp/worker-230547-7899/t.43 == env: VC3_ROOT_PKG_CONFIG=/usr/ == env: PARROT_ENABLED=TRUE == env: VC3_VERSION_FULL_SED=4.2.2 == env: LOBSTER_OSG_VERSION=3.4 == env: SRT_OPAL_PREFIX_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/openmpi/2.1.2rc4 == env: VC3_VERSION_WHICH=2.20 == env: LANG=C == env: SRT_QTDIR_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7 == env: MODULEPATH=/afs/crc.nd.edu/x86_64_linux/Modules/modules/development_tools_and_libraries:/afs/crc.nd.edu/x86_64_linux/Modules/modules/general_software:/afs/crc.nd.edu/x86_64_linux/Modules/modules/restricted_software:/afs/crc.nd.edu/x86_64_linux/Modules/modules/system_modules:/afs/crc.nd.edu/x86_64_linux/Modules/modules/deprecated_software:/afs/crc.nd.edu/x86_64_linux/Modules/current/modulefiles == env: VC3_MACHINE_GLIBC_VERSION=2.17 == env: SRT_G4NEUTRONHPDATA_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4NDL/4.5/data/G4NDL4.5 == env: X509_VOMS_DIR=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/etc/grid-security/vomsdir == env: LOBSTER_LCG_CP= == env: SRT_SHERPA_SHARE_PATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/share/SHERPA-MC == env: WORK_QUEUE_SANDBOX=/disk/vc3-root/tmp/worker-230547-7899 == env: CASCADE_PDFPATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/cascade/2.2.04-ghjeda3/share == env: SRT_ROOT_GCC_TOOLCHAIN_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0 == env: SRT_CASCADE_PDFPATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/cascade/2.2.04-ghjeda3/share == env: WORKER_DATA_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/blackhat/0.9.9-mlhled2/share/blackhat/datafiles/ == env: GIT_TEMPLATE_DIR=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/git/1.8.3.1/share/git-core/templates == env: G4SAIDXSDATA=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4SAIDDATA/1.1/data/G4SAIDDATA1.1 == env: SRT_DBSCMD_HOME_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/dbs-client/DBS_2_1_9-mlhled2/lib/DBSAPI == env: VC3_ROOT_CCTOOLS_WQ_WORKER=/disk/vc3-root/x86_64/redhat7.4.1708/cctools-wq-worker/v999.999.999 == env: SRT_CMSSW_FWLITE_INCLUDE_PATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/hepmc/2.06.07/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/clhep/2.3.4.2-mlhled2/include:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/boost/1.63.0-mlhled2/include == env: SRT_PYTHONPATH_SCRAMRT=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/python:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/python:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/coral/CORAL_2_3_21-njopjo6/slc7_amd64_gcc630/python:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/coral/CORAL_2_3_21-njopjo6/slc7_amd64_gcc630/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/professor2/2.2.1-ghjeda6/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pyqt/4.11.4-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/rivet/2.5.2-ghjeda4/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/python-ldap/2.4.10-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-matplotlib/1.5.2-njopjo/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sip/4.17-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/llvm/4.0.0-njopjo/lib64/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-sqlalchemy/1.1.4-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-lint/0.25.1-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-dxr/1.0-ghjeda6/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-cx-oracle/5.2.1-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-numpy/1.12.1-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/frontier_client/2.8.20-mlhled2/python/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/das_client/v03.01.00-mlhled2/bin:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/xrootd/4.6.1/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/yoda/1.6.5-ghjeda4/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/lcg/root/6.10.04-ghjeda/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pyminuit2/0.0.1-ghjeda4/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-PyYAML/3.11-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-pygithub/1.23.0-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-pip/9.0.1-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-dablooms/0.9.1-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-pippkgs_depscipy/3.0-ghjeda7/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-pippkgs/5.0-ghjeda6/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/professor/1.4.0-ghjeda4/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/dbs-client/DBS_2_1_9-mlhled2/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/dbs-client/DBS_2_1_9-mlhled2/lib/DBSAPI:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/cvs2git/5419-mlhled2/lib == env: VC3_MACHINE_TARGET=x86_64/redhat7.4.1708 == env: VC3_ROOT_SED=/usr/ == env: OASIS_CERTIFICATES=/cvmfs/oasis.opensciencegrid.org/mis/certificates == env: ROOTSYS=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/lcg/root/6.10.04-ghjeda/ == env: SRT_PYTHIA8DATA_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pythia8/230-ghjeda/share/Pythia8/xmldoc == env: SRT_HERWIGPATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/herwigpp/2.7.1-ghjeda/share/Herwig++ == env: CXX=g++ == env: SRT_QTLIB_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/lib == env: SHERPA_LIBRARY_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/lib/SHERPA-MC == env: HOME=/disk/vc3-root/vc3-home == env: SHLVL=3 == env: CMSSW_SEARCH_PATH=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/src:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/external/slc7_amd64_gcc630/data:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/src:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/external/slc7_amd64_gcc630/data == env: VC3_VERSION_FULL_GZIP=1.5.0 == env: VC3_VERSION_MAKE=3.82 == env: DISK=1000 == env: PARROT_CACHE=/disk/vc3-root/tmp/worker-230547-7899/cache/tmp == env: PARROT_CVMFS_ALIEN_CACHE=/disk/vc3-root/tmp/worker-230547-7899/cache/tmp/cvmfs == env: ROOT_TTREECACHE_PREFILL=0 == env: GENREFLEX=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/lcg/root/6.10.04-ghjeda/bin/genreflex == env: VINCIADATA=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/vincia/1.2.02-ghjeda3/share/Vincia/xmldoc == env: SRT_G4LEDATA_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4EMLOW/6.48/data/G4EMLOW6.48 == env: SRT_GENREFLEX_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/lcg/root/6.10.04-ghjeda/bin/genreflex == env: MALLOC_CONF=lg_chunk:18,lg_dirty_mult:4 == env: SRT_ROOT_TTREECACHE_SIZE_SCRAMRTDEL=0 == env: SRT_EVTGENDATA_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/evtgen/1.6.0-ghjeda3/share == env: GFAL_CONFIG_DIR=/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/etc/gfal2.d/ == env: VC3_VERSION_FULL_WHICH=2.20.0 == env: MEMORY=1200 == env: HTTP_PROXY=http://eddie.crc.nd.edu:3128 == env: SRT_CMSSW_DATA_PATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms == env: HEPPDT_PARAM_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/heppdt/3.03.00 == env: SRT_LD_LIBRARY_PATH_SCRAMRT=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/biglib/slc7_amd64_gcc630:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/external/slc7_amd64_gcc630/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/biglib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/external/slc7_amd64_gcc630/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/llvm/4.0.0-njopjo/lib64:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/lib64:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/gcc/6.3.0/lib == env: TMP=/disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS == env: PYTHONPATH=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/python:/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/python:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_6/lib/slc7_amd64_gcc630:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/coral/CORAL_2_3_21-njopjo6/slc7_amd64_gcc630/python:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/coral/CORAL_2_3_21-njopjo6/slc7_amd64_gcc630/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/professor2/2.2.1-ghjeda6/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pyqt/4.11.4-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/rivet/2.5.2-ghjeda4/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/python-ldap/2.4.10-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-matplotlib/1.5.2-njopjo/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sip/4.17-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/llvm/4.0.0-njopjo/lib64/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-sqlalchemy/1.1.4-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-lint/0.25.1-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-dxr/1.0-ghjeda6/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-cx-oracle/5.2.1-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-numpy/1.12.1-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/frontier_client/2.8.20-mlhled2/python/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/das_client/v03.01.00-mlhled2/bin:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/xrootd/4.6.1/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/yoda/1.6.5-ghjeda4/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/lcg/root/6.10.04-ghjeda/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pyminuit2/0.0.1-ghjeda4/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-PyYAML/3.11-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-pygithub/1.23.0-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-pip/9.0.1-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-dablooms/0.9.1-mlhled2/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-pippkgs_depscipy/3.0-ghjeda7/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/py2-pippkgs/5.0-ghjeda6/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/professor/1.4.0-ghjeda4/lib/python2.7/site-packages:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/dbs-client/DBS_2_1_9-mlhled2/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/dbs-client/DBS_2_1_9-mlhled2/lib/DBSAPI:/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/cvs2git/5419-mlhled2/lib:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib/python2.7/site-packages:/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/lib64/python2.7/site-packages:python:python: == env: QTLIB=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/qt/4.8.7/lib == env: SCONS_LIB_DIR=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/scons/1.2.0-mlhled2/lib == env: VC3_INSTALL_USER_HOME=/disk/vc3-root/vc3-home == env: VC3_VERSION_FULL_VC3_BUILDER_HANDS=1.0.0 == env: FRONTIER_CLIENT=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/frontier_client/2.8.20-mlhled2/ == env: SRT_G4NEUTRONXSDATA_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4NEUTRONXS/1.4/data/G4NEUTRONXS1.4 == env: SRT_SHERPA_LIBRARY_PATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/lib/SHERPA-MC == env: VC3_ROOT_TAR=/usr/ == env: VC3_VERSION_PKG_CONFIG=0.27 == env: CLHEP_PARAM_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/clhep/2.3.4.2-mlhled2 == env: CCTOOLS_ND=lobster-171-cd5e3e2c-cvmfs-70dfa0d6 == env: VC3_SH_PROFILE_WRAPPER=/disk/vc3-root/vc3-home/.vc3_sh-8C5grX.wrapper == env: HERWIGPATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/herwigpp/2.7.1-ghjeda/share/Herwig++ == env: PYTHIA8DATA=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/pythia8/230-ghjeda/share/Pythia8/xmldoc == env: SRT_FRONTIER_CLIENT_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/frontier_client/2.8.20-mlhled2/ == env: SRT_CSC_TRACK_FINDER_DATA_DIR_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/cms/CSCTrackFinderEmulation/1.2/data/ == env: SRT_ROOT_PATH_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/madgraph5amcatnlo/2.4.3-ghjeda6 == env: CC=gcc == env: TEXMFHOME=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/rivet/2.5.2-ghjeda4/share/Rivet/texmf == env: SRT_OPENBLAS_NUM_THREADS_SCRAMRTDEL=1 == env: SRT_GIT_TEMPLATE_DIR_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/git/1.8.3.1/share/git-core/templates == env: SRT_G4SAIDXSDATA_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/geant4-G4SAIDDATA/1.1/data/G4SAIDDATA1.1 == env: CMSSW_VERSION=CMSSW_9_3_6 == env: SRT_LANG_SCRAMRTDEL=C == env: CCTOOLS_RESOURCE_MONITOR_INFO=1033 == env: VC3_VERSION_FULL_MAKE=3.82.0 == env: CORES=1 == env: PARROT_DEBUG_FLAGS= == env: SHERPA_SHARE_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/sherpa/2.2.4-ghjeda2/share/SHERPA-MC == env: VC3_MACHINE_ARCH=x86_64 == env: VC3_ROOT_VC3_BUILDER_HANDS=/disk/vc3-root/x86_64/redhat7.4.1708/vc3-builder-hands/v1.0.0 == env: CMS_OPENLOOPS_PREFIX=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/openloops/1.3.1-mlhled2 == env: GIT_EXEC_PATH=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/git/1.8.3.1/libexec/git-core == env: LOCALRT=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6 == env: CMSSW_BASE=/disk/vc3-root/tmp/worker-230547-7899/t.43/CMSSW_9_3_6 == env: SRT_VALGRIND_LIB_SCRAMRTDEL=/cvmfs/cms.cern.ch/slc7_amd64_gcc630/external/valgrind/3.12.0/lib/valgrind == env: VC3_COMMAND_LINE=/disk/vc3-builder --var CCTOOLS_ND=lobster-171-cd5e3e2c-cvmfs-70dfa0d6 --require cctools-wq-worker -- work_queue_worker -M lobster_kmohrman_EFT_ALL_genOnly.* -t 300 -C catalog.cse.nd.edu,backup-catalog.cse.nd.edu -d all -o worker.log --cores=12 --memory=16000 --disk=17780 == env: _=/usr/bin/env === wrapper ready @ Sun Aug 16 18:51:55 EDT 2020 ==== working directory before execution @ Sun Aug 16 18:51:55 EDT 2020 ==== == dir: total 868 == dir: drwxr-xr-x 17 kmohrman campus 201 Aug 16 18:51 CMSSW_9_3_6 == dir: -rw-r--r-- 9 kmohrman campus 8037 Aug 16 18:50 GEN-00000-ttHJets_1_base-0_cfg.py == dir: drwxr-xr-x 2 kmohrman campus 71 Aug 16 18:50 bin == dir: -rwxr-xr-x 11 kmohrman campus 627272 Aug 16 18:50 cctools-monitor == dir: -rw-r--r-- 1 kmohrman campus 0 Aug 16 18:50 cctools-monitor.summary == dir: drwxrwxrwx 2 kmohrman campus 6 Aug 16 18:50 cctools-temp-t.43.YBXcWS == dir: -rw------- 1 kmohrman campus 51 Aug 16 18:51 cvmfschecksum.cms-ib.cern.ch == dir: -rw------- 1 kmohrman campus 51 Aug 16 18:51 cvmfschecksum.cms.cern.ch == dir: -rw------- 1 kmohrman campus 51 Aug 16 18:51 cvmfschecksum.oasis.opensciencegrid.org == dir: drwxr-xr-x 2 kmohrman campus 33 Aug 16 18:50 lib == dir: -rwxrwxrwx 1 kmohrman campus 146912 Aug 16 18:50 librmonitor_helper.so.3N9QXP == dir: -rw-r--r-- 2 kmohrman campus 2010 Aug 16 18:50 parameters.json == dir: -rw------- 2 kmohrman campus 9884 Aug 16 18:50 proxy == dir: drwxr-xr-x 3 kmohrman campus 20 Aug 16 18:50 python == dir: -rw-r--r-- 11 kmohrman campus 570 Aug 16 18:50 report.json.in == dir: -rw-r--r-- 11 kmohrman campus 449 Aug 16 18:50 sandbox-CMSSW_9_3_6-slc7_amd64_gcc630.tar.bz2 == dir: drwxr-xr-x 4 kmohrman campus 37 Aug 16 18:50 siteconf == dir: -rw-r--r-- 1 kmohrman campus 11 Aug 16 18:51 t_wrapper_ready == dir: -rw-r--r-- 1 kmohrman campus 11 Aug 16 18:51 t_wrapper_start == dir: -rwxr-xr-x 11 kmohrman campus 47217 Aug 16 18:50 task.py == dir: -rwxr-xr-x 11 kmohrman campus 5986 Aug 16 18:50 wrapper.sh >>> data is {u'files': {u'info': {}, u'output_info': {}, u'skipped': []}, u'cpu_time': 0, u'exe_exit_code': 0, u'events_written': 0, u'output_size': 0, u'task_exit_code': 0, u'output_storage_element': u'', u'cache': {u'start_size': 0, u'type': 2, u'end_size': 0}, u'stageout_exit_code': 0, u'output_bare_size': 0, u'transfers': defaultdict(, {}), u'task_timing': {u'wrapper_ready': 0, u'wrapper_start': 0, u'prologue_end': 0, u'epilogue_end': 0, u'stage_out_end': 0, u'stage_in_end': 0, u'processing_end': 0}, u'events_per_run': 0} @ Sun Aug 16 18:52:12 2020 >>> Trying hdfs client access method @ Sun Aug 16 18:52:12 2020 >>> executing 'timeout 300 hdfs dfs -fs hdfs://eddie.crc.nd.edu:19000 -get /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_157.root HIG-RunIIFall17wmLHE-00000ND_157.root' @ Sun Aug 16 18:52:12 2020 >> using /disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS/tmpGcg5p9 to store command output >> cmd: timeout: failed to run command 'hdfs': No such file or directory >>>>> Unable to copy input with hdfs client @ Sun Aug 16 18:52:12 2020 >>> Trying xrootd access method @ Sun Aug 16 18:52:12 2020 >>> executing 'env XRD_LOGLEVEL=Debug timeout 300 xrdfs deepthought.crc.nd.edu stat /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_157.root' @ Sun Aug 16 18:52:12 2020 >> using /disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS/tmpPozN7x to store command output >> cmd: [2020-08-16 18:52:13.481690 -0400][Warning][Utility ] Unable to process global config file: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:52:13.483709 -0400][Debug ][Utility ] Unable to process user config file: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:52:13.484060 -0400][Debug ][PlugInMgr ] Initializing plug-in manager... >> cmd: [2020-08-16 18:52:13.484141 -0400][Debug ][PlugInMgr ] No default plug-in, loading plug-in configs... >> cmd: [2020-08-16 18:52:13.484219 -0400][Debug ][PlugInMgr ] Processing plug-in definitions in /etc/xrootd/client.plugins.d... >> cmd: [2020-08-16 18:52:13.484374 -0400][Debug ][PlugInMgr ] Unable to process directory /etc/xrootd/client.plugins.d: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:52:13.484969 -0400][Debug ][PlugInMgr ] Processing plug-in definitions in /tmp/runos_singularity_home-228402745628656325426833164096691803151/.xrootd/client.plugins.d... >> cmd: [2020-08-16 18:52:13.485125 -0400][Debug ][PlugInMgr ] Unable to process directory /tmp/runos_singularity_home-228402745628656325426833164096691803151/.xrootd/client.plugins.d: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:52:13.485548 -0400][Debug ][App ] Executing: stat /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_157.root >> cmd: [2020-08-16 18:52:13.485769 -0400][Debug ][Poller ] Available pollers: built-in >> cmd: [2020-08-16 18:52:13.485843 -0400][Debug ][Poller ] Attempting to create a poller according to preference: built-in >> cmd: [2020-08-16 18:52:13.485912 -0400][Debug ][Poller ] Creating poller: built-in >> cmd: [2020-08-16 18:52:13.485995 -0400][Debug ][Poller ] Creating and starting the built-in poller... >> cmd: [2020-08-16 18:52:13.486702 -0400][Debug ][Poller ] Using 1 poller threads >> cmd: [2020-08-16 18:52:13.486892 -0400][Debug ][TaskMgr ] Starting the task manager... >> cmd: [2020-08-16 18:52:13.487331 -0400][Debug ][TaskMgr ] Task manager started >> cmd: [2020-08-16 18:52:13.487539 -0400][Debug ][JobMgr ] Starting the job manager... >> cmd: [2020-08-16 18:52:13.488704 -0400][Debug ][JobMgr ] Job manager started, 3 workers >> cmd: [2020-08-16 18:52:13.488873 -0400][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2020-08-16 18:52:13 -0400] >> cmd: [2020-08-16 18:52:13.489089 -0400][Debug ][PostMaster ] Creating new channel to: deepthought.crc.nd.edu:1094 1 stream(s) >> cmd: [2020-08-16 18:52:13.489264 -0400][Debug ][PostMaster ] [deepthought.crc.nd.edu:1094 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Widnow: 1800 >> cmd: [2020-08-16 18:52:13.491678 -0400][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: deepthought.crc.nd.edu:1094" to be run at: [2020-08-16 18:52:28 -0400] >> cmd: [2020-08-16 18:52:13.494291 -0400][Debug ][PostMaster ] [deepthought.crc.nd.edu:1094] Found 1 address(es): [::ffff:129.74.85.55]:1094 >> cmd: [2020-08-16 18:52:13.494701 -0400][Debug ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Attempting connection to [::ffff:129.74.85.55]:1094 >> cmd: [2020-08-16 18:52:13.494945 -0400][Debug ][Poller ] Adding socket 0x11f48a0 to the poller >> cmd: [2020-08-16 18:52:13.496061 -0400][Debug ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Async connection call returned >> cmd: [2020-08-16 18:52:13.496403 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Sending out the initial hand shake + kXR_protocol >> cmd: [2020-08-16 18:52:13.498332 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Got the server hand shake response (type: manager [], protocol version 400) >> cmd: [2020-08-16 18:52:13.498876 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] kXR_protocol successful (type: manager [], protocol version 400) >> cmd: [2020-08-16 18:52:13.505682 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Sending out kXR_login request, username: kmohrman, cgi: ?xrd.cc=us&xrd.tz=-5&xrd.appname=xrdfs&xrd.info=&xrd.hostname=qa-v100-001.crc.nd.edu&xrd.rn=v4.6.1, dual-stack: false, private IPv4: true, private IPv6: false >> cmd: [2020-08-16 18:52:13.507100 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Logged in, session: 296a0000c00d000001080000a56b0000 >> cmd: [2020-08-16 18:52:13.507363 -0400][Debug ][PostMaster ] [deepthought.crc.nd.edu:1094 #0] Stream 0 connected. >> cmd: [2020-08-16 18:52:13.507488 -0400][Debug ][Utility ] Monitor library name not set. No monitoring >> cmd: [2020-08-16 18:52:13.510446 -0400][Debug ][PostMaster ] Creating new channel to: primeradiant03.crc.nd.edu:1096 1 stream(s) >> cmd: [2020-08-16 18:52:13.510619 -0400][Debug ][PostMaster ] [primeradiant03.crc.nd.edu:1096 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Widnow: 1800 >> cmd: [2020-08-16 18:52:13.513459 -0400][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: primeradiant03.crc.nd.edu:1096" to be run at: [2020-08-16 18:52:28 -0400] >> cmd: [2020-08-16 18:52:13.515618 -0400][Debug ][PostMaster ] [primeradiant03.crc.nd.edu:1096] Found 1 address(es): [::ffff:149.165.243.11]:1096 >> cmd: [2020-08-16 18:52:13.516083 -0400][Debug ][AsyncSock ] [primeradiant03.crc.nd.edu:1096 #0.0] Attempting connection to [::ffff:149.165.243.11]:1096 >> cmd: [2020-08-16 18:52:13.516331 -0400][Debug ][Poller ] Adding socket 0xb40016a0 to the poller >> cmd: [2020-08-16 18:52:13.516663 -0400][Debug ][AsyncSock ] [primeradiant03.crc.nd.edu:1096 #0.0] Async connection call returned >> cmd: [2020-08-16 18:52:13.516965 -0400][Debug ][XRootDTransport ] [primeradiant03.crc.nd.edu:1096 #0.0] Sending out the initial hand shake + kXR_protocol >> cmd: [2020-08-16 18:52:13.517953 -0400][Debug ][XRootDTransport ] [primeradiant03.crc.nd.edu:1096 #0.0] Got the server hand shake response (type: server [], protocol version 400) >> cmd: [2020-08-16 18:52:13.518462 -0400][Debug ][XRootDTransport ] [primeradiant03.crc.nd.edu:1096 #0.0] kXR_protocol successful (type: server [], protocol version 400) >> cmd: [2020-08-16 18:52:13.524427 -0400][Debug ][XRootDTransport ] [primeradiant03.crc.nd.edu:1096 #0.0] Sending out kXR_login request, username: kmohrman, cgi: ?xrd.cc=us&xrd.tz=-5&xrd.appname=xrdfs&xrd.info=&xrd.hostname=qa-v100-001.crc.nd.edu&xrd.rn=v4.6.1, dual-stack: false, private IPv4: true, private IPv6: false >> cmd: [2020-08-16 18:52:13.525455 -0400][Debug ][XRootDTransport ] [primeradiant03.crc.nd.edu:1096 #0.0] Logged in, session: b6350000887a0000820f0000b6350000 >> cmd: [2020-08-16 18:52:13.525646 -0400][Debug ][XRootDTransport ] [primeradiant03.crc.nd.edu:1096 #0.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:ba240aa8.0|f5f0dfc2.0 >> cmd: [2020-08-16 18:52:13.525777 -0400][Debug ][XRootDTransport ] [primeradiant03.crc.nd.edu:1096 #0.0] Sending authentication data >> cmd: [2020-08-16 18:52:13.543044 -0400][Debug ][XRootDTransport ] [primeradiant03.crc.nd.edu:1096 #0.0] Trying to authenticate using gsi >> cmd: [2020-08-16 18:52:13.710812 -0400][Debug ][XRootDTransport ] [primeradiant03.crc.nd.edu:1096 #0.0] Sending more authentication data for gsi >> cmd: [2020-08-16 18:52:13.735163 -0400][Debug ][XRootDTransport ] [primeradiant03.crc.nd.edu:1096 #0.0] Authenticated with gsi. >> cmd: [2020-08-16 18:52:13.735415 -0400][Debug ][PostMaster ] [primeradiant03.crc.nd.edu:1096 #0] Stream 0 connected. >> cmd: [2020-08-16 18:53:14.036759 -0400][Debug ][AsyncSock ] [primeradiant03.crc.nd.edu:1096 #0.0] Closing the socket >> cmd: [2020-08-16 18:53:14.037016 -0400][Debug ][Poller ] <[::ffff:10.32.88.127]:59696><--><[::ffff:149.165.243.11]:1096> Removing socket from the poller >> cmd: [2020-08-16 18:53:14.037429 -0400][Debug ][PostMaster ] [primeradiant03.crc.nd.edu:1096 #0] Recovering error for stream #0: [ERROR] Socket error. >> cmd: [2020-08-16 18:53:14.037559 -0400][Debug ][PostMaster ] [primeradiant03.crc.nd.edu:1096 #0] Reporting disconnection to queued message handlers. >> cmd: [2020-08-16 18:53:14.037705 -0400][Debug ][XRootD ] [primeradiant03.crc.nd.edu:1096] Handling error while processing kXR_stat (path: /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_157.root, flags: none): [ERROR] Socket error. >> cmd: [2020-08-16 18:53:20.761145 -0400][Debug ][TaskMgr ] Registering task: "WaitTask for: 0x0x11eeea0" to be run at: [2020-08-16 18:53:35 -0400] >> cmd: [2020-08-16 18:53:35.511869 -0400][Debug ][TaskMgr ] Done with task: "WaitTask for: 0x0x11eeea0" >> cmd: [2020-08-16 18:53:35.514272 -0400][Debug ][PostMaster ] Creating new channel to: primeradiant04.crc.nd.edu:1096 1 stream(s) >> cmd: [2020-08-16 18:53:35.514469 -0400][Debug ][PostMaster ] [primeradiant04.crc.nd.edu:1096 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Widnow: 1800 >> cmd: [2020-08-16 18:53:35.518114 -0400][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: primeradiant04.crc.nd.edu:1096" to be run at: [2020-08-16 18:53:50 -0400] >> cmd: [2020-08-16 18:53:35.520622 -0400][Debug ][PostMaster ] [primeradiant04.crc.nd.edu:1096] Found 1 address(es): [::ffff:149.165.243.12]:1096 >> cmd: [2020-08-16 18:53:35.520988 -0400][Debug ][AsyncSock ] [primeradiant04.crc.nd.edu:1096 #0.0] Attempting connection to [::ffff:149.165.243.12]:1096 >> cmd: [2020-08-16 18:53:35.521218 -0400][Debug ][Poller ] Adding socket 0xc0001540 to the poller >> cmd: [2020-08-16 18:55:36.083964 -0400][Error ][AsyncSock ] [primeradiant04.crc.nd.edu:1096 #0.0] Socket error while handshaking: [ERROR] Socket timeout >> cmd: [2020-08-16 18:55:36.084207 -0400][Debug ][AsyncSock ] [primeradiant04.crc.nd.edu:1096 #0.0] Closing the socket >> cmd: [2020-08-16 18:55:36.084351 -0400][Debug ][Poller ] <--> Removing socket from the poller >> cmd: [2020-08-16 18:55:36.087300 -0400][Debug ][PostMaster ] [primeradiant04.crc.nd.edu:1096] Found 1 address(es): [::ffff:149.165.243.12]:1096 >> cmd: [2020-08-16 18:55:36.087718 -0400][Debug ][AsyncSock ] [primeradiant04.crc.nd.edu:1096 #0.0] Attempting connection to [::ffff:149.165.243.12]:1096 >> cmd: [2020-08-16 18:55:36.087973 -0400][Debug ][Poller ] Adding socket 0xc0001540 to the poller >> cmd: [2020-08-16 18:55:37.089981 -0400][Debug ][AsyncSock ] [primeradiant04.crc.nd.edu:1096 #0.0] Async connection call returned >> cmd: [2020-08-16 18:55:37.090401 -0400][Debug ][XRootDTransport ] [primeradiant04.crc.nd.edu:1096 #0.0] Sending out the initial hand shake + kXR_protocol >>> xrootd access to input file unavailable @ Sun Aug 16 18:57:12 2020 >>> Trying srm access method @ Sun Aug 16 18:57:12 2020 >>> executing '/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/bin/gfal-copy gsiftp://deepthought.crc.nd.edu/store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_157.root HIG-RunIIFall17wmLHE-00000ND_157.root' @ Sun Aug 16 18:57:12 2020 >> using /disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS/tmp1hfIfR to store command output >> cmd: gfal-copy error: 13 (Permission denied) - Could not stat the source: globus_ftp_client: the server responded with an error 530 530-Login incorrect. : /builddir/build/BUILD/gridftp-hdfs-1.1.1/src/gridftp_hdfs.c:hdfs_start:756: 530-Server is cancelling transfer due to over-load limit (host=deepthought.crc.nd.edu, user=kmohrman, path=(null)) 530 End. >>>>> Unable to copy input with SRM @ Sun Aug 16 18:57:20 2020 >>> Trying srm access method @ Sun Aug 16 18:57:20 2020 >>> executing '/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.4/3.4.53/el7-x86_64/usr/bin/gfal-copy gsiftp://deepthought.crc.nd.edu/store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_157.root HIG-RunIIFall17wmLHE-00000ND_157.root' @ Sun Aug 16 18:57:20 2020 >> using /disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS/tmpbWz8OQ to store command output >> cmd: gfal-copy error: 13 (Permission denied) - Could not stat the source: globus_ftp_client: the server responded with an error 530 530-Login incorrect. : /builddir/build/BUILD/gridftp-hdfs-1.1.1/src/gridftp_hdfs.c:hdfs_start:756: 530-Server is cancelling transfer due to over-load limit (host=deepthought.crc.nd.edu, user=kmohrman, path=(null)) 530 End. >>>>> Unable to copy input with SRM @ Sun Aug 16 18:57:25 2020 >>>>>> no stage out method succeeded for: kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_157.root @ Sun Aug 16 18:57:25 2020 >>> Trying hdfs client access method @ Sun Aug 16 18:57:25 2020 >>> executing 'timeout 300 hdfs dfs -fs hdfs://eddie.crc.nd.edu:19000 -get /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root HIG-RunIIFall17wmLHE-00000ND_888.root' @ Sun Aug 16 18:57:25 2020 >> using /disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS/tmpgXPQf5 to store command output >> cmd: timeout: failed to run command 'hdfs': No such file or directory >>>>> Unable to copy input with hdfs client @ Sun Aug 16 18:57:25 2020 >>> Trying xrootd access method @ Sun Aug 16 18:57:25 2020 >>> executing 'env XRD_LOGLEVEL=Debug timeout 300 xrdfs deepthought.crc.nd.edu stat /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root' @ Sun Aug 16 18:57:25 2020 >> using /disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS/tmpmqDohB to store command output >> cmd: [2020-08-16 18:57:25.917898 -0400][Warning][Utility ] Unable to process global config file: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:57:25.919705 -0400][Debug ][Utility ] Unable to process user config file: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:57:25.920027 -0400][Debug ][PlugInMgr ] Initializing plug-in manager... >> cmd: [2020-08-16 18:57:25.920104 -0400][Debug ][PlugInMgr ] No default plug-in, loading plug-in configs... >> cmd: [2020-08-16 18:57:25.920179 -0400][Debug ][PlugInMgr ] Processing plug-in definitions in /etc/xrootd/client.plugins.d... >> cmd: [2020-08-16 18:57:25.920324 -0400][Debug ][PlugInMgr ] Unable to process directory /etc/xrootd/client.plugins.d: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:57:25.920898 -0400][Debug ][PlugInMgr ] Processing plug-in definitions in /tmp/runos_singularity_home-228402745628656325426833164096691803151/.xrootd/client.plugins.d... >> cmd: [2020-08-16 18:57:25.921041 -0400][Debug ][PlugInMgr ] Unable to process directory /tmp/runos_singularity_home-228402745628656325426833164096691803151/.xrootd/client.plugins.d: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:57:25.921451 -0400][Debug ][App ] Executing: stat /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root >> cmd: [2020-08-16 18:57:25.921666 -0400][Debug ][Poller ] Available pollers: built-in >> cmd: [2020-08-16 18:57:25.921737 -0400][Debug ][Poller ] Attempting to create a poller according to preference: built-in >> cmd: [2020-08-16 18:57:25.921803 -0400][Debug ][Poller ] Creating poller: built-in >> cmd: [2020-08-16 18:57:25.921882 -0400][Debug ][Poller ] Creating and starting the built-in poller... >> cmd: [2020-08-16 18:57:25.922565 -0400][Debug ][Poller ] Using 1 poller threads >> cmd: [2020-08-16 18:57:25.922663 -0400][Debug ][TaskMgr ] Starting the task manager... >> cmd: [2020-08-16 18:57:25.922913 -0400][Debug ][TaskMgr ] Task manager started >> cmd: [2020-08-16 18:57:25.922990 -0400][Debug ][JobMgr ] Starting the job manager... >> cmd: [2020-08-16 18:57:25.923590 -0400][Debug ][JobMgr ] Job manager started, 3 workers >> cmd: [2020-08-16 18:57:25.923687 -0400][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2020-08-16 18:57:25 -0400] >> cmd: [2020-08-16 18:57:25.923831 -0400][Debug ][PostMaster ] Creating new channel to: deepthought.crc.nd.edu:1094 1 stream(s) >> cmd: [2020-08-16 18:57:25.923947 -0400][Debug ][PostMaster ] [deepthought.crc.nd.edu:1094 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Widnow: 1800 >> cmd: [2020-08-16 18:57:25.925641 -0400][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: deepthought.crc.nd.edu:1094" to be run at: [2020-08-16 18:57:40 -0400] >> cmd: [2020-08-16 18:57:25.927432 -0400][Debug ][PostMaster ] [deepthought.crc.nd.edu:1094] Found 1 address(es): [::ffff:129.74.85.55]:1094 >> cmd: [2020-08-16 18:57:25.927679 -0400][Debug ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Attempting connection to [::ffff:129.74.85.55]:1094 >> cmd: [2020-08-16 18:57:25.927843 -0400][Debug ][Poller ] Adding socket 0x67f8a0 to the poller >> cmd: [2020-08-16 18:57:25.928952 -0400][Debug ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Async connection call returned >> cmd: [2020-08-16 18:57:25.929228 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Sending out the initial hand shake + kXR_protocol >> cmd: [2020-08-16 18:57:25.931287 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Got the server hand shake response (type: manager [], protocol version 400) >> cmd: [2020-08-16 18:57:25.931753 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] kXR_protocol successful (type: manager [], protocol version 400) >> cmd: [2020-08-16 18:57:25.938497 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Sending out kXR_login request, username: kmohrman, cgi: ?xrd.cc=us&xrd.tz=-5&xrd.appname=xrdfs&xrd.info=&xrd.hostname=qa-v100-001.crc.nd.edu&xrd.rn=v4.6.1, dual-stack: false, private IPv4: true, private IPv6: false >> cmd: [2020-08-16 18:57:25.939956 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Logged in, session: 51700000c00d0000c4000000d2710000 >> cmd: [2020-08-16 18:57:25.940141 -0400][Debug ][PostMaster ] [deepthought.crc.nd.edu:1094 #0] Stream 0 connected. >> cmd: [2020-08-16 18:57:25.940244 -0400][Debug ][Utility ] Monitor library name not set. No monitoring >> cmd: [2020-08-16 18:57:25.944304 -0400][Debug ][PostMaster ] Creating new channel to: primeradiant01.crc.nd.edu:1096 1 stream(s) >> cmd: [2020-08-16 18:57:25.944471 -0400][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1096 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Widnow: 1800 >> cmd: [2020-08-16 18:57:25.947897 -0400][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: primeradiant01.crc.nd.edu:1096" to be run at: [2020-08-16 18:57:40 -0400] >> cmd: [2020-08-16 18:57:25.950214 -0400][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1096] Found 1 address(es): [::ffff:149.165.243.5]:1096 >> cmd: [2020-08-16 18:57:25.950568 -0400][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1096 #0.0] Attempting connection to [::ffff:149.165.243.5]:1096 >> cmd: [2020-08-16 18:57:25.950811 -0400][Debug ][Poller ] Adding socket 0x300016a0 to the poller >> cmd: [2020-08-16 18:57:25.951121 -0400][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1096 #0.0] Async connection call returned >> cmd: [2020-08-16 18:57:25.951431 -0400][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1096 #0.0] Sending out the initial hand shake + kXR_protocol >> cmd: [2020-08-16 18:57:25.952474 -0400][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1096 #0.0] Got the server hand shake response (type: server [], protocol version 400) >> cmd: [2020-08-16 18:57:25.953023 -0400][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1096 #0.0] kXR_protocol successful (type: server [], protocol version 400) >> cmd: [2020-08-16 18:57:25.959090 -0400][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1096 #0.0] Sending out kXR_login request, username: kmohrman, cgi: ?xrd.cc=us&xrd.tz=-5&xrd.appname=xrdfs&xrd.info=&xrd.hostname=qa-v100-001.crc.nd.edu&xrd.rn=v4.6.1, dual-stack: false, private IPv4: true, private IPv6: false >> cmd: [2020-08-16 18:57:25.960035 -0400][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1096 #0.0] Logged in, session: eee20000242b00006e0c0000eee20000 >> cmd: [2020-08-16 18:57:25.960168 -0400][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1096 #0.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:ba240aa8.0|f5f0dfc2.0 >> cmd: [2020-08-16 18:57:25.960300 -0400][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1096 #0.0] Sending authentication data >> cmd: [2020-08-16 18:57:25.974920 -0400][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1096 #0.0] Trying to authenticate using gsi >> cmd: [2020-08-16 18:57:26.115861 -0400][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1096 #0.0] Sending more authentication data for gsi >> cmd: [2020-08-16 18:57:26.137397 -0400][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1096 #0.0] Authenticated with gsi. >> cmd: [2020-08-16 18:57:26.137673 -0400][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1096 #0] Stream 0 connected. >> cmd: [2020-08-16 18:58:26.356165 -0400][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1096 #0.0] Closing the socket >> cmd: [2020-08-16 18:58:26.356428 -0400][Debug ][Poller ] <[::ffff:10.32.88.127]:58448><--><[::ffff:149.165.243.5]:1096> Removing socket from the poller >> cmd: [2020-08-16 18:58:26.356860 -0400][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1096 #0] Recovering error for stream #0: [ERROR] Socket error. >> cmd: [2020-08-16 18:58:26.357001 -0400][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1096 #0] Reporting disconnection to queued message handlers. >> cmd: [2020-08-16 18:58:26.357134 -0400][Debug ][XRootD ] [primeradiant01.crc.nd.edu:1096] Handling error while processing kXR_stat (path: /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root, flags: none): [ERROR] Socket error. >> cmd: [2020-08-16 18:58:28.359253 -0400][Debug ][TaskMgr ] Registering task: "WaitTask for: 0x0x679ea0" to be run at: [2020-08-16 18:58:43 -0400] >> cmd: [2020-08-16 18:58:43.947311 -0400][Debug ][TaskMgr ] Done with task: "WaitTask for: 0x0x679ea0" >> cmd: [2020-08-16 18:58:43.951022 -0400][Debug ][PostMaster ] Creating new channel to: primeradiant02.crc.nd.edu:1096 1 stream(s) >> cmd: [2020-08-16 18:58:43.951265 -0400][Debug ][PostMaster ] [primeradiant02.crc.nd.edu:1096 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Widnow: 1800 >> cmd: [2020-08-16 18:58:43.954946 -0400][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: primeradiant02.crc.nd.edu:1096" to be run at: [2020-08-16 18:58:58 -0400] >> cmd: [2020-08-16 18:58:43.957841 -0400][Debug ][PostMaster ] [primeradiant02.crc.nd.edu:1096] Found 1 address(es): [::ffff:149.165.243.6]:1096 >> cmd: [2020-08-16 18:58:43.958232 -0400][Debug ][AsyncSock ] [primeradiant02.crc.nd.edu:1096 #0.0] Attempting connection to [::ffff:149.165.243.6]:1096 >> cmd: [2020-08-16 18:58:43.958484 -0400][Debug ][Poller ] Adding socket 0x3c001540 to the poller >> cmd: [2020-08-16 18:58:43.958889 -0400][Debug ][AsyncSock ] [primeradiant02.crc.nd.edu:1096 #0.0] Async connection call returned >> cmd: [2020-08-16 18:58:43.959220 -0400][Debug ][XRootDTransport ] [primeradiant02.crc.nd.edu:1096 #0.0] Sending out the initial hand shake + kXR_protocol >> cmd: [2020-08-16 18:58:43.960286 -0400][Debug ][XRootDTransport ] [primeradiant02.crc.nd.edu:1096 #0.0] Got the server hand shake response (type: server [], protocol version 400) >> cmd: [2020-08-16 18:58:43.960848 -0400][Debug ][XRootDTransport ] [primeradiant02.crc.nd.edu:1096 #0.0] kXR_protocol successful (type: server [], protocol version 400) >> cmd: [2020-08-16 18:58:43.966638 -0400][Debug ][XRootDTransport ] [primeradiant02.crc.nd.edu:1096 #0.0] Sending out kXR_login request, username: kmohrman, cgi: ?xrd.cc=us&xrd.tz=-5&xrd.appname=xrdfs&xrd.info=&xrd.hostname=qa-v100-001.crc.nd.edu&xrd.rn=v4.6.1, dual-stack: false, private IPv4: true, private IPv6: false >> cmd: [2020-08-16 18:58:43.967627 -0400][Debug ][XRootDTransport ] [primeradiant02.crc.nd.edu:1096 #0.0] Logged in, session: e8b200008c2e00000c060000e9b20000 >> cmd: [2020-08-16 18:58:43.967763 -0400][Debug ][XRootDTransport ] [primeradiant02.crc.nd.edu:1096 #0.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:ba240aa8.0|f5f0dfc2.0 >> cmd: [2020-08-16 18:58:43.967888 -0400][Debug ][XRootDTransport ] [primeradiant02.crc.nd.edu:1096 #0.0] Sending authentication data >> cmd: [2020-08-16 18:58:43.969979 -0400][Debug ][XRootDTransport ] [primeradiant02.crc.nd.edu:1096 #0.0] Trying to authenticate using gsi >> cmd: [2020-08-16 18:58:44.011385 -0400][Debug ][XRootDTransport ] [primeradiant02.crc.nd.edu:1096 #0.0] Sending more authentication data for gsi >> cmd: [2020-08-16 18:58:44.021970 -0400][Debug ][XRootDTransport ] [primeradiant02.crc.nd.edu:1096 #0.0] Authenticated with gsi. >> cmd: [2020-08-16 18:58:44.022196 -0400][Debug ][PostMaster ] [primeradiant02.crc.nd.edu:1096 #0] Stream 0 connected. >> cmd: Path: /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root >> cmd: Id: 0 >> cmd: Size: 17890517 >> cmd: MTime: 2020-06-18 03:57:10 >> cmd: Flags: 24 (Offline|IsReadable) >> cmd: [2020-08-16 18:58:44.256311 -0400][Debug ][JobMgr ] Stopping the job manager... >> cmd: [2020-08-16 18:58:44.258201 -0400][Debug ][JobMgr ] Job manager stopped >> cmd: [2020-08-16 18:58:44.258362 -0400][Debug ][TaskMgr ] Stopping the task manager... >> cmd: [2020-08-16 18:58:44.259008 -0400][Debug ][TaskMgr ] Task manager stopped >> cmd: [2020-08-16 18:58:44.259141 -0400][Debug ][Poller ] Stopping the poller... >> cmd: [2020-08-16 18:58:44.260113 -0400][Debug ][TaskMgr ] Requesting unregistration of: "TickGeneratorTask for: deepthought.crc.nd.edu:1094" >> cmd: [2020-08-16 18:58:44.260261 -0400][Debug ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Closing the socket >> cmd: [2020-08-16 18:58:44.260388 -0400][Debug ][Poller ] <[::ffff:10.32.88.127]:34294><--><[::ffff:129.74.85.55]:1094> Removing socket from the poller >> cmd: [2020-08-16 18:58:44.260626 -0400][Debug ][PostMaster ] [deepthought.crc.nd.edu:1094 #0] Destroying stream >> cmd: [2020-08-16 18:58:44.260758 -0400][Debug ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Closing the socket >> cmd: [2020-08-16 18:58:44.260895 -0400][Debug ][TaskMgr ] Requesting unregistration of: "TickGeneratorTask for: primeradiant01.crc.nd.edu:1096" >> cmd: [2020-08-16 18:58:44.261020 -0400][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1096 #0.0] Closing the socket >> cmd: [2020-08-16 18:58:44.261145 -0400][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1096 #0] Destroying stream >> cmd: [2020-08-16 18:58:44.261279 -0400][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1096 #0.0] Closing the socket >> cmd: [2020-08-16 18:58:44.261403 -0400][Debug ][TaskMgr ] Requesting unregistration of: "TickGeneratorTask for: primeradiant02.crc.nd.edu:1096" >> cmd: [2020-08-16 18:58:44.261527 -0400][Debug ][AsyncSock ] [primeradiant02.crc.nd.edu:1096 #0.0] Closing the socket >> cmd: [2020-08-16 18:58:44.261651 -0400][Debug ][Poller ] <[::ffff:10.32.88.127]:50802><--><[::ffff:149.165.243.6]:1096> Removing socket from the poller >> cmd: [2020-08-16 18:58:44.261864 -0400][Debug ][PostMaster ] [primeradiant02.crc.nd.edu:1096 #0] Destroying stream >> cmd: [2020-08-16 18:58:44.261988 -0400][Debug ][AsyncSock ] [primeradiant02.crc.nd.edu:1096 #0.0] Closing the socket >>> will stream using xrootd instead of copying @ Sun Aug 16 18:58:44 2020 >>> Trying hdfs client access method @ Sun Aug 16 18:58:44 2020 >>> executing 'timeout 300 hdfs dfs -fs hdfs://eddie.crc.nd.edu:19000 -get /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_527.root HIG-RunIIFall17wmLHE-00000ND_527.root' @ Sun Aug 16 18:58:44 2020 >> using /disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS/tmpDSlyXF to store command output >> cmd: timeout: failed to run command 'hdfs': No such file or directory >>>>> Unable to copy input with hdfs client @ Sun Aug 16 18:58:44 2020 >>> Trying xrootd access method @ Sun Aug 16 18:58:44 2020 >>> executing 'env XRD_LOGLEVEL=Debug timeout 300 xrdfs deepthought.crc.nd.edu stat /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_527.root' @ Sun Aug 16 18:58:44 2020 >> using /disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS/tmpjOiH78 to store command output >> cmd: [2020-08-16 18:58:44.696244 -0400][Warning][Utility ] Unable to process global config file: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:58:44.697992 -0400][Debug ][Utility ] Unable to process user config file: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:58:44.698310 -0400][Debug ][PlugInMgr ] Initializing plug-in manager... >> cmd: [2020-08-16 18:58:44.698383 -0400][Debug ][PlugInMgr ] No default plug-in, loading plug-in configs... >> cmd: [2020-08-16 18:58:44.698457 -0400][Debug ][PlugInMgr ] Processing plug-in definitions in /etc/xrootd/client.plugins.d... >> cmd: [2020-08-16 18:58:44.698614 -0400][Debug ][PlugInMgr ] Unable to process directory /etc/xrootd/client.plugins.d: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:58:44.699162 -0400][Debug ][PlugInMgr ] Processing plug-in definitions in /tmp/runos_singularity_home-228402745628656325426833164096691803151/.xrootd/client.plugins.d... >> cmd: [2020-08-16 18:58:44.699296 -0400][Debug ][PlugInMgr ] Unable to process directory /tmp/runos_singularity_home-228402745628656325426833164096691803151/.xrootd/client.plugins.d: [ERROR] OS Error: No such file or directory >> cmd: [2020-08-16 18:58:44.699726 -0400][Debug ][App ] Executing: stat /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_527.root >> cmd: [2020-08-16 18:58:44.699933 -0400][Debug ][Poller ] Available pollers: built-in >> cmd: [2020-08-16 18:58:44.699999 -0400][Debug ][Poller ] Attempting to create a poller according to preference: built-in >> cmd: [2020-08-16 18:58:44.700065 -0400][Debug ][Poller ] Creating poller: built-in >> cmd: [2020-08-16 18:58:44.700142 -0400][Debug ][Poller ] Creating and starting the built-in poller... >> cmd: [2020-08-16 18:58:44.700861 -0400][Debug ][Poller ] Using 1 poller threads >> cmd: [2020-08-16 18:58:44.700957 -0400][Debug ][TaskMgr ] Starting the task manager... >> cmd: [2020-08-16 18:58:44.701208 -0400][Debug ][TaskMgr ] Task manager started >> cmd: [2020-08-16 18:58:44.701285 -0400][Debug ][JobMgr ] Starting the job manager... >> cmd: [2020-08-16 18:58:44.701949 -0400][Debug ][JobMgr ] Job manager started, 3 workers >> cmd: [2020-08-16 18:58:44.702033 -0400][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2020-08-16 18:58:44 -0400] >> cmd: [2020-08-16 18:58:44.702178 -0400][Debug ][PostMaster ] Creating new channel to: deepthought.crc.nd.edu:1094 1 stream(s) >> cmd: [2020-08-16 18:58:44.702297 -0400][Debug ][PostMaster ] [deepthought.crc.nd.edu:1094 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Widnow: 1800 >> cmd: [2020-08-16 18:58:44.703972 -0400][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: deepthought.crc.nd.edu:1094" to be run at: [2020-08-16 18:58:59 -0400] >> cmd: [2020-08-16 18:58:44.705723 -0400][Debug ][PostMaster ] [deepthought.crc.nd.edu:1094] Found 1 address(es): [::ffff:129.74.85.55]:1094 >> cmd: [2020-08-16 18:58:44.706127 -0400][Debug ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Attempting connection to [::ffff:129.74.85.55]:1094 >> cmd: [2020-08-16 18:58:44.706373 -0400][Debug ][Poller ] Adding socket 0x6bd8a0 to the poller >> cmd: [2020-08-16 18:58:44.707129 -0400][Debug ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Async connection call returned >> cmd: [2020-08-16 18:58:44.707507 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Sending out the initial hand shake + kXR_protocol >> cmd: [2020-08-16 18:58:46.533445 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Got the server hand shake response (type: manager [], protocol version 400) >> cmd: [2020-08-16 18:58:46.534133 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] kXR_protocol successful (type: manager [], protocol version 400) >> cmd: [2020-08-16 18:58:46.542494 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Sending out kXR_login request, username: kmohrman, cgi: ?xrd.cc=us&xrd.tz=-5&xrd.appname=xrdfs&xrd.info=&xrd.hostname=qa-v100-001.crc.nd.edu&xrd.rn=v4.6.1, dual-stack: false, private IPv4: true, private IPv6: false >> cmd: [2020-08-16 18:59:00.320678 -0400][Debug ][XRootDTransport ] [deepthought.crc.nd.edu:1094 #0.0] Logged in, session: 01710000c00d0000140000007b720000 >> cmd: [2020-08-16 18:59:00.321031 -0400][Debug ][PostMaster ] [deepthought.crc.nd.edu:1094 #0] Stream 0 connected. >> cmd: [2020-08-16 18:59:00.321179 -0400][Debug ][Utility ] Monitor library name not set. No monitoring >> cmd: [2020-08-16 18:59:00.429310 -0400][Debug ][PostMaster ] Creating new channel to: primeradiant05.crc.nd.edu:1096 1 stream(s) >> cmd: [2020-08-16 18:59:00.429511 -0400][Debug ][PostMaster ] [primeradiant05.crc.nd.edu:1096 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Widnow: 1800 >> cmd: [2020-08-16 18:59:00.433352 -0400][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: primeradiant05.crc.nd.edu:1096" to be run at: [2020-08-16 18:59:15 -0400] >> cmd: [2020-08-16 18:59:00.436116 -0400][Debug ][PostMaster ] [primeradiant05.crc.nd.edu:1096] Found 1 address(es): [::ffff:149.165.243.13]:1096 >> cmd: [2020-08-16 18:59:00.436508 -0400][Debug ][AsyncSock ] [primeradiant05.crc.nd.edu:1096 #0.0] Attempting connection to [::ffff:149.165.243.13]:1096 >> cmd: [2020-08-16 18:59:00.436763 -0400][Debug ][Poller ] Adding socket 0x800016a0 to the poller >> cmd: [2020-08-16 18:59:00.437090 -0400][Debug ][AsyncSock ] [primeradiant05.crc.nd.edu:1096 #0.0] Async connection call returned >> cmd: [2020-08-16 18:59:00.437431 -0400][Debug ][XRootDTransport ] [primeradiant05.crc.nd.edu:1096 #0.0] Sending out the initial hand shake + kXR_protocol >> cmd: [2020-08-16 18:59:00.438536 -0400][Debug ][XRootDTransport ] [primeradiant05.crc.nd.edu:1096 #0.0] Got the server hand shake response (type: server [], protocol version 400) >> cmd: [2020-08-16 18:59:00.439101 -0400][Debug ][XRootDTransport ] [primeradiant05.crc.nd.edu:1096 #0.0] kXR_protocol successful (type: server [], protocol version 400) >> cmd: [2020-08-16 18:59:00.444765 -0400][Debug ][XRootDTransport ] [primeradiant05.crc.nd.edu:1096 #0.0] Sending out kXR_login request, username: kmohrman, cgi: ?xrd.cc=us&xrd.tz=-5&xrd.appname=xrdfs&xrd.info=&xrd.hostname=qa-v100-001.crc.nd.edu&xrd.rn=v4.6.1, dual-stack: false, private IPv4: true, private IPv6: false >> cmd: [2020-08-16 18:59:00.445747 -0400][Debug ][XRootDTransport ] [primeradiant05.crc.nd.edu:1096 #0.0] Logged in, session: 693500008f1800007f06000069350000 >> cmd: [2020-08-16 18:59:00.445883 -0400][Debug ][XRootDTransport ] [primeradiant05.crc.nd.edu:1096 #0.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:ba240aa8.0|f5f0dfc2.0 >> cmd: [2020-08-16 18:59:00.446019 -0400][Debug ][XRootDTransport ] [primeradiant05.crc.nd.edu:1096 #0.0] Sending authentication data >> cmd: [2020-08-16 18:59:00.462651 -0400][Debug ][XRootDTransport ] [primeradiant05.crc.nd.edu:1096 #0.0] Trying to authenticate using gsi >> cmd: [2020-08-16 18:59:00.584041 -0400][Debug ][XRootDTransport ] [primeradiant05.crc.nd.edu:1096 #0.0] Sending more authentication data for gsi >> cmd: [2020-08-16 18:59:00.603682 -0400][Debug ][XRootDTransport ] [primeradiant05.crc.nd.edu:1096 #0.0] Authenticated with gsi. >> cmd: [2020-08-16 18:59:00.603990 -0400][Debug ][PostMaster ] [primeradiant05.crc.nd.edu:1096 #0] Stream 0 connected. >> cmd: Path: /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_527.root >> cmd: Id: 0 >> cmd: Size: 17892110 >> cmd: MTime: 2020-06-18 03:44:21 >> cmd: Flags: 24 (Offline|IsReadable) >> cmd: [2020-08-16 18:59:44.336380 -0400][Debug ][JobMgr ] Stopping the job manager... >> cmd: [2020-08-16 18:59:44.338510 -0400][Debug ][JobMgr ] Job manager stopped >> cmd: [2020-08-16 18:59:44.338733 -0400][Debug ][TaskMgr ] Stopping the task manager... >> cmd: [2020-08-16 18:59:44.339478 -0400][Debug ][TaskMgr ] Task manager stopped >> cmd: [2020-08-16 18:59:44.339627 -0400][Debug ][Poller ] Stopping the poller... >> cmd: [2020-08-16 18:59:44.340619 -0400][Debug ][TaskMgr ] Requesting unregistration of: "TickGeneratorTask for: deepthought.crc.nd.edu:1094" >> cmd: [2020-08-16 18:59:44.340770 -0400][Debug ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Closing the socket >> cmd: [2020-08-16 18:59:44.340904 -0400][Debug ][Poller ] <[::ffff:10.32.88.127]:34338><--><[::ffff:129.74.85.55]:1094> Removing socket from the poller >> cmd: [2020-08-16 18:59:44.341134 -0400][Debug ][PostMaster ] [deepthought.crc.nd.edu:1094 #0] Destroying stream >> cmd: [2020-08-16 18:59:44.341260 -0400][Debug ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Closing the socket >> cmd: [2020-08-16 18:59:44.341393 -0400][Debug ][TaskMgr ] Requesting unregistration of: "TickGeneratorTask for: primeradiant05.crc.nd.edu:1096" >> cmd: [2020-08-16 18:59:44.341516 -0400][Debug ][AsyncSock ] [primeradiant05.crc.nd.edu:1096 #0.0] Closing the socket >> cmd: [2020-08-16 18:59:44.341651 -0400][Debug ][Poller ] <[::ffff:10.32.88.127]:48600><--><[::ffff:149.165.243.13]:1096> Removing socket from the poller >> cmd: [2020-08-16 18:59:44.341859 -0400][Debug ][PostMaster ] [primeradiant05.crc.nd.edu:1096 #0] Destroying stream >> cmd: [2020-08-16 18:59:44.341982 -0400][Debug ][AsyncSock ] [primeradiant05.crc.nd.edu:1096 #0.0] Closing the socket >>> will stream using xrootd instead of copying @ Sun Aug 16 18:59:44 2020 >>> modified input files @ Sun Aug 16 18:59:44 2020 >> input: root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root >> input: root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_527.root >>> updated parameters are @ Sun Aug 16 18:59:44 2020 >> json: { >> json: "accelerate stage-in": 3, >> json: "arguments": [], >> json: "cores": 1, >> json: "default ce": "T3_US_NotreDame", >> json: "default host": "earth.crc.nd.edu", >> json: "default se": "T3_US_NotreDame", >> json: "disable streaming": false, >> json: "epilogue": null, >> json: "executable": "cmsRun", >> json: "file map": { >> json: "root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_527.root": "kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_527.root", >> json: "root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root": "kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root" >> json: }, >> json: "gridpack": false, >> json: "input": [ >> json: "hdfs://eddie.crc.nd.edu:19000/store/user/", >> json: "root://deepthought.crc.nd.edu//store/user/", >> json: "gsiftp://deepthought.crc.nd.edu/store/user/", >> json: "gsiftp://deepthought.crc.nd.edu/store/user/" >> json: ], >> json: "mask": { >> json: "events": -1, >> json: "files": [ >> json: "root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root", >> json: "root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_527.root" >> json: ], >> json: "lumis": null >> json: }, >> json: "monitoring": { >> json: "monitorid": null, >> json: "syncid": null, >> json: "taskid": "lobster_kmohrman_EFT_ALL_genOnly_20200816_1832_313cabba8ca41f29" >> json: }, >> json: "output": [ >> json: "hdfs://eddie.crc.nd.edu:19000/store/user/kmohrman/genOnly_step/tests/lobster_20200816_1832/v3/", >> json: "root://deepthought.crc.nd.edu//store/user/kmohrman/genOnly_step/tests/lobster_20200816_1832/v3/", >> json: "gsiftp://deepthought.crc.nd.edu/store/user/kmohrman/genOnly_step/tests/lobster_20200816_1832/v3", >> json: "gsiftp://deepthought.crc.nd.edu/store/user/kmohrman/genOnly_step/tests/lobster_20200816_1832/v3", >> json: "file:///hadoop/store/user/kmohrman/genOnly_step/tests/lobster_20200816_1832/v3/" >> json: ], >> json: "output files": [ >> json: [ >> json: "GEN-00000.root", >> json: "gen_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/GEN-00000_43.root" >> json: ] >> json: ], >> json: "prologue": null, >> json: "pset": "GEN-00000-ttHJets_1_base-0_cfg.py", >> json: "want summary": true >> json: } >>> using sync CE T3_US_NotreDame @ Sun Aug 16 18:59:44 2020 >>> config file fragment @ Sun Aug 16 18:59:45 2020 >> pset: >> pset: import FWCore.ParameterSet.Config as cms >> pset: process.Timing = cms.Service("Timing", >> pset: useJobReport = cms.untracked.bool(True), >> pset: summaryOnly = cms.untracked.bool(True)) >> pset: >> pset: process.maxEvents = cms.untracked.PSet(input = cms.untracked.int32(-1)) >> pset: >> pset: import os >> pset: _, major, minor, _ = os.environ["CMSSW_VERSION"].split('_', 3) >> pset: if int(major) >= 7 and int(minor) >= 4: >> pset: xrdstats = cms.Service("XrdAdaptor::XrdStatisticsService", cms.untracked.PSet(reportToFJR = cms.untracked.bool(True))) >> pset: process.add_(xrdstats) >> pset: >> pset: for prod in process.producers.values(): >> pset: if prod.hasParameter('nEvents') and prod.type_() == 'ExternalLHEProducer': >> pset: prod.nEvents = cms.untracked.uint32(-1) >> pset: >> pset: process.source.fileNames = cms.untracked.vstring(['root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root', 'root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_527.root']) >> pset: if hasattr(process, 'options'): >> pset: process.options.wantSummary = cms.untracked.bool(True) >> pset: else: >> pset: process.options = cms.untracked.PSet(wantSummary = cms.untracked.bool(True)) >> pset: >> pset: if hasattr(process, 'options'): >> pset: process.options.numberOfThreads = cms.untracked.uint32(1) >> pset: process.options.numberOfStreams = cms.untracked.uint32(0) >> pset: else: >> pset: process.options = cms.untracked.PSet( >> pset: numberOfThreads = cms.untracked.uint32(1), >> pset: numberOfStreams = cms.untracked.uint32(0) >> pset: ) >>> executing 'cmsRun -j report.xml GEN-00000-ttHJets_1_base-0_cfg_mod.py' @ Sun Aug 16 18:59:45 2020 >> using /disk/vc3-root/tmp/worker-230547-7899/t.43/cctools-temp-t.43.YBXcWS/tmp4_b3nd to store command output >> cmd: 16-Aug-2020 19:00:12 EDT Initiating request to open file root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root >> cmd: [2020-08-16 19:00:57.957788 -0400][Error ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Socket error while handshaking: [ERROR] Socket timeout >> cmd: [2020-08-16 19:01:43.081905 -0400][Error ][AsyncSock ] [deepthought.crc.nd.edu:1094 #0.0] Socket error while handshaking: [ERROR] Socket timeout >> cmd: [2020-08-16 19:01:43.082562 -0400][Error ][PostMaster ] [deepthought.crc.nd.edu:1094 #0] Unable to recover: [ERROR] Socket timeout. >> cmd: [2020-08-16 19:01:43.082828 -0400][Error ][XRootD ] [deepthought.crc.nd.edu:1094] Impossible to send message kXR_prepare (flags: kXR_stage , priority: 0, paths: /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root). Trying to recover. >> cmd: [2020-08-16 19:01:43.083149 -0400][Error ][XRootD ] [deepthought.crc.nd.edu:1094] Impossible to send message kXR_prepare (flags: kXR_stage , priority: 0, paths: /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_527.root). Trying to recover. >> cmd: [2020-08-16 19:01:43.083284 -0400][Error ][XRootD ] [deepthought.crc.nd.edu:1094] Impossible to send message kXR_query (code: kXR_Qconfig, arg length: 9). Trying to recover. >> cmd: [2020-08-16 19:01:43.083437 -0400][Error ][XRootD ] [deepthought.crc.nd.edu:1094] Impossible to send message kXR_open (file: /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. >> cmd: %MSG-w XrdAdaptorInternal: file_open 16-Aug-2020 19:01:43 EDT pre-events >> cmd: Failed to open file at URL root://deepthought.crc.nd.edu:1094//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root. >> cmd: %MSG >> cmd: [2020-08-16 19:01:43.086753 -0400][Error ][XRootD ] [deepthought.crc.nd.edu:1094] Unable to send the message kXR_open (file: /store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root?tried=, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL] Socket timeout >> cmd: ----- Begin Fatal Exception 16-Aug-2020 19:01:43 EDT----------------------- >> cmd: An exception of category 'FileOpenError' occurred while >> cmd: [0] Constructing the EventProcessor >> cmd: [1] Constructing input source of type PoolSource >> cmd: [2] Calling RootFileSequenceBase::initTheFile() >> cmd: [3] Calling StorageFactory::open() >> cmd: [4] Calling XrdFile::open() >> cmd: Exception Message: >> cmd: Failed to open the file 'root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root' >> cmd: Additional Info: >> cmd: [a] Input file root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root could not be opened. >> cmd: [b] XrdCl::File::Open(name='root://deepthought.crc.nd.edu//store/user/kmohrman/LHE_step/2019_04_19/ttXJet_HanV4-QED1-noQCDconstraints/v1/lhe_step_ttHJet_HanV4DIM61QED1noQEDconstraintwithRwgt_run1/HIG-RunIIFall17wmLHE-00000ND_888.root', flags=0x10, permissions=0660) => error '[FATAL] Socket timeout' (errno=0, code=103) >> cmd: [c] Remote server already encountered a fatal error; no redirections were performed. >> cmd: ----- End Fatal Exception ------------------------------------------------- >>> executable returned with exit code 84. @ Sun Aug 16 19:01:43 2020 >> trace: Traceback (most recent call last): >> trace: File "task.py", line 268, in wrapper >> trace: result = fct(data, *args, **kwargs) >> trace: File "task.py", line 1052, in run_command >> trace: raise subprocess.CalledProcessError(p.returncode, cmd) >> trace: CalledProcessError: Command '[u'cmsRun', '-j', 'report.xml', u'GEN-00000-ttHJets_1_base-0_cfg_mod.py']' returned non-zero exit status 84 >>>>> call to 'run_command' failed, exiting with exit code 185 @ Sun Aug 16 19:01:44 2020 Sun Aug 16 18:59:44 2020: ApMon[ERROR]: Error resolving cms-jobmon.cern.ch: [Errno -2] Name or service not known Sun Aug 16 18:59:45 2020: ApMon[WARNING]: Not sending parameters since no destination is defined. Sun Aug 16 18:59:45 2020: ApMon[DEBUG]: Closing UDP socket on ApMon object destroy. Sun Aug 16 19:01:43 2020: ApMon[ERROR]: Error resolving cms-jobmon.cern.ch: [Errno -2] Name or service not known Sun Aug 16 19:01:44 2020: ApMon[WARNING]: Not sending parameters since no destination is defined. Sun Aug 16 19:01:44 2020: ApMon[DEBUG]: Closing UDP socket on ApMon object destroy. >> Execution time -1597618285 >> Exiting with code 185 >> Reporting ExeExitCode 8020 >> Reporting StageOutSE >> Reporting StageOutExitCode 0 Sun Aug 16 19:01:44 2020: ApMon[ERROR]: Error resolving cms-jobmon.cern.ch: [Errno -2] Name or service not known Sun Aug 16 19:01:45 2020: ApMon[WARNING]: Not sending parameters since no destination is defined. Sun Aug 16 19:01:45 2020: ApMon[DEBUG]: Closing UDP socket on ApMon object destroy. ==== working directory after execution @ Sun Aug 16 19:01:45 EDT 2020 ==== == dir: total 892 == dir: drwxr-xr-x 17 kmohrman campus 201 Aug 16 18:51 CMSSW_9_3_6 == dir: -rw-r--r-- 9 kmohrman campus 8037 Aug 16 18:50 GEN-00000-ttHJets_1_base-0_cfg.py == dir: -rw-r--r-- 1 kmohrman campus 9673 Aug 16 18:59 GEN-00000-ttHJets_1_base-0_cfg_mod.py == dir: drwxr-xr-x 2 kmohrman campus 71 Aug 16 18:50 bin == dir: -rwxr-xr-x 11 kmohrman campus 627272 Aug 16 18:50 cctools-monitor == dir: -rw-r--r-- 1 kmohrman campus 0 Aug 16 18:50 cctools-monitor.summary == dir: drwxrwxrwx 2 kmohrman campus 6 Aug 16 19:01 cctools-temp-t.43.YBXcWS == dir: -rw------- 1 kmohrman campus 51 Aug 16 18:51 cvmfschecksum.cms-ib.cern.ch == dir: -rw------- 1 kmohrman campus 51 Aug 16 18:51 cvmfschecksum.cms.cern.ch == dir: -rw------- 1 kmohrman campus 51 Aug 16 18:51 cvmfschecksum.oasis.opensciencegrid.org == dir: drwxr-xr-x 2 kmohrman campus 33 Aug 16 18:50 lib == dir: -rwxrwxrwx 1 kmohrman campus 146912 Aug 16 18:50 librmonitor_helper.so.3N9QXP == dir: -rw-r--r-- 2 kmohrman campus 2010 Aug 16 18:50 parameters.json == dir: -rw------- 2 kmohrman campus 9884 Aug 16 18:50 proxy == dir: drwxr-xr-x 3 kmohrman campus 20 Aug 16 18:50 python == dir: -rw-r--r-- 1 kmohrman campus 808 Aug 16 19:01 report.json == dir: -rw-r--r-- 11 kmohrman campus 570 Aug 16 18:50 report.json.in == dir: -rw-r--r-- 1 kmohrman campus 1573 Aug 16 19:01 report.xml == dir: -rw-r--r-- 1 kmohrman campus 649 Aug 16 19:01 report.xml.gz == dir: -rw-r--r-- 11 kmohrman campus 449 Aug 16 18:50 sandbox-CMSSW_9_3_6-slc7_amd64_gcc630.tar.bz2 == dir: drwxr-xr-x 4 kmohrman campus 37 Aug 16 18:50 siteconf == dir: -rw-r--r-- 1 kmohrman campus 11 Aug 16 18:51 t_wrapper_ready == dir: -rw-r--r-- 1 kmohrman campus 11 Aug 16 18:51 t_wrapper_start == dir: -rwxr-xr-x 11 kmohrman campus 47217 Aug 16 18:50 task.py == dir: -rwxr-xr-x 11 kmohrman campus 5986 Aug 16 18:50 wrapper.sh === wrapper done @ Sun Aug 16 19:01:45 EDT 2020 === final return status = 185 @ Sun Aug 16 19:01:45 EDT 2020