[kmohrman@earth: ~]$ xrdcp -d 2 -f root://primeradiant01.crc.nd.edu//store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root /dev/null [2022-02-09 08:08:32.570272 -0500][Debug ][Utility ] CopyProcess: 2 jobs to prepare [2022-02-09 08:08:32.570537 -0500][Debug ][Utility ] Creating a classic copy job, from root://primeradiant01.crc.nd.edu:1094//store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root to file://localhost/dev/null [2022-02-09 08:08:32.570582 -0500][Debug ][Utility ] Monitor library name not set. No monitoring [2022-02-09 08:08:32.570817 -0500][Debug ][Poller ] Available pollers: built-in [2022-02-09 08:08:32.570844 -0500][Debug ][Poller ] Attempting to create a poller according to preference: built-in [2022-02-09 08:08:32.570866 -0500][Debug ][Poller ] Creating poller: built-in [2022-02-09 08:08:32.570894 -0500][Debug ][Poller ] Creating and starting the built-in poller... [2022-02-09 08:08:32.571196 -0500][Debug ][Poller ] Using 1 poller threads [2022-02-09 08:08:32.571252 -0500][Debug ][TaskMgr ] Starting the task manager... [2022-02-09 08:08:32.571351 -0500][Debug ][TaskMgr ] Task manager started [2022-02-09 08:08:32.571379 -0500][Debug ][JobMgr ] Starting the job manager... [2022-02-09 08:08:32.571546 -0500][Debug ][JobMgr ] Job manager started, 3 workers [2022-02-09 08:08:32.571576 -0500][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2022-02-09 08:08:32 -0500] [2022-02-09 08:08:32.571614 -0500][Debug ][Utility ] Opening root://primeradiant01.crc.nd.edu:1094//store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root for reading [2022-02-09 08:08:32.571794 -0500][Debug ][File ] [0x1b5de30@root://primeradiant01.crc.nd.edu:1094//store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?xrdcl.requuid=4eb878a6-8270-4fa9-8c4b-82015d268a81] Sending an open command [2022-02-09 08:08:32.571893 -0500][Debug ][ExDbgMsg ] [primeradiant01.crc.nd.edu:1094] MsgHandler created: 0x1b62a80 (message: kXR_open (file: /store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2022-02-09 08:08:32.571979 -0500][Debug ][PostMaster ] Creating new channel to: primeradiant01.crc.nd.edu:1094 1 stream(s) [2022-02-09 08:08:32.572030 -0500][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1094 #0] Stream parameters: Network Stack: IPAuto, Connection Window: 120, ConnectionRetry: 5, Stream Error Window: 1800 [2022-02-09 08:08:32.572552 -0500][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: primeradiant01.crc.nd.edu:1094" to be run at: [2022-02-09 08:08:47 -0500] [2022-02-09 08:08:32.573071 -0500][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1094] Found 1 address(es): [::ffff:149.165.243.5]:1094 [2022-02-09 08:08:32.573120 -0500][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1094 #0.0] Attempting connection to [::ffff:149.165.243.5]:1094 [2022-02-09 08:08:32.573169 -0500][Debug ][Poller ] Adding socket 0x1b638b0 to the poller [2022-02-09 08:08:32.573359 -0500][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1094 #0.0] Async connection call returned [2022-02-09 08:08:32.573463 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Sending out the initial hand shake + kXR_protocol [2022-02-09 08:08:32.598812 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Got the server hand shake response (type: server [], protocol version 500) [2022-02-09 08:08:32.598864 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] kXR_protocol successful (type: server [proxy], protocol version 500) [2022-02-09 08:08:32.600098 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Sending out kXR_login request, username: kmohrman, cgi: ?xrd.cc=us&xrd.tz=-5&xrd.appname=xrdcp&xrd.info=&xrd.hostname=earth.crc.nd.edu&xrd.rn=v4.12.6, dual-stack: false, private IPv4: true, private IPv6: false [2022-02-09 08:08:32.600317 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Logged in, session: 01000000b86300001d00000001000000 [2022-02-09 08:08:32.600349 -0500][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1094 #0] Stream 0 connected. [2022-02-09 08:08:32.600434 -0500][Debug ][ExDbgMsg ] [primeradiant01.crc.nd.edu:1094] Moving MsgHandler: 0x1b62a80 (message: kXR_open (file: /store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2022-02-09 08:09:32.661048 -0500][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1094 #0.0] Closing the socket [2022-02-09 08:09:32.661130 -0500][Debug ][Poller ] <[::ffff:10.32.77.18]:29334><--><[::ffff:149.165.243.5]:1094> Removing socket from the poller [2022-02-09 08:09:32.661256 -0500][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1094 #0] Recovering error for stream #0: [ERROR] Socket error. [2022-02-09 08:09:32.661287 -0500][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1094 #0] Reporting disconnection to queued message handlers. [2022-02-09 08:09:32.661327 -0500][Debug ][XRootD ] [primeradiant01.crc.nd.edu:1094] Handling error while processing kXR_open (file: /store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Socket error. [2022-02-09 08:09:32.661365 -0500][Info ][XRootD ] [primeradiant01.crc.nd.edu:1094] Retrying request: kXR_open (file: /store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). [2022-02-09 08:09:32.661474 -0500][Debug ][ExDbgMsg ] [primeradiant01.crc.nd.edu:1094] Retry at server MsgHandler: 0x1b62a80 (message: kXR_open (file: /store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2022-02-09 08:09:32.662163 -0500][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1094] Found 1 address(es): [::ffff:149.165.243.5]:1094 [2022-02-09 08:09:32.662256 -0500][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1094 #0.0] Attempting connection to [::ffff:149.165.243.5]:1094 [2022-02-09 08:09:32.662320 -0500][Debug ][Poller ] Adding socket 0x1b638b0 to the poller [2022-02-09 08:09:32.662452 -0500][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1094 #0.0] Async connection call returned [2022-02-09 08:09:32.662492 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Sending out the initial hand shake + kXR_protocol [2022-02-09 08:09:32.662681 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Got the server hand shake response (type: server [], protocol version 500) [2022-02-09 08:09:32.662723 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] kXR_protocol successful (type: server [proxy], protocol version 500) [2022-02-09 08:09:32.663784 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Sending out kXR_login request, username: kmohrman, cgi: ?xrd.cc=us&xrd.tz=-5&xrd.appname=xrdcp&xrd.info=&xrd.hostname=earth.crc.nd.edu&xrd.rn=v4.12.6, dual-stack: false, private IPv4: true, private IPv6: false [2022-02-09 08:09:32.664011 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Logged in, session: 02000000b86300001e00000002000000 [2022-02-09 08:09:32.664050 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Sending out kXR_endsess for session: 01000000b86300001d00000001000000 [2022-02-09 08:09:32.664337 -0500][Info ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Got wait response to kXR_endsess: session still active [2022-02-09 08:09:32.664374 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Sending out kXR_endsess for session: 01000000b86300001d00000001000000 [2022-02-09 08:09:32.664424 -0500][Debug ][TaskMgr ] Registering task: "WaitTask for: 0x0x7ff16c001120" to be run at: [2022-02-09 08:09:36 -0500] [2022-02-09 08:09:36.578879 -0500][Debug ][TaskMgr ] Done with task: "WaitTask for: 0x0x7ff16c001120" [2022-02-09 08:09:36.579415 -0500][Info ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Got wait response to kXR_endsess: session still active [2022-02-09 08:09:36.579489 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Sending out kXR_endsess for session: 01000000b86300001d00000001000000 [2022-02-09 08:09:36.579529 -0500][Debug ][TaskMgr ] Registering task: "WaitTask for: 0x0x7ff16c001160" to be run at: [2022-02-09 08:09:42 -0500] [2022-02-09 08:09:42.579746 -0500][Debug ][TaskMgr ] Done with task: "WaitTask for: 0x0x7ff16c001160" [2022-02-09 08:09:42.580246 -0500][Info ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Got wait response to kXR_endsess: session still active [2022-02-09 08:09:42.580310 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Sending out kXR_endsess for session: 01000000b86300001d00000001000000 [2022-02-09 08:09:42.580377 -0500][Debug ][TaskMgr ] Registering task: "WaitTask for: 0x0x7ff16c000950" to be run at: [2022-02-09 08:09:50 -0500] [2022-02-09 08:09:50.580813 -0500][Debug ][TaskMgr ] Done with task: "WaitTask for: 0x0x7ff16c000950" [2022-02-09 08:09:50.581273 -0500][Info ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Got wait response to kXR_endsess: session still active [2022-02-09 08:09:50.581355 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Sending out kXR_endsess for session: 01000000b86300001d00000001000000 [2022-02-09 08:09:50.581389 -0500][Debug ][TaskMgr ] Registering task: "WaitTask for: 0x0x7ff16c001120" to be run at: [2022-02-09 08:10:00 -0500] [2022-02-09 08:10:00.582198 -0500][Debug ][TaskMgr ] Done with task: "WaitTask for: 0x0x7ff16c001120" [2022-02-09 08:10:00.582620 -0500][Info ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Got wait response to kXR_endsess: session still active [2022-02-09 08:10:00.582708 -0500][Debug ][XRootDTransport ] [primeradiant01.crc.nd.edu:1094 #0.0] Sending out kXR_endsess for session: 01000000b86300001d00000001000000 [2022-02-09 08:10:00.582743 -0500][Debug ][TaskMgr ] Registering task: "WaitTask for: 0x0x7ff16c001160" to be run at: [2022-02-09 08:10:12 -0500] [2022-02-09 08:10:01.087305 -0500][Error ][AsyncSock ] [primeradiant01.crc.nd.edu:1094 #0.0] Socket error while handshaking: [ERROR] Socket error [2022-02-09 08:10:01.087369 -0500][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1094 #0.0] Closing the socket [2022-02-09 08:10:01.087398 -0500][Debug ][Poller ] <[::ffff:10.32.77.18]:29368><--><[::ffff:149.165.243.5]:1094> Removing socket from the poller [2022-02-09 08:10:01.087461 -0500][Error ][PostMaster ] [primeradiant01.crc.nd.edu:1094 #0] elapsed = 29, pConnectionWindow = 120 seconds. [2022-02-09 08:10:01.087501 -0500][Info ][PostMaster ] [primeradiant01.crc.nd.edu:1094 #0] Attempting reconnection in 91 seconds. [2022-02-09 08:10:01.087528 -0500][Debug ][TaskMgr ] Registering task: "StreamConnectorTask for primeradiant01.crc.nd.edu:1094 #0" to be run at: [2022-02-09 08:11:32 -0500] [2022-02-09 08:10:12.584084 -0500][Warning][Poller ] <--> Socket is not registered [2022-02-09 08:10:12.584172 -0500][Error ][AsyncSock ] [primeradiant01.crc.nd.edu:1094 #0.0] Socket error while handshaking: [FATAL] Poller error [2022-02-09 08:10:12.584189 -0500][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1094 #0.0] Closing the socket [2022-02-09 08:10:12.584205 -0500][Error ][PostMaster ] [primeradiant01.crc.nd.edu:1094 #0] elapsed = 40, pConnectionWindow = 120 seconds. [2022-02-09 08:10:12.584252 -0500][Error ][PostMaster ] [primeradiant01.crc.nd.edu:1094 #0] Unable to recover: [FATAL] Poller error. [2022-02-09 08:10:12.584269 -0500][Error ][XRootD ] [primeradiant01.crc.nd.edu:1094] Impossible to send message kXR_open (file: /store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. [2022-02-09 08:10:12.584286 -0500][Debug ][XRootD ] [primeradiant01.crc.nd.edu:1094] Handling error while processing kXR_open (file: /store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL] Poller error. [2022-02-09 08:10:12.584319 -0500][Debug ][ExDbgMsg ] [primeradiant01.crc.nd.edu:1094] Passing to the thread-pool MsgHandler: 0x1b62a80 (message: kXR_open (file: /store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2022-02-09 08:10:12.584346 -0500][Debug ][TaskMgr ] Done with task: "WaitTask for: 0x0x7ff16c001160" [2022-02-09 08:10:12.584545 -0500][Debug ][ExDbgMsg ] [primeradiant01.crc.nd.edu:1094] Calling MsgHandler: 0x1b62a80 (message: kXR_open (file: /store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [FATAL] Poller error. [2022-02-09 08:10:12.584674 -0500][Debug ][File ] [0x1b5de30@root://primeradiant01.crc.nd.edu:1094//store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?xrdcl.requuid=4eb878a6-8270-4fa9-8c4b-82015d268a81] Open has returned with status [FATAL] Poller error [2022-02-09 08:10:12.584693 -0500][Debug ][File ] [0x1b5de30@root://primeradiant01.crc.nd.edu:1094//store/mc/RunIISummer20UL17NanoAODv9/WZTo3LNu_TuneCP5_13TeV-amcatnloFXFX-pythia8/NANOAODSIM/106X_mc2017_realistic_v9-v2/250000/3259A23F-EBAD-7B4A-A0AF-098263FDE109.root?xrdcl.requuid=4eb878a6-8270-4fa9-8c4b-82015d268a81] Error while opening at primeradiant01.crc.nd.edu:1094: [FATAL] Poller error [2022-02-09 08:10:12.584767 -0500][Debug ][XRootD ] Redirect trace-back: [2022-02-09 08:10:12.584767 -0500][Debug ][XRootD ] 0. Retrying: root://primeradiant01.crc.nd.edu:1094/ [2022-02-09 08:10:12.584783 -0500][Debug ][ExDbgMsg ] [primeradiant01.crc.nd.edu:1094] Destroying MsgHandler: 0x1b62a80. [0B/0B][100%][==================================================][0B/s] Run: [FATAL] Poller error: (source) [2022-02-09 08:10:12.585306 -0500][Debug ][JobMgr ] Stopping the job manager... [2022-02-09 08:10:12.585676 -0500][Debug ][JobMgr ] Job manager stopped [2022-02-09 08:10:12.585706 -0500][Debug ][TaskMgr ] Stopping the task manager... [2022-02-09 08:10:12.585874 -0500][Debug ][TaskMgr ] Task manager stopped [2022-02-09 08:10:12.585900 -0500][Debug ][Poller ] Stopping the poller... [2022-02-09 08:10:12.586138 -0500][Debug ][TaskMgr ] Requesting unregistration of: "TickGeneratorTask for: primeradiant01.crc.nd.edu:1094" [2022-02-09 08:10:12.586170 -0500][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1094 #0.0] Closing the socket [2022-02-09 08:10:12.586202 -0500][Debug ][PostMaster ] [primeradiant01.crc.nd.edu:1094 #0] Destroying stream [2022-02-09 08:10:12.586243 -0500][Debug ][AsyncSock ] [primeradiant01.crc.nd.edu:1094 #0.0] Closing the socket