text:
"distributed.utils - ERROR" after PBS jobs start

While trying to use PBS scheduler on HPC , I get the "distributed.utils - ERROR" once the requested job starts. It was working previously but now it throws this error :
distributed.utils - ERROR - addresses should be strings or tuples, got None
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/utils.py", line 656, in log_errors
    yield
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/scheduler.py", line 2208, in remove_worker
    address = self.coerce_address(address)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/scheduler.py", line 4946, in coerce_address
    raise TypeError("addresses should be strings or tuples, got %r" % (addr,))
TypeError: addresses should be strings or tuples, got None

Here's what I my request code looks like :

Anything else we need to know?:
The error report in one of the dask worker's file looks something like this :
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.20.12.16:41556'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.20.12.16:42703'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.20.12.16:41949'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.20.12.16:45256'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.20.12.16:41740'
distributed.worker - INFO -       Start worker at:   tcp://172.20.12.16:35810
distributed.worker - INFO -          Listening to:   tcp://172.20.12.16:35810
distributed.worker - INFO -          dashboard at:         172.20.12.16:46589
distributed.worker - INFO - Waiting to connect to:   tcp://172.20.9.175:35077
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                       12 B
distributed.worker - INFO -       Local Directory: /scratch/pbs/pbs.1158744.pbshpc.x8z/dask-worker-space/dask-worker-space/worker-f812nex4
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at:   tcp://172.20.12.16:36070
distributed.worker - INFO -          Listening to:   tcp://172.20.12.16:36070
distributed.worker - INFO -          dashboard at:         172.20.12.16:45435
distributed.worker - INFO - Waiting to connect to:   tcp://172.20.9.175:35077
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at:   tcp://172.20.12.16:39425
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                       12 B
distributed.worker - INFO -          Listening to:   tcp://172.20.12.16:39425
distributed.worker - INFO -       Local Directory: /scratch/pbs/pbs.1158744.pbshpc.x8z/dask-worker-space/dask-worker-space/worker-1n98k0_q
distributed.worker - INFO -          dashboard at:         172.20.12.16:44500
distributed.worker - INFO - Waiting to connect to:   tcp://172.20.9.175:35077
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                       12 B
distributed.worker - INFO -       Local Directory: /scratch/pbs/pbs.1158744.pbshpc.x8z/dask-worker-space/dask-worker-space/worker-zxkfhqd4
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.20.12.16:35810
distributed.worker - INFO - Stopping worker at tcp://172.20.12.16:36070
distributed.worker - INFO - Closed worker has not yet started: Status.undefined
distributed.worker - INFO - Closed worker has not yet started: Status.undefined
distributed.worker - INFO - Stopping worker at tcp://172.20.12.16:39425
distributed.worker - INFO - Closed worker has not yet started: Status.undefined
distributed.utils - ERROR - local variable 'buf' referenced before assignment
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 280, in _
    await asyncio.wait_for(self.start(), timeout=timeout)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1094, in start
    await self._register_with_scheduler()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 832, in _register_with_scheduler
    response = await future
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/utils.py", line 656, in log_errors
    yield
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1127, in close
    await r.close_gracefully()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.utils - ERROR - local variable 'buf' referenced before assignment
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 280, in _
    await asyncio.wait_for(self.start(), timeout=timeout)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1094, in start
    await self._register_with_scheduler()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 832, in _register_with_scheduler
    response = await future
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/utils.py", line 656, in log_errors
    yield
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1127, in close
    await r.close_gracefully()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.utils - ERROR - local variable 'buf' referenced before assignment
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 280, in _
    await asyncio.wait_for(self.start(), timeout=timeout)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1094, in start
    await self._register_with_scheduler()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 832, in _register_with_scheduler
    response = await future
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/utils.py", line 656, in log_errors
    yield
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1127, in close
    await r.close_gracefully()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 280, in _
    await asyncio.wait_for(self.start(), timeout=timeout)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1094, in start
    await self._register_with_scheduler()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 832, in _register_with_scheduler
    response = await future
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 766, in run
    await worker
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 283, in _
    await self.close(timeout=1)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1127, in close
    await r.close_gracefully()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 280, in _
    await asyncio.wait_for(self.start(), timeout=timeout)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1094, in start
    await self._register_with_scheduler()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 832, in _register_with_scheduler
    response = await future
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 766, in run
    await worker
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 283, in _
    await self.close(timeout=1)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1127, in close
    await r.close_gracefully()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 280, in _
    await asyncio.wait_for(self.start(), timeout=timeout)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1094, in start
    await self._register_with_scheduler()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 832, in _register_with_scheduler
    response = await future
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 766, in run
    await worker
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 283, in _
    await self.close(timeout=1)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1127, in close
    await r.close_gracefully()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - ERROR - Failed while trying to start worker process: local variable 'buf' referenced before assignment
distributed.nanny - ERROR - Failed while trying to start worker process: local variable 'buf' referenced before assignment
distributed.nanny - ERROR - Failed while trying to start worker process: local variable 'buf' referenced before assignment
distributed.nanny - INFO - Closing Nanny at 'tcp://172.20.12.16:42703'
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x2b7d10dc46a0>>, <Task finished name='Task-20' coro=<Nanny._on_exit() done, defined at /home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py:440> exception=UnboundLocalError("local variable 'buf' referenced before assignment")>)
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 443, in _on_exit
    await self.scheduler.unregister(address=self.worker_address)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - INFO - Closing Nanny at 'tcp://172.20.12.16:41556'
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x2b7d10dc46a0>>, <Task finished name='Task-23' coro=<Nanny._on_exit() done, defined at /home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py:440> exception=UnboundLocalError("local variable 'buf' referenced before assignment")>)
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 443, in _on_exit
    await self.scheduler.unregister(address=self.worker_address)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - INFO - Closing Nanny at 'tcp://172.20.12.16:41740'
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x2b7d10dc46a0>>, <Task finished name='Task-26' coro=<Nanny._on_exit() done, defined at /home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py:440> exception=UnboundLocalError("local variable 'buf' referenced before assignment")>)
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 443, in _on_exit
    await self.scheduler.unregister(address=self.worker_address)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.worker - INFO -       Start worker at:   tcp://172.20.12.16:46337
distributed.worker - INFO -          Listening to:   tcp://172.20.12.16:46337
distributed.worker - INFO -          dashboard at:         172.20.12.16:36561
distributed.worker - INFO - Waiting to connect to:   tcp://172.20.9.175:35077
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                       12 B
distributed.worker - INFO -       Local Directory: /scratch/pbs/pbs.1158744.pbshpc.x8z/dask-worker-space/dask-worker-space/worker-61bwz0l8
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.20.12.16:46337
distributed.worker - INFO - Closed worker has not yet started: Status.undefined
distributed.utils - ERROR - local variable 'buf' referenced before assignment
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 280, in _
    await asyncio.wait_for(self.start(), timeout=timeout)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1094, in start
    await self._register_with_scheduler()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 832, in _register_with_scheduler
    response = await future
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/utils.py", line 656, in log_errors
    yield
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1127, in close
    await r.close_gracefully()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 280, in _
    await asyncio.wait_for(self.start(), timeout=timeout)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1094, in start
    await self._register_with_scheduler()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 832, in _register_with_scheduler
    response = await future
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 766, in run
    await worker
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 283, in _
    await self.close(timeout=1)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1127, in close
    await r.close_gracefully()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - ERROR - Failed while trying to start worker process: local variable 'buf' referenced before assignment
distributed.worker - INFO -       Start worker at:   tcp://172.20.12.16:41698
distributed.worker - INFO -          Listening to:   tcp://172.20.12.16:41698
distributed.worker - INFO -          dashboard at:         172.20.12.16:34770
distributed.worker - INFO - Waiting to connect to:   tcp://172.20.9.175:35077
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                       12 B
distributed.worker - INFO -       Local Directory: /scratch/pbs/pbs.1158744.pbshpc.x8z/dask-worker-space/dask-worker-space/worker-w907saoo
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.20.12.16:41698
distributed.worker - INFO - Closed worker has not yet started: Status.undefined
distributed.utils - ERROR - local variable 'buf' referenced before assignment
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 280, in _
    await asyncio.wait_for(self.start(), timeout=timeout)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1094, in start
    await self._register_with_scheduler()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 832, in _register_with_scheduler
    response = await future
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/utils.py", line 656, in log_errors
    yield
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1127, in close
    await r.close_gracefully()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 280, in _
    await asyncio.wait_for(self.start(), timeout=timeout)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1094, in start
    await self._register_with_scheduler()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 832, in _register_with_scheduler
    response = await future
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 766, in run
    await worker
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 283, in _
    await self.close(timeout=1)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1127, in close
    await r.close_gracefully()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - ERROR - Failed while trying to start worker process: local variable 'buf' referenced before assignment
distributed.nanny - INFO - Closing Nanny at 'tcp://172.20.12.16:45256'
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x2b7d10dc46a0>>, <Task finished name='Task-43' coro=<Nanny._on_exit() done, defined at /home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py:440> exception=UnboundLocalError("local variable 'buf' referenced before assignment")>)
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 443, in _on_exit
    await self.scheduler.unregister(address=self.worker_address)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - INFO - Closing Nanny at 'tcp://172.20.12.16:41949'
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x2b7d10dc46a0>>, <Task finished name='Task-52' coro=<Nanny._on_exit() done, defined at /home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py:440> exception=UnboundLocalError("local variable 'buf' referenced before assignment")>)
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 881, in _read_to_buffer
    buf = bytearray(self.read_chunk_size)
TypeError: cannot convert 'float' object to bytearray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 443, in _on_exit
    await self.scheduler.unregister(address=self.worker_address)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.20.12.16:42703'
distributed.dask_worker - INFO - End worker
Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 280, in _
    await asyncio.wait_for(self.start(), timeout=timeout)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 295, in start
    response = await self.instantiate()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 365, in instantiate
    result = await asyncio.wait_for(
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 581, in start
    msg = await self._wait_until_connected(uid)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 696, in _wait_until_connected
    raise msg["exception"]
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/nanny.py", line 766, in run
    await worker
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 283, in _
    await self.close(timeout=1)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/worker.py", line 1127, in close
    await r.close_gracefully()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 441, in read_bytes
    self._try_inline_read()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 857, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 770, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/iostream.py", line 907, in _read_to_buffer
    del buf
UnboundLocalError: local variable 'buf' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 450, in <module>
    go()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 446, in go
    main()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 432, in main
    loop.run_sync(run)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/tornado/ioloop.py", line 532, in run_sync
    return future_cell[0].result()
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 426, in run
    await asyncio.gather(*nannies)
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/asyncio/tasks.py", line 684, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/home/civil/phd/cez198621/miniconda3/envs/dask/lib/python3.8/site-packages/distributed/core.py", line 284, in _
    raise TimeoutError(
asyncio.exceptions.TimeoutError: Nanny failed to start in 60 seconds
distributed.process - INFO - reaping stray process <SpawnProcess name='Dask Worker process (from Nanny)' pid=21340 parent=21255 started daemon>

Environment:
Dask version: 2.19.0
Python version: 3.7.7
Operating System: CentOS Linux 7 (Core)
Install method (conda, pip, source): conda and pip

Seems like there was a wrong entry at "memory=64", replacing that with memory="64GB", resolved the issue.

repo: dask/dask-jobqueue


text:
BUG: set_index causes type-information to be lost?

This is a strange one...
In [1]: import dask
   ...: import string
   ...: import numpy as np
   ...: import pandas as pd
   ...: import dask.dataframe as dd
   ...:
   ...: # setup
   ...: dask.set_options(get=dask.get)
   ...:
   ...: df = pd.DataFrame(np.random.choice(list(string.ascii_letters), size=(100, 4)))
   ...: df = df.apply(lambda x: x.astype('category'))
   ...: df.index.name = 'idx'
   ...: df = df.reset_index()
   ...: df['idx'] = df['idx'].astype(str)
   ...:
   ...:

In [2]: df.head()
Out[2]:
  idx  0  1  2  3
0   0  q  G  n  E
1   1  N  A  T  b
2   2  x  v  P  F
3   3  u  t  a  p
4   4  Z  h  X  G

In [3]: df.dtypes
Out[3]:
idx      object
0      category
1      category
2      category
3      category
dtype: object

So let's make a dask DataFrame from df
In [4]: a = dd.from_pandas(df, npartitions=2)
   ...:

In [5]: a
Out[5]: dd.DataFrame<from_pa..., npartitions=2, divisions=(0, 50, 99)>

In [6]: a.dtypes
Out[6]:
idx      object
0      category
1      category
2      category
3      category
dtype: object

Everything looks good. Now set_index
In [7]: a2 = a.set_index('idx')

In [8]: a2.dtypes
Out[8]:
0    category
1    category
2    category
3    category
dtype: object

Still good, but
In [9]: a2.head().dtypes

gives
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-9-969290535361> in <module>()
----> 1 a2.head().dtypes

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/dask/dask/dataframe/core.py in head(self, n, npartitions, compute)
    612
    613         if compute:
--> 614             result = result.compute()
    615         return result
    616

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/dask/dask/base.py in compute(self, **kwargs)
     84             Extra keywords to forward to the scheduler ``get`` function.
     85         """
---> 86         return compute(self, **kwargs)[0]
     87
     88     @classmethod

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/dask/dask/base.py in compute(*args, **kwargs)
    177         dsk = merge(var.dask for var in variables)
    178     keys = [var._keys() for var in variables]
--> 179     results = get(dsk, keys, **kwargs)
    180
    181     results_iter = iter(results)

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/dask/dask/async.py in get_sync(dsk, keys, **kwargs)
    525     queue = Queue()
    526     return get_async(apply_sync, 1, dsk, keys, queue=queue,
--> 527                      raise_on_exception=True, **kwargs)
    528
    529

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/dask/dask/async.py in get_async(apply_async, num_workers, dsk, result, cache, queue, get_id, raise_on_exception, rerun_exceptions_locally, callbacks, **kwargs)
    485                 f(key, res, dsk, state, worker_id)
    486             while state['ready'] and len(state['running']) < num_workers:
--> 487                 fire_task()
    488     except KeyboardInterrupt:
    489         for cb in started_cbs:

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/dask/dask/async.py in fire_task()
    460             # Submit
    461             apply_async(execute_task, args=[key, dsk[key], data, queue,
--> 462                                             get_id, raise_on_exception])
    463
    464         # Seed initial tasks into the thread pool

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/dask/dask/async.py in apply_sync(func, args, kwds)
    513 def apply_sync(func, args=(), kwds={}):
    514     """ A naive synchronous version of apply_async """
--> 515     return func(*args, **kwds)
    516
    517

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/dask/dask/async.py in execute_task(key, task, data, queue, get_id, raise_on_exception)
    261     """
    262     try:
--> 263         result = _execute_task(task, data)
    264         id = get_id()
    265         result = key, result, None, id

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/dask/dask/async.py in _execute_task(arg, cache, dsk)
    243         func, args = arg[0], arg[1:]
    244         args2 = [_execute_task(a, cache) for a in args]
--> 245         return func(*args2)
    246     elif not ishashable(arg):
    247         return arg

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/dask/dask/dataframe/shuffle.py in shuffle_group_3(df, col, npartitions, p)
    374     g = df.groupby(col)
    375     d = {i: g.get_group(i) for i in g.groups}
--> 376     p.append(d, fsync=True)
    377
    378

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/partd/encode.py in append(self, data, **kwargs)
     21
     22     def append(self, data, **kwargs):
---> 23         data = valmap(self.encode, data)
     24         data = valmap(frame, data)
     25         self.partd.append(data, **kwargs)

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/toolz/dicttoolz.py in valmap(func, d, factory)
     82     """
     83     rv = factory()
---> 84     rv.update(zip(iterkeys(d), map(func, itervalues(d))))
     85     return rv
     86

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/partd/pandas.py in serialize(df)
    106     blocks, index, index_name, columns, placement = to_blocks(df)
    107     b_blocks = [pnp.compress(pnp.serialize(block), block.dtype)
--> 108                 for block in blocks]  # this can be slightly faster if we merge both operations
    109     b_index = pnp.compress(pnp.serialize(index), index.dtype)
    110     frames = [pickle.dumps(index_name),

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/partd/pandas.py in <listcomp>(.0)
    106     blocks, index, index_name, columns, placement = to_blocks(df)
    107     b_blocks = [pnp.compress(pnp.serialize(block), block.dtype)
--> 108                 for block in blocks]  # this can be slightly faster if we merge both operations
    109     b_index = pnp.compress(pnp.serialize(index), index.dtype)
    110     frames = [pickle.dumps(index_name),

/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/partd/numpy.py in serialize(x)
     99         return frame(pickle.dumps(x.tolist(), protocol=pickle.HIGHEST_PROTOCOL))
    100     else:
--> 101         return x.tobytes()
    102
    103

AttributeError: 'Categorical' object has no attribute 'tobytes'

In my real example, I'm reading from gzipped CSVs, so I need to use read_csv().set_index(). However, the actual behavior there isn't this attribute error. Instead the computation proceeds, but some of my dtypes are object instead of Categorical, so I guess the metadata is re-inferred?

This is a bug in partd, the micro-library we use for on-disk shuffles.  The code in it that handles Pandas categoricals is incorrectly treating them like numpy arrays instead of special-pandas-things.

Thanks (looks like an interesting library). I'm just going to work around this for now by writing my index column to a separate key in my HDFStore. From reading the docs, it sounds like that'll have better performance since it'll avoid a set_index shuffle.

There is also a sorted=True option to set_index I believe

@TomAugspurger, sorry for just getting to this. I think you might have an old version of partd installed, the following passes for me on both pandas 0.19 and 0.18.1, on both python 2 and 3. Using partd version 0.3.6.
import string
import numpy as np
import pandas as pd
import dask.dataframe as dd
from dask.dataframe.utils import eq

df = pd.DataFrame(np.random.choice(list(string.ascii_letters), size=(100, 4)))
df = df.apply(lambda x: x.astype('category'))
df.index.name = 'idx'
df = df.reset_index()
df['idx'] = df['idx'].astype(str)

ddf = dd.from_pandas(df, npartitions=2)

assert eq(ddf.set_index('idx'), df.set_index('idx'))

👍 upgraded to 3.6 and it works. Thanks.

source: gharchive/issue

authors: TomAugspurger jcrist mrocklin

repo: dask/dask

license_type: permissive

license_source: github-api


text:
Compute forward difference with Dask DataFrame

I'm really excited about Dask.  A huge THANK YOU to all the contributors :)
I'd like to make a feature request for a new method: dask.DataFrame.diff(). It would work like pandas.DataFrame.diff().
Mathematically, the operation is very simple: subtract a column vector from a copy of itself shifted by one or more rows.
I have tried implementing diff() using Dask in the following ways, none of which works (yet):

df - df.shift(periods=1) works in Pandas.  But Dask DataFrame doesn't have a shift() method.
df.values[:-1] - df.values[1:] works in Pandas.  But I can't see how to index into a Dask DataFrame by position.

My current best idea for implementing diff would be to wrap some custom code in dask.dataframe.rolling.wrap_rolling, as suggested in this stack overflow answer (although I haven't been able to find any documentation on how to do this).  Or wrap some custom code using Dask Delayed?  Any other thoughts?

It would be nice to have a higher level function around rolling that accepted user defined functions, much in the same way we do for reduction today.

That would be amazing!  The project I'm working on (NILMTK) requires quite a few custom 'rolling' functions.  So it would be amazing to be able to supply these custom functions to Dask in a simple way, without having to reach into Dask's innards for each new function.

Sure, I can take this on. Definitely agree that a general purpose function would be good here.

See #1769.

authors: JackKelly jcrist mrocklin


text:
read_csv from s3 without listbuckets access

Hi!
I am trying to run the following:
import dask.dataframe as dd

data = dd.read_csv("s3://bucket-name/file-name.csv")
print(data.head())

But it fails due to botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied`. My credentials does not have access to list buckets, but has access to list objects inside the specified bucket.
Full traceback:
Traceback (most recent call last):
  File "dask_test.py", line 8, in <module>
    "s3://bucket-name/file-name.csv"
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/dask/dataframe/io/csv.py", line 579, in read
    **kwargs
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/dask/dataframe/io/csv.py", line 406, in read_pandas
    **(storage_options or {})
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/dask/bytes/core.py", line 116, in read_bytes
    size = logical_size(fs, path, compression)
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/dask/bytes/core.py", line 624, in logical_size
    return fs.size(path)
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/dask/bytes/s3.py", line 54, in size
    return self.info(path)["Size"]
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/s3fs/core.py", line 470, in info
    return super().info(path)
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/fsspec/spec.py", line 467, in info
    out = self.ls(self._parent(path), detail=True, **kwargs)
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/s3fs/core.py", line 488, in ls
    out = self._ls(self._parent(path), refresh=refresh)
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/s3fs/core.py", line 423, in _ls
    return self._lsbuckets(refresh)
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/s3fs/core.py", line 376, in _lsbuckets
    files = self.s3.list_buckets()['Buckets']
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/antonfr/.virtualenvs/dtc/lib/python3.6/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

pip freeze output:
bokeh==1.3.0
boto3==1.9.193
botocore==1.12.193
Click==7.0
cloudpickle==1.2.1
dask==2.1.0
distributed==2.1.0
docutils==0.14
fsspec==0.3.4
HeapDict==1.0.0
Jinja2==2.10.1
jmespath==0.9.4
locket==0.2.0
MarkupSafe==1.1.1
msgpack==0.6.1
numpy==1.16.4
packaging==19.0
pandas==0.25.0
partd==1.0.0
Pillow==6.1.0
pkg-resources==0.0.0
psutil==5.6.3
pyparsing==2.4.1
python-dateutil==2.8.0
pytz==2019.1
PyYAML==5.1.1
s3fs==0.3.0
s3transfer==0.2.1
six==1.12.0
sortedcontainers==2.1.0
tblib==1.4.0
toolz==0.10.0
tornado==6.0.3
urllib3==1.25.3
zict==1.0.0

The reason seems to be that the read_csv function does not pass the full s3 path (s3://bucket-name/file-name.csv) into the s3fs.core module which results in s3fs.core._ls issuing s3fs.core._lsbuckets instead of s3fs.core_lsdirs. https://github.com/dask/s3fs/blob/master/s3fs/core.py#L405-L425
I consider this a bug since since most s3 credentials would not give away access to list other buckets on the account.
Expected behavior
That dask.dataframe.read_csv('s3://bucket/key whould not need ListBuckets access rights but instead only access the single bucket specified.

cc @martindurant

authors: AntonFriberg jcrist


text:
Sequential calls of client.get result in the same delayed object from the cluster

The client.get API when submitted with a delayed method sequentially gets back the same delayed object from the cluster
import dask
from dask.distributed import Client, LocalCluster

import time

cluster = LocalCluster()
client = Client(cluster)

@dask.delayed
def inc(x):
    x = x + 1
    return x

graph_inc_5 = {
    "inc": (inc, 5)
}

graph_inc_10 = {
    "inc": (inc, 10)
}

f1 = client.get(graph_inc_5, "inc")
f2 = client.get(graph_inc_10, "inc")

client.compute(f1).result() # Result is 6
client.compute(f2).result() # Result is 6. Should be 11

However, if I introduce a time delay the results are what I expect
f1 = client.get(graph_inc_5, "inc")
time.sleep(1)
f2 = client.get(graph_inc_10, "inc")

client.compute(f1).result() # Result is 6
client.compute(f2).result() # Result is 11

You are using delayed functions inside of custom task graphs.  This is very strange.  I suspect that you are trying to use too much technology here.  Instead, I recommend that you just use Dask delayed, and not use custom graphs / dictionaries.

Hi Matt!
My colleague's example is a bit contrived here for the sake of brevity.
What we are doing is using Dask in Production for a flexible data processing workflow.
For this, we have defined a set of Function classes in our codebase. Each Function class takes in a Dask data structure (or a list of), does transformations, and returns a Dask data structure (or a list of).
It's natural that some of these Functions take in a list of dask.Delayed objects, and output a transformed list of dask.Delayed objects.
At the time of triggering the workflow, we create a custom graph of these functions following this article ( https://docs.dask.org/en/latest/graphs.html ) then persist/compute it on our cluster.
Below is an example of Functions and our workflow trigger code.
Dataset Load Function
def my_load_function(dataset_specification):
    # dataset_specification contains data-identifying information.
    dataset_data = []

    for url in get_raw_file_urls(..)
        dataset_data.append(dask.delayed(url_load_function(url))

    return Dataset(data=dataset_data)

Dataset Transform Function
def my_transform_function(dataset):
    # dataset.data is a list of dask.Delayed objects
    transformed_dataset_data = []

    for d in dataset.data:
        dataset_data.append(dask.delayed(apply_my_transform(d))

    return Dataset(data=transformed_dataset_data)

Workflow Trigger Code
    graph = {
        "load": (my_load_function, dataset_specification_id),
        "transform": (my_transform_function, "load")
    }

result = compute.graph(graph, "transform") # 'get' is the entry point
client.compute(result.data) # compute the delayed values

Do you see our usage pattern as "too much technology"? If so, will you recommend us on how we can alter our usage pattern?
Thank you!

Tagging @mrocklin since I cannot reopen this issue.

I recommend that you just use Python to organize your many dask collections.  I recommend that you do not use custom graphs.
@delayed
def inc(x):
    return x + 1

x = inc(5)
y = inc(10)

dask.compute(x, y)


text:
roll operation changes chunksize

Is this a bug?
When applying  rolling operation the chunksize changes.
I would expect it to stay constant.
Example:
import dask.array as da
from dask.array import roll
import numpy as np
chunks=(64,64,64)
data=da.from_array(np.random.randn(125, 194, 125).astype('single'),chunks=chunks)
print(data.chunksize)
data_r=roll(data,[-2,-2,-2],axis=[0,1,2])
print(data_r.chunksize)

Output:
(64, 64, 64)
(62, 64, 62)

Possibly related to this
Dask version is 2.16.0

This is not a bug. The .chunksize attribute can be a bit misleading, since dask arrays can be composed of multiple arrays, each with different chunks. I recommend looking at the chunks attribute instead:
In [10]: data.chunks
Out[10]: ((64, 61), (64, 64, 64, 2), (64, 61))

In [11]: data_r.chunks
Out[11]: ((62, 61, 2), (62, 64, 64, 2, 2), (62, 61, 2))

Each tuple is the chunking along one axis. Here we can see that the roll operation took a chunk of 2 off the beginning of each axis (64 -> 62), and appended it to the end of each axis (the new 2 on the end).
We don't want to rechunk so that things are aligned automatically, as that would require moving all of the data around (which could be expensive!). For most uses this is fine - dask array should be agnostic to chunking. If you do need the output to be chunked explicitly, you can use array.rechunk to do so yourself.
Does that make sense?

That makes sense. So chunksize is displayed as max(chunks) ? And can I therefore fully fragment the array by repeated calls to roll?

The chunksize attribute is defined as:
    @property
    def chunksize(self):
        return tuple(max(c) for c in self.chunks)

And can I therefore fully fragment the array by repeated calls to roll?

Meaning break an array that was chunked in large chunks into many smaller chunks by repeatedly calling roll? Yeah. Since that would be application specific, we'd leave it up to you to periodically rechunk to avoid having many many small chunks. Dask works with small chunks, but ideally you want each task to take at least a 1 ms (so the overhead of dask is low in comparison), and small array tasks generally run much faster than that.

Alright, thanks taking the time to explain this.
I'll close the issue.

authors: MaximilianHoffmann jcrist


text:
Adding support for multidimensional histograms with dask.array.histogramdd

This is an attempt to add the equivalent of numpy.histogramdd to Dask (which should make it easy to support numpy.histogram2d, the NumPy version of histogram2d is dependent on histogramdd).
(moved from https://github.com/dask/dask/pull/7346 because of a broken rebase)

I've gone ahead and removed the draft label on the PR, @jakirkham since it looks like you've taken a peek would you be the right person to ask for a proper review at this point? The function is complete w.r.t. being a dask version of what numpy.histogramdd calculates. I view making compatibility with delayed range/bin arguments as a separate PR.

Yep, I'll put it on my todo list. Thanks for working on this Doug! 😄

quick ping @jakirkham to see if you can give this is a first pass when you get a chance

Thanks @jakirkham! and no worries. I'll get to answering them soon

Thanks Doug! 😄

source: gharchive/pull-request

authors: douglasdavis jakirkham


text:
Fix typo in configuration docs

Just a small typo that caught me out!

Can one of the admins verify this patch?

add to allowlist

Thanks @robalar -- merging in

authors: GPUtester quasiben robalar


text:
Lazy computation of array shape for statistical tests

Updates the computation of the number of data points in statistical tests to be lazy, allowing them to handle nan dimensional inputs

[x] Closes #9546
[x] Tests added / passed: 3728 passed, 230 skipped, 8 xfailed, 40 xpassed, 1 warning
[x] Passes pre-commit run --all-files

Can one of the admins verify this patch?
Admins can comment ok to test to allow this one PR to run or add to allowlist to allow all future PRs from the same author to run.

ok to test

authors: GPUtester lrast phobson


text:
Add configuration for key effect and color

Thanks to @Mubramaj comment, I was able to figure out and submit this PR which adds configuration options for effect and color.
Fixes #4.

Thanks @crstauf  for this PR.
Will deploy the new version later today.

authors: Mubramaj crstauf

repo: daskeyboard/daskeyboard-applet--github

license: MIT


text:
Cherry pick for release/0.16 - Unit test of PLUGIN-191

BigQuery Multisink fails on table names with special characters
PR: #538
Unit Tests: #564
JIRA: https://cdap.atlassian.net/browse/PLUGIN-191

/gcbrun

authors: CuriousVini flakrimjusufi

repo: data-integrations/google-cloud

license_source: bigquery


text:
Help on Adding more Methods in API , for grpc

I am using grpc (started with grpc example), and need to add some more methods in API part (in the example, there is set and get methods, I want to add set_extra1, set_extra2, etc.). I changed proto files and everything looks good, but I don't know what should be set in  D part of TypeConfig.
openraft::declare_raft_types!(
pub TypeConfig:
D = THIS ACCEPT JUST ONE METHOD,
R = Response,
Node = Node,
SnapshotData = StateMachineData,
);

D is not a method, it is a type.
https://github.com/datafuselabs/openraft/blob/6908a3cb0f3f7d4481c382bfe68cf4831004cadf/openraft/src/type_config.rs#L60
You should make the type for D a oneof type or something else:
message Requeest {
  oneof request {
    Foo foo = 1;
    Bar bar = 2;
    // ...
  }
}

authors: alidehghan drmingdrmer

repo: databendlabs/openraft


text:
Add DBT_DATABRICKS_UC_INITIAL_CATALOG Option

Resolves #

Description
During the test, I revised the template so that I can provide a catalog of profiles to be used.
https://github.com/databricks/dbt-databricks/blob/main/tests/profiles.py#L53
https://github.com/databricks/dbt-databricks/blob/main/CONTRIBUTING.MD#functional--integration-tests

Checklist

[x] I have run this code in development and it appears to resolve the stated issue
[ ] This PR includes tests, or tests are not required/relevant for this PR
[ ] I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

Thanks for the submission

authors: benc-db case-k-git

repo: databricks/dbt-databricks


text:
Support Series.str.len()

df['col_sum'] = df['col'].str.slice(2, 2+7).replace('0', '').len()
AttributeError: 'Series' object has no attribute 'len'

Series.str.len() is already implemented: https://github.com/databricks/koalas/blob/0fd088e9f142177ef2a80b41d095a49c40320b2a/databricks/koalas/strings.py#L1244-L1276

Thanks.

authors: LSturtew RainFung

repo: databricks/koalas


text:
Introduce default index with new three index types

This PR proposes a default index so that we can now forget about the case when index is missing in Koalas DataFrame  - when Koalas DataFrame is directly created from Spark DataFrame.
There are three types of default index that can be controlled by DEFAULT_INDEX environment variable.

one-by-one: It implements an one-by-one sequence by Window function without
specifying partition. Therefore, it ends up with whole partition in single node.
This index type should be avoided when the data is large. This is default.
TL;DR: Window with row_number without partitioning spec

distributed-one-by-one: It implements an one-by-one sequence by group-by and
group-map approach. It still generates a one-by-one sequential index globally.
If the default index must be an one-by-one sequence in a large dataset, this
index has to be used.
TL;DR: groupby(partition_id).count().collect() and groupby(partition_id).apply(f).

distributed: It implements a monotonically increasing sequence simply by using
Spark's monotonically_increasing_id function. If the index does not have to be
a one-by-one sequence, this index should be used. Performance-wise, this index
almost does not have any penalty comparing to other index types.
TL;DR: moninically_increasing_id().

will fix the tests soon.

Codecov Report

Merging #639 into master will decrease coverage by 2.66%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master     #639      +/-   ##
==========================================
- Coverage   92.95%   90.29%   -2.67%
==========================================
  Files          31       31
  Lines        5093     5119      +26
==========================================
- Hits         4734     4622     -112
- Misses        359      497     +138

Impacted Files
Coverage Δ

databricks/koalas/sql.py
93.75% <ø> (-1.05%)
:arrow_down:

databricks/koalas/namespace.py
87.6% <ø> (-1.61%)
:arrow_down:

databricks/koalas/frame.py
92.81% <ø> (-1.83%)
:arrow_down:

databricks/koalas/internal.py
96.11% <85.71%> (-3.21%)
:arrow_down:

databricks/koalas/usage_logging/__init__.py
23.14% <0%> (-74.08%)
:arrow_down:

databricks/koalas/usage_logging/usage_logger.py
50% <0%> (-50%)
:arrow_down:

databricks/koalas/__init__.py
76.31% <0%> (-7.9%)
:arrow_down:

databricks/conftest.py
90.69% <0%> (-6.98%)
:arrow_down:

databricks/koalas/utils.py
97.43% <0%> (-2.57%)
:arrow_down:

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 82e2e41...4978f80. Read the comment docs.

I am merging this too. Let me know if you guys have more comments.

authors: HyukjinKwon codecov-io