Re: [Starpu-devel] Deadlock with starpu_calibrate

[Samuel Thibault <samuel.thibault@inria.fr>]

Mathieu Faverge, on Sun 04 Dec 2016 19:53:01 +0100, wrote:
> Here it is:
> #0  0x00007ffff7ff9b24 in clock_gettime ()
> #1  0x00007ffff7858ea6 in clock_gettime () from /lib64/librt.so.1
> #2  0x00007ffff3be2f1e in ?? () from /usr/lib64/nvidia/libcuda.so.1
> #3  0x00007ffff3c70325 in ?? () from /usr/lib64/nvidia/libcuda.so.1
> #4  0x00007ffff3ba890e in ?? () from /usr/lib64/nvidia/libcuda.so.1
> #5  0x00007ffff3ba8b01 in ?? () from /usr/lib64/nvidia/libcuda.so.1
> #6  0x00007ffff3ae192a in ?? () from /usr/lib64/nvidia/libcuda.so.1
> #7  0x00007ffff3c0b648 in cuCtxSynchronize () from
> /usr/lib64/nvidia/libcuda.so.1
> #8  0x00007ffff6dca179 in ?? () from /opt/cuda/8.0/lib64/libcudart.so.8.0
> #9  0x00007ffff6deb5f9 in cudaThreadSynchronize () from
> /opt/cuda/8.0/lib64/libcudart.so.8.0
...

> And if you do a strace, the number of calls to clock_gettime is just insane.

So it really looks like CUDA getting stuck in there.  Not StarPU's fault
then.  You could try to set

export STARPU_ENABLE_CUDA_GPU_GPU_DIRECT=0

in case CUDA has troubles with GPU direct.

Mathieu Faverge, on Sun 04 Dec 2016 19:59:55 +0100, wrote:
> Ok I'll check but it might just be that I didn't use the same cuda on the
> compilation and compute node.

That could be a problem indeed, I'm not really surprised by CUDA bugs
any more :)

Samuel