Re: [Starpu-devel] Problems with memory in CUDA

[Cyril Bordage <cyril.bordage@inria.fr>]

The "out of memory" can occur at different locations.
I have found one example again : in cudaMemset for the initialization of
a redux handle.
#0  0x00007fffec9271e5 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fffec92a398 in __GI_abort () at abort.c:90
#2  0x00007fffec920272 in __assert_fail_base (fmt=0x7fffeca5d9c0
"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x4ae460 "_b == cudaSuccess",
    file=file@entry=0x4ae435 "utils/starpu.c", line=line@entry=157,
function=function@entry=0x4ae640 <__PRETTY_FUNCTION__.14205>
"init_vector_cuda_func") at assert.c:92
#3  0x00007fffec920322 in __GI___assert_fail (assertion=0x4ae460 "_b ==
cudaSuccess", file=0x4ae435 "utils/starpu.c", line=157,
    function=0x4ae640 <__PRETTY_FUNCTION__.14205>
"init_vector_cuda_func") at assert.c:101
#4  0x000000000043e16a in init_vector_cuda_func (buffers=0x7fff7f244978,
cl_arg=0x0) at utils/starpu.c:157
#5  0x00007ffff491f1b4 in _starpu_redux_init_data_replicate
(handle=0x7fff7f2446f0, replicate=0x7fff7f244960, workerid=0) at
datawizard/reduction.c:78
#6  0x00007ffff490ec37 in _starpu_fetch_task_input (j=0x7fff7591a8b0,
mask=0) at datawizard/coherency.c:703
#7  0x00007ffff493bd5a in execute_job_on_cuda (j=0x7fff7591a8b0,
args=0x7ffff4b71230 <config+1040>) at drivers/cuda/driver_cuda.c:335
#8  0x00007ffff493c51e in _starpu_cuda_driver_run_once
(d=0x7fffeb71de50) at drivers/cuda/driver_cuda.c:477
#9  0x00007ffff493c6db in _starpu_cuda_worker (arg=0x7ffff4b71230
<config+1040>) at drivers/cuda/driver_cuda.c:537
#10 0x00007fffecca5e0e in start_thread (arg=0x7fffeb71e700) at
pthread_create.c:311
#11 0x00007fffec9da9ed in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113


Cyril.


Le 21/10/2013 15:26, Samuel Thibault a écrit :
> Cyril Bordage, le Mon 21 Oct 2013 02:09:45 +0200, a écrit :
>> I call starpu_data_unregister_submit to free memory. It works well on
>> CPUs, but only on CPUs, not on GPUs. To be more precise, I see, with
>> nvidia-smi, only increasing in memory consumption. And it ends with a
>> "error: out of memory". If I split my program into two passes, it works
>> like a charm (no out of memory).
> 
> Well, it is expected that StarPU does not release memory immediately,
> for two reasons:
> 
> - it releases GPU memory from the GPU-driving thread, not from the thread
> making the free call. Releasing data is triggered when the GPU is low on
> memory.  What is returning an out of memory error exactly?  That part
> should be triggered releasing data, if it isn't there's a bug there.
> - for GPUs there is an allocation cache, to allow free/alloc data
> reuses.  Are your data all the same, or of varying sizes & types?  In
> the latter case, the cache will not be very useful, but again, it should
> get flushed on low memory condition.  You can try to avoid it completely
> by passing --disable-allocation-cache to configure.
> 
> Samuel
>