Re: [Starpu-devel] StarPU will cope by trying to purge

[Kadir Akbudak <kadir.akbudak@kaust.edu.sa>]

Dear Samuel,
Thanks for your fast response.

My current application is in-memory and should stay in-memory.

I also used 4 nodes (4nodes x 128GiB/node=512GiB) and I am still
getting the same error for matrix size of 166464-by-166464. When I run
on 8 nodes (1024GiB), the error disappears.

On 4 nodes, when I use the following environment variables
export STARPU_DISK_SWAP=/tmp
export STARPU_DISK_SWAP_BACKEND=unistd_o_direct
I get lots of the following error and my program stalls forever:
[starpu][_starpu_mktemp] Could not create temporary file in directory
'/tmp/448/455', mskostemp failed with error 'Invalid argument'

The reason may be that the compute nodes of the Cray XC40 system that
I use may not have /tmp. In fact compute nodes do not have disks at
all.


Again on 4 nodes, when I use export STARPU_LIMIT_CPU_MEM=0, I get the
following message only then my program stalls forever:
[starpu][_starpu_memory_reclaim_generic] Not enough memory left on
node RAM 0. Your application data set seems too huge to fit on the
device, StarPU will cope by trying to purge 32735 MiB out. This
message will not be printed again for further purges


I will investigate throttling task submission with the
STARPU_LIMIT_MAX_SUBMITTED_TASKS environment variable.

Regarding the shared memory system with 256GiB, I only run unix top
program so I might have missed the spikes in memory consumption. What
is the best way to show real memory consumption of StarPU?
I tried to use starpu_data_display_memory_stats(), but it prints only
"Memory stats :" and nothing else. I used the following configure
command:

CFLAGS=-fPIC CXXFLAGS=-fPIC CC=cc CXX=CC FC=ftn ./configure
--prefix=/akbudak/codes/starpu-1.2.1/install --disable-cuda
--disable-opencl --with-mpicc=/opt/cray/craype/2.4.1/bin/cc
--disable-shared --disable-build-doc --disable-export-dynamic
--disable-mpi-check  --enable-memory-stats


Thanks,
Kadir


On Tue, Jun 20, 2017 at 3:09 PM, Samuel Thibault
<samuel.thibault@inria.fr> wrote:
> Hello,
>
> Kadir Akbudak, on mar. 20 juin 2017 11:28:52 +0300, wrote:
>> [starpu][starpu_memchunk_tidy] Low memory left on node RAM 0 (6545MiB
>> over 130940MiB).
>
> So it's too tight.
>
>> Can I make disable Out-Of-Core (OOC) support of StarPU so that StarPU
>> does not try to purge memory?
>
> Well, purging memory is the cure, not the disease :)
>
> OOC is not enabled by default actually, so enabling it would actually
> fix the issue by letting StarPU push the data that doesn't fit in memory
> onto disk.
>
> export STARPU_DISK_SWAP=/tmp
> export STARPU_DISK_SWAP_BACKEND=unistd_o_direct
>
> Another way is to let StarPU just allocate all the memory it needs:
> export STARPU_LIMIT_CPU_MEM=0
> it will then not care that it doesn't fit, and it's the Operating System
> which will start swapping data out.
>
>> I have 31 threads on a single node. I use 2 nodes. There is 1 MPI
>> process on each node.
>
> Beware that in the MPI case, on each node there is memory needed for the
> reception of the data produced by the other node.
>
>> The following larger experiment (B) generates the above-mentioned warning:
>> M=N=166464 and MB=NB=1156. Total memory for matrices~=206GB,
>
> So that's only a few dozen GB for the MPI receptions, that may not be
> enough. One thing you can do is try to throttle task submission with the
> STARPU_LIMIT_MAX_SUBMITTED_TASKS environment variable. See our research
> paper on the memory consumption issue on
>
> https://hal.inria.fr/hal-01284004
>
> We plan to make this automatic according to memory consumption, but
> that's still on its way.
>
>> I ran Experiment (B) on a system with 256GB RAM. It worked without any
>> warning. On this system, I also ran for 1 MPI process and 2 MPI
>> processes. The amount of memory required for each process scales as
>> expected. That is 206 GB is required for 1 MPI process and 103GB is
>> required for each of 2 MPI processes.
>
> How did you measure it?
>
> There might be spikes that you didn't catch, and that StarPU wants to
> avoid.  In the two-process case, I guess you didn't tell StarPU it has
> to use only half of the memory, so each instance thought it could affort
> 256GB, and was thus happy to, and the OS just had to swap data out so
> that it can actually fit.
>
> Samuel

-- 

------------------------------
This message and its contents, including attachments are intended solely 
for the original recipient. If you are not the intended recipient or have 
received this message in error, please notify me immediately and delete 
this message from your computer system. Any unauthorized use or 
distribution is prohibited. Please consider the environment before printing 
this email.