[Starpu-devel] StarPU will cope by trying to purge

[Kadir Akbudak <kadir.akbudak@kaust.edu.sa>]

Dear StarPU Developers,
I am getting the following error/warning and my program stalls forever:

[starpu][starpu_memchunk_tidy] Low memory left on node RAM 0 (6545MiB
over 130940MiB). Your application data set seems too huge to fit on
the device, StarPU will cope by trying to purge 6548 MiB out. This
message will not be printed again for further purges. The thresholds
can be tuned using the STARPU_MINIMUM_AVAILABLE_MEM and
STARPU_TARGET_AVAILABLE_MEM environment variables.
[starpu][_starpu_memory_reclaim_generic] Not enough memory left on
node RAM 0. Your application data set seems too huge to fit on the
device, StarPU will cope by trying to purge 32735 MiB out. This
message will not be printed again for further purges

I tried to play with these environment variables but, they did not help:
STARPU_MINIMUM_AVAILABLE_MEM
STARPU_TARGET_AVAILABLE_MEM
STARPU_DISK_SWAP_SIZE=0
STARPU_LIMIT_CPU_MEM=110000

I am using Cray XC40 system. Each node has 128GB RAM.
I tried StarPU 1.2.1 and StarPU 1.2.2.
I have 31 threads on a single node. I use 2 nodes. There is 1 MPI
process on each node.

The following experiment (A) does NOT generate any warning:
M=N=139876 and MB=NB=1156. Total memory for matrices~=146GB, each
codelet requires  3/2 MB^2 + 3MB= STARPU_SCRATCH memory. So 31 threads
on a node require 0.015GB*31~=0.5GB temporary buffer.

The following larger experiment (B) generates the above-mentioned warning:
M=N=166464 and MB=NB=1156. Total memory for matrices~=206GB, each
codelet requires  3/2 MB^2 + 3MB=0.015GB STARPU_SCRATCH memory. So 31
threads on a node require 0.015GB*31~=0.5GB temporary buffer.

There are 2x128GB=256GB RAM in total on 2 nodes. Experiment (B) needs
206.5GB. This size is well below the total amount of memory 256GB.
Would you please suggest me how to find out why my program stalls? Can
I make disable Out-Of-Core (OOC) support of StarPU so that StarPU does
not try to purge memory?


I ran Experiment (B) on a system with 256GB RAM. It worked without any
warning. On this system, I also ran for 1 MPI process and 2 MPI
processes. The amount of memory required for each process scales as
expected. That is 206 GB is required for 1 MPI process and 103GB is
required for each of 2 MPI processes.

I appreciate your help on this issue.

Thanks a lot.

Best regards,
Kadir Akbudak
Extreme Computing Research Center

-- 

------------------------------
This message and its contents, including attachments are intended solely 
for the original recipient. If you are not the intended recipient or have 
received this message in error, please notify me immediately and delete 
this message from your computer system. Any unauthorized use or 
distribution is prohibited. Please consider the environment before printing 
this email.