Re: [Starpu-devel] StarPU will cope by trying to purge

[Samuel Thibault <samuel.thibault@inria.fr>]

Hello,

Kadir Akbudak, on mar. 20 juin 2017 11:28:52 +0300, wrote:
> [starpu][starpu_memchunk_tidy] Low memory left on node RAM 0 (6545MiB
> over 130940MiB).

So it's too tight.

> Can I make disable Out-Of-Core (OOC) support of StarPU so that StarPU
> does not try to purge memory?

Well, purging memory is the cure, not the disease :)

OOC is not enabled by default actually, so enabling it would actually
fix the issue by letting StarPU push the data that doesn't fit in memory
onto disk.

export STARPU_DISK_SWAP=/tmp
export STARPU_DISK_SWAP_BACKEND=unistd_o_direct

Another way is to let StarPU just allocate all the memory it needs:
export STARPU_LIMIT_CPU_MEM=0
it will then not care that it doesn't fit, and it's the Operating System
which will start swapping data out.

> I have 31 threads on a single node. I use 2 nodes. There is 1 MPI
> process on each node.

Beware that in the MPI case, on each node there is memory needed for the
reception of the data produced by the other node.

> The following larger experiment (B) generates the above-mentioned warning:
> M=N=166464 and MB=NB=1156. Total memory for matrices~=206GB,

So that's only a few dozen GB for the MPI receptions, that may not be
enough. One thing you can do is try to throttle task submission with the
STARPU_LIMIT_MAX_SUBMITTED_TASKS environment variable. See our research
paper on the memory consumption issue on 

https://hal.inria.fr/hal-01284004

We plan to make this automatic according to memory consumption, but
that's still on its way.

> I ran Experiment (B) on a system with 256GB RAM. It worked without any
> warning. On this system, I also ran for 1 MPI process and 2 MPI
> processes. The amount of memory required for each process scales as
> expected. That is 206 GB is required for 1 MPI process and 103GB is
> required for each of 2 MPI processes.

How did you measure it?

There might be spikes that you didn't catch, and that StarPU wants to
avoid.  In the two-process case, I guess you didn't tell StarPU it has
to use only half of the memory, so each instance thought it could affort
256GB, and was thus happy to, and the OS just had to swap data out so
that it can actually fit.

Samuel