Re: [Starpu-devel] execution time takes forever with cuda+large matrices+ooc test case

[Amani Alonazi <amani.alonazi@kaust.edu.sa>]

I tried it on V100+spinning desk/P100+SSDs/V100+SSDs. Unfortunately, it does freeze with large matrices.

t appears to be connected to three things:

1) [starpu][starpu_interface_end_driver_copy_async] Warning: the submission of asynchronous transfer from NUMA 0 to Disk 0 took a very long time (0.452278 ms)
For proper asynchronous transfer overlapping, data registered to StarPU must be allocated with starpu_malloc() or pinned with starpu_memory_pin()

2) Right after, the message about updating the perfmodel history:

[starpu][_starpu_update_perfmodel_history] Too big deviation for model ssrc on cpu0_impl0 (Comb1): 44.715000 vs average 15.980000, 1 such errors against 1 samples (+179.818523%), flushing the performance model. Use the STARPU_HISTORY_MAX_ERROR environement variable to control the threshold (currently 50%)

3) Out-of-core using dmda scheduler. 

Many thanks for your time.

A.

On Mon, Feb 11, 2019 at 11:56 AM Amani Alonazi <amani.alonazi@kaust.edu.sa> wrote:

Dear starpu-dev,

A deadlock case (or extremely very very bad execution) is occurring from time to time. I'm sharing here a test case example that takes two matrices A and U of size M * NK, one matrix R of size NK * NK, and matrix W of size nM * NK with OOC enabled. You can run the application using this command:

STARPU_LIMIT_CPU_MEM=109600 STARPU_GENERATE_TRACE=1 STARPU_STATS=1 STARPU_WORKER_STATS=1 STARPU_PROFILING=1  STARPU_DISK_SWAP=$SWAP_DIR STARPU_DISK_SWAP_BACKEND=unistd STARPU_BUS_STATS=1 STARPU_SILENT=0 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 ./tb-modeling -i 1000 --freq=10 -m 250 -n 250 -k 250

I attach the code here. Please help.

Thanks,

Amani AlOnazi
PhD Student
Extreme Computing Research Center

King Abdullah University of Science and Technology
Al-Khawarizmi (Bldg.1)
Floor 0, Office 0203-WS02
Email: amani.alonazi@kaust.edu.sa

www.kaust.edu.sa

---

This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.