Re: [Starpu-devel] Starpu stats about disk, OOC, and swapping

[Amani Alonazi <amani.alonazi@kaust.edu.sa>]

Nice!

I did a benchmark of my node I/O bandwidth and it's 90 MB/s and also confirmed by StarPU benchmark at the beginning of the execution. Am I missing something?

Data transfer speed for /tmp/swap (node 2):
0 -> 2: 79 MB/s
1 -> 2: 78 MB/s
2 -> 0: 79 MB/s
2 -> 1: 78 MB/s
0 -> 2: 34219 µs
1 -> 2: 34230 µs
2 -> 0: 34219 µs
2 -> 1: 34230 µs

 

Data transfer statistics:
*************************
        0 -> 1  256292.63 MB    134.42MB/s      (transfers : 946 - avg 270.92 MB)
        1 -> 0  285238.24 MB    149.60MB/s      (transfers : 1049 - avg 271.91 MB)
        0 -> 2  273821.92 MB    143.61MB/s      (transfers : 1009 - avg 271.38 MB)
        2 -> 0  228862.16 MB    120.03MB/s      (transfers : 842 - avg 271.81 MB)
Total transfers: 1044214.94 MB

Many thanks!

A.

On Mon, Jan 14, 2019 at 7:01 PM Samuel Thibault <samuel.thibault@inria.fr> wrote:

Amani Alonazi, le lun. 14 janv. 2019 17:26:35 +0300, a ecrit:
> Where does it appear as memory node? I see node 0 and node 1 but I can't see
> disk in the stats output.

It should. How do you enable OOC exactly? For instance,

STARPU_BUS_STATS=1 STARPU_DISK_SWAP=/tmp STARPU_DISK_SWAP_SIZE=200  ./examples/basic_examples/vector_scal

gives

Data transfer statistics:
*************************
        0 -> 1  0.00 MB 0.00MB/s        (transfers : 0 - avg -nan MB)
        1 -> 0  0.00 MB 0.00MB/s        (transfers : 0 - avg -nan MB)

I don't have a GPU, so here node 1 is the disk.

In StarPU 1.3 the output will contain the node name:

        Disk 0 -> NUMA 0        0.0000 GB       0.0000 MB/s     (transfers : 0 - avg -nan MB)
        NUMA 0 -> Disk 0        0.0000 GB       0.0000 MB/s     (transfers : 0 - avg -nan MB)

> Speaking of traces, does it take the time spent on GPUs?

It should, yes.

> I don't see it in my fxt output traces. 

Could you post them so we can see what it looks like?

> The execute_on_all_wrapper time sometimes increases dramatically, any idea why?

That could be just the CUBLAS initialization that happens badly
for some CUDA-ish reason which we can't do anything about. You
could check the timing of the starpu_cublas_init() call in
./runtime/starpu/control/runtime_control.c

Samuel

---

This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.