Re: [Starpu-devel] StarPU on IBM Power8 systems

[<vedran.novakovic@stfc.ac.uk>]

Hi Samuel,

Good news:

StarPU (SVN version 21567) indeed builds on Power8LE, with CUDA support.

Configured as:

NVCC_CC=`which xlC_r` NVCC="`which nvcc` -arch sm_60" ./configure 
--prefix=... --enable-dependency-tracking --disable-mpi --enable-long-check 
--disable-valgrind --enable-maxcudadev=4 --disable-opencl 
--disable-fstack-protector-all --enable-verbose=extra --enable-model-debug 
--enable-memory-stats --disable-glpk --disable-ayudame1 --disable-ayudame2 
--enable-openmp --disable-build-examples --enable-blas-lib=none 
--disable-starpufft --disable-build-doc --with-gnu-ld --with-cuda-dir=... 
--without-x

with IBM XL C 13.1.5, Fortran 15.1.5, and CUDA 8.0.

Bad news (but not really bad at all):

Unless some debugging flags that XL compilers don't understand are removed, 
as with:

find . -name 'Makefile' -exec sed -i -e 's/\-gdwarf\-2//g' -e 's/\-g3/-g/g' 
{} \;

(i.e., -gdwarf-2 -g3 to -g)
they fail with

xlc_r: error: 1501-230 Internal compiler error...

With such change the build completes successfully.

Running `make check` gives me these failures so far, for your information (it 
takes a long time to run):

FAIL: datawizard/variable_size

>From variable_size.log:

[starpu][17592869712080][_starpu_create_request_to_fetch_data:datawizard/coherency.c@546]
 <-- (data available)
[starpu][starpu_memchunk_tidy] Low memory left on node RAM 0 (-2MiB over 
1000MiB). Your application data set seems too huge to fit on the device, 
StarPU will cope by trying to purge 2 MiB out. This message will not be 
printed again for further purges. The thresholds can be tuned using the 
STARPU_MINIMUM_AVAILABLE_MEM and STARPU_TARGET_AVAILABLE_MEM environment 
variables.
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(+0x1cab04)[0x10000023ab04]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(starpu_unistd_o_direct_global_async_write+0x130)[0x10000023a3b0]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(_starpu_disk_write+0x194)[0x1000000f39d4]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(_starpu_disk_copy_src_to_disk+0x80)[0x100000185620]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(starpu_interface_copy+0x128)[0x100000197048]
.../starpu-svn/tests/datawizard/.libs/variable_size[0x10001db8]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(+0x126a1c)[0x100000196a1c]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(_starpu_driver_copy_data_1_to_1+0x150)[0x100000195ed0]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(+0x12395c)[0x10000019395c]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(+0x120150)[0x100000190150]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(__starpu_datawizard_progress+0x138)[0x100000195078]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(_starpu_datawizard_progress+0x1c)[0x10000019537c]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(_starpu_wait_data_request_completion+0x7c)[0x10000018efdc]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(+0x13a79c)[0x1000001aa79c]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(+0x137d2c)[0x1000001a7d2c]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(+0x134ea0)[0x1000001a4ea0]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(starpu_memchunk_tidy+0x1c0)[0x1000001a3ba0]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(__starpu_datawizard_progress+0x118)[0x100000195058]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(_starpu_cuda_driver_run_once+0x250)[0x1000002355f0]
.../starpu-svn/src/.libs/libstarpu-1.3.so.0(_starpu_cuda_worker+0x60)[0x100000237020]
/lib64/libpthread.so.0(+0x8728)[0x1000035a8728]
/lib64/libc.so.6(clone+0x98)[0x100003c9d210]

[starpu][starpu_unistd_o_direct_global_async_write][assert failure] The 
unistd_o_direct variant can only write a multiple of page size 65536 Bytes 
(Here 2183168). Use the non-o_direct unistd variant if your data is not a 
multiple of 65536

variable_size: core/disk_ops/disk_unistd_o_direct.c:95: void 
*starpu_unistd_o_direct_global_async_write(void *, void *, void *, off_t, 
size_t): Assertion `(size % getpagesize()) == 0' failed.
[error] `./datawizard/variable_size' killed with signal 6; test marked as 
failed
while looking for core file of ./datawizard/variable_size: core.124854: No 
such file or directory
while looking for core file of ./datawizard/variable_size: core: No such file 
or directory
#Execution_time_in_seconds 9.174380 ./datawizard/variable_size
FAIL datawizard/variable_size (exit status: 1)

FAIL: disk/mem_reclaim

>From mem_reclaim.log:
[starpu][17592186347216][_starpu_submit_job:core/task.c@378] <--
[error] `./disk/mem_reclaim' killed with signal 11; test marked as failed
while looking for core file of ./disk/mem_reclaim: core.2331: No such file or 
directory
while looking for core file of ./disk/mem_reclaim: core: No such file or 
directory
#Execution_time_in_seconds 6.591819 ./disk/mem_reclaim
FAIL disk/mem_reclaim (exit status: 1)

with these tests skipped or expected to fail:

SKIP: main/static_restartable_using_initializer
SKIP: datawizard/interfaces/multiformat/advanced/multiformat_cuda_opencl
SKIP: datawizard/readonly
XFAIL: errorcheck/invalid_blocking_calls

Thank you for a quick response,

Vedran
________________________________________
From: Samuel Thibault [samuel.thibault@inria.fr]
Sent: 20 June 2017 14:53
To: Novakovic, Vedran (STFC,DL,HC)
Cc: starpu-devel@lists.gforge.inria.fr
Subject: Re: [Starpu-devel] StarPU on IBM Power8 systems

vedran.novakovic@stfc.ac.uk, on mar. 20 juin 2017 13:45:55 +0000, wrote:
> Has anyone had any experience with StarPU on IBM Power8 systems (little 
> endian, running Red Hat, using IBM's XL compilers)?
>
> Does it build on such configuration at all and are there any special 
> tweaking efforts or source patches needed?
> How about CUDA and MPI support?

Nothing special should be needed. StarPU is tested on all Debian
architectures, for instance.

Samuel