[Starpu-devel] segfault MPI execution and starpu_fxt_tool

[Florent Pruvost <florent.pruvost@inria.fr>]

This is a multi-part message in MIME format.
Hi,

I've got a segfault with the starpu_fxt_tool program for prof_files 
generated with a starpu+mpi program:

I work on an experimental platform from Inria Bordeaux, PlaFRIM2, see 
attached file for the node characteristics

I use starpu branch 1.1 at rev 15648, linked with fxt-0.3.1 and 
openmpi-1.8.4 all compiled with gcc-4.8.4, see config.log.

I have four nodes, my execution is the following (a starpu mpi example):

$ mpirun -pernode ./mpi/examples/stencil/stencil5

Then I want to use starpu_fxt_tool to generate the paje trace file and I 
get the segfault

$ gdb --args 
~/work/install/starpu-1.1/mpi-fxt-debug/gcc-4.8.4/bin/starpu_fxt_tool -i 
~/prof_file_pruvost_*

(gdb) r
Starting program: 
/home/pruvost/work/install/starpu-1.1/mpi-fxt-debug/gcc-4.8.4/bin/starpu_fxt_tool 
-i /home/pruvost/prof_file_pruvost_0 /home/pruvost/prof_file_pruvost_1 
/home/pruvost/prof_file_pruvost_2 /home/pruvost/prof_file_pruvost_3
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff63c4a94 in vfprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install 
glibc-2.17-78.el7.x86_64 libxml2-2.9.1-5.el7_1.2.x86_64 
xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
(gdb) bt
#0  0x00007ffff63c4a94 in vfprintf () from /lib64/libc.so.6
#1  0x00007ffff63cdb77 in fprintf () from /lib64/libc.so.6
#2  0x00007ffff7b7ece6 in _starpu_fxt_dag_set_task_done 
(job_id=<optimized out>, label=<optimized out>, color=<optimized out>) 
at ../../src/debug/traces/starpu_fxt_dag.c:97
#3  0x00007ffff7b7cf70 in handle_task_done (options=0x6020a0 <options>, 
ev=0x7fffffffc6d0) at ../../src/debug/traces/starpu_fxt.c:1058
#4  starpu_fxt_parse_new_file (filename_in=<optimized out>, 
options=options@entry=0x6020a0 <options>) at 
../../src/debug/traces/starpu_fxt.c:1531
#5  0x00007ffff7b7e3a7 in starpu_fxt_generate_trace 
(options=options@entry=0x6020a0 <options>) at 
../../src/debug/traces/starpu_fxt.c:2009
#6  0x0000000000400bea in main (argc=6, argv=0x7fffffffca88) at 
../../tools/starpu_fxt_tool.c:121

I have the same problem with the 1.2 branch.

Any idea? Thanks for your help.

Florent

Attachment: miriel.info
Description: application/info

Machine (128GB total)
  Package L#0
    NUMANode L#0 (P#0 32GB)
      L3 L#0 (15MB)
        L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 
(P#0)
        L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 
(P#2)
        L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 
(P#4)
        L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 
(P#6)
        L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 
(P#8)
        L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 
(P#10)
      HostBridge L#0
        PCIBridge
          PCI 1077:7322
            Net L#0 "ib0"
            OpenFabrics L#1 "qib0"
        PCIBridge
          PCI 8086:10fb
            Net L#2 "em1"
          PCI 8086:10fb
            Net L#3 "em2"
        PCI 8086:8d62
          Block(Disk) L#4 "sda"
        PCIBridge
          PCI 8086:1521
            Net L#5 "em3"
          PCI 8086:1521
            Net L#6 "em4"
        PCIBridge
          PCIBridge
            PCIBridge
              PCIBridge
                PCI 102b:0534
                  GPU L#7 "card0"
                  GPU L#8 "controlD64"
        PCI 8086:8d02
    NUMANode L#1 (P#2 32GB) + L3 L#1 (15MB)
      L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 
(P#12)
      L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 
(P#14)
      L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 
(P#16)
      L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 
(P#18)
      L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU 
L#10 (P#20)
      L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU 
L#11 (P#22)
  Package L#1
    NUMANode L#2 (P#1 32GB) + L3 L#2 (15MB)
      L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + PU 
L#12 (P#1)
      L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + PU 
L#13 (P#3)
      L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + PU 
L#14 (P#5)
      L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + PU 
L#15 (P#7)
      L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16 + PU 
L#16 (P#9)
      L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17 + PU 
L#17 (P#11)
    NUMANode L#3 (P#3 32GB) + L3 L#3 (15MB)
      L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18 + PU 
L#18 (P#13)
      L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19 + PU 
L#19 (P#15)
      L2 L#20 (256KB) + L1d L#20 (32KB) + L1i L#20 (32KB) + Core L#20 + PU 
L#20 (P#17)
      L2 L#21 (256KB) + L1d L#21 (32KB) + L1i L#21 (32KB) + Core L#21 + PU 
L#21 (P#19)
      L2 L#22 (256KB) + L1d L#22 (32KB) + L1i L#22 (32KB) + Core L#22 + PU 
L#22 (P#21)
      L2 L#23 (256KB) + L1d L#23 (32KB) + L1i L#23 (32KB) + Core L#23 + PU 
L#23 (P#23)
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by StarPU configure 1.1.5, which was
generated by GNU Autoconf 2.69.  Invocation command line was

  $ ../configure --with-fxt --prefix=/home/pruvost/work/install/starpu-1.1/mpi-fxt-debug/gcc-4.8.4

## --------- ##
## Platform. ##
## --------- ##

hostname = devel11
uname -m = x86_64
uname -r = 3.10.0-229.4.2.el7.x86_64
uname -s = Linux
uname -v = #1 SMP Wed May 13 10:06:09 UTC 2015

/usr/bin/uname -p = x86_64
/bin/uname -X     = unknown

/bin/arch              = x86_64
/usr/bin/arch -k       = unknown
/usr/convex/getsysinfo = unknown
/usr/bin/hostinfo      = unknown
/bin/machine           = unknown
/usr/bin/oslevel       = unknown
/bin/universe          = unknown