Re: [Starpu-devel] StarPU's limitations

[Samuel Thibault <samuel.thibault@ens-lyon.org>]

Hello,

Chris Hennick, le Tue 21 Feb 2012 10:59:46 +0100, a écrit :
> The impression I get from the manual is that StarPU's main limitations
> are these:

Among a lot others :)

>   • New drivers can't be added modularly, because they each have a different
>     *_funcs[] member in struct starpu_codelet.

Yes. There are several reasons why we didn't try to make it modular:
- with dynamically-loaded drivers, the way to bind the content of funcs
members in starpu_codelet becomes way more complex, and applications
would not be able to use static initializers any more.
- each interface needs to declare how to transfer data between memory
nodes. We however plan on making this general, we just haven't got
around doing it yet.
- The automatic format conversion according to PU type also comes into
play, with yet more hardcoded foo_to_bar members which would be more
difficult to use with dynamic loading.
- We don't need to add new drivers that often.

>   • StarPU-MPI, which seems to be the only framework available for 
> distributed
>     systems, can't transparently handle replication

We actually have experimental code to have a cache in the MPI transfers:
as long as some data is not written to, its copies on the various nodes
are kept and no MPI transfer is done. The code is however not working at
the moment, and thus disabled for now.

> or failover,

You mean machine failover?  Well, yes, and I believe we'd need to
rethink quite a bit the structure of StarPU if it had to cope with tasks
which failed.

> and performance would probably suffer if these were implemented at a
> lower level (e.g. by running on top of MOSIX).

Yes. The big advantage of doing communications at the StarPU level is
that it since it knows the task/data relations, it is able to transfer
whole pieces of data in just one bloc, and not page per page.

>   • It can't optimize the choice of scheduling policy or performance models
>     based on the trade-off between accuracy and overhead.

Do you mean e.g. single precision vs double precision?  A problem I see
is that this involves the application quite a bit, making the interface
more complex. But that could be interesting to have a look at for e.g.
cg/gmres etc.

>   • It can't dynamically switch an Nvidia GPU between CUDA and OpenCL (or, I
>     assume, a CPU between OpenCL and native mode). 

Indeed. One problem is that we can't use a GPU both in CUDA and OpenCL
mode, so we'd have to completely switch it. Another issue is that some
applications may at initialization probe the architecture, and wouldn't
cope with changes there.

That said, when an application has both CUDA and OpenCL functions for
a codelet, or both CPU and OpenCL functions for a codelet (e.g. OpenCL
for everything, and CUDA/CPU-optimized for some of them), it could be
interesting to do the switch. That would require some changes here and
there, but nothing too fundamental.

Of course, the hard research problem then becomes how to decide the
switch :)

>   • It won't support "CPU-assisted GPGPU" (see 
> http://arstechnica.com/business/
>     news/2012/02/
>     
> researchers-boost-processor-performance-by-getting-cpu-and-gpu-to-collaborate.ars
>     ), because parallel tasks can't specify that a particular implementation
>     requires a shared cache and a split across multiple specific drivers (in
>     this case, CPU and OpenCL).

Yes, parallel tasks are for now quite simple, but with not too much
changes, special cases could probably be made to handle these.

>   • In power-based scheduling, there's no provision for dynamically 
> adjusting
>     gamma (e.g. if we're on a smart-metered power grid, or a laptop that's
>     sometimes unplugged),

Well, there is no programming interface for it, but gamma is a
variable in the scheduler, which can be changed and the scheduler will
follow. Actually, the StarPU-Top GUI permits to do it.

> nor for optional best-effort tasks that aren't always
>     worth their power cost.

I've been wondering about possibly cancelling tasks for some time. The
difficult part here is how to trade "worth" and "power cost". How to
quantify "worth"? :)

>   • The same StarPU instance can't securely serve multiple users.

We rarely needs this in the HPC context where users have dedicated
CPUs/GPUs anyway indeed :)

Users can't share one StarPU process since their codelets would then run
in the same addressing space, which is no good. So it has to be somehow
distributed.

One thing that we consider is to have communication between instances of
StarPU, which would declare CPU/GPU reservation, probably with a server
which centralizes reservations. That way, e.g. HEFT would be notified
of the use of CPUs/GPUs by others, and schedule and make reservations
accordingly.

> Have I missed any

We have a lot of things on the TODO list which we aren't working on
already :) For instance,
- partial data_memcpy() using user-provided layout
- asynchronous partition/unpartition (we already have something
experimental on it)
- use an initial value for scratch access
- include MPI nodes in the generated DAGs
- have a scheduling "window": make starpu_submit block when 'n' tasks
have already been submitted, to avoid filling memory with myriads of
tasks when the computation is huge (expected to last months).
- ...

> Which of them do the team consider high priorities, and which ones are
> being worked on? I don't want to duplicate effort, and I'd like my
> thesis to be useful to the "real world". Thanks in advance.

Well, we already work on high priority items :)

Among what is mentioned here, I'd say the priority could go to fixing
the MPI cache, CUDA/OpenCL CPU/OpenCL switch, centralized reservations.

Samuel