[Numpy-discussion] calling NumPy from Julia - a plea for fewer macros

[Steven G. Johnson]

Dear NumPy developers,

I've been working on a glue package that allows the Julia language 
(http://julialang.org/) to call Python routines easily
	https://github.com/stevengj/PyCall.jl
and I'm using NumPy to pass multidimensional arrays between languages.

Julia has the ability to call C functions directly (without writing C 
glue), and I've been exploiting this to write PyCall purely in Julia. 
(This is nice for a number of reasons; besides programming and linking 
convenience, it means that I can dynamically load different Python 
versions on the same machine, and don't need to recompile if e.g. NumPy 
is updated.) However, calling NumPy has been a challenge, because of 
NumPy's heavy reliance on macros in its C API.

I wanted to make a couple of suggestions to keep in mind as you plan for 
NumPy 2.0:

1) Dynamically linking to NumPy's C API was challenging, to say the 
least.  Assuming you stick with the PyArray_API lookup table of 
pointers, it would be much easier to call from other languages if you 
include e.g. a numpy.core.multiarray._ARRAY_API_NAMES variable in the 
Python module that is a list of strings giving the symbol names 
corresponding to the  numpy.core.multiarray._ARRAY_API pointer.  (Plus 
documentation, of course.)   Currently I need to parse 
__multiarray_api.h to extract this information, which is somewhat hackish.

2) Please provide non-macro equivalents (exported in the _ARRAY_API 
symbol table or otherwise) of PyArray_NDIM etcetera to access 
PyArrayObject members.  (e.g. call them PyArray_ndim etc.  Note that 
inline functions are not enough, since they are not loadable 
dynamically.)  Right now, the only ways[*] I can see to access this 
information are either to use C glue (which I want to avoid for the 
reasons above) or to call Python to access the __array_interface__ 
attribute (which is suboptimal from a performance standpoint).

Thanks for all your efforts! Any feedback on PyCall would be welcome, too.

--SGJ

[*] A third way would be to parse ndarraytypes.h to extract the format 
of the PyArrayObject_fields structure, and use upcoming Julia support 
for accessing C struct types to read the fields.  This is likely to 
require tracking NumPy releases carefully to avoid breakage, however, as 
well as involving some care with the PyObject_HEAD macro.

PS. If you want to try out PyCall with NumPy, note that a patch to Julia 
is currently required for this to work: 
https://github.com/JuliaLang/julia/pull/2317