Since the TRICAP2018 conference, where it was decided that the signal processing community should try to work together to produce, among others, an efficient software for tensor problems (see the presentation here), nothing has really been done to actually start the development of such a toolbox. This is probably due to the immense work that is required to lay the first stones to this project. In this post, I would like to draw the attention of our community on a newly developped package for Python: TensorLy. I will first present the package, then discuss its advantages and flaws with respect to the other toolboxes available for MATLAB such as Tensorlab or Tensor Toolbox, and finally I will try to provide a guide map for how to improve TensorLy to match our goals.
Note: If any statement in the following discussion appears to be even partially incorrect, please mention it asap in the comments below, or email me at tensor(dot)world(at)gmail(dot)com.
TensorLy, a front-end library for Python
Although this was new to me before I started doing some research on tensor packages this summer (2018), it appears the question of lacking tensor decompositions software on free platforms such as Python has been around for quite some time.
In particular, this article from 2015 already states partially the problem of lacking efficient tensor software, especially for the machine learning community, in spite of obvious interest in such tools. Anima Anandkumar, who was interviewed at the time, and has been promoting the use of tensors in machine learning for quite some time, recently associated with Jean Kossaifi and Yannis Panagakis to try and fill this gap.
TensorLy is the result of this process. It is a Python library built upon widely used libraries, in particular NumPy and PyTorch. This means that, when using the TensorLy syntax to implement tensor factorizations and manipulations, one does not need to worry about the way basic operations are handled. In particular, using PyTorch as a back end, it is possible to make use of GPUs to speed up computations in a completely transparent manner to the user.
TensorLy is relatively recent project, thus only a few functions of interest for signal processing people are implemented, namely PARAFAC, Tucker and their nonnegative counterparts.
What is missing for TensorLy
Before stating what I think is a very important chunk of content missing from TensorLy, I want to point out that it does a good job, in its current version, at handling tensor operations such as vectorizations, matricizations, tensor computation from the Kruskal form, Kronecker products and so on. This is because TensorLy relies on other efficient implementations of very basic tensor operations (provided for instance by PyTorch). To illustrate how TensorLy works, here is a shamelessly taken image from the official publication link:
However, for those familiar with tensor decomposition problems, we know that the set of mathematical tools required is quite large. In short, tensor decompositions are based on, at least, four ingredients:
- data visualization, in order to asses the type of tensor model to apply, to estimate the rank, to remove corrupted data, and so on.
- data manipulation, which was discussed above.
- optimization, to efficiently identify the parameters of decomposition models. This is a crucial point, that has been extensively discussed in the tensor literature and has known quite important improvements over the years.
- modeling, which means being able to specify complex structures on the tensor data, as well as the tensor decomposition parameters. The champion toolbox for flexible modeling is the MATLAB tensorlab v3.0 and further, see the docs for many detailed examples.
Sadly, TensorLy falls somewhat short in anything related to visualisation, optimization and modeling. To illustrate this, I have used the example of Nonnegative CPD/PARAFAC (which I know quite well, thus the choice). See this Jupyter Notebook for a tentative comparison of the native TensorLy solution with the numpy nonnegative least squares solver used alternatively.
We should help each other!
These missing pieces in TensorLy are exactly what us, signal processing researchers, are good at. Personally, I think trying to start from scratch to make yet another tensor toolbox on Python would be quite stupid, since libraries such as TensorLy and PyTorch already offer a lot of flexibility.
Therefore, I think the (curently void) toolbox part of the TensorWorld project should merge with the TensorLy project, to incorporate the previously mentioned features. In particular, I myself am very sensitive to the optimization issue. Right now, TensorLy implements ALS PARAFAC and Tucker, and multiplicative update (ugh) for nonnegative PARAFAC. It is possible to use methods inherited from the back-ends, such as BFGS and Stochastic gradient. However, I would love the following to be easily doable in TensorLy:
- Using CVXPY, which makes optimization easy to write and is efficiently implemented. CVXPY is based on conic optimization which is extremely flexible and powerful. Recent works have shown that proximal methods used in an alternating fashion perform very well for a wide variety of tensor decomposition problems. Therefore, CVXPY can be used straightforwardly on the per-mode derivation of most decomposition problems, see the example notebook.
- Using personal code for specific problems, such as nonnegative least squares. For instance, I like the A-HALS method for solving NNLS. Supposing this method is implemented in a separate module, I would like to be able to call it to perform the NNLS steps in Nonnegative PARAFAC.
I am not a Python expert nor a hardcore programmer, but at the very least, using CVXPY as a back-end seems reasonable. In any case, we need more constrained decompositions available. Also, multiplicative update is not a great algorithm in most scenarios, see for instance this article for the matrix case.
- TensorLy is a cool project, but could be much cooler with more people contributing (a.k.a. this community).
- It is a starting point that we can build upon, not just a pile of words with no actions following up. This makes our life easier.
- Contributing is accessible to anyone who knows Python, and the developers have a detailed description on how to contribute.
Let me know what you think of all this in the comments below! Also, please take some time (2 minutes) to complete the survey on this website!
EDIT (12/10/2018): after actually testing CVXPY for tensor decompositions, I am starting to think that maybe implementing methods such as NNLS, or AODAMM for a more general framework, would be very beneficial. It does not change the fact that optimization is one key point we could start working on alongside the Tensorly People.