Zotero Reader for Android

Posted on by

Zotero Reader web

A few months back, I announced a web application that gives you access to your Zotero library from mobile devices. I’m thrilled to announce that Zotero Reader for the Web is now available for free at zoteroreader.com. It offers the following features:

  • View, star, mark items as read in your Zotero library
  • Download PDFs, upload edited versions
  • Collections, tags, and full text search
  • Access to your public shared libraries
  • Automatically find PDFs for orphaned items

It’s available for free at zoteroreader.com – since it’s a web app, it runs in a web browser, so it works on Android phones and tablets, iPhone/iPad, as well as your desktop web browser (not IE).

Fundraising campaign

With the help of web developer and longtime buddy Philippe Vachon-Rivard, we’re simultaneously launching a fundraising campaign on Indiegogo for Zotero Reader for Android. This will be a native Android app, based on the web version. The native app will be immensely better than the (already pretty good) web version. Most exciting is background syncing, which will allow the app to seamlessly fetch your library and PDFs in advance; that way, you’ll get offline access to your papers: in the subway, on the plane, in a conference with shifty Wi-Fi.

Breaking out of the browser sandbox will also allow the app to upload your edited PDFs to Zotero without your intervention: your notes and highlights will be synced across your electronic devices. The app will also be much faster thanks to a native UI. The code for the app will be released on github.

All in all, our goal is to make it as easy to read papers on your tablet as it is to read a book on an ereader.

Our funding goal is 6,000$. That money will be going into Phil’s development time and to fund the server for about a year. It runs until June 20th.

This is an all-or-nothing funding campaign – like Kickstarter, if we don’t reach the funding goal by the end date, you get refunded. That means you have nothing to lose – you will only get debited if the campaign is succesful and we can go ahead with development. The final app will be free – every dollar you contribute will help you and your fellow scientist keep up with the scientific literature.

If you’re broke, you can still contribute! Just tweet about the campaign or link to it on your blog or Facebook.

Whiten images in Matlab

Posted on by

Previously, I showed how to whiten a matrix in Matlab. This involves finding the inverse square root of the covariance matrix of a set of observations, which is prohibitively expensive when the observations are high-dimensional – for instance, high-resolution natural images.

Thankfully, it’s possible to whiten a set of natural images approximately by multiplying the data by the inverse of the square root of the mean power spectrum in the Fourier domain. Assume that X is a matrix of size (nimages,height,width), then:

mX = bsxfun(@minus,X,mean(X)); %remove mean
fX = fft(fft(mX,[],2),[],3); %fourier transform of the images
spectr = sqrt(mean(abs(fX).^2)); %Mean spectrum
wX = ifft(ifft(bsxfun(@times,fX,1./spectr),[],2),[],3); %whitened X

Why this works

It’s not obvious at all that this should give similar results to the other method, which involved finding the inverse square root of the covariance matrix M. Let’s assume that the images are stationary – that is, the second order statistics are similar at any point in the image. That means that the elements of the covariance matrix should only depend on the relative position between two pixels. And of course, M is a symmetric matrix.

That means that M is a matrix such that the product Mx performs the convolution between x and the kernel associated with M – call this kernel m. That kernel is actually the sum of the autocorrelation of the images. If Mx = m * x , then by the convolution theorem, Mx = F^-1(F(m)\cdot F(x)). The Fourier transform F(x) is a linear mapping from an n-dimensional complex vector to another n-dimensional complex vector. Therefore, F(x) = \hat F x, where \hat F is an n-by-n dimensional matrix.

Then, we have that Mx = \hat F^{-1} \text{diag}(\hat F m) \hat F x. If we take images X and transform them via Y' = \hat F^{-1} \text{diag}(1 \// ({\hat F m})^{-1 \// 2}) \hat F X', then it’s easy to verify that the covariance of Y will now be a scalar times the identity matrix. Therefore, modulo edge effects, which break the stationarity requirement, images can be whitened by dividing them in the Fourier domain by the inverse of the root of the mean power spectrum.

Canopy scientific Python editor for Windows

Posted on by

In my last post on IDEs for scientific Python, I couldn’t install, and therefore couldn’t properly review Canopy, a commercial IDE developed by Enthought, who sponsor SciPy. I had a chance to install it on Windows and try it.

sc1

Canopy’s main screen shows three options: Editor, Package Manager and documentation browser. The package manager offers a graphical interface to perform much the same tasks as easy_install and pip. The documentation browser offers shortcuts for the online docs for Scipy, matplotlib, and more; it would be preferable, IMHO, if the installation included offline copies of this documentation.

sc2

The main editor interface is uncluttered and fairly basic. ipython is used as the interpreter, and interestingly, the interface offers an option for including matplotlib graphics inline via SVG rather than in separate windows. Canopy’s editor is fairly smart, offering autocompletion and basic introspection, on par with that of Spyder but not as advanced as that in PyDev. docstrings pop up in a tooltip on hitting the Tab key after an opening parenthesis. The editor supports running both a complete file and a selection via run tools.

And – that’s pretty much it. There’s no function browser, project manager or graphical debugger – although post-hoc debugging is supported in the ipython console via the debug command. Although I appreciate that the interface is solid and uncluttered – this was the main positive I mentioned when discussing IEP – I was expecting more from a commercial product, especially one that costs 199$ for the 64-bit Windows version. Although free versions are available for 32-bit platforms and academic users, support is limited, AFAIK, to StackOverflow.

This is a new product, and it seems that Enthought wants to push in the direction of making scientific Python more accessible for novices by removing the main barriers to entry – the next version promises to have integrated e-learning tutorials in the interface. This is a noble goal, although I can see other, perhaps more productive ways to accomplish the same goal, for instance:

  • Interactive notebooks on specific themes, running in ipython Notebook, similar to Maple’s tutorials
  • Graphical access to plotting commands and common analysis commands, as in Matlab and RKWard
  • A Matlab-to-Python table of equivalence integrated in the documentation
  • Wizards for common tasks (for instance, a data import tool)
  • Written introduction to Python with emphasis on science in the documentation, like here.

I like the editor, and in terms of bare-bones, smooth editors it’s a little better than IEP. In terms of features, however, Spyder is better. It’s a good option if you can get it for free (academic or 32-bit versions).

  • Editor: 3.5/5
  • Interpretor integration: 4/5
  • Plotting: 4/5
  • Debugging: 2/5
  • Smooth: 4/5

Overall score: 3.5 if you can get it for free, 2.5 otherwise.

SciTrends

Posted on by

scitrends

I’ve just put the finishing touches on SciTrends, a new website which allows you to browse science articles which are being talked about on Twitter and read on CiteULike. Kind of like hackernews or 9gag, but for science instead of tech or stupid BS. It’s a thin layer over Altmetric – watch out for the spinning logo. Try it today!

 

Deinterlace a movie in Matlab

Posted on by

Here’s a simple function to deinterlace a movie that was interlaced to go from 24fps to 30fps:

Woody deinterlaced

function [As] = deinterlace(As)
    %function [As] = deinterlace(As)
    %Deinterlaces a video input by inverse 3:2 pulldown
    %As a 3D matrix of size (vsize x hsize x nframes)
    %The function assumes that for every 5 frames, 2 of these are
    %interlaced (transfer from a 24fps source to a 30fps output)
    %When it detects two consecutive frames are interlaced:
    %    im1 = As(:,:,jj);
    %    im2 = As(:,:,jj+1);
    %It tries to reassemble them in two ways:
    %    im0 = im1;
    %    im0(2:2:end,:) = im2(2:2:end,:);
    %
    %    im3 = im1;
    %    im3(1:2:end,:) = im2(1:2:end,:);
    %Looks at which looks best via computing the sum of the squares of the 
    %vertical second order derivatives, and replaces each frame with a copy
    %of whichever assembly looked best:
    %
    %if s0 < s3
    %    As(:,:,jj) = im0;
    %    As(:,:,jj+1) = im0;
    %else
    %    As(:,:,jj) = im3;
    %    As(:,:,jj+1) = im3;
    %end
    %
    %Author: Patrick Mineault
    %http://xcorr.net

    
    %Detect which frames are interlaced
    %assumes every 5th + offset and 5th + offset + 1'th frames are
    %interlaced
    interidx = sum(sum(convn(As,[-1,2,-1]','same').^2,1),2);
    interidx = interidx(1:floor(length(interidx)/5)*5);
    n = median(reshape(interidx,5,length(interidx)/5),2);
    [~,idx] = sort(median(n,2));
    thewhich = idx(end-1:end);
    
    if(min(thewhich) == max(thewhich) -1 )
        offset = min(thewhich);
    elseif(max(thewhich) == 5 && min(thewhich)==1)
        offset = max(thewhich);
    else
        error('WTF???');
    end
    
    %And deinterlace
    for jj = offset:5:size(As,3)-1
        im1 = As(:,:,jj);
        im2 = As(:,:,jj+1);
        
        %try both ways
        im0 = im1;
        im0(2:2:end,:) = im2(2:2:end,:);

        im3 = im1;
        im3(1:2:end,:) = im2(1:2:end,:);
        
        s0 = sum(sum(conv2(im0,[-1,2,-1]','same').^2));
        s3 = sum(sum(conv2(im3,[-1,2,-1]','same').^2));

        %replace
        if s0 < s3
            As(:,:,jj) = im0;
            As(:,:,jj+1) = im0;
        else
            As(:,:,jj) = im3;
            As(:,:,jj+1) = im3;
        end
    end
end

Online courses in computational neuroscience, machine learning, stats

Posted on by

Browsing through the courses available on Coursera, I noticed a class on Neural Nets by none other than Geoffrey Hinton. Lots more courses have been added since I looked at it last year, including:

Memory management in gnumpy

Posted on by

I’m working on a project using gnumpy, which is a wrapper for CUDAmat, a toolbox for GPU computation in Python. I was having memory management issues, where the script would crash after the memory on the graphics card would fill up. I kept trying to delete the variables via del but this seemed to have no effect.

Digging into the internals of gnumpy, once del is called it does not actually delete the object; rather, it marks it as overwritable. I assume this is because allocating memory on the GPU is expensive, so rather than deleting the object it sets it up for recycling; clever. However, it will only reuse an existing deleted array if the new array is of the same size. I was using batches of data of unequal sizes, and pushing them on the GPU as I went along; after a few batches, the GPU was filled up with arrays marked as overwritable.

There are three possible solutions:

  • Make the batches of equal size. This is the most efficient, if least flexible solution – it does not involve allocating and deallocating memory
  • Call gnumpy.free_reuse_cache – this deallocates memory and triggers garbage collection. This is slow
  • Deallocate the memory associated with a given array manually. This may be done as follows:
import gnumpy as gpu
def force_deallocate(thesize):
    try:
        gpu._cmsForReuse[thesize].pop()
        gpu.__memoryInUse -= thesize*4
        del gc.garbage[:]
    except IndexError:
        #Can't deallocate if it don't exist, can ya?
        pass

#assuming you have an array called a
thesize = a.size
del a #delete variable
force_deallocate(thesize) #deallocate the corresponding memory