Tagged: ipython Toggle Comment Threads | Keyboard Shortcuts

  • raj 11:44 pm on June 26, 2013 Permalink | Reply
    Tags: ipython, uploader   

    Plotting uploads/day 

    We are beta testing a HTML5 Uploader, and I wanted to see how much use it was getting. Here is how I plotted the usage. First, install ipython html notebook, then install matplotlib 1.3rc2 to use the xkcd style, then start iPython notebook:

    $ source ~/pyenvs/notebook/bin/activate
    $ ipython notebook --profile=nbserver

    Then, in iPython notebook, use the archive.org advanced search engine to see how many uploads per day we were getting:

    import json
    import urllib
    import datetime as dt
    urlbase = 'http://archive.org/advancedsearch.php?%s'
    date = dt.datetime.utcnow()
    for i in range(90):
        date -= dt.timedelta(days=1)
        params = {'q':'scanner:"Internet Archive HTML5 Uploader" AND publicdate:'+date.strftime('%Y-%m-%d'),
        url = urlbase % urllib.urlencode(params)
        f = urllib.urlopen(url)
        o = json.load(f)

    Now that I have the data, I can plot it, using xkcd-style:

    #Turn on XKCD style
    plot(x, y)
    plt.title("HTML Uploader Items/Day")
    #Save PNG
    fig = plt.gcf()

    It looks like this:

  • raj 11:04 pm on June 19, 2013 Permalink | Reply
    Tags: ipython, , xkcd   

    I wanted to update the virtualenv I use… 

    I wanted to update the virtualenv I use for iPython HTML Notebook with the latest matplotlib, which includes XKCD-style plotting. I first set up the virtualenv using these instructions, and then used pip to install the RC2 version of matplotlib using the tarball (since 1.3.0rc2 is not in pypi):

    $ pip install https://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.3.0rc2/matplotlib-1.3.0rc2.tar.gz
    #pip installs matplotlib from tarball, but half-uninstalls the dateutil package that we previously installed
    $ pip uninstall python-dateutil
    $ pip install python-dateutil

    Now, we need to install the Humor Sans font. I downloaded the ttf and put it in ~/.fonts/Humor-Sans.ttf

    After installing the font, we need to make sure the fontList.cache file is rebuilt (Thanks Georg!):

    $ rm ~/.matplotlib/fontList.cache

    We should now be able to produce xkcd-style plots just by running this command in our notebook before plotting:


    However, there are a couple errors with matplotlib 1.3.0rc2:

    • matplotlib.font_manager’s findfont() can’t find Humor-Sans.ttf, which I placed in matplotlib.font_manager.path, so manaully set the fonts for now
    • The XKCD artist style doesn’t work with plot_time() very well, so handle x-axis dates manaully as well

    I downloaded some data from graphite using &rawData=true and pasted that into my ipython notebook. This is sample graph of what it looks like:
    matplotlib xkcd

    Here is the code I used:

    Update: The code below contains workarounds for a font issue that is now fixed (Thanks Georg!). See this post for a better example

    #Use the XKCD style
    #Manually set the font, since I can't seem to get matplotlib to load Humor Sans automatically
    ax = pylab.axes()
    prop = matplotlib.font_manager.FontProperties(fname=matplotlib.font_manager.path+'/Humor-Sans.ttf', size=16)
    #Set the Y axis
    def fmt_mb(x, pos):
        if x > 14e6:
            return ""
        return '%0.0fMB/min' % (x*1e-6)
    yfmt = FuncFormatter(fmt_mb)
    #Set the X axis
    start = 1371584040
    end = 1371670440
    timestamps = numpy.linspace(start, end, len(x))
    dates=[datetime.datetime.utcfromtimestamp(ts) for ts in timestamps]
    xfmt = DateFormatter('%m-%d %H:%M')
    #Set the title, and manually specify font due to matplotlib findfont issue
    plt.title("HTML5 Uploader MB/min for the last 24 hours", fontproperties=prop) 
    #Plot the moving average
    values = matplotlib.mlab.movavg(x, 15)
    ax.plot(dates[:len(values)], values)
    #Manually set font for the labels, due to findfont issue
    for label in ax.get_xticklabels():
    for label in ax.get_yticklabels():
    #Save PNG
    fig = matplotlib.pyplot.gcf()
    • Georg Nebehay 1:03 pm on July 8, 2013 Permalink | Reply

      I ran into the same issues with matplotlib not recognising Humor Sans but this thread here http://matplotlib.1069221.n5.nabble.com/getting-matplotlib-to-recognize-a-new-font-td40500.html
      provided me with a solution. After installing a new font it is necessary to regenerate the matplotlib font list cache, which can be done by deleting ~/.matplotlib/fontList.cache on Unix systems. I also had to rename the file Humor_Sans.ttf to Humor Sans.ttf, and now it works like a charm.

      • raj 11:30 pm on July 17, 2013 Permalink | Reply

        Thanks, Georg! Removing ~/.matplotlib/fontList.cache worked for me, without having to rename my font file, which on my system is ~/.fonts/Humor-Sans.ttf

        I’ll update the instructions above.

  • raj 8:40 pm on February 2, 2012 Permalink | Reply
    Tags: ipython, , numpy   

    Installing ipython notebook to replace matlab 

    Here is a recipe for installing the amazing ipython notebook. This is for Ubuntu 10.04 Lucid:

    virtualenv ~/pyenvs/notebook
    source ~/pyenvs/notebook/bin/activate
    #install zeromq using these hints: http://johanharjono.com/archives/633
    sudo apt-get install python-software-properties
    sudo add-apt-repository ppa:chris-lea/zeromq
    sudo add-apt-repository ppa:chris-lea/libpgm
    sudo apt-get update
    sudo apt-get install libzmq1
    sudo apt-get install libzmq-dev
    sudo apt-get install libpgm-5.1-0
    pip install pyzmq
    pip install tornado
    pip install --upgrade ipython
    pip install numpy
    pip install matplotlib

    Now, we want to create a profile for running a notebook server on a public port. Use this command:

    ipython profile create nbserver

    This will create a directory named ~/.ipython/profile_nbserver with a bunch of files in it. Since we are running public mode, we need a hashed password. Generate one like so:

    In [1]: from IPython.lib import passwd
    In [2]: passwd()
    Enter password:
    Verify password:
    Out[2]: 'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'

    Now edit the file named ~/.ipython/profile_nbserver/ipython_notebook_config.py. This file will already be populated by a lot of configuration, most of which is commented out. You will want to uncomment/edit these lines:

    c = get_config()
    # When disabled, equations etc. will appear as their untransformed TeX source.                             
    c.NotebookApp.enable_mathjax = True
    # The string should be of the form type:salt:hashed-password.                                              
    c.NotebookApp.password = u'sha1:xxx:yyyyy' #from above
    # The IP address the notebook server will listen on.                                                       
    c.NotebookApp.ip = '*'
    # Pre-load matplotlib and numpy for interactive use, selecting a particular                                
    # matplotlib backend and loop integration.                                                                 
    c.IPKernelApp.pylab = 'inline'

    Also, you might want these:

    c.NotebookApp.certfile = u'/absolute/path/to/your/certificate/mycert.pem'
    c.NotebookApp.open_browser = False

    Now, start your server like so:

    ipython notebook --profile=nbserver

    Connect to your server with firefox, and you should be good to go!

    • raj 5:32 am on February 11, 2012 Permalink | Reply

      Even Visual Studio has inline matplotlib graphs now!

    • raj 8:26 pm on March 19, 2012 Permalink | Reply

      • raj 6:51 am on March 21, 2012 Permalink | Reply

        installing a modern numpy and matplotlib is crazy hard. used git repos for each.

        numpy instructions from here: http://blog.hyperjeff.net/?p=160

        the env vars helped me build numpy 1.6.1, but unit tests fail. ugh.

        I used matplotlib build steps from the included make.osx file.

    • tymm 9:43 pm on March 21, 2012 Permalink | Reply

      To quickly generate an ssl key via openssl (based on instructions from http://www.madboa.com/geek/openssl/#cert-self )

      $ mkdir ~/.ipython/profile_nbserver/ssl
      $ cd ~/.ipython/profile_nbserver/ssl
      $ openssl req -x509 -nodes -days 365 -subj ‘/C=US/ST=California/L=San Francisco/CN=NA’ -newkey rsa:2048 -keyout ipynb.pem -out ipynb.pem

      (feel free to change the argument to -subj as best suits your organization)

      then use the aforementioned c.NotebookApp.certfile setting in ipython_notebook_config.py to tell ipython notebook to look for it there.

    • od1 4:15 pm on March 27, 2012 Permalink | Reply

      Great writeup — thanks for posting. I had to install the following w support libraries to get matplotlib to compile:

      apt-get install libfreetype6-dev libpng-dev

    • Mark 1:39 am on September 3, 2012 Permalink | Reply

      These instructions worked well. Thank you. I had to add an sudo in front of the “pip install xxxx” instructions to make it work.

      • raj 3:05 am on September 12, 2012 Permalink | Reply

        You can also use virtualenv if you don’t have sudo privileges, or if you want to isolate your ipython installation from the rest of the system packages.

    • drnealaggarwal 8:36 am on October 24, 2012 Permalink | Reply

      Thanks for this Raj. Works brilliantly on Linux Mint 12 only had to sudo the pip commands. Neal

    • cbellingan 7:32 am on November 29, 2012 Permalink | Reply

      Thanks for the info, there were a couple of extra deps for the Ubuntu 12.04 AMI running in EC2, full instructions:

      sudo apt-get install python-software-properties
      sudo add-apt-repository ppa:chris-lea/zeromq
      sudo add-apt-repository ppa:chris-lea/libpgm
      sudo apt-get update
      sudo apt-get install libzmq1
      sudo apt-get install libzmq-dev
      sudo apt-get install libpgm-5.1-0
      sudo apt-get install python-pip
      sudo apt-get install gcc
      sudo apt-get install python-dev
      sudo apt-get install g++
      sudo apt-get install libpng-dev libjpeg8-dev libfreetype6-dev

      sudo pip install pyzmq

      sudo pip install tornado
      sudo pip install –upgrade ipython
      sudo pip install numpy
      sudo pip install matplotlib

    • Gavin 4:45 am on January 10, 2013 Permalink | Reply

      I have automated the set of instructions for ipython notebook using Puppet scripts in a virtual machine. This reduces the installation to just 4 step. I have a github repository that has the code with installation instructions


Compose new post
Next post/Next comment
Previous post/Previous comment
Show/Hide comments
Go to top
Go to login
Show/Hide help
shift + esc