Updates from raj Toggle Comment Threads | Keyboard Shortcuts

  • raj 8:55 pm on January 6, 2015 Permalink | Reply

    Open Library dev quickstart 

    Setting up a dev instance

    • Install virtualbox
    • Install vagrant
    • Fork the Open Library repo to your own github account
    • Clone your forked repo to your local machine:
      • git clone git://github.com/YOURNAME/openlibrary.git
    • switch into the directory that you just cloned:
      • cd openlibrary
    • Start up the dev instance using vagrant:
      • vagrant up
    • You can now view your running instance by loading localhost:8080 in a web browser.
    • If you have previously set up the dev instance, and want to start it up again, you will need to force vagrant to reprovision the instance:
      • vagrant up --provision
    • You can log into the OL test instance as an admin, with the username openlibrary, password openlibrary.
    • If you need to ssh into the vagrant dev box, type:
      • vagrant ssh
    • If you need to copy a file out of the vagrant dev box, you can use scp:
      • vagrant ssh-config > ./vagrant.ssh.config
        scp -F vagrant.ssh.config default://var/log/openlibrary/ol-errors/2015-01-06/211111402777.html /tmp



    Logs for the upstart services will be in /var/log/upstart/. The app server logs will be in /var/log/upstart/ol-web.log



    • If you want to add a new user to the admin group, you can do that at https://localhost:8080/usergroup/admin
    • The admin interface is available at http://localhost:8080/admin


    Routing and Templates


    Copying documents

    You can copy test data from the live openlibrary.org site into your dev instance.

    `vagrant ssh` into your dev instance, and run the copydocs.py script in /openlibrary/scripts. If you want to add a book, you must first copy an author record, then the work record, and then the book record.

    $ cd /openlibrary/scripts
    vagrant@ol-dev:/openlibrary/scripts$ $ ./copydocs.py /authors/OL1385865A
    fetching ['/authors/OL1385865A']
    saving ['/authors/OL1385865A']
    [{'key': '/authors/OL1385865A', 'revision': 1}]
    vagrant@ol-dev:/openlibrary/scripts$ ./copydocs.py /works/OL14906539W 
    fetching ['/works/OL14906539W']
    saving ['/works/OL14906539W']
    [{'key': '/works/OL14906539W', 'revision': 1}]
    vagrant@ol-dev:/openlibrary/scripts$ ./copydocs.py /books/OL24966433M
    fetching ['/books/OL24966433M']
    saving ['/books/OL24966433M']
    [{'key': '/books/OL24966433M', 'revision': 1}]


    Creating users

    If you create a user, you will have to verify the email address, but you will not be able to send email from your vagrant dev instance. Instead, you can find the verification link in the app server log, which should be in /var/log/upstart/ol-web.log. The verification link should look like:


    The hash you see will be different that above. Just load that link and the user will be created in your dev instance.





    Infobase queries get cached in memcache. In the vagrant dev instance, there is a single-node memcache cluster that you can test by connecting to your test instance using `vagrant ssh` and then typing:

    #in vagrant test instance
    $ cd /openlibrary
    $ python
    Python 2.7.6 (default, Mar 22 2014, 22:59:56)
    [GCC 4.8.2] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    import yaml
    from openlibrary.utils import olmemcache
    y = yaml.safe_load(open('/openlibrary/conf/openlibrary.yml'))
    c = olmemcache.Client(y['memcache_servers'])
    '{"bio": {"type": "/type/text", "value": "Mark Twain, was an American author and humorist. Twain is noted for his novels Adventures of Huckleberry Finn (1884), which has been called \\"the Great American Novel\\", and The Adventures of Tom Sawyer (1876). He is extensively quoted. Twain was a friend to presidents, artists, industrialists, and European royalty. ([Source][1].)\\r\\n\\r\\n[1]:http://en.wikipedia.org/wiki/Mark_Twain"}, "photograph": "/static/files//697/OL2622189A_photograph_1212404607766697.jpg", "name": "Mark Twain", "marc": ["1 \\u001faTwain, Mark,\\u001fd1835-1910.\\u001e"], "alternate_names": ["Mark TWAIN", "M. Twain", "TWAIN", "Twain", "Twain, Mark (pseud)", "Twain, Mark (Spirit)", "Twain, Mark, 1835-1910", "Mark (Samuel L. Clemens) Twain", "Samuel Langhorne Clemens (Mark Twain)", "Samuel Langhorne Clemens", "mark twain "], "death_date": "21 April 1910", "wikipedia": "http://en.wikipedia.org/wiki/Mark_Twain", "created": {"type": "/type/datetime", "value": "2013-03-28T07:50:47.897206"}, "last_modified": {"type": "/type/datetime", "value": "2013-03-28T07:50:47.897206"}, "latest_revision": 1, "key": "/authors/OL18319A", "birth_date": "30 November 1835", "title": "(pseud)", "personal_name": "Mark Twain", "type": {"key": "/type/author"}, "revision": 1}'
  • raj 6:24 pm on December 22, 2014 Permalink | Reply
    Tags: ,   

    Nikon J4 support is now in gphoto 2.5.6… 

    Nikon J4 support is now in gphoto 2.5.6

    The J4 camera id was added in commit r15269

  • raj 10:38 pm on June 24, 2014 Permalink | Reply
    Tags: ,   

    Using the Nikon J3 with gphoto and ubuntu 

    Some notes on using the Nikon J3 for remote capture:

    • We added Nikon J3 support to gphoto, for use with IA tabletop bookscanners. Most these changes came in r149415. Version r14922 has been tested with this camera on both Ubuntu 12.04 and Ubuntu 14.04.
    • A git-svn mirror of photo (at r14922) is available here: https://github.com/rajbot/gphoto
    • Ubuntu tries to mount the camera as a filesystem using `GVFS` as soon as the camera is connected via USB. On older versions of Ubuntu, a gvfs-gphoto bug causes the camera to lock up. The easiest way to disable automounting of the camera is to `chmod -x` the gvfs-gphoto2-volume-monitor:
      sudo chmod -x /usr/lib/gvfs/gvfs-gphoto2-volume-monitor

    Update: Nikon 1 support has now been integrated into gPhoto 2.5.5!

  • raj 7:46 pm on June 18, 2014 Permalink | Reply
    Tags: littefreelibrary   

    Leawood [Kansas] man faces citation for putting Little Free Library in his front yard

  • raj 4:33 am on June 10, 2014 Permalink | Reply

    Some questions when considering a camera for remote capture 

    • Does the camera support a standard for remote capture, such as PTP?
    • Does gphoto already support this camera? If not, is there an existing SDK for linux? If not, is there existing remote capture software on any platform that works with this camera?
    • Can you capture and download an image without a memory card installed in the camera?
    • How fast can you capture and download an image?
    • Is there an API for enabling/disabling autofocus?
    • Is there a mechanical mirror? If so, can you lock it in position?
    • Is the shutter electronic or manual? (Are both first and second curtain electronic?)
    • How many shots can you take before the camera needs repair?
    • Does the remote capture API work over USB? or only wifi? How many cameras can we connect to the same computer?

    There are lots of questions about image quality to consider as well. Also, we are not concerned about video, but it would be worth knowing what video operations are supported via API.

  • raj 10:02 pm on May 2, 2014 Permalink | Reply

    Compiling gphoto on Ubuntu 14.04 LTS 

    Here are instructions on how to compile a minimal gphoto build into /usr/local on a fresh Ubuntu 14.04 Trusty desktop install:

    # install dependencies and checkout gphoto source
    $ sudo apt-get install automake autopoint gettext libtool libusb-dev libpopt-dev subversion
    $ svn checkout svn://svn.code.sf.net/p/gphoto/code/trunk gphoto
    # compile libphoto2 and install into /usr/local/lib
    $ cd gphoto/libgphoto2/
    $ autoreconf --install --symlink
    $ ./configure
    $ make
    $ sudo make install
    # compile gphoto2 and install into /usr/local/bin
    $ cd ../gphoto2
    $ autoreconf --install --symlink
    $ ./configure
    $ make
    $ sudo make install

    In order to run ghoto with the version of libgphoto2 that you just built (as opposed to the one that came with Ubuntu 14.04), be sure to set LD_LIBRARY_PATH:

    $ LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/gphoto2 --version
    gphoto2 2.5.4
    Copyright (c) 2000-2014 Lutz Mueller and others
    gphoto2 comes with NO WARRANTY, to the extent permitted by law. You may
    redistribute copies of gphoto2 under the terms of the GNU General Public
    License. For more information about these matters, see the files named COPYING.
    This version of gphoto2 is using the following software versions and options:
    gphoto2         2.5.4          gcc, popt(m), no exif, no cdk, no aa, no jpeg, no readline
    libgphoto2        all camlibs, gcc, ltdl, no EXIF
    libgphoto2_port 0.10.0         gcc, ltdl, USB, serial without locking
  • raj 7:23 pm on March 5, 2014 Permalink | Reply
    Tags: PyInstaller, PyQt,   

    Loading DLLs from a PyInstaller-packaged PyQt standalone application 

    I wanted to package a PyQt script as a standalone Windows .exe using PyInstaller, and I wanted the app to display a jpeg.

    It turned out that displaying a PNG was fine, but displaying a JPEG took a while to get working. PyQt uses a dll called `qjpeg4.dll` for decoding the jpeg, and PyInstaller doesn’t automatically bundle it into the package, so we have to tell PyInstaller to do so.

    There is still one problem though: when you run the .exe, where does PyQt try to load `qjpeg4.dll` from? It turns out that *it depends*..

    If you use the `–onedir` option for PyInstaller, the `imageformats/qjpeg4.dll` just needs to be in a directory called `imageformats`. No problem.

    If you use the `–onefile` option, PyInstaller unzips the binaries into a temp directory, and loads dlls from there. But for some reason, in the `–onefile` case, PyQt tries to load the dll from `qt4_plugins/imageformats/qjpeg4.dll`.

    I used SysInternals Process Monitor to figure this out…

    To make things easy, I just added an `imageformats` directory to my repo, and checked in the qjpeg4.dll.

    I also wanted to load a dll called `discid.dll`, required by python-discid. I checked into the root level of the repo, but couldn’t get the python package to load the dll properly, until I modified the PATH environment variable:

    #fix for loading discid.dll
    if getattr(sys, 'frozen', None):
         BASE_DIR = sys._MEIPASS
         BASE_DIR = os.path.dirname(__file__)
    os.environ['PATH'] = BASE_DIR + '\;' + os.environ.get('PATH', '')
    import discid

    My PyInstaller .spec file for the `–onefile` case looks like this:

    # -*- mode: python -*-
    a = Analysis(['wizard.py'],
    onefile_binaries = a.binaries + [('discid.dll', 'discid.dll', 'BINARY'),
                             ('qt4_plugins/imageformats/qjpeg4.dll', 'imageformats/qjpeg4.dll', 'BINARY'),
    pyz = PYZ(a.pure)
    exe = EXE(pyz,
              Tree('images', prefix='images'),
              console=True )
  • raj 11:46 pm on February 12, 2014 Permalink | Reply

    Thinking about the Wayback Machine 

    wayback boxes and arrows

  • raj 11:58 pm on February 7, 2014 Permalink | Reply

    BookReader usage in 2013 

    We measured whenever someone opened the IA BookReader. Usage more than doubled in 2013, to more than 5 people opening up the BookReader every second!


    For 2014, we won’t be able to produce the same data, since we now embed the BookReader on archive.org details pages, and a pageview now registers as a “bookreader open” event, even if the user doesn’t actually read the book.

  • raj 7:16 pm on October 31, 2013 Permalink | Reply
    Tags: , wordpress   

    BookReader embeds on wordpress.com 

    Blogs hosted at wordpress.com can now use BookReader embeds!


Compose new post
Next post/Next comment
Previous post/Previous comment
Show/Hide comments
Go to top
Go to login
Show/Hide help
shift + esc