Tagged: openlibrary Toggle Comment Threads | Keyboard Shortcuts

  • raj 8:55 pm on January 6, 2015 Permalink | Reply
    Tags: openlibrary   

    Open Library dev quickstart 

    Setting up a dev instance

    • Install virtualbox
    • Install vagrant
    • Fork the Open Library repo to your own github account
    • Clone your forked repo to your local machine:
      • git clone git://github.com/YOURNAME/openlibrary.git
    • switch into the directory that you just cloned:
      • cd openlibrary
    • Start up the dev instance using vagrant:
      • vagrant up
    • You can now view your running instance by loading localhost:8080 in a web browser.
    • If you have previously set up the dev instance, and want to start it up again, you will need to force vagrant to reprovision the instance:
      • vagrant up --provision
    • You can log into the OL test instance as an admin, with the username openlibrary, password openlibrary.
    • If you need to ssh into the vagrant dev box, type:
      • vagrant ssh
    • If you need to copy a file out of the vagrant dev box, you can use scp:
      • vagrant ssh-config > ./vagrant.ssh.config
        scp -F vagrant.ssh.config default://var/log/openlibrary/ol-errors/2015-01-06/211111402777.html /tmp



    Logs for the upstart services will be in /var/log/upstart/. The app server logs will be in /var/log/upstart/ol-web.log



    • If you want to add a new user to the admin group, you can do that at https://localhost:8080/usergroup/admin
    • The admin interface is available at http://localhost:8080/admin


    Routing and Templates


    Copying documents

    You can copy test data from the live openlibrary.org site into your dev instance.

    `vagrant ssh` into your dev instance, and run the copydocs.py script in /openlibrary/scripts. If you want to add a book, you must first copy an author record, then the work record, and then the book record.

    $ cd /openlibrary/scripts
    vagrant@ol-dev:/openlibrary/scripts$ $ ./copydocs.py /authors/OL1385865A
    fetching ['/authors/OL1385865A']
    saving ['/authors/OL1385865A']
    [{'key': '/authors/OL1385865A', 'revision': 1}]
    vagrant@ol-dev:/openlibrary/scripts$ ./copydocs.py /works/OL14906539W 
    fetching ['/works/OL14906539W']
    saving ['/works/OL14906539W']
    [{'key': '/works/OL14906539W', 'revision': 1}]
    vagrant@ol-dev:/openlibrary/scripts$ ./copydocs.py /books/OL24966433M
    fetching ['/books/OL24966433M']
    saving ['/books/OL24966433M']
    [{'key': '/books/OL24966433M', 'revision': 1}]


    Creating users

    If you create a user, you will have to verify the email address, but you will not be able to send email from your vagrant dev instance. Instead, you can find the verification link in the app server log, which should be in /var/log/upstart/ol-web.log. The verification link should look like:


    The hash you see will be different that above. Just load that link and the user will be created in your dev instance.





    Infobase queries get cached in memcache. In the vagrant dev instance, there is a single-node memcache cluster that you can test by connecting to your test instance using `vagrant ssh` and then typing:

    #in vagrant test instance
    $ cd /openlibrary
    $ python
    Python 2.7.6 (default, Mar 22 2014, 22:59:56)
    [GCC 4.8.2] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    import yaml
    from openlibrary.utils import olmemcache
    y = yaml.safe_load(open('/openlibrary/conf/openlibrary.yml'))
    c = olmemcache.Client(y['memcache_servers'])
    '{"bio": {"type": "/type/text", "value": "Mark Twain, was an American author and humorist. Twain is noted for his novels Adventures of Huckleberry Finn (1884), which has been called \\"the Great American Novel\\", and The Adventures of Tom Sawyer (1876). He is extensively quoted. Twain was a friend to presidents, artists, industrialists, and European royalty. ([Source][1].)\\r\\n\\r\\n[1]:http://en.wikipedia.org/wiki/Mark_Twain"}, "photograph": "/static/files//697/OL2622189A_photograph_1212404607766697.jpg", "name": "Mark Twain", "marc": ["1 \\u001faTwain, Mark,\\u001fd1835-1910.\\u001e"], "alternate_names": ["Mark TWAIN", "M. Twain", "TWAIN", "Twain", "Twain, Mark (pseud)", "Twain, Mark (Spirit)", "Twain, Mark, 1835-1910", "Mark (Samuel L. Clemens) Twain", "Samuel Langhorne Clemens (Mark Twain)", "Samuel Langhorne Clemens", "mark twain "], "death_date": "21 April 1910", "wikipedia": "http://en.wikipedia.org/wiki/Mark_Twain", "created": {"type": "/type/datetime", "value": "2013-03-28T07:50:47.897206"}, "last_modified": {"type": "/type/datetime", "value": "2013-03-28T07:50:47.897206"}, "latest_revision": 1, "key": "/authors/OL18319A", "birth_date": "30 November 1835", "title": "(pseud)", "personal_name": "Mark Twain", "type": {"key": "/type/author"}, "revision": 1}'
  • raj 4:43 am on June 3, 2011 Permalink | Reply
    Tags: evergreen, openlibrary   

    “The wonderful new OpenLibrary Read API and Evergreen integration”


  • raj 2:38 am on March 24, 2011 Permalink | Reply
    Tags: anand, george, noufal, openlibrary   

    Pics from Anand and Noufal’s visit 

  • raj 12:56 am on March 3, 2011 Permalink | Reply
    Tags: opds, openlibrary   

    Open Library OPDS 

    I’ve started adding OPDS/BookServer support to Open Library, starting with OL Edition records. OPDS is an Atom-based specification for distribution of ebook metadata. You can read more at http://opds-spec.org.

    For each edition record in OL, you can add “.odps” to the end of the edition key to retrieve the OPDS version. For example the OPDS entry for this edition of Peter Rabbit can be retrieved from


    OPDS provides a good way of programmatically extracting and harvesting OL edition data. Currently, the OPDS record contains more metadata than the JSON version of an edition record, which makes it easy to grab author and subject data without multiple requests.

    For example, to get an author name for an OL edition using JSON, you would first have to get the edition’s JSON record, find the work key, then request the work’s JSON record, then find the author key, then request the author’s JSON record, and then you would be able find the author name. The OPDS entry for an edition will contain Work-level data, so you can avoid jumping through hoops.

    Also, for edition records added before Works were introduced in OL, the JSON edition record sometimes shows author and subject data which is old and sometimes incorrect!

    We are still discussing how to support external identifiers, indirect acquisition, and DAISY links with the ODPS community, so the format of the OPDS record might change slightly in the future.

    The template for the OPDS XML is at the end of this very long url:


    This is the first step in closing this bug that Matt filed two years ago.. Poor Matt!

  • raj 6:33 am on February 18, 2011 Permalink | Reply
    Tags: openlibrary   

    How to connect an Open Library template to a page 

    Since Open Library is a wiki, you can create a new page in almost any namespace you want (assuming you are an admin). We created http://openlibrary.org/libraries by going to that URL and clicking “Create it”.

    So how can we get /libraries to use the /libraries/index.html template?

    Anand helped by adding some python code in openlibrary/plugins/openlibrary/libraries.py:

    class libraries(delegate.page):
        def GET(self):
            return render_template("libraries/index")

    If the path of the page isn’t at the top level of the namespace, you can specify a path:

    class locations(delegate.page):
        path = "/libraries/locations.txt"
        def GET(self):
            libraries = inlibrary.get_libraries()
            return render_template("libraries/locations")
Compose new post
Next post/Next comment
Previous post/Previous comment
Show/Hide comments
Go to top
Go to login
Show/Hide help
shift + esc