REST with Python: awesome API development tools and docs

Here goes a collection of links I find very useful to keep handy regarding RESTful APIs and Python:

Frameworks/libraries

• eve: Python REST API Framework (Flask, MongoDB and Redis)
• Partial JSON API implementation in Python
• JSONAPI implementation for use with SQLAlchemy
• Django REST framework
• Tastypie (for Django)
• Flask-RESTful
• Flask API (lots of future and nice projection, not the best documentation)
• Flask-Restless (only for SQLAlchemy, but with a lot of features like pagination, filters, pre/post-processors, etc.)
• Restless is a lightweight API framework that aims to be framework agnostic.
• Implementation of JSON Schema for Python
• A Practical API Design Framework

Videos/blog posts/Tutorials

• Developing RESTful Web APIs with Flask and mongodb (from the author of eve).
• Versioned and versionless APIs
• Designing a RESTful Web API
• HTTP API Design Guide
• REST API Tutorial
• 18F API Standards
• Designing a Beautiful REST+JSON API (blog post)

Others

• API Testing for Humans
• Very good/interesting: http://raml.org/ Tools: http://raml.org/projects.html
• http://www.intridea.com/blog/2010/4/29/rest-isnt-what-you-think-it-is

Dealing with speed

• json, from the Python standard library, is slow and performance can be improved using ujson
• SQLAlchemy has an option to cache connections: connection pool.

Special thanks to Juan Norris for creating and letting me share the original document here.

Get file extension from filename with Python

Several times I had to put some code together some Python code to get the extension from a filename.

It was required for it to work OK with complex extensions, like ".tar.gz", and remain functional when you have dots in the filename, like in this case: "some.filename.1.v2.tar.gz".

After trying several options, the one that did the trick was using regular expressions, wrapped in a little function.

Here the case for keeping the ".", and getting the extension in the fashion ".tar.gz" or ".png":



Here the case for not keeping the ".", and getting the extension in the fashion "tar.gz" or "png":




Et voilà! :D

Tech Meetup en Santex: slides y código de la charla

Acá comparto el material de la Tech Meetup que di en Santex la semana pasada.

Slides: Link.
Código fuente de proyecto de ejemplo: https://github.com/matiasherranz/tech_meetup_django

Installing MySQL-Python in Mac OS X Mavericks

If you are in Mac OS X Mavericks and while trying to install MySQL-Python you run into this error:



This is a what makes the trick:



Then, pip works its way towards installing MySQL-Python OK:



Voilà!

Create a global GIT hook to check Flake8 before each commit

First, let's install Flake8:

Note It is important to set the right version when installing Flake8, as there there are some versions that are bugged and will prevent the GIT hook to work (like the 2.1.0 version).

Now, let's setup global git commit hooks:

1. Enable git templates:

This tells git to copy everything in ~/.git-templates to your per-project .git/ directory when you run git init

2. Create a directory to hold the global hooks:

3. Write your hooks in ~/.git-templates/hooks.

For example, here's a post-commit hook (located in ~/.git-templates/hooks/post-commit):

4. Make sure the hook is executable.

5. Re-initialize git in each existing repo you'd like to use this in:

NOTE if you already have a hook defined in your local git repo, this will not overwrite it.

Now, let's create a global git pre-commit hook:

Finally, let's take it for a spin!

Go to some of your repos and re-initialize it to take the pre-commit hook:

Now edit some of your Python files, introducing some violation (like a >80 columns line, only one blank line between function/class definitions, etc.) and try to commit it:

Voilà!

Custom sorting function as parameter to sorted() in Python

So, I am working with Python and I come across a list of image names,  and I'd like to have them sorted. So, I call the sorted function with this list:



HA! It doesn't work as I would have liked it to. So, what can we do to achieve the sorting of the list as we want it?

Python's standard library sorted function, as you can see here, has an optional cmp parameter, through which you can pass a custom comparison function.

Going back to my example, I needed the images to be compared by the number in their name. I made this little function to achieve this goal:



And now let's put altogether:



TA DA!

Mostrar el branch de GIT y el status actual en el shell prompt y más!

Algunos trucos para configurar nuestro setup local de GIT:

Hace algunos años encontré algo como eso y, con el tiempo, armé una serie de scripts (algunas cosas que hice yo, algunas cosas que fui encontrando en la web), que hacen varias cosas con bash, que están piolas.

¿Qué cosas?

- que se detecte automáticamente si se está en una carpeta perteneciente a un repo git, y que, en el caso que sí lo sea, el prompt muestre información relevante del repo en que se está: branch y estado. Ejemplo:

en ese caso, estoy en el branch "master" y tengo archivos modificados (por eso la "m") y archivos que borré (por eso la "d").

- otra cosa muy piola es tener tab completion (que al apretar tab, con un comando a medio escribir, se muestren opciones para completar ese comando -o que se completen, si hubiera una opción posible). Imaginen eso aplicado a nombres de branch, comandos git y comandos de django (como "python manage.py runserver", por ejemplo).

- Algunas cosas más, como:

-- aumentar el tamaño del history de bash, pero eliminando duplicados, para que al hacer "ctrl r" en la consola, podamos buscar en el historial de los comandos que hicimos y encontrar resultados interesantes.

-- Una larga lista de colores definidos, para directamente poder usarlos en la definición del prompt.

Etc!

Les paso adjuntas las scripts para hacer todo esto:

•  Los archivos:
• .git_completion.sh: https://gist.github.com/matiasherranz/6683058
• .django_bash_completion: https://gist.github.com/matiasherranz/6683069
• .profile / .bashrc: https://gist.github.com/matiasherranz/6683083
• ZIP con los 3 archivos:
• http://www.mediafire.com/?31hsm27r7r59h5p

Las dos primeras las ponen derecho en el home. El .profile pueden ponerlo derecho en el home (si usan Mac y no tienen otro .profile armado) o copiar su contenido a su .profile existente. Si usan Linux, pueden hacer esto mismo con el archivo .bashrc.

(notar que tienen un underscore los archivos, a propósito, para que no sobreescriban cosas sin querer y para que no los tome el file browser como archivos ocultos).

Un abrazo y may the bash be with you (?),

Find out when a line was introduced or deleted in GIT history

Hi!

Did you ever come across a line of code that you d *really* like to know when was introduced/deleted ?

Well, if you are using GIT for your project this might come handy.

Let's say the piece of text you are investigating is this:

a = get_something(some_var)

and it is located in the file some_folder/some_file.py

Then, try this command:

git log -S "a = get_something(some_var)" some_folder/some_file.py

Then, GIT will look for the string we specified string within the commit history and output something like this:

commit
Author: Some Dude <some.dude@mail.com>
Date:   Fri Feb 15 14:09:18 2013 -0300
    The commit message of this commit.

This came quite handy to me recently. Hope it helps!

Santex Tech Meetup: Python en el mundo real

Python en en mundo real

Se expondrán brevemente proyectos y empresas que han encarado proyectos exitosos a nivel mundial con Python, comparando ventajas, ecosistema, comunidad y desventajas del lenguaje.
La charla apuntará a ser bien de alto nivel (en el sentido de que no requiere más que nociones básicas de programación) y se mostrarán ejemplos prácticos de programas y librerías de computación científica, motores para hacer juegos, aplicaciones web, frameworks, etc.

UPDATE:

Material de la charla! https://github.com/matiasherranz/talks/tree/master/Real_world_Python

Fotos del evento! link.

Working on django-notification (django 1.4 support, internationalization, etc.).

What's this all about?

Working on a Django project, I happened to come across with the need of a notifications app for Django. The first one to take a look at and perhaps the most popular the last years is django-notification. So, let's take a look at it, my friend.

What was the problem?

Well, to begin with, it does not work with Django  1.4 or above. That's the starting point for me to branch out from the original repo of the project. I tuned it up to work with Django 1.4.

Later down the road, we decided to use internationalized dates. And then again, django-notification did not handle them properly (does not suppor internationalization at all). And therefore a zillion of our tests started to throw warning.

Once again, I had some time to spare, and got myself hands on that.

Summarizing

Which changes did I make to django-notification? Let's take a look:

• Made it work with django 1.4
• [this is a big one] Made django-notification timezone aware and compliant.
• Made every single file PEP8 compliant
• Fixed a bunch of issues I found while going through the codebase (like missing imports)
• Quite a few code improvements (like the usage of "zip" or "file" as variable names)

Where's this code? How I install it for my project?

The code is available here, at my github account: https://github.com/matiasherranz/django-notification

You can install it with pip this way:

pip install  -e git+https://github.com/matiasherranz/django-notification.git#egg=django-notification

I'm open to suggestions, comments or pull requests to improve the app.

Hope it helps!

Material de mi charla sobre Django en el PyDay Córdoba 2012

Acá dejo el contenido de mi charla de Django del PyDay 2012:

Filminas y la aplicación que mostré, creada paso a paso:

• https://github.com/matiasherranz/scoobygalletas/tree/master/PyDay_Cba2012

Muchas gracias a todos/as los/as que vinieron!


Cualquier consulta, no duden en mandarme un email.


Salud!

Some date handling util functions I coded for Python/Django.

Repairing PostgreSQL after upgrading to Mac OSX Lion

I upgraded to Mac OSX to 10.7.3 (Lion) yesterday and had some issues with PostgreSQL.

For instance, when I tried to run the South migrations for the Django project I'm working at, I got this:

psycopg2.OperationalError: could not connect to server: Permission denied
    Is the server running locally and accepting
    connections on Unix domain socket "/var/pgsql_socket/.s.PGSQL.5432"?

How I fixed this?

1.  Edit PostgreSQL's config to define the unix_socket_directory setting:
   
   $ sudo nano /Library/PostgreSQL/9.0/data/postgresql.conf
   
   In my case I changed the setting to unix_socket_directory = '/var/pgsql_socket/'.
   
2. Exceute the following commands:
   
   

   $ sudo dscl . append /Groups/_postgres GroupMembership postgres
   $ sudo chmod g+w,o+rx /var/pgsql_socket/

   
   
3.  Restart the server:
   
   $ sudo -u postgres /Library/PostgreSQL/9.0/bin/pg_ctl -D /Library/PostgreSQL/9.0/data/ restart

Note: You may want to add an alias for this command in your ~/.profile file. If so, it is just a matter of adding a line like this:

alias restart_postgres='sudo -u postgres /Library/PostgreSQL/9.0/bin/pg_ctl -D /Library/PostgreSQL/9.0/data/ restart'

How to install GTK and PyGTK in Mac OS X Lion

I run into some problems trying to install GTK and PyGTK, but I found a workaround, so here's the trick:

You'll need to have brew installed in your system. If you don't have brew installed, you can installed just by running this command:

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"

 Now you are ready to go!

Install GTK: just run this command:

$ brew install gtk

Install PyGTK:

• Download this: link.
• It is a Mac installer package. Install it.

That's it!

You  can test your installation from a Python console this way:

$ python
>>> import pygtk

Procrastinando una atrocidad, inspirado por @ralsina :P

Acá en PyConAr Junín 2011, inspirado por unos comentarios que hizo @ralsina en una charla, surgiéronme estas atrocidades, que tenía que compartir:




:)

Después de haber laburado más de 10 horas hoy....

.... llegar al final del día logrando esto, es.... en un modo muy particular, maravilloso! jeje

Acá la cosa en cuestión que me hizo feliz ver:

----------------------------------------------------------------------
Ran 414 tests in 232.764s

OK
Destroying test database for alias 'default' ('test_cpi_mrp3')...

Sí señor! Sí señor! Ahora sí puedo hacer un push a origin e irme a dormir :)

Mac OS X Lion, Python, Django and how I'm happier now with my computer

I recently upgraded my MacBook Pro from Snow Leopard to Lion.

The idea of the upgrade unsettled me quite a bit since I use my computer to work and having it off due to OS or setup issues was not an option. Basically I was worried about having problems to install Python packages(I work mostly as a Django/Python developer, and I use a bunch of stuff from PyPi and quite a few Django pluggables) and I've already had some nasty problems installing, for instance, the MySQL connector in the past in Snow Leopard.

After some hesitation, I finally got myself to the task of upgrading.

After I upgraded, I did all the setup of the Python stuff: all the virtualenvs, Python/Django packages, the requirements for NINJA-IDE, everything WORKED PERFECT. I did not have a single issue. Nor even a single one. I'm happy this of not having to write a long, strange and tricky post saying something like "yes, it is a huge pain in the... patience", because it is just not. Everything work perfect.

I believe that the root cause of the difference relies in Mac OS Lion to came bundled with 64-bit versions of Python pre-installed:

It has, pre-installed:

• 32-bit Python 2.5
• 64-bit Python 2.6 (I use this one for work)
• 64-bit Python 2.7 (I use this for NINJA-IDE, both for running it and to develop the code I write for it)

That's good new that Apple got its OS with more Python magic :)

Installing cairo and pycairo in Mac OS X with python 2.6

Mac OS X includes an old version of Cairo(usually 1.8.6) which is too old for new software to link against. For the installation of py2cairo to succeed, we need 1.8.10.

Let's check which version we have:

$ pkg-config --atleast-version=1.8.10 cairo
$ echo $?
1

That means there's no a version 1.8.10 or above of the 'cairo' lib.

Let's install it using Brew:

$ brew install cairo

Let's now check again:

$ pkg-config --atleast-version=1.8.10 cairo
$ echo $?
1

No luck again! Why? We need to tell the system where the recently installed version of cairo lives:

$  export PKG_CONFIG_PATH=

Note: In my case, the path is: /usr/local/Cellar/cairo/1.10.2/lib/pkgconfig/


Let's check again:

$  pkg-config --atleast-version=1.8.10 cairo
$ echo $?
0

Awesome!

Let's now install py2cairo, the Python binding for cairo:

1. Download this.
2. Uncompress, go inside the directory.
3. Run the following command, expecting an output like the depicted below:

$ ./waf configure
  ./set_options
  ./init
  ./configure
Checking for program gcc or cc           : /usr/bin/gcc
Checking for program cpp                 : /usr/bin/cpp
Checking for program ar                  : /usr/bin/ar
Checking for program ranlib              : /usr/bin/ranlib
Checking for gcc                         : ok 
Checking for program python              : /Users/matias/Dev/CPI/Env/bin/python
Checking for Python version >= 2.6.0     : ok 2.6.1
Checking for library python2.6           : yes
Checking for program python2.6-config    : /usr/local/bin/python2.6-config
Checking for header Python.h             : yes
Checking for cairo >= 1.8.10             : yes
'configure' finished successfully (0.416s)
  ./shutdown

4. Now build:

$ ./waf build

5. Install:

$ ./waf install

6. Update your PYTHONPATH(you may want to add this line at the bottom of your ~/.profile):

export PYTHONPATH=/usr/local/lib/python2.6/site-packages/:$PYTHONPATH 

7. Check everything worked:

$ python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cairo
>>> cairo.version
'1.8.10'

 Happiness!

 

FYI: The output of the configure, build and install commands in my case was the following(I include it for reference, in case it might be of any help):

(Env)matias@[13:51:34]:pycairo-1.8.10 $ ./waf configure  ./set_options

  ./init
  ./configure
Checking for program gcc or cc           : /usr/bin/gcc
Checking for program cpp                 : /usr/bin/cpp
Checking for program ar                  : /usr/bin/ar
Checking for program ranlib              : /usr/bin/ranlib
Checking for gcc                         : ok 
Checking for program python              : /Users/matias/Dev/CPI/Env/bin/python
Checking for Python version >= 2.6.0     : ok 2.6.1
Checking for library python2.6           : yes
Checking for program python2.6-config    : /usr/local/bin/python2.6-config
Checking for header Python.h             : yes
Checking for cairo >= 1.8.10             : yes
'configure' finished successfully (0.416s)
  ./shutdown

(Env)matias@[13:52:07]:pycairo-1.8.10 $ ./waf build

  ./set_options
  ./init
Waf: Entering directory `/Users/matias/Dev/CPI/Stuff/pycairo-build'
  ./build
  src/build
[1/9] cc: src/cairomodule.c -> ../pycairo-build/default/src/cairomodule_2.o
[2/9] cc: src/context.c -> ../pycairo-build/default/src/context_2.o
[3/9] cc: src/font.c -> ../pycairo-build/default/src/font_2.o
[4/9] cc: src/path.c -> ../pycairo-build/default/src/path_2.o
[5/9] cc: src/pattern.c -> ../pycairo-build/default/src/pattern_2.o
[6/9] cc: src/matrix.c -> ../pycairo-build/default/src/matrix_2.o
[7/9] cc: src/surface.c -> ../pycairo-build/default/src/surface_2.o
[8/9] copy: pycairo.pc.in -> ../pycairo-build/default/pycairo.pc
In file included from /usr/X11/include/X11/Xlib.h:64,
                 from /usr/local/Cellar/cairo/1.10.2/include/cairo/cairo-xlib.h:44,
                 from ../pycairo-1.8.10/src/surface.c:1360:
/usr/X11/include/X11/Xosdefs.h:145:1: warning: "_DARWIN_C_SOURCE" redefined
In file included from /System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/Python.h:8,
                 from ../pycairo-1.8.10/src/surface.c:32:
/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/pyconfig.h:1075:1: warning: this is the location of the previous definition
[9/9] cc_link: ../pycairo-build/default/src/cairomodule_2.o ../pycairo-build/default/src/context_2.o ../pycairo-build/default/src/font_2.o ../pycairo-build/default/src/path_2.o ../pycairo-build/default/src/pattern_2.o ../pycairo-build/default/src/matrix_2.o ../pycairo-build/default/src/surface_2.o -> ../pycairo-build/default/src/_cairo.so
Waf: Leaving directory `/Users/matias/Dev/CPI/Stuff/pycairo-build'
'build' finished successfully (0.764s)
  ./shutdown

(Env)matias@[14:05:51]:pycairo-1.8.10 $ ./waf install

  ./set_options
  ./init
Waf: Entering directory `/Users/matias/Dev/CPI/Stuff/pycairo-build'
  ./build
  src/build
* installing src/pycairo.h as /usr/local/include/pycairo/pycairo.h
* installing /Users/matias/Dev/CPI/Stuff/pycairo-build/default/src/_cairo.so as /usr/local/lib/python2.6/site-packages/cairo/_cairo.so
* installing /Users/matias/Dev/CPI/Stuff/pycairo-build/default/pycairo.pc as /usr/local/lib/pkgconfig/pycairo.pc
Waf: Leaving directory `/Users/matias/Dev/CPI/Stuff/pycairo-build'
* installing src/__init__.py as /usr/local/lib/python2.6/site-packages/cairo/__init__.py
* byte compiling '/usr/local/lib/python2.6/site-packages/cairo/__init__.py'
'install' finished successfully (0.832s)
  ./shutdown

How to install MySQL Server and Python connector in Mac OS X Snow Leopard

================================================================
-> MySQL Server and Python connector(Mac OS X Snow Leopard):
================================================================

Here goes some tricky stuff. You may want to consider getting yourself a cup of
linden infusion :-P

--> If you have a previous MySQL version:

If you have a previous MySQL version, perhaps a 32 bit one, you may encounter several
problems to perform the upgrade to a newer, 64-bit version. But don't worry, the
following steps should get you going in no time.

Hands on!
To uninstall the previous version, do this *in order*:

- First you need to go to: /Library/Receipts and look for a file named
  'InstallHistory.plist' (It's just a regular property list)
- Open it and look for the entries regarding MySQL, and delete them.

    MacBookPro:~ matias$ cd /Library/Receipts/
    MacBookPro:Receipts matias$ sudo nano InstallHistory.plist
    ctrl+shift+w, look for the entries regarding MySQL and delete them.
    ctrl+o (save) ctrl+x (exit)

- As the installer receipts are actually located in a different place on Snow Leopard:

    MacBookPro:Receipts matias$ cd /private/var/db/receipts/
    MacBookPro:receipts matias$ ls *mysql*
    com.mysql.mysql.bom          com.mysql.mysqlstartup.bom
    com.mysql.mysql.plist        com.mysql.mysqlstartup.plist
    MacBookPro:receipts matias$ sudo rm  *mysql*


- Now we are ready to do the uninstall work per se, by doing these *in order*:

    * Use mysqldump or MySQL Admin to backup your databases to text files!
    * Stop the database server
    * sudo rm /usr/local/mysql
    * sudo rm -rf /usr/local/mysql*
    * sudo rm -rf /Library/StartupItems/MySQLCOM
    * sudo rm -rf /Library/PreferencePanes/My*
    * edit /etc/hostconfig and remove the line MYSQLCOM=-YES-
    * rm -rf ~/Library/PreferencePanes/My*
    * sudo rm -rf /Library/Receipts/mysql*
    * sudo rm -rf /Library/Receipts/MySQL*
    * sudo rm /etc/my.cnf


--> Install MySQL (server, startup item, pref pane):

- Install mysql's x86_64  version from the .dmg file. Download the .dmg from
  here: http://dev.mysql.com/downloads/mysql/5.1.html#downloads
  (I also put the installers under '3rdparty/MacOSX_Installers' folder).

--> Install MySQLdb in the virtualenv

- Turn the virtualenv on. This command does the trick:

    MacBookPro:~ matias$ laexpo_start
    (Env)MacBookPro:~ matias$

- Fetch MySQL-python-1.2.3 from here: http://sourceforge.net/projects/mysql-python/.
  I'll assume you downloaded the .tar.gz file to the Desktop.

- Unpack the tarball, enter the folder:

    (Env)MacBookPro:~ matias$ tar xzf MySQL-python-1.2.3.tar.gz
    (Env)MacBookPro:~ matias$ cd MySQL-python-1.2.3

- Open the 'site.cfg' file and edit the 'mysql_config' to poin to the path where
  you mysql_config resides. For me it is '/usr/local/mysql-5.1.50-osx10.6-x86_64/bin/mysql_config'

- Open the 'setup_posix.py' file and set the 'mysql_config.path' to the same path
  you did in the previous step(line 26). Should end up looking like this:

  mysql_config.path = "/usr/local/mysql-5.1.50-osx10.6-x86_64/bin/mysql_config"

- Compile the thing:

    (Env)MacBookPro:~ matias$ cd ~/Desktop/MySQL-python-1.2.3
    (Env)MacBookPro:MySQL-python-1.2.3 matias$ python setup.py clean
    (Env)MacBookPro:~ matias$ ARCHFLAGS='-arch x86_64' /usr/bin/python setup.py build
    (Env)MacBookPro:~ matias$ ARCHFLAGS='-arch x86_64' python setup.py install

* NOTE 1: It is critical to have the virtualenv activated before running the above
          listed command. If it is not, the installation of MySQLdb will be done in
          the system and no in the virtualenv and therefore MySQLdb will not be
          available when you turn the virtualenv on.

* NOTE 2: It is also critical to set the compilation flags correctly. If not set
          correctly, the compilation will default to 32-bit architecture and will
          not work afterwards with MySQL.

* NOTE 3: DON'T USE 'sudo' FOR THE COMPILLATION/INSTALL COMMANDS. If you do so, you'll
          be installing MySQLdb to the system and not to the virtualenv.


- Confirm the process worked ok:

    (Env)MacBookPro:MySQL-python-1.2.3 matias$ python
    Python 2.6.1 (r261:67515, Dec 17 2009, 00:59:15)
    [GCC 4.2.1 (Apple Inc. build 5646)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import MySQLdb
    >>> MySQLdb.version_info
    (1, 2, 3, 'final', 0)

How to determine if you are running a 32 or 64 bit version of Python?

Just run the this in a Python shell:

import ctypes
print ctypes.sizeof(ctypes.c_voidp)*8

In my case, it goes this way:

MacBookPro:~ matias$ python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> print ctypes.sizeof(ctypes.c_voidp)*8
64
>>>