Skip to content

Python value swap

Been looking (again) at XMPP recently. While browsing through existing source code and samples in several languages, there’s one pattern which comes back quite frequently in ‘echobot’ demos: when a message comes in, the to and from attributes are swapped, and the message is sent.

The most common approach is something like (pseudocode):

temp = from
from = to
to = temp

In Python there’s an easier approach though which seems to be unknown to several developers. It uses the multi-assignment/expansion syntax:

from, to = to, from

Basically, the tuple on the right (to, from) is constructed, then expanded to locals ‘from’ and ‘to’.

Just a hint :-) It’s a pretty elegant line of code IMHO.

Posted in Development.

Tagged with .

Sun -> Oracle

Sun acquired by Oracle. Yet another employer ;-)

Posted in Technology.

Tagged with , .

Erlang, Python and Twisted mashup using TwOTP

Recently, I’ve been toying around with Erlang again. After creating some simple apps I wanted to integrate some Erlang code inside a Python application (since that’s still my favorite day-to-day language, it’s used at work and I’m sort-of convinced Erlang would be a good choice for several of the applications we need to develop, integrated with our existing Python code). The most obvious solution would be to use an Erlang port, but this is IMHO rather cumbersome: it requires a developer to define a messaging format, parsing code for incoming messages, etc. There’s a tutorial available if you want to take this route.

A more elegant solution is creating a node using Python, similar to JInterface and equivalents. Luckily there’s an existing project working on a library to create Erlang nodes using Python and Twisted: TwOTP.

One downside: it’s rather underdocumented… So here’s a very quick demo how to call functions on an Erlang node from within a Twisted application.

First of all we’ll create 2 Erlang functions: one which returns a simple “Hello” message, one which uses an extra process to return ‘pong’ messages on calls to ‘ping’, and counts those.

The code:

-export([hello/1, ping/0, start/0]).

hello(Name) ->
    Message = "Hello, " ++ Name,
    io:format(Message ++ "~n", []),

ping_loop(N) ->
        {get_id, From} ->
            From ! {pong, N},
            ping_loop(N + 1)

ping() ->
    pingsrv ! {get_id, self()},
        {pong, N} -> ok
    {pong, N}.

start() ->
    Pid = spawn_link(fun() -> ping_loop(1) end),
    register(pingsrv, Pid).

This should be straight-forward if you’re familiar with Erlang (which I assume).

The Python code is not that hard to get either: it follows the basic Twisted pattern. First one should create a connection to EPMD, the Erlang Port Mapper Daemon (used to find other nodes), then a connection to the server node should be created, and finally functions can be called (calls happen the same way as Erlang’s RPC module).

Here’s the code. I’d advise to read it bottom-to-top:

import sys

from twisted.internet import reactor
import twotp

def error(e):
    '''A generic error handler'''
    print 'Error:'
    print e

def do_pingpong(proto):
    def handle_pong(result):
        # Parse the result
        # 'ping' returns a tuple of an atom ('pong') and an integer (the pong
        # id)
        # In TwOTP, an Atom object has a 'text' attribute, which is the string
        # form of the atom
        text, id_ = result[0].text, result[1]
        print 'Got ping result: %s %d' % (text, id_)
        # Recurse
        reactor.callLater(1, do_pingpong, proto)

    # Call the 'ping' function of the 'demo' module
    d = proto.factory.callRemote(proto, 'demo', 'ping')
    # Add an RPC call handler
    # And our generic error handler

def call_hello(proto, name):
    def handle_hello(result):
        print 'Got hello result:', result
        # Erlang strings are lists of numbers
        # The default encoding is Latin1, this might need to be changed if your
        # Erlang node uses another encoding
        text = ''.join(chr(c) for c in result).decode('latin1')
        print 'String form:', text
        # Start pingpong loop

    # Call the 'hello' function of the 'demo' module, and pass in argument
    # 'name'
    d = proto.factory.callRemote(proto, 'demo', 'hello', name)
    # Add a callback for this function call
    # And our generic error handler

def launch(epmd, remote, name):
    '''Entry point of our demo application'''
    # Connect to a node. This returns a deferred
    d = epmd.connectToNode(remote)
    # Add a callback, called when the connection to the node is established
    d.addCallback(call_hello, name)
    # And add our generic error handler

def main():
    remote = sys.argv[1]
    name = sys.argv[2]
    # Read out the Erlang cookie value
    cookie = twotp.readCookie()
    # Create a name for this node
    this_node = twotp.buildNodeName('demo_client')
    # Connect to EPMD
    epmd = twotp.OneShotPortMapperFactory(this_node, cookie)
    # Call our entry point function when the Twisted reactor is started
    reactor.callWhenRunning(launch, epmd, remote, name)
    # Start the reactor

if __name__ == '__main__':

Finally, to run it, you should first start a server node, and run the ‘pingsrv’ process:

MacBook:pyping nicolas$ erl -sname test@localhost
Erlang (BEAM) emulator version 5.6.5 [source] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.6.5  (abort with ^G)
(test@localhost)1> c(demo).
(test@localhost)2> demo:start().

Notice we started erl providing test@localhost as short node name.

Now we can launch our client:

(pythonenv)MacBook:pyping nicolas$ python 'test' Nicolas
Got hello result: [72, 101, 108, 108, 111, 44, 32, 78, 105, 99, 111, 108, 97, 115]
String form: Hello, Nicolas
Got ping result: pong 1
Got ping result: pong 2
Got ping result: pong 3

‘test’ is the shortname of the server node.

You can stop the ping loop using CTRL-C. If you restart the client afterwards, you can see the ping IDs were retained:

(pythonenv)MacBook:pyping nicolas$ python 'test' Nicolas
Got hello result: [72, 101, 108, 108, 111, 44, 32, 78, 105, 99, 111, 108, 97, 115]
String form: Hello, Nicolas
Got ping result: pong 4
Got ping result: pong 5

That’s about it. Using TwOTP you can also develop a node which exposes functions, which can be called from an Erlang node using rpc:call/4. Check the documentation provided with TwOTP for a basic example of this feature.

Combining Erlang applications as distributed, fault tolerant core infrastructure and Python/Twisted applications for ‘everyday coding’ can be an interesting match in several setups, an TwOTP provides all required functionalities to integrate the 2 platforms easily.

Posted in Development, Technology.

Tagged with , , , , .


It’s finally public: Q-layer, the company I’m employed at, has been acquired by Sun (VirtualBox, MySQL, Java, OpenSolaris, SPARC,… You know). Press release here.

Pretty exciting times ahead. The social event with several guys from the Sun Cloud computing department (among others) last night was really fun, looks like Sun is one of those companies ensuring a pleasant work environment (which ensures the continuation of the spirit at Q-layer before). I’m really thrilled to see how this adventure continues, since it’s the first time I’ll be involved in a large company.

Hi to all Sun colleagues reading this, by the way :-)

Interesting times…

Posted in Various.

Tagged with , .

Got to love the open source communities…

Hanging around in recently, this just passed by:

15:28  <eikke> is it possible in linux to access partitions in an losetup’ed raw image?
15:29  <carldani>  eikke: yes, I wrote a kernel patch for that
15:29  <eikke> ah, patch :p i did it using –offset now, but thats kinda hackish
15:29  <carldani>  eikke: the easiest way is to use kpartx
15:29  <carldani>  eikke: that was impossible back then
15:30  <eikke> i see
15:30  <carldani>  eikke: kpartx -a /dev/loop0
15:30  <carldani>  eikke: that will give you /dev/mapper/loop0p1 /dev/mapper/loop0p5 etc
15:30  <eikke> cool
15:30  <carldani>  eikke: are you the eikke responsible for ivman?
15:31  <eikke> a long time ago, yes
15:31  <eikke> dont start the flames please :p
15:31  <carldani>  eikke: thanks! I’m still using it on my laptop

A helps out B, B helps out A, A writes something useful for B, B wrote something used by A, etc. Got to love the open source movement.

Posted in Development.

Tagged with .

Python gotcha

Don’t ever do this unless it’s really what you want:

import os

def some_func(fd):
    f = os.fdopen(fd, 'w')

fd = get_some_fd()

Here’s what goes wrong: when some_func comes to an end, f (which is a file-like objects) goes out of scope, is destructed, which causes fd to be closed. I think this is pretty weird behavior (an object closing an fd it didn’t open itself), but well.

Here’s a better version, for reference:

def some_func(fd):
    f = os.fdopen(os.dup(fd), 'w')
    #Use f here

Try this on fd 0/1/2 in an (I)Python shell ;-)

Posted in Development.

Tagged with .

Scripting your app

Lots of buzz on adding scripting interfaces to applications on Planet GNOME recently, cool. Looks like Alexander Larsson hacked together a wrapper around SpiderMonkey (the Mozilla Engine) to get JavaScript integrated. Related to the jscore-python thing I blogged about before.

Not sure this is the best way to tackle this cool new opportunity though. JavaScript can be pretty hard to “get” for people not used to it, but more familiar with classical languages (wrt object oriented paradigms). I guess lots of current code contributors are not very familiar with JavaScript, but do have an in-depth knowledge of some other scripting language though (not listing any here, you certainly can name some).

So, wouldn’t it be nice if the GScript interface would be abstract for the application developer, who should just create a script context, put some objects in it, and launch a script, after which the GScript runtime figures out which interpreter to use for this script, so scripting languages become plugable?

Reminds me a little of my old OPluginManager work :-)

Posted in Desktop, Development.

Tagged with , , .

Embedding JavaScript in Python

Reading some posts about embedding languages/runtimes in applications on Planet GNOME reminded me I still had to announce some really quick and incomplete code blob I created some days after last GUADEC edition (which was insanely cool, thanks guys).

It takes WebKit‘s JavaScriptCore and allows you to embed it in some Python program, so you, as a Python developer, can allow consumers to write plugins using JavaScript. Don’t ask me whether it’s useful, maybe it’s not, but anyway.

There’s one catch: currently there is no support to expose custom Python objects to the JavaScript runtime: you’re able to use JavaScript objects and functions etc. from within Python, but not the other way around. I started working on this, but the JSCore API lacked some stuff to be able to implement this cleanly (or I missed a part of it, that’s possible as well), maybe it has changed by now… There is transparent translation of JavaScript base types: unicode strings, booleans, null (which becomes None in Python), undefined (which becomes jscore.UNDEFINED) and floats.

I did not work on the code for quite a long time because of too much real-job-work, maybe it no longer compiles, sorry… Anyway, it’s available in git here, patches welcome etc. I guess this is the best sample code around. It’s using Cython for compilation (never tried with Pyrex, although this might work as well). If anyone can use it, great, if not, too bad, I did learn Cython doing this ;-)

Posted in Development, Linux.

Tagged with , , , .

Free Software startups

At GUADEC last days, fun times. One thing I noticed (again) is the growth of the number of small companies, some of them driven by young guys (like, between 25 and 30) doing some really great stuff with Free Software, combining a steady income, whilst still providing valuable contributions to the community.

I’d want to write more about this seen, but here’s a quick idea: how about creating a mailing list for, at one side, these young startups or people interested in starting one, and on the other side people who already got their business running, or already established investors or managers, to discuss issues they encounter, potential business plans, how to get in contact with potential customers, how to combine open communication at one side and dealing with closed environments on the other,…

This could lower the barrier for people to start working full-time on the projects they love, it’d add value to Free Software because larger companies can start using it even more, because more expertise and consultancy is available on the market,…

I guess this is just a shot in the dark, but I think it could be pretty useful. Thoughts?

Posted in Technology.


Leaving for GUADEC ’08 with RubenV tomorrow. We’re staying in the Golden Horn Sultanahmet hotel. Packing starts in a minute, see you around.

Posted in Various.

Tagged with , .