Skip to content


Erlang, Python and Twisted mashup using TwOTP

Recently, I’ve been toying around with Erlang again. After creating some simple apps I wanted to integrate some Erlang code inside a Python application (since that’s still my favorite day-to-day language, it’s used at work and I’m sort-of convinced Erlang would be a good choice for several of the applications we need to develop, integrated with our existing Python code). The most obvious solution would be to use an Erlang port, but this is IMHO rather cumbersome: it requires a developer to define a messaging format, parsing code for incoming messages, etc. There’s a tutorial available if you want to take this route.

A more elegant solution is creating a node using Python, similar to JInterface and equivalents. Luckily there’s an existing project working on a library to create Erlang nodes using Python and Twisted: TwOTP.

One downside: it’s rather underdocumented… So here’s a very quick demo how to call functions on an Erlang node from within a Twisted application.

First of all we’ll create 2 Erlang functions: one which returns a simple “Hello” message, one which uses an extra process to return ‘pong’ messages on calls to ‘ping’, and counts those.

The code:

-module(demo).
-export([hello/1, ping/0, start/0]).

hello(Name) ->
    Message = "Hello, " ++ Name,
    io:format(Message ++ "~n", []),
    Message.

ping_loop(N) ->
    receive
        {get_id, From} ->
            From ! {pong, N},
            ping_loop(N + 1)
    end.

ping() ->
    pingsrv ! {get_id, self()},
    receive
        {pong, N} -> ok
    end,
    {pong, N}.

start() ->
    Pid = spawn_link(fun() -> ping_loop(1) end),
    register(pingsrv, Pid).

This should be straight-forward if you’re familiar with Erlang (which I assume).

The Python code is not that hard to get either: it follows the basic Twisted pattern. First one should create a connection to EPMD, the Erlang Port Mapper Daemon (used to find other nodes), then a connection to the server node should be created, and finally functions can be called (calls happen the same way as Erlang’s RPC module).

Here’s the code. I’d advise to read it bottom-to-top:

import sys

from twisted.internet import reactor
import twotp

def error(e):
    '''A generic error handler'''
    print 'Error:'
    print e
    reactor.stop()

def do_pingpong(proto):
    def handle_pong(result):
        # Parse the result
        # 'ping' returns a tuple of an atom ('pong') and an integer (the pong
        # id)
        # In TwOTP, an Atom object has a 'text' attribute, which is the string
        # form of the atom
        text, id_ = result[0].text, result[1]
        print 'Got ping result: %s %d' % (text, id_)
        # Recurse
        reactor.callLater(1, do_pingpong, proto)

    # Call the 'ping' function of the 'demo' module
    d = proto.factory.callRemote(proto, 'demo', 'ping')
    # Add an RPC call handler
    d.addCallback(handle_pong)
    # And our generic error handler
    d.addErrback(error)

def call_hello(proto, name):
    def handle_hello(result):
        print 'Got hello result:', result
        # Erlang strings are lists of numbers
        # The default encoding is Latin1, this might need to be changed if your
        # Erlang node uses another encoding
        text = ''.join(chr(c) for c in result).decode('latin1')
        print 'String form:', text
        # Start pingpong loop
        do_pingpong(proto)

    # Call the 'hello' function of the 'demo' module, and pass in argument
    # 'name'
    d = proto.factory.callRemote(proto, 'demo', 'hello', name)
    # Add a callback for this function call
    d.addCallback(handle_hello)
    # And our generic error handler
    d.addErrback(error)

def launch(epmd, remote, name):
    '''Entry point of our demo application'''
    # Connect to a node. This returns a deferred
    d = epmd.connectToNode(remote)
    # Add a callback, called when the connection to the node is established
    d.addCallback(call_hello, name)
    # And add our generic error handler
    d.addErrback(error)

def main():
    remote = sys.argv[1]
    name = sys.argv[2]
    # Read out the Erlang cookie value
    cookie = twotp.readCookie()
    # Create a name for this node
    this_node = twotp.buildNodeName('demo_client')
    # Connect to EPMD
    epmd = twotp.OneShotPortMapperFactory(this_node, cookie)
    # Call our entry point function when the Twisted reactor is started
    reactor.callWhenRunning(launch, epmd, remote, name)
    # Start the reactor
    reactor.run()

if __name__ == '__main__':
    main()

Finally, to run it, you should first start a server node, and run the ‘pingsrv’ process:

MacBook:pyping nicolas$ erl -sname test@localhost
Erlang (BEAM) emulator version 5.6.5 [source] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.6.5  (abort with ^G)
(test@localhost)1> c(demo).
{ok,demo}
(test@localhost)2> demo:start().
true

Notice we started erl providing test@localhost as short node name.

Now we can launch our client:

(pythonenv)MacBook:pyping nicolas$ python hello.py 'test' Nicolas
Got hello result: [72, 101, 108, 108, 111, 44, 32, 78, 105, 99, 111, 108, 97, 115]
String form: Hello, Nicolas
Got ping result: pong 1
Got ping result: pong 2
Got ping result: pong 3

‘test’ is the shortname of the server node.

You can stop the ping loop using CTRL-C. If you restart the client afterwards, you can see the ping IDs were retained:

(pythonenv)MacBook:pyping nicolas$ python hello.py 'test' Nicolas
Got hello result: [72, 101, 108, 108, 111, 44, 32, 78, 105, 99, 111, 108, 97, 115]
String form: Hello, Nicolas
Got ping result: pong 4
Got ping result: pong 5

That’s about it. Using TwOTP you can also develop a node which exposes functions, which can be called from an Erlang node using rpc:call/4. Check the documentation provided with TwOTP for a basic example of this feature.

Combining Erlang applications as distributed, fault tolerant core infrastructure and Python/Twisted applications for ‘everyday coding’ can be an interesting match in several setups, an TwOTP provides all required functionalities to integrate the 2 platforms easily.

Posted in Development, Technology.

Tagged with , , , , .


9 Responses

Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.

  1. Jean-Pierre says

    Don’t understand a hole of this.

    • Nicolas says

      Poor you. No need to share that feeling with the world though.

  2. Dave Bryson says

    Very nice. I just came across your work. I’ve been working on something similar for Ruby using Eventmachine. I haven’t had the opportunity to look though all the code yet; but have you tackled “links” yet?

  3. Thomas Hervé says

    Thanks a lot for this post! I’m Twotp author, and I’d be happy to help you if you have some questions. It’s a side project and I don’t spend much time on it nowadays, but I’d like to spend more!

  4. Alex Clemesha says

    Yes, thanks to Nicolas for posting this article – it prompted me to realize how awesome Twotp is. I’m personally going to be using it for some RabbitMQ management, but connecting (in a first class citizen type of way) to any sort of Erlang node from twisted (like Ejabberd, etc) is powerful. Also, I’d like to mention that last night I talked with Thomas Hervé (author of Twotp) on the #twisted irc channel (he is ‘therve’) and he was extremely helpful. Furthermore, I noticed by the next morning a small bug was fixed and a couple great examples were added to Twotp, so I recommend getting the latest source.

  5. sl63 says

    Yep, Erlang and Python could become like Peanut Butter and Jelly for web programming.

  6. Alex Clemesha says

    @sl63 I’m mostly in the same camp with the point you are making, but you might have presented it in a less snide way. The presentation _is_ excellent, so thanks for posting it. I’m interested in using Twopt to manage RabbitMQ *locally*, as using this technique is far better than, say, scripting shell calls to “rabbitmqctl”. Until other solutions come along, this is a perfect way of solving some immediate problems.

  7. Steve says

    Note for current readers: Since this blog entry was published, twotp introduced backwards incompatible changes. These changes invalidate this blog entry. Download the latest twotp release and look in the doc folder for a few samples to get you going.

  8. Steve says

    Also, if you’ve downloaded release 0.7, you’ll probably need to apply the fix suggested here:

    https://bugs.launchpad.net/twotp/+bug/767842



Some HTML is OK

or, reply to this post via trackback.