Long time no post, I should really have had something up for 3.4.0, and that was of course the plan but on short notice I went to Vientiane, Laos, for $dayjob, the week of the release. Timezone difference, internet connection at the hostel, and a busy schedule made it quite impossible to participate in the release. I didn't stay long but I had a really good time there, thanks again Chanesakhone and Jean Christophe.

Then, back in Brussels I had to spend time on my new apartement, various administrative tasks, arranging things for gas, electricity and water, keeping an eye on the roof where workers had to knock a chimney down, discussing plans with an architect friend, and so on.

Things finally settled down and two releases have now consecutively been done; 3.4.1 brought some important fixes, and improvements in accessibility, translations and documentation; then last week 3.5 opened the path to 3.6. A new adventure begins…

Swift is the software behind the OpenStack Object Storage service.

This service provides a simple storage service for applications using RESTful interfaces, providing maximum data availability and storage capacity.

I explain here how some parts of the storage and replication in Swift works, and show some of its current limitations.

If you don't know Swift and want to read a more "shallow" overview first, you can read John Dickinson's Swift Tech Overview.

How Swift storage works

If we refer to the CAP theorem, Swift chose availability and partition tolerance and dropped consistency. That means that you'll always get your data, they will be dispersed on many places, but you could get an old version of them (or no data at all) in some odd cases (like some server overload or failure). This compromise is made to allow maximum availability and scalability of the storage platform.

But there are mechanisms built into Swift to minimize the potential data inconsistency window: they are responsible for data replication and consistency.

The official Swift documentation explains the internal storage in a certain way, but I'm going to write my own explanation here about this.

Consistent hashing

Swift uses the principle of consistent hashing. It builds what it calls a ring. A ring represents the space of all possible computed hash values divided in equivalent parts. Each part of this space is called a partition.

The following schema (stolen from the Riak project) shows the principle nicely:

In a simple world, if you wanted to store some objects and distribute them on 4 nodes, you would split your hash space in 4. You would have 4 partitions, and computing hash(object) modulo 4 would tell you where to store your object: on node 0, 1, 2 or 3.

But since you want to be able to extend your storage cluster to more nodes without breaking the whole hash mapping and moving everything around, you need to build a lot more partitions. Let's say we're going to build 2¹⁰ partitions. Since we have 4 nodes, each node will have 210 ÷ 4 = 256 partitions. If we ever want to add a 5^th node, it's easy: we just have to re-balance the partitions and move 1⁄4 of the partitions from each node to this 5^th node. That means all our nodes will end up with 210 ÷ 5 ≈ 204 partitions. We can also define a weight for each node, in order for some nodes to get more partitions than others.

With 2¹⁰ partitions, we can have up to 2¹⁰ nodes in our cluster. Yeepee.

For reference, Gregory Holt, one of the Swift authors, also wrote an explanation post about the ring.

Concretely, when building one Swift ring, you'll have to say how much partitions you want, and this is what this value is really about.

Data duplication

Now, to assure availability and partitioning (as seen in the CAP theorem) we also want to store replicas of our objects. By default, Swift stores 3 copies of every objects, but that's configurable.

In that case, we need to store each partition defined above not only on 1 node, but on 2 others. So Swift adds another concept: zones. A zone is an isolated space that does not depends on other zone, so in case of an outage on a zone, the other zones are still available. Concretely, a zone is likely to be a disk, a server, or a whole cabinet, depending on the size of your cluster. It's up to you to chose anyway.

Consequently, each partitions has not to be mapped to 1 host only anymore, but to N hosts. Each node will therefore store this number of partitions:

number of partition stored on one node = number of replicas × total number of partitions ÷ number of node

Examples:

We split the ring in 2¹⁰ = 1024 partitions. We have 3 nodes. We want 3 replicas of data.
→ Each node will store a copy of the full partition space: 3 × 210 ÷ 3 = 210 = 1024 partitions.

We split the ring in 2¹¹ = 2048 partitions. We have 5 nodes. We want 3 replicas of data.
→ Each node will store 211 × 3 ÷ 5 ≈ 1129 partitions.

We split the ring in 2¹¹ = 2048 partitions. We have 6 nodes. We want 3 replicas of data.
→ Each node will store 211 × 3 ÷ 6 = 1024 partitions.

Three rings to rule them all

In Swift, there is 3 categories of thing to store: account, container and objects.

An account is what you'd expect it to be, a user account. An account contains containers (the equivalent of Amazon S3's buckets). Each container can contains user-defined key and values (just like a hash table or a dictionary): values are what Swift call objects.

Swift wants you to build 3 different and independent rings to store its 3 kind of things (accounts, containers and objects).

Internally, the two first categories are stored as SQLite databases, whereas the last one is stored using regular files.

Note that this 3 rings can be stored and managed on 3 completely different set of servers.

Data replication

Now that we have our storage theory in place (accounts, containers and objects distributed into partitions, themselves stored into multiple zones), let's go the replication practice.

When you put something in one of the 3 rings (being an account, a container or an object) it is uploaded into all the zones responsible for the ring partition the object belongs to. This upload into the different zones is the responsibility of the swift-proxy daemon.

But if one of the zone is failing, you can't upload all your copies in all zones at the upload time. So you need a mechanism to be sure the failing zone will catch up to a correct state at some point.

That's the role of the swift-{container,account,object}-replicator processes. This processes are running on each node part of a zone and replicates their contents to nodes of the other zones.

When they run, they walk through all the contents from all the partitions on the whole file system and for each partition, issue a special REPLICATE HTTP request to all the other zones responsible for that same partition. The other zone responds with information about the local state of the partition. That allows the replicator process to decide if the remote zone has an up-to-date version of the partition.

In case of account and containers, it doesn't check at the partition level, but check each account/container contained inside each partition.

If something is not up-to-date, it will be pushed using rsync by the replicator process. This is why you'll read that the replication updates are "push based" in Swift documentation.

# Pseudo code describing replication process for accounts
# The principle is exactly the same for containers
for account in accounts:
    # Determine the partition used to store this account
    partition = hash(account) % number_of_partitions
    # The number of zone is the number of replicas configured
    for zone in partition.get_zones_storing_this_partition():
        # Send a HTTP REPLICATE command to the remote swift-account-server process
        version_of_account = zone.send_HTTP_REPLICATE_for(account):
        if version_of_account  account.version()
            account.sync_to(zone)

This replication process is O(number of account × number of replicas). The more your number of account will increase and the more you will want replicas for your data, the more the replication time for your accounts will grow. The same rule applies for containers.

# Pseudo code describing replication process for objects
for partition in partitions_storing_objects:
    # The number of zone is the number of replicas configured
    for zone in partition.get_zones_storing_this_partition():
        # Send a HTTP REPLICATE command to the remote swift-object-server process
        verion_of_partition = zone.send_HTTP_REPLICATE_for(partition):
        if version_of_partition  partition.version()
            # Use rsync to synchronize the whole partition
            # and all its objects
            partition.rsync_to(zone)

This replication process is O(number of objects partitions × number of replicas). The more your number of objects partitions will increase, and the more you will want replicas for your data, the more the replication time for your objects will grow.

I think this is something important to know when deciding how to build your Swift architecture. Choose the right number the number of replicas, partitions and nodes.

Replication process bottlenecks

File accesses

The problem, as you might have guessed, is that to replicate, it walks through every damn things, things being accounts, containers, or object's partition hash files. This means it need to open and read (part of) a every file your node stores to check that data need or not to be replicated!

For accounts & containers replication, this is done every 30 seconds by default, but it will likely take more than 30 seconds as soon as you hit around 12 000 containers on a node (see measurements below). Therefore you'll end up checking consistency of accounts & containers on each all node all the time, using obviously a lot of CPU time.

For reference, Alex Yang also did an analysis of that same problem.

TCP connections

Worst, the HTTP connections used to send the REPLICATE commands are not pooled: a new TCP connection is established each time something has to be checked against the same thing stored on a remote zone.

This is why you'll see in the Swift's Deployment Guide this lines listed under "general system tuning":

# disable TIME_WAIT.. wait..
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1

# double amount of allowed conntrack
net.ipv4.netfilter.ip_conntrack_max = 262144

In my humble opinion, this is more an ugly hack than a tuning. If you don't activate this and if you have a lot of containers on your node, you'll end up soon with thousands of connections in TIME_WAIT state, and you indeed risk to overload the IP conntrack module.

Container deletion

We also should talk about container deletion. When a user deletes a container from its account, the container is marked as deleted. And that's it. It's not deleted. Therefore the SQLite database file representing the container will continue to be checked for synchronization, over and over.

The only way to have a container permanently deleted is to mark an account as deleted. This way the swift-account-reaper will delete all its containers and, finally, the account.

Measurement

On a pretty big server, I measured the replications to be done at a speed of around 350 {account,container,object-partitions}/second, which can be a real problem if you chose to build a lots of partition and you have a low number_of_node ⁄ number_of_replicas ratio.

For example, the default parameters runs the container replication every 30 seconds. To check replication status of 12 000 containers stored on one node at the speed of 350 containers/seconds, you'll need around 34 seconds to do so. In the end, you'll never stop checking replication of your containers, and the more you'll have containers, the more your inconsistency window will increase.

Conclusion

Until some of the code is fixed (the HTTP connection pooling probably being the "easiest" one), I warmly recommend to chose correctly the different Swift parameters for your setup. The replication process optimization consists in having the minimum amount of partitions per node, which can be done by:

• decreasing the number of partitions
• decreasing the number of replicas
• increasing the number of node

For very large setups, some code to speed up accounts and containers synchronization, and remove deleted containers will be required, but this does not exist yet, as far as I know.

The opinions expressed here represent my own and not those of my employer.

Today I release a Python client library to query Munin servers.

I wrote it as part of some experiments I did a few weeks ago. I discovered there was no client library to query a Munin server. There's PyMunin or python-munin which help developing Munin plugins, but nothing to access the munin-node and retrieve its data.

So I decided to write a quick and simple one, and it's released under the name of PyMuninCli, providing the munin.client Python module.

Mise en ligne des photos de mon voyage en Islande.

Apache 2.4 being out, I noticed that my good old mod_defensible did not compile anymore.

The changes in the new Apache 2.4 API were small for its concern, so it was pretty easy to update this software to make it compile again.

Honestly, I'm not sure that this module is really used into the wild, but I still think that it can serve as a good prototype for doing other things so I like keeping it around. :-)

All this has been triggered by the Apache 2.4 arrival into Debian experimental. Therefore I've updated the mod_defensible package to use the new dh_apache2, and imported it into Git at the same time.

It took a while to get it out, but finally, 3 years after the latest release (1.2), the version of 1.3 of xpyb (the XCB Python bindngs) is out.

This version has a lot of improvement, and major bug fixes (memory corruption and memory leak were tracked down and fixed).

One amazing feature that is now shipped with that release, is my code to export the xpyb API to other Python modules, allowing to draw with Pycairo in Python using XCB.

Here is an example of a Python program that draws a spiral in a window using xpyb and Pycairo. You need xpyb >= 1.3 and Pycairo >= 1.10 to make this works.

import cairo
import xcb
from xcb.xproto import *

WIDTH, HEIGHT = 600, 600

def draw_spiral(ctx, width, height):
    """Draw a spiral with lines!"""
    wd = .02 * width
    hd = .02 * height

    width -= 2
    height -= 2

    ctx.move_to (width + 1, 1-hd)
    for i in range(9):
    ctx.rel_line_to (0, height - hd * (2 * i - 1))
    ctx.rel_line_to (- (width - wd * (2 *i)), 0)
    ctx.rel_line_to (0, - (height - hd * (2*i)))
    ctx.rel_line_to (width - wd * (2 * i + 1), 0)

    ctx.set_source_rgb (0, 0, 1)
    ctx.stroke()

# Connect to the X server
conn = xcb.connect()
# Get the X server setup
setup = conn.get_setup()
# Generate X ID for our X "objects"
window = conn.generate_id()
pixmap = conn.generate_id()
gc = conn.generate_id()
# Create a new window
conn.core.CreateWindow(setup.roots[0].root_depth, window,
                       # Parent is the root window
                       setup.roots[0].root,
                       0, 0, WIDTH, HEIGHT, 0, WindowClass.InputOutput,
                       setup.roots[0].root_visual,
                       CW.BackPixel | CW.EventMask,
                       [ setup.roots[0].white_pixel, EventMask.ButtonPress | EventMask.EnterWindow | EventMask.LeaveWindow | EventMask.Exposure ])

# Create a pixmap: it will be used to draw with cairo
conn.core.CreatePixmap(setup.roots[0].root_depth, pixmap, setup.roots[0].root,
                       WIDTH, HEIGHT)

# We just need a GC to copy later the pixmap on the window, so create one
# very simple
conn.core.CreateGC(gc, setup.roots[0].root, GC.Foreground | GC.Background,
                   [ setup.roots[0].black_pixel, setup.roots[0].white_pixel ])

# Create a cairo surface
surface = cairo.XCBSurface (conn, pixmap,
                            setup.roots[0].allowed_depths[0].visuals[0], WIDTH, HEIGHT)
# Create a cairo context with that surface
ctx = cairo.Context(surface)

# Paint everything in white
ctx.set_source_rgb (1, 1, 1)
ctx.set_operator (cairo.OPERATOR_SOURCE)
ctx.paint()

# Draw our spiral
draw_spiral (ctx, WIDTH, HEIGHT)

# Map the window on the screen so it gets visible
conn.core.MapWindow(window)

# Flush all X requests to the X server
conn.flush()

while True:
    try:
        event = conn.wait_for_event()
    except xcb.ProtocolException, error:
        print "Protocol error %s received!" % error.__class__.__name__
        break
    except:
        break

    # ExposeEvent are received when we need to refresh the content of the
    # window, so we copy the content of the pixmap (where cairo drew) in the
    # window
    if isinstance(event, ExposeEvent):
        conn.core.CopyArea(pixmap, window, gc, 0, 0, 0, 0, WIDTH, HEIGHT)
    # You click, I quit.
    elif isinstance(event, ButtonPressEvent):
        break
    conn.flush()

Seeing the complexity it is to draw something simple with this technology, I somehow understand why nobody bothered to release or use the code during the last 3 years.

But hey, now that it's out, you can build the next Python based desktop environment with bleeding edge technologies. :-)

Top départ à 10h17, on se voit, dernières marches de l'escalator, on se dira bonjour plus tard, on sprinte ! 10h18, heure officielle de départ du train. 10h19, arrivée sur le quai, il est toujours là, on saute dedans. On souffle.

Lille. Puis Dunkerque. Retrouver et rencontrer des personnes, se perdre dans les habits, tenter mille combinaisons, se maquiller, sortir.

Carnaval.

Lignes. Chahuts. Harengs. Rigodon.

Le lendemain se balader, retraverser la frontière, La Panne, saluer tout le monde, rater un train, courir à Gand, arriver à Bruxelles, passer encore du temps ensemble, attraper le dernier métro.

Moments qui seront chéris. N.

Ten years ago, I joined the Debian project as a developer.

At that time, I was 18 and in my first year at university, hanging out with the TuxFamily system administrators, which included 3 french Debian developers (sjg, igenibel and creis).

I was learning Debian packaging while working on VHFFS, and decided to package one or two non-yet-packaged software for Debian. My friends pushed me into the NM process, and less than 2 months later I was a Debian developer. One have to admit that back in the days, the NM process was really fast if you were able to reply to the questions quickly. :-) I think I became the youngest developer among Debian's ones.

That was my first steps in a Free Software project, and it was really exciting.

In 10 years, I've been doing a lot of different things for Debian. Sure, I've been using it all the years long, but let's recap a bit what I did, from what I recall.

My first Debian only project was apt-build around 2003, and later rebuildd in 2007.

I built the Xen packaging team in 2005, I've been a Stable Release Manager for a year in 2006, and did heavy bug squashing to release Etch that same year.

I also was an Application Manager in 2006 and managed the application of 2 Debian developers (Jose Parrella and Damián Viano).

I admit I've been less active in Debian after 2007, mainly because I was busy working on awesome, GNU Emacs and others software.

Since 2011, I joined the OpenStack packaging team and I'm working on OpenStack on a (almost) daily basis.

I don't know how many packages I touched, managed or updated, but that should be one or two hundreds. I still maintain 53 of them.

After all, the adventure has been really pleasant, and I had the chance to work with and meet fabulous and smart people. I always liked this project and what it's trying to do.

After all these years, I'm definitively staying! See you in another 10 years, folks! :)

After several months on hiatus, then some January evenings to process the backlog, I am happy to have the commit digests back to the present day.

What now? I'll try to get back to the weekly updates, whatever the weather.

Of course you can help; whenever you see a noteworthy commit, whenever you make a noteworthy commit, just send me an email, or ping me on IRC, this will help me, and could also bring other perspectives on what constitutes a “noteworthy” commit. And if you love the commit digests, if you have time on your hands, you can help extending the projects to new heights, got an interest in statistics? got an interest in interviews? there's a place for you.

Happy reading!

…

À trainer à gauche à droite j'ai forcément lu quelques bilans 2011, je passe mon tour (dyslexique j'aurais écrit « je passe mon trou » et ça nous aurait fait rire un peu) mais je m'étais dit que dans mon billet sur Montréal (ses gens, ses bars, la casa del popolo) j'aurais une place pour les lectures, et ce billet, il n'arrivera pas, tant pis pour les photos, mais quand même, bilans 2011, listes de lecture, prétexte et phrase trop longue.

Capitalisme, désir et servitude, de Frédéric Lordon. Énorme. Oublié dans l'avion au retour, à nouveau acheté lors des emplettes de Noël pour en lire les dernières pages. Sous-titré « Marx et Spinoza ». Et de ce dernier, la mise en exergue de cette phrase, « Par réalité et par perfection, j'entends la même chose », qui est quand même la phrase illico recopiée lors de ma lecture de l'Éthique…

Un livre, beaucoup(?) d'autres, et après avoir lu De onze à douze je me suis motivé à enfin faire l'inventaire de ma bibliothèque (en sous-texte il y a la perspective d'un déménagement…), goodreads, quelques soirées d'encodage, bien sûr ponctuées d'écroulages de piles, mais en sortie, enfin, un tableau. Et un tas de statistiques amusantes à faire. Une autre fois.

Et pour finir dans le sous-texte, première lecture de l'année.

I use Google Calendar to manage my calendars, and I really missed something to warn me whenever I have an appointment with an alert set.

So here is an example of a Python program to do such a thing. It is written using the Google Data APIs Python client library and pynotify.

I'll detail the code here, so you can build your own and adapt it to your needs.

First, we need to import GTK+ and pynotify, and initialize it.

import gtk
import pynotify
pynotify.init(sys.argv[0])

Then, we need to import gdata Calendar API and connect to the calendar. I'll use the simple email/password way to login, which is clearly not the best, but it's also the simplest. Feel free to use OAuth 2.0. :-)

calendar_service = gdata.calendar.service.CalendarService()
calendar_service.email = 'mygooglelogin'
calendar_service.password = 'mygooglepassword'
calendar_service.ProgrammaticLogin()

Now we're ready to request stuff and notify! First, request the events from the default calendar.

feed = calendar_service.GetCalendarEventFeed()

Now we can iterate over feed and do various checks.

for event in feed.entry:
    # If the event status is not confirmed, go to the next event.
    if event.event_status.value != "CONFIRMED":
        continue
    # Now iterate over all the event dates (usually it has one)
    for when in event.when:
        # Parse start and end time
        try:
            start_time = datetime.datetime.strptime(when.start_time.split(".")[0], "%Y-%m-%dT%H:%M:%S")
            end_time = datetime.datetime.strptime(when.end_time.split(".")[0], "%Y-%m-%dT%H:%M:%S")
        except ValueError:
            # ValueError happens on parsing error. Parsing errors
            # usually happen for "all day" events since they have
            # not time, but we do not care about this events.
            continue
        now = datetime.datetime.now()
        # Check that the event hasn't already ended
        if end_time > now:
            # Check each alert
            for reminder in when.reminder:
                # We handle only reminders with method "alert"
                # and whose start time minus the reminder delay has passed
                if reminder.method == "alert" \
                        and start_time - datetime.timedelta(0, 60 * int(reminder.minutes))  now:
                    # Build the notification
                    notification = pynotify.Notification(summary=event.title.text,
                                                         message=event.content.text)
                    # Set an icon from the GTK+ stock icons
                    notification.set_icon_from_pixbuf(gtk.Label().render_icon(gtk.STOCK_DIALOG_INFO,
                                                                              gtk.ICON_SIZE_LARGE_TOOLBAR))
                    notification.set_timeout(0)
                    # Show the notification
                    notification.show()

Running this program, you should see a notification if an appointment has an alert to be raised at that time.

This should be enough to start to build something.

If you don't want to program this into Python, you might want to take a look at gcalcli.

It took me a while to find this, so I'm just blogging it so other people will be able to find it.

I wanted to send a desktop notification using pynotify, but using a GTK+ stock icons.

With the following snippet, I managed to do it.

import pynotify
pynotify.init("myapp")
import gtk
n = pynotify.Notification(summary="Summary", message="Message!")
n.set_icon_from_pixbuf(gtk.Label().render_icon(gtk.STOCK_HARDDISK, gtk.ICON_SIZE_LARGE_TOOLBAR))
n.show()

Note that the use of a Label is just to have a widget instanciated to use the render_icon() method. It could be any widget type as far as I understand.

Like I already wrote here last week, I've been heavily working on OpenStack for the last weeks.

My first assignment was to package OpenStack for Debian. The packages already present in unstable were mainly done by Thomas Goirand, who based its work on the one done in Ubuntu. Therefore, the packages where not in a very good shape for Debian.

Today Ghe Rivero and I (members of the OpenStack Debian packaging team) managed to push the OpenStack Essex 2 milestone into unstable with great success. You can now test and deploy OpenStack Essex 2 very easily!

Packaging OpenStack made me write several patches, mainly related to packaging, patches which were all accepted and merged by upstream. This is nice because most of the OpenStack Debian packages lost their debian/patches directories now!

Finally, I've finished to implement one blueprint I really missed: the ability to boot from an ISO image using libvirt. The code still needs a review, but it should be included in the Essex 3 milestone if everything's right.

It has been a while since I blogged but I've been very busy, with my new job and this new blog!

New job!

I quitted my job last September, and found another one that I started in October. I'm now the lead developer of eNovance Labs, where I work on the OpenStack project. So far, this allowed me to contribute heavily to the Debian packaging of OpenStack.

New blog!

In the meantime, I took some time to redesign my personal homepage and this blog, which is now using Hyde, the Python equivalent of Jekyll, which is in Ruby. Since I dislike Ruby (sorry), I preferred to use a Python based generator, and I admit Hyde is really cool. Since I really suck at Web design, this one is obviously based on Twitter's bootstrap

C'était ce week-end, journées du logiciel libre à Lyon, c'était le week-end dernier, Ubuntu Party à Paris, c'était il y a déjà plus d'un mois, Montréal, c'était il y a, oh, cinq mois… déjà.

C'était, calibré, rythme de six mois, Ubuntu Party à Paris; c'était, rythme d'un an, JDLL l'année dernière, et voilà donc qu'il y a maintenant plus d'un an j'étais ainsi, assis, sur les pentes d'une croix rousse, à discuter, à ne pas savoir. Entre… et …

Pour, aujourd'hui, une question en moins. Mais je ne sais quoi.

Mise en ligne du journal de bord de ma randonnée à vélo Tours - Bouchemaine - Dompierre-sur-Mer.

The date came late, and it was definitely not at the best time wrt some projets at work, but I decided to go nevertheless, and have to give my thanks to the GNOME Foundation, and the travel committee, for quickly accepting when I asked for sponsorship.

Probably because of the short notice it felt like some important teams didn't have enough representation, and while this gave ample place for some topics (building gnome!) I wish we had enough teams for a roundup of the different aspects of GNOME. On the positive side this wide cooperation is happening in the mailing list discussion on freezes, with translators, documentation team, release team and other interested parties.

Still, back to Montreal and the summit, I spent much of the first day testing and reviewing jhbuild patches, and wrapping the day with the presentation of Baserock by Lars Wirzenius. The second day was more diverse, and more intense, with (I heard) a very interesting discussion on GNOME strategy (Tiffany wrote about it in details) that happened at the same time as a jhbuild (and more) session lead by Colin, and later in the afternoon a good serie of questions asked by Xan about our (lack of a clear) developer platform.

And the Collabora party, of course.

Then on Monday, more patch reviews, including (at last) Bug 654872 - Delete no longer shipped files at install time but the day was short as many people had to leave early, so it ended with random hacking and bug filing, with the good luck of hitting a bug in glib-networking with Nicolas Dufresne sitting just behind.

All in all this was my first summit and it went well, it would sure benefit from some earlier planning (both dates, and sessions), but this was a nice chance to see new heads (and known heads, of course), especially as I was not in Berlin this summer.

Six mois, presque, depuis les derniers messages, du temps, des lieux, pas de photos, des quenelles, et pour encore quelques jours, le Québec.

I finally finished a thing I was really missing: accessing my Google Contacts from Emacs.

That's now possible, thanks to my new google-contacts.el package.

It includes searching for any contact and displaying the result in a window. You can also jump to a contact from Gnus by pressing a key, and complete e-mail addresses while composing a mail.

This week, I've finished my OAuth 2.0 client implementation for GNU Emacs.

I have imported it into GNU ELPA so Emacs 24 users will be soon able to install it using the new Emacs packaging system.

OAuth 2.0 can be used to access, among others, Google APIs or the Facebook Graph API.

Mise en ligne du journal de bord de ma randonnée à vélo Tours - Saumur.

Mise en ligne de quelques-uns de mes morceaux sur le site Grooveshark.

After more than 5 years at Easter-eggs as a system engineer, I'll be leaving my job soon.

It has been a fabulous adventure, also due to the "cooperative" nature of the company. I've enjoyed worked here, with great people. I do wish them luck for the future. Looking at the numerous things I did for the past years, it has been quite productive!

Therefore, I'll be looking for a new job in the next weeks, which will probably keep me busy a bit. :-)

Mise en ligne des photos de mon voyage en Grèce (sans vélo cette fois-ci ;-) ).

La bibliothèque a fait son entrée sur pypi en version 0.9, le code pour HOTP et TOTP est complet et comprend une suite de test reprenant les vecteurs de test des RFC.

Une nouvelle spécification a fait son entrée: OCRA ou RFC-6287. OCRA ne définit pas un mais toute une suite d’algorithmes d’authentification simple ou mutuelle ainsi que de signature, à base de challenge sur les valeurs renvoyées par la fonction de hachage défini par HOTP. Chaque algorithme est défini par une chaine comme par exemple:

OCRA-1:HOTP-SHA256-6:QN08

Cette chaîne se lit comme il suit:

• selon la syntaxe V1 des spécifications OCRA: OCRA-1
• utilisant l’agorithme HMAC/HOTP, via la fonction de hachage SHA256 et un résultat décimal à six caractères: HOTP-SHA256-6,
• en réponse à un challenge de type numérique de au maximum 8 chiffres: QN08.

C’est prévu pour couvrir un grand nombre de besoins et en même temps permettre à plusieurs implémentations d’être facilement interopérable, le format de description permettant de vérifier et configurer l’implémentation.

La bibliothèque inclut le code d’analyse de ces chaînes ainsi que le code de calcul des condensés (hash). La prochaine étape est l’importation des vecteurs de test depuis la RFC et l’écriture de tests unitaires les utilisant. Cela devrait donner la version 1.0.

Mise en ligne du journal de bord de ma randonnée à vélo Tours - Île de Groix.

Mise en ligne du journal de bord de ma randonnée à vélo Tours - Rochefort-en-Terre.

This week I lost some time playing with Python's sets.

After digging into Python source code, I finally discovered there is what seems to be little bug. Anyway, it has been "fixed" in Python 3, fortunately. I did not find if it was reported somewhere, but since it's fixed, it's not a big deal.

Python 2.7.1+ (default, Apr 20 2011, 10:53:33) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class A(object):
...     def __eq__(self, other):
...             return True
... 
>>> A() == A()
True
>>> [A()] == [A()]
True
>>> set([A()]) == set([A()])
False

This clearly did not make any sense to me. I've then tested under Python 3.2:

Python 3.2.1a0 (default, May  4 2011, 19:59:25) 
[GCC 4.6.1 20110428 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class A(object):
...     def __eq__(self, other):
...             return True
... 
>>> set([A()]) == set([A()])
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unhashable type: 'A'

At least, raising an error is saner. It actually helped me to understand what I needed to do to have my sets working correctly with Python 2:

Python 2.7.1+ (default, Apr 20 2011, 10:53:33) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class A(object):
...     def __eq__(self, other):
...             return True
...     def __hash__(self):
...             return 123456789
... 
>>> set([A()]) == set([A()])
True

Since my latest announcement of the Lua workshop, I received a couple of emails asking why I discourage the use of Lua.

Actually, I already wrote out many of the things I dislike about Lua. I won't come back on this technical issues here, but since Lua 5.2 is not yet released (it's still at alpha stage), they are still relevant nowadays.

Stack based API is harder

The ease of integration of Lua into a C program is one of the point of Lua. They claim it's very easy to integrate Lua into your C application, because it does not use pointer, nor reference counting, nor anything that requires a minimum amount of skills to be used.

It uses a virtual stack based approach. You push or pop things on a stack, and refers to them using a relative or absolute index.

In order to people who never wrote Lua code to understand, here's a quick example on how this work. The L pointer is a Lua environment.

/* Create a table on the stack: index 1 */
lua_newtable(L);
/* Push a string on the stack: index 2 */
lua_pushstring(L, "hello");
/* Push a number on the stack: index 3 */
lua_pushnumber(L, 123);
/* Set newtable["hello"] = 123 */
lua_settable(L, -3);

You first push a table (in Lua, a table is almost equivalent to what you'd call a hash table in other language), then push the key, the value, and do the assignment operation. In the settable, we use -3 as index, meaning the "3rd item on the stack counting from top". We could also have written lua_settable(L, 1), since the table is also the first item on the stack from the bottom.

So far, so good.

Problems arise when you do more complicated stuff. My previous example is what you would typically find in a tutorial, but of course, real life is different, and usually more complex. If you cut the things in different parts, it can start to be more complicated.

Let's take a look at the following:

/* Create a table on the stack: index 1 */
lua_newtable(L);
/* Push a string on the stack: index 2 */
lua_pushstring(L, "hello");
/* Push a number on the stack: index 3 */
lua_pushnumbe(L, mycomputingfunction());
/* Set newtable["hello"] = 123 */
lua_settable(L, -3);

Here, we do exactly the same thing, but we do not push 123 directly: we compute it.

And here's the trick: if your computing function is also using the Lua stack, things can become very messy. As long as your computing function use the stack cleanly by pushing and poping all its item, and returning the stack in the same state it was before, you're safe. The problem is that in a complex program, you also write bugs. You do not chose to, but you do. And sometimes, you forget to pop one of the item you fetched from a table.

Imagine that mycomputingfunction is:

int
mycomputingfunctiong(void)
{
  /* Just push the table we want to fetch
     the number from on the stack */
  pushatableonstack(L);
  lua_pushstring(L, "mykey");
  lua_gettable(L, -2);
  return lua_tonumber(L, -1);
}

This function works perfectly. It pushes a table, then a key ("mykey"), then fetches mytable["mykey"] and pops the key (lua_gettable does push value/pop key itself), and then returns the numeric value of the last item (the fetched one) of the stack.

However, this function has a bug: it does not pop the table! This does not prevent the function to work. It does not raise a segmentation fault. It does not show any problem under gdb. It does not show any leak under Valgrind. It does now show any problem under any standard C debugging tool.

But when you'll start using it, your program will start to do weird things, and you'll have to spend a huge amount of time debugging it manually, dumping the stack content at each step of your program to watch out what's wrong.

Another bad thing, that can happen, is some code poping accidentally an item from the stack, or worst, from an empty stack. This does not raise any error on the Lua side, but will break your program in very unfunny way.

Even if I've been very meticulous writing awesome, but we hit that problem regularly.

The easiest workaround is to use lua_settop(L, 0) to reset the stack to 0 element. Doing this regularly (like after each program event or treatment) can remove left-over items and avoid the never ending stack grow you may experience if your left-over items continue to pile up. Did I tell you I dislike work-around?

You could also use lua_call(), which would avoid such an error, but this would require a huge amount of indirection, and would make write more (useless) code.

This kind of problem does not exists with pointer based API. If you screw things up, the problem will cause a segmentation fault or leak memory, or cause things you can (easily) debug with standard tools like gdb or Valgrind.

No reference counting is a pain in the ass

Userdata objects are variable Lua size objects embedding a C struct you define. It's the equivalent of an object in object oriented language.

Lua does not provide any reference counting for the userdata objects. That means you can push this objects on the stack, use them, but they cannot directly reference each others. If you have a "car" userdata and a "wheel" one, the car cannot hold directly a reference to the wheel. This is not possible because userdata are allocated and garbage collected by Lua, and there's no way to increase the reference counting yourself.

So the common hack is to store the wheel into a table as a value, and store the table index as an integer into the car data structure.

This obviously makes memory leaks tracking harder, add huge level of reference indirection in usage (still more code), and does not make the whole process less error prone (at least in my opinion).

No paradigm makes you lose time

Lua is proud to come with no paradigm and to provide metatables. I already showed 3 years ago that it has big flaws.

To me, this ain't no good. Lua is not functional, nor it is object oriented.

Most people, including me, want one of this paradigm, or any else. Plain old imperative is not enough.

So you'll start to build more, or to use something like LOOP, which implements an object model. You'll implement your paradigm. I say life is too short to (re)write a paradigm.

In awesome we wanted to have an object oriented approach (this is kind of typical in such a graphical application context), so we tried to build one. To me, this started to be a show stopper when I realized that I've ended writing Python object model into Lua while developing awesome (which aims to be a window manager, not a language). This is one of the reason I stopped hacking on Lua things.

I liked Python object model and wanted to have it in Lua, and spending time rewriting Python is just not worth it. I probably should have chose Python, not Lua. YMMV.

Embedding may not be a good choice

This is not Lua related, but I want to mention it. Googling for "embedding vs extending" will probably tell you more about why you should double check that you really need to embed Lua rather than to extend it.

Being small is not an excuse

One common argument to choose Lua is that it has a small footprint. Yeah, that's true, but that's useless. Bummer! When I program, I don't have any resource usage pressure. People who have such pressure are either paranoids or playing in the world of embedded computers. This is also a no more existing conception since quad core processors equiped phones are coming into the market. I'm rather confident that what we used to call embedded devices are just dead and are now plain computers. But as usual, YMMV.

So start to forget about it, run in your underpants and yelling "yay we killed that shit!", and then use real computers stuff. :-)

Even if benchmarks show how Lua is damn fast, remember what a benchmark proves: that you can do useless things very fast.

Too few extension modules

This is not directly Lua's fault, but there's too few extension modules for Lua. The community is quite small compared to other big languages' ones.

So think twice

before you choose Lua (or any other language). My recommendations these days would be not to embed, but to extend. If you really have no choice and need to embed a language into your application, GNU Guile is probably worth considering, because it's a Scheme and therefore a functional language :-), and because it can provides also different languages.

Including Lua.

Mise en ligne du journal de bord de ma randonnée à vélo Tours - Île d'Oléron.

Vincent is taking his release team hat off and dropped it on my head. I am a bit sad because the real blue hat has been lost, but I am very happy to be here at this time, GNOME 3 is out, people loves it.

For 3.2 we will continue to have our work driven by design, and we are making adjustments to our schedule and processes to keep on going with a global vision, there have been a few emails about feature planning on desktop-devel-list, we will expand on that soon, but for now, I wanted this post to give all my thanks to Vincent Untz (plenty of time for icecreams now), Lucas Rocha (don't forget to add ajax support to the board), and Frédéric Crozat (we will continue harassing you for live usb images), who are leaving the team, and to welcome our new members,

• Luca Ferreti, he was a team member already but Vincent gave him a trainee badge as no one was leaving at that time; he has already been helping with releases;
• Javier Jardón, he arrived on #gnome-love someday, got hooked fixing build failures and went on to lead wide goals to improve our modules, and more;
• Alejandro Piñeiro Iglesias, I met him in the build brigade, but really he is now an accessibility guy, and his expertise in the domain will be immensely valuable;
• Colin Walters, shell developer, involved with gobject introspection from the beginning, his latest feat has been to push for a standalone spidermonkey release from our friends in Mozilla.

Let's now go to 3.2, and beyond!

Mise en ligne du journal de bord de ma randonnée à vélo Tours - Bouchemaine.

With no consideration for Bangalore timezone or my sleep schedule, GNOME 3.0 has now been released! Live images are already updated (go try them) and packages are flowing into distributions.

I originally had plans for some tourism in Bangalore after GNOME.Asia (fanstastic event) but didn't do much in the end as everyone was working hard on the release, and I certainly didn't want to let it happen without my part of the effort.

And that effort has been concentrated on the documentation websites, library.gnome.org, updated to the new website look, and developer.gnome.org, a revived site dedicated to developers, working on and with GNOME technologies. It couldn't have happened without the Berlin Development and Documentation Tools hackfest, and the collective effort of numerous hackers, inspired by the immense work Shaun McCance has been doing for years.

Thanks everyone for making it happen, and let's all step into the GNOME 3 era.

As you may know, Apache's mod_authzn_ldap allows to authenticate users in Apache HTTP server using an LDAP server. Unfortunately, it has a little implementation flaw.

The filter used to authenticate the user is built by abusing the RFC 2255 which specifies the LDAP URL format. This format has an "attribute" field which is normally used to specify which attributes should be returned. But mod_authzn_ldap uses this attribute to compare with the username given by the client. That means that you have to have an attribute in your LDAP entries which matches the username, and you have to use it in the "attribute" part of the URL to get things working.

Therefore, I wrote a patch to add a format string in the LDAP URL in order to user the provided username in the filter, and ignore the attribute part of the URL, which has no use in such a context anyway.

The bug has been opened in ASF Bugzilla and has number #51005, with the patch. The patch is backward compatible with the current configuration format, which is not the best choice in theory, but probably the more pragmatic.

I've no clue on the typical delay for patches in clusion in Apache HTTP server, so le t's just wait'n see.

The Heise Online just published an article, Mozilla kills embedding support for Gecko layout engine, without making any fuss it starts with "Mozilla has officially ended support for embedding the Gecko layout engine in applications other than Mozilla core applications", then it links to Benjamin Smedberg post on mozilla.dev.embedding but it doesn't offer much details, or reasons (other than "our product is firefox, we have to focus ressources there.").

This article ends with an open question, about applications that are currently using Gecko, but it erroneously cites Devhelp. Devhelp has been ported to Webkit a long time ago (I had a post title "Devhelp with Webkit back in 2007).

No worries for Devhelp then; but while it talks about Gecko only this decision may be of concern to us, if it was extended to Mozilla Javascript engine (SpiderMonkey); a few months ago Colin Walters was actually quite positive ("Actually we're discussing this upstream again very productively; there's renewed interest in supporting embedders, and I'm in the process of getting some patches in to help here.") but who knows... Mozilla certainly keeps on ignoring some of our needs, I can't count the number of times jhbuild had to be updated because a xulrunner tarball was removed from their mirrors. (last time? two days ago, bug 645971).

Fourth day of the Bangalore release hackfest and things are going smoothly; after the Intel offices in the beginning of the week (thanks Intel!) we are now at the GNOME.Asia summit venue, the Dayananda Sagar Institutions, still hard working at "release team" stuff, and more.

Today, between testing and approving patches (go read the On the road to GNOME 3.0 post of Olav for details) and work on the future library.gnome.org and developer.gnome.org, I managed to reach the mythical "100% building" status on my build slave on build.gnome.org, 199 out of 199 modules built correctly, at the same time.

Good sign for the forthcoming release!

The server hosting this forge has just been upgraded to Debian Squeeze. Tell us if you found any new problem.

First day of the hackfest and Allan Day wrote we would do release team things, but what would those "release team things" be? Many things!

Just go and see all the things Andre did today, or the gnome-panel branch of Vincent (299 files changed, 17334 insertions(+), 27275 deletions(-)).

"Now let me break stuff", "3.0-freeze-break", noticed the pattern? But don't take it seriously, we are making 3.0 rock, together with the release team members that couldn't join us and are also working hard.

What about me, some details about what I did? I went for boring things, like pushing a preliminary set of modules for our second release candidate, which was initially planned for today, then building and smoketesting it, including the gnome-panel changes mentioned above. For the record that release will finally happen tomorrow, and I blame the timezone for this.

That's it, you should now imagine a "Sponsored by the GNOME Foundation" badge here (thanks!), and a "meet me at GNOME.Asia summit 2011" image on the other side; and go over to read what the other participants did, they've done so much already.

Thanks to my recent promotion allowing me to commit directly in Org-mode, I've moved Org-contacts into the contrib directory of the Orgmode distribution.

This is a quiet Sunday, perfect to spend time reading blogs, and doing so made me want to look in Bugzilla if requests for supporting appindicators had been mistreated, but the first bug report I read was about Empathy, and I read the following comment:

« My understanding of the GNOME release process is that the release team prefers to see a library used as a configure time option of several projects before accepting it as a dependency. This removes the "I built a library that sounds good but no one really uses" problem of adding libraries to the platform. For instance, libnotify has been used by many programs before it was a blessed dependency for 2.26. »

—Ted Gould, in bug 574744, "Empathy could take advantage of the Messaging Indicator", 2009-04-06.

So I was derailed and went looking for the case of libnotify, which I had forgotten, and indeed, libnotify was proposed for 2.20, and refused,

+ libnotify
  - mixed feelings within the release team.
  - this is being used more and more, so it's pretty clear
    there's a demand for this.
  - some of us still strongly feel that it should live in one
    of the libraries we already have in our stack (probably
    GTK+). We'd like to see people working on integrating
    this in our stack, or explaining why it's not possible to
    do so.
  - what might be worth is accepting the dbus API: this is
    something that will happen more and more in the future
    (think Xesam, for example). The API probably needs to be
    standardized, though.

— Vincent Untz, in New module decisions for 2.20, 2007-08-10

And it was finally in 2.26 that is was approved, and it was still noted at that time that it would be better to have it in GTK+.

+ libnotify (external dependency)
 - widely used
 - would be nice to have a more active development
 - feature that should live in GTK+ in the future (when dbus
   can be used there)
 => approved
   The release team wants to stress out that it should really
   not be abused (as it tends to be).

— Vincent Untz, in New module decisions for 2.26, 2009-01-21

For reference I found this list of requests for appindicator support: Empathy, Ekiga, Epiphany, Vino, and GNOME Control Center, in case someone wants to look for mistreatments.

I'll be at the OrgCamp April 2011 in Paris next month. However, I'm not yet sure of what I'll talk about this time.

I'll be at the OrgCamp April 2011 in Paris next month. However, I'm not yet sure of what I'll talk about this time.

Flattr me!

I spend too much time writing Emacs Lisp code these days. Unfortunately, the more I do the more I find new useful tools to improve my work-flow and save time for doing more Lisp. D'oh.

I did not work on any big thing these last weeks, so I'm thinking it's a good time to talk about the various code and patches I sent to multiple Emacs packages.

el-get

el-get, a fabulous tool that installs and handles all the external Emacs packages I use. A friendly war started on the development list about autoloads handling. The discussion was overall pointless, since we had a very hard time to communicate our ideas, and we did not understand each others several times.

In the end, el-get now supports autoload correctly and do not load automatically all your packages, improving the startup time, and using the Emacs way to do things. Which is always better, obviously.

git-commit-mode

I've started to use git-commit-mode some times ago. I usually use git-commit with the -v option to see what I'm committing. I though it would be useful to color the diff with diff-mode, so I wrote a patch just to do that, which was merged today by Florian.

magit

Some weeks ago, I decided to give a try to magit, and loved it. I am not always using it, but for basic operations it is very useful. But I really soon found some things I did not like and therefore send patches to enhance it.

First, I've added a patch to honor status.showUntrackedFiles which I use in my home directory. In the mean time, I've also added a patch to allow adding an arbitrary file.

Yesterday, I sent another pull request, not closed for now, which adds the possibility to visit files in another window from a diff file, and the support for add-change-log-entry directly from the displayed diff. Useful for these old projects still using ChangeLog files but accessible through git (hi Emacs & Gnus!).

Gnus

Nothing remarkable, but I write a couple of fixes and enhancements to the Sieve manage mode, to the Gravatar code and cleaned-up some very very old code. Also added the possibility to set list-identifier as a group parameter.

Org-mode

I spent most of my time working on my jd/agenda-format branch, which is soon to be merged. I've also just got developer access to the Org-mode patch work and repository, so I'll be able to break things even more! ;-)

ERC

I fixed the bug that annoyed me for a long time. Now erc-track does not reset the last channel status on window visibility changes not made by the user.

Flattr me!

git-annex is a recent tool allowing to manage files with git without having theirs contents checked into git. I've been looking at it with some interest in the past few weeks in order to use it to handle my music collection on the various computers I use.

My laptop just got a new SSD drive which is quite small and cannot handle the whole collection. By using git-annex, I'm now able to only check out part of my music collection and keeping it synchronized with all my others computers.

Using git-annex is pretty trivial once you understand the concepts.

I've set up a a git repository in my Music directory, like that:

% git init
Initialized empty Git repository in /home/jd/Music

Then I initialized git annex:

% git annex init "Music on keller"

So now I'm able to import all my date into git annex. git-annex offers several backends to store data. WORM is the default, but is not a good choice for file that could be modified, and music files can be, at least when editing the metadata. Therefore, I chose the slow but more reliable SHA1 backend.

% git annex add --backend=SHA1 .

That can be long, so you should be patient. Then you should commit the fact you annexed files:

% git commit -a -m "Imported my music collection"
[master 82d060f] Imported my music collection

Now, it's done. You can go on another computer and clone the repo:

% git clone ssh://keller/~/Music
% cd Music
% git annex get "Incubus - Light Grenades"

And then let the magic happens. It will retrieve the whole directory for you. Do not forgot to commit the fact that you got those album on that machine!

The concept used by git-annex is quite simple. All files are replaced by symlinks pointing to .git-annex/SHA1:filechecksum.log. This file contains lines, whose format is:

retrieval-timestamp is-file-present uuid-of-the-repository

Indicating if the file is present in a repository. This logs file are commited into the repository, whereas the real content is stored in the .git/annex directory and therefore not commited.

I recommend to take a look to the walkthrough page to get more idea and what you can and cannot do with git-annex.

Flattr me!

Publication papier du recueil de poèmes Teneurs du vide aux éditions ILV-Edition.

Fiche - Commander

gurlchecker est un vérificateur graphique de sites web pour GNU/Linux et autres systèmes POSIX. Il fonctionne sur un site entier, une page locale ou un fichier de signets.

- Added option to export IP address. - Added option to export only external links. - Fixed a bug in header request. - Fixed memory leaks. - Fixed a problem with libclamav6 >= 0.96 (do not call cl_init() twice). - Fixed want_*_path detection for LDFLAGS and yes/no logic in configuration. - Fixed a build problem with po.

Debian Squeeze was released a few days ago, this makes it a good opportunity to post about that distribution. Nowadays, even if I have more duties in GNOME than in Debian, I still believe Debian may be one step closer to me in terms of core values.

Free Software. Debian litterally stands for Free Software, the Debian Free Software Guidelines were the actual basis for the Open Source Definition, years ago. This is still going strong and an enormous effort hasa come to fruition with the release of Squeeze with a kernel clean of closed firmwares.

Or the discussion that gave birth to the Freedom Box project, that happened this year in Debconf10, (Eben Moglen talked about this during the FOSDEM, there is an article on lwn.net: Moglen on Freedom Box and making a free net).

Universality. While it was written that Linux is not about choice Debian still sports the inspiring "Universal Operating System" motto.

And it's true in so many areas, the vast number of supported architectures (and this has practical uses, ARM or MIPS devices are more common these days), or the attention to translations, from the website to the package descriptions (the joy of demonstrating a "babel box", demonstrating how the debian installer supported lots of languages, years ago).

I'd even say that this goes way further, and a projet like Debian Woman came to Debian before GNOME or Mozilla in part because of that "universality" tagline.

I will be updating various servers and other computers to Squeeze in the coming weeks, I don't worry about those updates, and I know they will keep working without causing me any stress for two more years, at least.

As of my personal computer, it will continue to run the Sid branch, just like it has been doing for more than ten years. And it will run GNOME 3, as Emilio Pozuelo Monfort, and others, are already doing a fantastic job packaging it.

Thank you Debian community.

When I started to use Emacs, I got hooked by many stuff like Gnus and Org-mode. One thing I quickly started to hate is how the Lisp code can be old and unmaintained. That especially applies to BBDB, which has been unmaintained for years, and has very very very old and obsolete code.

Therefore I've decided to develop my own BBDB replacement based on my lovely Org-mode. It's called org-contacts, and it allows you to handle your contact like anything you would handle in Org. This way you can manage them the way you want, without any preset fields or any assumptions like BBDB has.

I had the chance to present it at the Paris OrgCamp a couple of weeks ago, and due to the enthusiastic audience I had, I'm now releasing it to the wide Internet.

I often post Emacs screenshots on this blog, and consequently receive a bunch of request for my Emacs theme. Therefore I decided to publish it.

Emacs snapshot packages for Ubuntu are now available, thanks to Dimitri Fontaine.

Yesterday afternoon, I was at the first OrgCamp in Paris. It was my first attendance to a BarCamp, and I really liked it. It's basically the first geek event I do not find boring nor useless.

There was about 18-20 persons participating, which was quite high, since we all initially though we would have been only 5.

We had several presentations of various features and personal usages of Org-mode. For my part, I've quickly presented the agenda, and my BBDB replacement named org-contacts (I'll probably talk about it on this blog in another post later).

The only downside was that Bastien (the new Org-mode maintainer) was not able to come and join us. On the other side, there were so much to tell for a first time, I did not have so much time to code. I only have been able to fix one bug reported during my agenda presentation.

In the end, the overall atmosphere was very enthusiastic and friendly, which was extremely pleasant. The #org-mode-fr IRC channel has been created on Freenode, following this event. Feel free to join us.

Since people liked it so badly, it seems there should be another barcamp in the next months. Stay tuned.

I've added code fontification using Org src blocks inside Gnus.

This interprets the block as an Org buffer and fontify it accordingly if org-src-fontify-natively it set to t.

Thanks to Sébastien Vauban for the original idea and implementation. Now it works out of the box without any customization.

I'll be at the OrgCamp 2011 in Paris next Saturday.

I've decided to take over the maintenance of the unofficial emacs-snapshot Debian packages that were maintained by Romain Francoise.

They are available on a dedicated page.

Johannes did a nice wrap-up of the hackfest, I'll just point to a few things.

1. There is a library.gnome.org featuring the new gnome.org design almost ready to go, but it won't get out before the real gnome.org goes live, I am confident it won't take five more years. That new site also has a new "GNOME Developer Center" page, featuring quick access to important ressources and links to the demos and tutorials we have been working on during the hackfest. This is the page that is currently screenshoted on developer.gnome.org.
2. The demos and tutorials are also up on the existing library website, but not integrated in any way, http://library.gnome.org/devel/demos/nightly/ It is all automated already, it will get updated every night.
3. There's a test server, running as lgo.0d.be, I just added some HTTP authentication, to make sure it doesn't get indexed in any way. There's no automation there, and it doesn't feature the live search in API symbols.

Thanks again to everyone involved, and a special mention to Chris for having me.

We had some glühwein this evening in freezing Berlin, so of course the hackfest is going well… I came with my very own tasklist, mostly centered on improvements to library.gnome.org, this means I didn't participate in the writing of the 10-minute tutorials (they are progressing nicely) and mostly went hacking on my own instead.

My first objective was to improve Mallard support, as I wrote initial support last year and it needed a serious refresh to work with current documents. Basically it meant switching to yelp-xsl and updating the generation of Mallard cache file. It's already pushed to the repository, and you can have a look at the Empathy documentation for an example.

Next thing was to get live API search, something I actually did three years ago, but missing SCGI packages on RHEL meant that it was never deployed. Now that we have been getting used to deploy Django applications (snowy, shell-perf, damned-lies) it made sense to rewrite this using the Django framework. It's now waiting for a GNOME sysadmin to deploy, I hope you'll all be able to enjoy this soon.

Then I went on updating the library.gnome.org theme to match what is being done for the future gnome.org website (amazing work by Vinicius and Andreas on this), this is still a work in progress, and it won't be deployed before the gnome.org website goes live, but it was also the opportunity to move pages around, create a more clear API references page, and so on, different things that will make it easier to be used for a future "GNOME Developer Center".

To sum it up it's really nice to have dedicated time to work on GNOME, something which is quite difficult in normal days, between work and other activities, so thanks a lot to the GNOME foundation for sponsoring my travel!

I finally took some time to finish my color contrast corrector.

It's now able to compare two colors and to tell if they are readable when used as foreground and background color for text rendering. If they are too close, the code corrects both colors so to they'll become distant enough to be readable.

To do that, it uses color coordinates in the CIE L*a*b* colorspace. This allows to determine the luminance difference between 2 colors very easily by comparing the L component of the coordinates. The default threshold used to determine readability based on luminance difference is 40 (on 100), which seems to give pretty good results so far.

Then it uses the CIE Delta E 2000 formula to obtain the distance between colors. A distance of 6 is considered to be enough for the colors to be distinctive in our case, but that can be adjusted anyway. That depends on reader's eyes.

If both the color and luminance distances are big enough, the color pair is considered readable when used upon each other.

If these criteria are not satisfied, the code simply tries to correct the color by adjusting the L (luminance) component of the colors so their difference is 40. Optionally, the background color can be fixed so only the foreground color would be adjusted; this is especially handy when the color background is not provided by any external style, but it the screen one (like the Emacs frame background in my case).

Here is an example result generated over 10 pairs of random colors. Left colors are randomly generated, and right colors are the corrected one.

bg: DarkSeaGreen4 fg: gray67 ->                             bg: #4a6b4b fg: #cccccc
bg: SlateGray4 fg: forest green ->                          bg: #9faec0 fg: #005700
bg: grey13 fg: grey36 ->                                    bg: #131313 fg: #6c6c6c
bg: MediumPurple2 fg: honeydew ->                           bg: #9e78ed fg: #f0fff0
bg: grey43 fg: chartreuse3 ->                               bg: #5e5e5e fg: #79de25
bg: linen fg: DeepPink2 ->                                  bg: linen fg: DeepPink2
bg: CadetBlue4 fg: blue1 ->                                 bg: #6c9fa4 fg: #0000e1
bg: gray33 fg: NavajoWhite3 ->                              bg: #525252 fg: #cfb58c
bg: chartreuse1 fg: RosyBrown3 ->                           bg: #9cff38 fg: #b28282
bg: medium violet red fg: DeepPink1 ->                      bg: #9c0060 fg: #ff55b9

All this has been written in Emacs Lisp. The code is now available in Gnus (and therefore in Emacs 24) in the packages color-lab and shr-color.

A future work would be to add support for colour blindness.

As a side note, several people pointed me at the WCAG formulas to determine luminance and contrast ratio. These are probably good criteria to choose your color when designing a user interface. However, they are not enough to determine if displayed color will be readable. This means you can use them if you are a designer, but IMHO they are pretty weak for detecting and correcting colors you did not choose.