What's the title of this blog ?

from hat import rabbit

Wed, 01 Sep 2021 00:00:00 +0000

Get your finest top hat and dive into the rabbit hole with me for a magic trick like only the best snakes can do: create Python modules out of thin air. We’ll import Python modules that don’t exist and yet, Presto, here they are.

Presented at PyConlineAU 2021, this talk explores both how the Python import machinery actually works, and how we can use it to make dynamic Python modules.

Sources:

How the Python import system works, by Victor Skvortsov
Python documentation
Procrastinate, the project we speak about
The migration magic which inspired this talk.

Art by Goulwen Reboux, a.k.a @nautilebleu

Maintaning a Django project after 10.000 commits

Sat, 13 Apr 2019 00:00:00 +0000

Twidi and I dive into the challenges that arise when a Django project starts to become huge. This talk was presented at DjangoCon Europe 2019 in Copenhagen.

Django is extremely effective for creating a website quickly with top notch features out of the box. But in some codebases, after a while, new developments can become harder and harder.

In this talk, we’ll examine some design decisions that have, and haven’t, scaled successfully, in the hope that the next time you start your big scale Django project, you won’t end up cursing your past self after three years.

Among the subjects we’ll tackle:

Third parties, the problems they bring and a few solutions;
Where to put your business logic to keep your sanity;
How to archiecture your code for fun and profit;
Testing strategies for large projects

Here are the slides:

And here is the video:

The right tool for the job

Sat, 13 Oct 2018 00:00:00 +0000

One year after our PyconFR packaging talk, we sat down again with Twidi, and wonder what other aspect of tooling would be worth a talk. We couldn’t decide, so we did them all :D

We gave this talk at PyConFr 2018 in Lille, and this time, we needed a gadget loving guide. So follow us on the steps of the world’s most famous (and thus worst) secret agent.

Thank you, and happy tooling. Feel free to ping me on Twitter if you have any feedback!

Video in French is comming soon.

A migration journey with Django

Fri, 15 Jun 2018 00:00:00 +0000

A talk (in French) at DjangoCong (a.k.a DjangoCon France) 2018 in Lille, about a whole new direction to go with Django migrations. Meet Django-North, and your migrations won’t go south!

DjangoCong was a success! I was thrilled to have the opportunity to organize this year’s conference in my (current) home town, Lille. We put some serious effort into making this welcoming and inclusive, and even if we still have a long way to go, I was really happy to see a lot of new faces, a lot of old faces, and most of all, a lot of face who want to come back next year for more!

Also, because I’m kinda crazy, I submitted a proposal for a talk and I was accepted (… ok, partly by myself, but for my defense we accepted every proposal). This talk was based on an unpublished (as of today) 3-part blog article I’ve prepared for my employer tech blog. At the time of the proposal, I had written the first part and was about to start preparing the 2 other parts with my colleagues. Currently, we’re still there, but since the talk is now written and public, I guess the blog articles should be easier to write :)

Now, to the talk itself. I’m talking about a way to rethink ownership on your database migrations by having Database Admins (DBAs, database experts whose job, craft and passion are aligned towards making postgres and other databases a better place) write them with you. This is, literally, a story that let us sleep at night!

The slides in French:

The video in French:

As usual, I’ll publish the slides in English if I get to do this talk in an international context, or if you ask me kindly on Twitter.

Also, a better video might exist at some point, and I’ll update this.

Finally, I’ll add the blog article from my employer’s tech blog once it’s published.

See ya!

Packaging my Python code - again, better

Fri, 29 Sep 2017 00:00:00 +0000

I felt that the packaging talk I told you about could be expanded and improved, so with the help of the original writter, Twidi, we did so.

We gave this talk at PyConFr 2017 in Toulouse, and because we felt adventurous, we decided that Indiana Jones shall be the guide to our Python packaging discoveries!

The slides are currently being translated in English.

These are the French slides:

Thank you, and happy packaging. Ping me on Twitter for typos and imprecisions.

Video:

Packaging my Python code

Mon, 03 Jul 2017 00:00:00 +0000

I gave another talk at my local Python meetup about Python packaging, and how to do it properly in 2017.

The new hype is to put everything in your setup.cfg. That’s clean, that’s beautiful, that’s moderm and I :heart: it. For an example of how that looks for a “real” project, look at Raincoat !

Here’s the French slides. Thanks again to Twidi for letting me use his work ! You can find the original slides here and the source there.

English version will follow as soon as I give this presentation in an international context (or if you ask for it)!

The sources for my talk are there.

I'd like to contribute to Open Source but...

Tue, 09 May 2017 00:00:00 +0000

I gave a talk at my local Python meetup about the diversity of Open Source contributions.

Here’s the French slides.

English version will follow as soon as I give this presentation in an international context (or if you ask for it)!

A tale of Python and Adrenaline

Mon, 28 Nov 2016 00:00:00 +0000

The full story behind the talk I gave tonight at the Paris Python Meetup on Python 3.6

So first, if you want the slides, they’re in French, I’ll translate on demand (please, really, fell free !) or if I make this talk in an international context.

You can have them here : Ladies and Gentlemen, the Amazing Python 3.6 !

The original Jupyter notebook with the original typos is available here : Jupyter Notebook

Disclamer

This is an only sightly romanticized version of the facts as they happened.

So what happened ?

First the context : I was supposed to give this talk tomorrow. I wrote it yesterday and planned to spend a few more hours on it tonight. At 5PM, the Meetup fantastic organizers annonced that one of the planned speakers was unavailable. Given I was to give this talk (in another meetup) on the following day, I volonteered.

Another bit of context : I recently got my hands on (or more precisely got one hand encircled by) a connected watch.

Another bit of context : I’m a Jupyter fanboy. But that part you already know if you’ve been following this blog.

Lastly: When adrenaline flows in my brain, I become convinced I can do anything.

So… What happened !?

Well I realized it would have been a nice opportunity to try using my watch as a slides remote control. But that would have needed planning in advance, developing, reserching etc.

“Heck, I’m using Python. My brain is flowing adrenaline I can do anything. So let’s go.”, did I think. At 6.45 PM, 15 minutes before the meetup would start, my brain did.

I was using Jupyter for my presentation, which meant my slides would be exported directly as HTML slides using Reveal.js. The export command was :

$ jupyter nbconvert Python\ 3.6.ipynb --to slides --post serve

(I originally added --reveal-prefix reveal.js with a locally downloaded reveal.js to make sure it would work even without Internet access but that proved useless, as the internet from my phone was enough)

Then a quick skim at the Reveal.js README told me that a javascript call to Reveal.next() would trigger the next slide.

Then I rememberd about IFTTT and their Do button was compatible with my watch. So I knew I could have my watch trigger an API call on a public server.

At that point, the first talk started, about “Creating your Flask API in 5 minutes”. I remebered that I had once deployed a small flask website using Python Anywhere which definitely lived up to its name.

I cloned my earlier project and quicky repurposed it for its new goal : have 2 endpoints : one that would set a flag (“Go to next slide”) and one that would read and reset it. It took 3 commits to works, including one aptly named “haaaaaands” but then it worked. It’s obviously bad code, but time was of the essence here. Notice : being coded in 5 minutes was not an excuse to use HTTP, so it’s using HTTPS, because I have principles (and it was already coded in the first place)

Here’s the code, BTW:

from flask import Flask, Response
from flask_sslify import SSLify


app = Flask(__name__)

SSLify(app)

file = "bla.txt"


def empty():
    with open(file, "w"):
        pass

empty()


@app.route("/get")
def get():
    with open(file, "r") as f:
        c = f.read()
    empty()
    resp = Response(c)
    resp.headers['Access-Control-Allow-Origin'] = '*'
    return resp


@app.route("/set")
def set():
    with open(file, "w") as f:
        f.write("true")

    resp = Response("true")
    resp.headers['Access-Control-Allow-Origin'] = '*'
    return resp

I then created an IFTTT “do button > Maker” reciepe that would call my “set” api when I’d tap on my watch. I would have shared the reciepe with you but meh. But then it’s not really hard to do.

Lastly wrote a small bit of javascript that would poll every 800ms and call Reveal.next() whenever the polled result was “true”. Yes I know that in 2016, websockets are a thing, and I’m a man of the past but hey.

<script>
a = function(){$.get("https://ewjoachim.pythonanywhere.com/get", function(data){if (data == "true"){Reveal.next()}})};
setInterval(a, 800);
</script>

Of course it failed because at first I didn’t think about those pesky Access-Control-Allow-Origin but once that was taken care of, it worked. With a 10 second delay, but i worked nethertheless.

And then it was my turn to go on stage.

And it worked on stage.

And I delivered my talk, feeling filled with so much energy I could have zapped the light bulbs with my confidence. And maybe I actually did but I’m not even sure I’d have noticed. I was elsewhere.

Conclusion

Tinker away. Do things. Make stuff. Useless stuff. The useless stuff of today will be a brick in the new stuff you do tomorrow and all in all, you’ll end up happier. Be happy. Contribute to Open source software. Be happier. Don’t listen to the advice of random folks on the Internet telling you how to live your life.

And thank you to all developers worldwide who made this possible.

Yours, truly,

Joachim

Django Readonly Field - Chapter 3 - Testing and Continuous Integration

Mon, 31 Oct 2016 00:00:00 +0000

Hello folks !

This is the third article on a series I’m trying to make about Django Readonly Field. This one is not about Django Readonly field. Or more precisely it’s not specific to Django Readonly Field. I’ll talk about all the (quite classical) tools I use to be confident that a package I’m making works. I might also talk about project organization, docs and such.

Tests

Coding habits include testing your code. Good coding habits include writing automated testing. Great coding habits include writing your tests even before you write your code. On that matter, I have to recommand Harry Percival’s excellent Test Driven Development with Python (Obey the testing goat !)

:warning: I’m not an expert in writing a good quality maintainable test suite (not any more than anything in this blog, even if I probably make a good job pretending so). Don’t take my word for it on anything. If you know I’m wrong, please correct me. If you’re unsure, find reliable sources :smile:

A bit of theory.

There are several species of tests

Coverage

Tox

Makefile

Travis

DJ-Database-URL

Django Readonly Field - Chapter 2 - The code

Sat, 22 Oct 2016 00:10:00 +0000

This article is part of a series on Django Readonly Field. Here, we see how it works.

Exploring the project

First, you can see that the GitHub project contains quite a few files, but the really interesting part for today is the django_readonly_field subdirectory. So that’s only 3 files and 59 lines of code according to my coverage data.

Let’s look at those files.

`init.py`, the entry point

In: [ ]

__version__ = '1.0.1'

default_app_config = "django_readonly_field.apps.Readonly"

You can see that there’s not much done here. The version is useful to have here for the lib to easily introspect its own version if need be. The default_app_config is interesting. According to the Django documentation, this variable will be used if the module it’s in is placed in the list of INSTALLED_APPS. So now we kind of have a idea how the app is to be used.

The value seems to be a class defined in apps.py. Let’s go and check that.

`apps.py`, switching the compiler

In: [ ]

from __future__ import unicode_literals

from django.apps import AppConfig


class Readonly(AppConfig):
    name = 'django_readonly_field'

    def ready(self):
        from django.db import connection
        from django.db import utils

        readonly_compiler_module = "django_readonly_field.compiler"

        # Change the current value (this is mostly important for the tests)
        connection.ops.compiler_module = readonly_compiler_module

        original_load_backend = utils.load_backend

        def custom_load_backend(*args, **kwargs):
            backend = original_load_backend(*args, **kwargs)

            class ReadOnlyBackend(object):
                @staticmethod
                def DatabaseWrapper(*args2, **kwargs2):
                    connection = backend.DatabaseWrapper(*args2, **kwargs2)
                    connection.ops.compiler_module = readonly_compiler_module
                    return connection

            return ReadOnlyBackend

        # Make sure all future values will be changed too
        # (this is mostly important for the real life)
        utils.load_backend = custom_load_backend

Let’s dive in !

In: [ ]

class Readonly(AppConfig):
    name = 'django_readonly_field'

    def ready(self):

From the Django documentation above, we learn that the AppConfig.ready method is called as part of django.start() which is called at the very beginning of a django process.

In: [ ]

from django.db import connection

readonly_compiler_module = "django_readonly_field.compiler"

# Change the current value (this is mostly important for the tests)
connection.ops.compiler_module = readonly_compiler_module

So the main step here is that we replace the string in django.db.connection.ops.compile_module with 'django_readonly_field'.

This string is used here.

In: [ ]

# django/django/db/backends/base/operations.py in BaseDatabaseOperations
def compiler(self, compiler_name):
    """
    Returns the SQLCompiler class corresponding to the given name,
    in the namespace corresponding to the `compiler_module` attribute
    on this backend.
    """
    if self._cache is None:
        self._cache = import_module(self.compiler_module)
    return getattr(self._cache, compiler_name)

It is the main link between the DatabaseWrapper object which represent a connection between Django and its Database, and the SQLCompiler class which is how django transforms methods calls into SQL.

The idea is that we’ll use our very own compiler that will remove the fields marked as readonly from write queries.

The thread problem

One thing that was hard to foresee is that the object we’re modifying here (actually, the whole DatabaseWrapper) is completely recreated in every thread (it’s local to the thread), and Django will use a new thread for every connection. This means that the object we’re modifying here will be thrown away shortly, except in the tests that are mono-threaded.

Consequently, we need to make the same modification in every new version of the DatabaseWrapper. We’ll do this by modifying the load_backend function that is defined here and used here. The main advantage of modifying this function in particular is that it’s not thread local (there will be only one instance of this function, whatever the number of threads used) but all the threads will use this function.

The next question is how do we make load_backend return a patched DatabaseWrapper?

In: [ ]

 original_load_backend = utils.load_backend

def custom_load_backend(*args, **kwargs):
    backend = original_load_backend(*args, **kwargs)

    class ReadOnlyBackend(object):
        @staticmethod
        def DatabaseWrapper(*args2, **kwargs2):
            connection = backend.DatabaseWrapper(*args2, **kwargs2)
            connection.ops.compiler_module = readonly_compiler_module
            return connection

    return ReadOnlyBackend

# Make sure all future values will be changed too
# (this is mostly important for the real life)
utils.load_backend = custom_load_backend

If we follow the original code:

load_backend is called and returns an object (more precisely a module);
in this object, a DatabaseWrapper is called and returns a DatabaseWrapper instance;
In DatabaseWrapper.__init__(), it defines the op object whose compiler_module we want to change.

So we use Python duck-typing capabilities :

Our own custom_load_backend needs to return an object. Instead of a module like the original function, we’ll return a class;
The returned object needs to contain an object named DatabaseWrapper, which is the case, but instead of a class in the original function, our object is a staticmethod;
When that object is called, it’s supposed to return the DatabaseWrapper. In the original function, this would simply a call to the class constructor. In our cas, it’s a call to the static method. The static method will instanciate a real DatabaseWrapper, then patch it and then return it.

The magic of Python duck-typing is that python has no need to know that it just manipulated a class and a staticmethod instead of a module and a class like it was originally written for, because both had the same interfaces.

So now, all that’s left for us to see is what exactly our SQLCompiler does.

`compiler.py`, the real work

In: [ ]

from django.db.models.sql.compiler import SQLCompiler
from django.db.models.sql.compiler import SQLInsertCompiler as BaseSQLInsertCompiler  # noqa
from django.db.models.sql.compiler import SQLDeleteCompiler
from django.db.models.sql.compiler import SQLUpdateCompiler as BaseSQLUpdateCompiler  # noqa
from django.db.models.sql.compiler import SQLAggregateCompiler

SQLCompiler = SQLCompiler
SQLDeleteCompiler = SQLDeleteCompiler
SQLAggregateCompiler = SQLAggregateCompiler

The SQL compiler module is expected to contain classes named in a very specific way. The classes that we’re not modifying, we’ll just import them and don’t touch them.

For the ones we’ll modify, we’ll actually subclass them and add the a mixin that reads as follow:

In: [ ]

class ReadonlySQLCompilerMixin(object):

    @property
    def readonly_field_names(self):
        try:
            readonly_meta = getattr(self.query.model, "ReadonlyMeta")
        except AttributeError:
            return ()
        else:
            fields = getattr(readonly_meta, "_cached_readonly", None)
            if not fields:
                readonly_meta._cached_readonly = fields = frozenset(
                    getattr(readonly_meta, "readonly", ()))
            return fields

    def as_sql(self):
        readonly_field_names = self.readonly_field_names
        if readonly_field_names:
            self.remove_readonly_fields(readonly_field_names)
        return super(ReadonlySQLCompilerMixin, self).as_sql()

The readonly_field_names will explore the Model associated to the query, and if there’s a class named ReadonlyMeta defined here, it will read its readonly property containing the names of the fields we want to be readonly. We make a frozenset out of it, because that’s the best structure Python provides for our use (unordered iterable with unique values that we never have to update in the course of the program). These field names are returned.

The as_sql method is the entrypoint for all things SQL, so that’s where we’re going to hit. Sadly, the format of the fields is not the same for SQLInsertCompiler and SQLUpdateCompiler, so we’ll let the subclass add all the necessary details.

The subclasses are just doing the expected work : removing the fields from the query :

In: [ ]

class SQLUpdateCompiler(ReadonlySQLCompilerMixin, BaseSQLUpdateCompiler):

    def remove_readonly_fields(self, readonly_field_names):
        """
        Remove the values from the query which correspond to a
        readonly field
        """
        values = self.query.values

        # The tuple is (field, model, value) where model if used for FKs.
        values[:] = (
            (field, _, __) for (field, _, __) in values
            if field.name not in readonly_field_names
        )


class SQLInsertCompiler(ReadonlySQLCompilerMixin, BaseSQLInsertCompiler):

    def _exclude_readonly_fields(self, fields, readonly_field_names):
        for field in fields:
            if field.name not in readonly_field_names:
                yield field

    def remove_readonly_fields(self, readonly_field_names):
        """
        Remove the fields from the query which correspond to a
        readonly field
        """
        fields = self.query.fields

        try:
            fields[:] = self._exclude_readonly_fields(
                fields, readonly_field_names)
        except AttributeError:
            # When deserializing, we might get an attribute error because this
            # list shoud be copied first :

            # "AttributeError: The return type of 'local_concrete_fields'
            # should never be mutated. If you want to manipulate this list for
            # your own use, make a copy first."

            self.query.fields = list(self._exclude_readonly_fields(
                fields, readonly_field_names))

There’s not much to say here. A few tricks are used :

my_list[:] = new_list is quite the same as my_list = new_list except that the former re-uses the same list, while the latter creates a new list in memory.
The 2 forms of generators are used here. The comprehension generator and the one that uses yield. If you don’t know about those, go and learn about it, it’s one of Python’s really nice features.

Conclusion

Well that’s the whole of it ! We patch the compiler to remove the readonly fields from the sql requests. We do this both in the main thread and in the subthreads created afterwards. And just with this, it works.

But this is only a part of Django Readonly Field. In order for it to be usable, we need many other parts. If you’re interested in other articles on, say, the tests, the CI, the scaffolding, the packaging etc, let me know !

What's the title of this blog ?

from hat import rabbit

Maintaning a Django project after 10.000 commits

The right tool for the job

A migration journey with Django

Packaging my Python code - again, better

Packaging my Python code

I'd like to contribute to Open Source but...

A tale of Python and Adrenaline

Disclamer

So what happened ?

So… What happened !?

Conclusion

Django Readonly Field - Chapter 3 - Testing and Continuous Integration

Tests

A bit of theory.

Coverage

Tox

Makefile

Travis

DJ-Database-URL

Django Readonly Field - Chapter 2 - The code

Exploring the project

__init__.py, the entry point

apps.py, switching the compiler

The thread problem

compiler.py, the real work

Conclusion

`init.py`, the entry point

`apps.py`, switching the compiler

`compiler.py`, the real work