2General

Want to know more about us? Visit 2general.com »

An easier way to change Django cache backend between test cases

In Changing Django cache backend between test cases I showed how to use the Mock library to activate a different cache backend for individual tests.

In the comments for that article, Diederik van der Boor pointed out that the same effect can be achieved in a cleaner way by using a custom “proxy” cache backend.

I took the challenge and created a proxy cache backend and a decorator for switching the effective backend on the fly.

First, let’s look at how to use this beast. Activate the proxy backend in the settings.py you use for running tests:

CACHES = {
    'default': {
        'BACKEND': 'cache_switch.CacheSwitch',
        'LOCATION': 'dummy://'
    }
}

Thanks to a trick adapted from the Mock library, the decorator can be used to decorate test case classes, test methods and functions. It also works as a context manager. Here are examples of each of those use cases:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
from cache_switch import cache_switch
from django.test import TestCase

@cache_switch('locmem://')
def test_decorated_function_runs_with_locmem_cache():
    # test code

class MethodDecoratorTestCase(TestCase):
    @cache_switch('locmem://')
    def test_decorated_method_runs_with_locmem_cache(self):
        # test code

@cache_switch('locmem://')
class ClassDecoratorTestcase(TestCase):
    def test_method_of_decorated_class_run_with_locmem_cache(self):
        # test code

class ContextManagerTestCase(TestCase):
    def test_code_inside_context_manager_runs_with_locmem_cache(self):
        with cache_switch('locmem://'):
            # test code

Note that in the class decorator case, setup/teardown methods would be run with the locmem cache as well, but the cache is cleaned between each method call.

The function decorator is only useful with a test runner which discovers and runs test functions. Django’s own test runner only runs methods in test case classes.

You can also activate a other cache backend than the locmem cache and provide the same options that are accepted in the CACHES setting:

@cache_switch('django.core.cache.backends.memcached.MemcachedCache',
              LOCATION='127.0.0.1:11211',
              TIMEOUT=30)
def test_cache_options():
    # ...

It also accepts the old URI notation:

@cache_switch('memcached://127.0.0.1:11211')
def test_uri_notation():
    # ...

Note that the flush_all command will be executed on your memcached server before each test case!

Here’s cache_switch.py:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
from django.core.cache.backends.base import BaseCache
from functools import wraps
import sys


class OldStyleClass:
    pass
ClassType = type(OldStyleClass)


inPy3k = sys.version_info[0] == 3

if inPy3k:
    class_types = type
else:
    class_types = (type, ClassType)


class CacheSwitch(BaseCache):
    def __init__(self, _location, params):
        BaseCache.__init__(self, params)
        self._current_cache = None

    def get_current_cache(self):
        if not self._current_cache:
            from django.core import cache
            self._current_cache = cache.get_cache('dummy://')
        return self._current_cache

    def set_current_cache(self, backend, **kwargs):
        if isinstance(backend, BaseCache):
            self._current_cache = backend
        else:
            from django.core import cache
            self._current_cache = cache.get_cache(backend, **kwargs)
            self._current_cache.clear()

    def get(self, key, default=None, version=None):
        cache = self.get_current_cache()
        return cache.get(key, default=default, version=version)

    def set(self, key, value, timeout=None, version=None):
        cache = self.get_current_cache()
        return cache.set(key, value, timeout=timeout, version=version)

    def delete(self, key):
        cache = self.get_current_cache()
        return cache.delete(key)

    def clear(self):
        cache = self.get_current_cache()
        return cache.clear()


class cache_switch(object):
    def __init__(self, backend, **kwargs):
        self.backend = backend
        self.kwargs = kwargs

    def __call__(self, f):
        if isinstance(f, class_types):
            return self.decorate_class(f)

        @wraps(f)
        def _inner(*args, **kw):
            self._cache_switch()
            try:
                return f(*args, **kw)
            finally:
                self._cache_unswitch()

        return _inner

    def decorate_class(self, klass):
        for attr in dir(klass):
            attr_value = getattr(klass, attr)
            if attr.startswith("test") and hasattr(attr_value, "__call__"):
                decorator = cache_switch(self.backend, **self.kwargs)
                decorated = decorator(attr_value)
                setattr(klass, attr, decorated)
        return klass

    def __enter__(self):
        self._cache_switch()

    def _get_cache_switch_backend(self):
        from django.core import cache
        if not isinstance(cache.cache, CacheSwitch):
            raise TypeError("Can't change effective cache backend "
                            "since CacheSwitch is not the default backend")
        return cache.cache

    def _cache_switch(self):
        switch = self._get_cache_switch_backend()
        self._old_cache = switch.get_current_cache()
        switch.set_current_cache(self.backend, **self.kwargs)

    def _cache_unswitch(self):
        switch = self._get_cache_switch_backend()
        switch.set_current_cache(self._old_cache)

    def __exit__(self, *args):
        self._cache_unswitch()
        return False

    start = __enter__
    stop = __exit__

This pattern should be a bit more intuitive to use than the Mock-based one.

For brevity, the proxy backend presented above only supports the get(), set(), clear() and delete() cache methods. As such, it doesn’t qualify as a full-featured replacement, but it should be enough for most testing needs.

For maximum testing pleasure we already have Eric Holscher’s Django Test Utils, Gareth Rushgrove’s django-test-extensions and a bunch of other packages listed in the Testing tools grid on Django Packages. I wonder if this cache switching tool, after some refinement, could find a good home in one of those packages.

Thanks for Diederik for the proxy cache backend idea and Michael Foord for the “decorator and context manager” pattern!