Set up Django to use Aiven for Valkey™

Django is a fully-featured high-level Python web framework. Read on to see how to get it to cache its data using Aiven for Valkey™.

Subscribe to RSS

In a previous article we looked at using an Aiven for PostgreSQL® database as the backend for Django. But Aiven also provides other managed services, including Aiven for Valkey, a key-value store using the Redis serialization protocol. So in this article we are going to look at adding a Valkey cache to the previous example.

Summary of the previous article

The basics

If you just worked through the previous article, then great, you're all ready to continue with this tutorial, and can skip the rest of this section.

Otherwise, if you just want to work with this article, then you need to get Oscar setup. Here's a summary of the basics:

You will need node and npm (on my Mac I use brew install npm to install them). Then:

 git clone --branch sandbox-requirements-update \
     https://github.com/TibsAtWork/django-oscar.git
 cd django-oscar
 python3 -m venv venv
 source venv/bin/activate
 make sandbox

You can test that by running the following command and going to http://127.0.0.1:8000/ in a web browser to see the Oscar Sandbox store front.

sandbox/manage.py runserver

Talking to Aiven for PostgreSQL^®

This leaves you with Oscar storing its data in a local SQLite database. To change to using an Aiven for PostgreSQL® instance, follow the instructions at Create a PostgreSQL® database and Use PostgreSQL as the backend database, from the previous article.

Of course, you can leave Oscar talking to the local SQLite database, although running a cache in the cloud for a local database is not something one would expect to do in real life.

Continue using the Oscar e-commerce example

Note

If you are carrying on from the previous example (so skipped to here) then let's just check everything is still set up correctly.

Remember to make sure you're in the django-oscar directory - if necessary, cd into it:

cd django-oscar

Also make sure that the Python virtual environment is enabled. If not, do:

source venv/bin/activate

Check everything still works by running the sandbox application again with:

sandbox/manage.py runserver

Make sure that the bookshop appears at http://127.0.0.1:8000/.

Then check if the DATABASE_ environment variables are still set, to tell Oscar to use the remote PostgreSQL database:

printenv | grep DATABASE_

which should show something like:

DATABASE_PORT=10143
DATABASE_NAME=defaultdb
DATABASE_PASSWORD=YOUR_DATABASE_PASSWORD_HERE
DATABASE_USER=avnadmin
DATABASE_HOST=tibs-django-pg-project-tibs.aivencloud.com
DATABASE_ENGINE=django.db.backends.postgresql_psycopg2

(that's not a real password there, and your host name should be different as
well).

Note

If you're using Aiven for PostgreSQL, then all of those environment variables should match the values in the service's Overview page in the Aiven Console.

The Oscar Sandbox
documentation
warns that: "The sandbox has Django Debug Toolbar enabled by default, which will affect its performance. You can disable it by setting INTERNAL_IPS to an empty list in your local settings."

Since the point of using Aiven for Caching is to improve performance, we should do that.

Edit sandbox/settings.py and search for INTERNAL_IPS. It's probably around line 364. Change the line:

INTERNAL_IPS = ['127.0.0.1', '::1']

INTERNAL_IPS = []

Create an Aiven for Valkey service

First we need to start a new Aiven for Valkey service using the Aiven Console.

I want the same cloud (Google Cloud) and location (google-europe-north1) as for the PostgreSQL service
Again, Service Plan "Hobbyist" should do for this demo
Following my previous naming convention, I named it "tibs-django-valkey"

Export the Aiven for Valkey Service URI from the Aiven Console to the following environment variable:

export VALKEY_SERVICE_URI='<Service URI>'

Install the Valkey CLI

We want to be able to "talk" to the Aiven for Valkey server, so let's install the Valkey
command line tool, valkey-cli, as described at connect with valkey-cli

For instance, on on my Mac I can install Valkey locally:

brew install valkey

Then run the command using the Aiven for Valkey's service's URL from the service overview page:

valkey-cli -u $VALKEY_SERVICE_URI

This will leave us at a prompt naming the HOST and PORT:

HOST:PORT>

In my case, it looked something like:

tibs-django-valkey-project-tibs.aivencloud.com:10144>

We can then ask what keys are in my Aiven for Valkey datastore:

INFO KEYSPACE

At this stage, the keystore is empty:

# Keyspace

We can quit valkey-cli using:

QUIT

Tip

It's useful to be able to run valkey-cli from a different terminal window, so you can watch how things change while interacting with the running Oscar web app. Don't forget to set VALKEY_SERVICE_URI in that second terminal as well!

About Oscar and Django versions

Oscar 3.2 uses Django 4.2.

Oscar 3.2 is a Long Time Support (LTS) version, which will be supported until January 2026.

How can you tell which versions of Oscar and Django you've got? You can check the version of Django by typing:

pip show django | grep Version

You can do the same for the version of Oscar:

pip show django-oscar | grep Version

This documentation has been checked against Oscar 3.2.4.

The relevant Python libraries

Setting up the environment for Oscar (in the make sandbox step) will already have installed the necessary Python libraries to allow the application to talk to Aiven for Valkey.

Note

Actually, Oscar sets things up for talking to Redis®*, but this will work as Valkey uses the same Redis serialization protocol. You can see the specification Oscar uses in requirements.txt - look for the string "redis" in the Sandbox section.

If we were using plain Django, we'd need to install the Valkey Python library ourselves. It's OK to install the Python library for Valkey instead of that for Redis:

pip install valkey[hiredis]

The [hiredis] also installs the compiled response parser from the hiredis package, which primarily speeds up parsing of multi bulk replies, and is recommended by Django.

Tell Django we're going to be doing caching

Edit the file sandbox/settings.py.

Find the MIDDLEWARE = [ definition, and add the following line to the start of the list:

    'django.middleware.cache.UpdateCacheMiddleware',

Then add the following line to the end of the list:

    'django.middleware.cache.FetchFromCacheMiddleware',

After that, it should look like:

MIDDLEWARE = [
    'django.middleware.cache.UpdateCacheMiddleware',
    'debug_toolbar.middleware.DebugToolbarMiddleware',

    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',

    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.contrib.flatpages.middleware.FlatpageFallbackMiddleware',

    # Allow languages to be selected
    'django.middleware.locale.LocaleMiddleware',
    'django.middleware.http.ConditionalGetMiddleware',
    'django.middleware.common.CommonMiddleware',

    # Ensure a valid basket is added to the request instance for every request
    'oscar.apps.basket.middleware.BasketMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',
]

Note

The details of the listed middleware may be different in your setup, but the location of the new entries is the important part.

For more on the ordering of the middleware, see the documentation for per-site caching.

Tell Django to use (this) Aiven for Valkey as its cache

We need to tell Django to actually use Aiven for Caching for caching, and where to find the Aiven for Caching service.

Django 4 comes with Redis serialization protocol based database support built in.

Still in sandbox/settings.py, find the CACHES = { definition, and change it from:

CACHES = {
    'default': env.cache(default='locmemcache://'),
}

to:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.redis.RedisCache',
        'LOCATION': os.environ.get('VALKEY_SERVICE_URI'),
    },
}

This tells it to use Aiven for Valkey as a backend cache, and to find the instance of Aiven for Valkey at the URL specified by that environment variable (which we set earlier).

Use variables rather than constants

Although the Django cache framework documentation section on Caching shows setting the LOCATION explicitly to a specific string, it's better practice not to embed such strings into (what might be) production code. In this instance, we know that the VALKEY_SERVICE_URI includes a password, as well as other connection details, so we don't want to commit those details to version control, or even expose them unnecessarily in our CI (continuous integrations). Using an environment variable allows us to add this information at run time instead, and use a secrets manager in the actual production environment.

Set the cache behavior

Still in sandbox/settings.py, add the following settings after the CACHES
definition (add them on lines after the final closing } of the CACHES definition -
these lines should not be indented):

CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_KEY_PREFIX = ''
CACHE_MIDDLEWARE_SECONDS = 60 * 10

CACHE_MIDDLEWARE_ALIAS is the cache connection to use for the cache middleware, default is the default
CACHE_MIDDLEWARE_SECONDS is the the cache timeout, in seconds - how long before a cached page expires. The default is 600.
CACHE_MIDDLEWARE_KEY_PREFIX is a string used to prefix all the cache keys. The default is an empty string.

This is actually setting the default behavior, but it seems sensible to be explicit about our choices, and the Django per-site
cache documentation recommends adding these settings.

Show it in action

Let's restart the sandbox application:

sandbox/manage.py runserver

Then log out and back in, add another book or two to the basket, and open the basket.

Run valkey-cli again:

valkey-cli -u $VALKEY_SERVICE_URI

Ask it again about the keys it is storing:

INFO KEYSPACE

This time we should see something like this in response:

db0:keys=32,expires=32,avg_ttl=187151332590

In this case, it's telling us that:

there are now 32 keys, so 32 cached items (the actual number will vary according to what you did)
all 32 of them have expiration times set

The last value, avg_ttl, isn't terribly useful in this case, as it indicates the average TTL for all of those keys, and some of them have very large values set, presumably to stop them expiring.

Getting more detailed, type:

KEYS *

We should then see a list similar to the following:

 1) ":1:CATEGORY_URL_en-gb_3"
 2) ":1:oscar-sandbox||image||ff719c9a5892bf5b4680eb86da85f5aa"
 3) ":1:CATEGORY_URL_en-gb_4"
 4) ":1:oscar-sandbox||image||20242e100a811986ece16c1fac3b2a57"
 5) ":1:views.decorators.cache.cache_header..8ac2d55b8eaa00e52c563c9a18db6736.en-gb.Europe/London"
 6) ":1:oscar-sandbox||image||f544ccac158defae9a7da221d5a79d61"
 7) ":1:views.decorators.cache.cache_header..357698008328fc178c9adfab49a0d197.en-gb.Europe/London"
 8) ":1:CATEGORY_URL_en-gb_7"
 9) ":1:CATEGORY_URL_en-gb_6"
10) ":1:oscar-sandbox||image||05221781d86552e0ab294b1ba2b4d977"
11) ":1:oscar-sandbox||image||0daf8fe2289a1bccdad5cad5a97abefa"
12) ":1:oscar-sandbox||image||eee159e62c22e44531117720e9d9e3e7"
13) ":1:oscar-sandbox||image||f53672bae89061349c4beb1d55721858"
14) ":1:views.decorators.cache.cache_page..GET.357698008328fc178c9adfab49a0d197.95da522d651f032384984b71dab9c668.en-gb.Europe/London"
15) ":1:oscar-sandbox||image||c08074a3afe5aba9e44b7b2a725266c8"
16) ":1:views.decorators.cache.cache_page..GET.8ac2d55b8eaa00e52c563c9a18db6736.7ff9cc28df2e7c550de2e09525c9bf06.en-gb.Europe/London"
17) ":1:oscar-sandbox||image||146cc1138e35c43b8f5a8dc41370dda9"
18) ":1:views.decorators.cache.cache_page..GET.8ac2d55b8eaa00e52c563c9a18db6736.ed66662513025556335a676d702540bc.en-gb.Europe/London"

Let's choose one that sounds likely to represent a page view (a key with
view.decorators.cache.cache_page..GET in its name).

Getting the value for the key I chose:

GET ":1:views.decorators.cache.cache_page..GET.357698008328fc178c9adfab49a0d197.95da522d651f032384984b71dab9c668.en-gb.Europe/London"

Gives me back a "Django template response" with an HTML page embedded in it. The response starts with:

"\x80\x05\x95\x98b\x00\x00\x00\x00\x00\x00\x8c\x18django.template.response\x94\x8c\x10TemplateResponse\x94

And ends with the following:

<title>\n    Basket | Oscar - Sandbox\n</title>

We can also ask for the TTL:

TTL ":1:views.decorators.cache.cache_page..GET.357698008328fc178c9adfab49a0d197.95da522d651f032384984b71dab9c668.en-gb.Europe/London"

Which returns something similar to the following:

(integer) 474

And if you repeat the TTL command for the same key, you will see the time decreasing.

In fact, if you wait long enough, all of the page view keys will disappear, as they time out, leaving only the image keys - here we can see the first four of those.

 1) ":1:oscar-sandbox||image||ff719c9a5892bf5b4680eb86da85f5aa"
 2) ":1:oscar-sandbox||image||20242e100a811986ece16c1fac3b2a57"
 3) ":1:oscar-sandbox||image||f544ccac158defae9a7da221d5a79d61"
 4) ":1:oscar-sandbox||image||05221781d86552e0ab294b1ba2b4d977"

Change the TTL

Stop the application and edit sandbox/settings.py to change the cache TTL to be 20 (twenty seconds):

CACHE_MIDDLEWARE_SETTINGS = 20

Then restart the application and refresh the current page (which in my case was still showing the basket). Use redis-clis to ask for the TTL of a content page again. For example:

TTL ":1:views.decorators.cache.cache_page..GET.357698008328fc178c9adfab49a0d197.95da522d651f032384984b71dab9c668.en-gb.Europe/London"

This time, the values returned should be lower - in my case, I saw 11 because
I wasn't quick enough to see it at 20!

(integer) 11

In other words, all of the page keys will expire within that shorter TTL.

What we've achieved

In the last post, I explored how to use an Aiven service (Aiven for
PostgreSQL®) as a backend for a web platform I already knew,
Django. To avoid the lengthy process of
setting up my own Django application, I decided to use an existing one, the
Oscar sandbox.

In this post, I showed how to add an Aiven for Valkey cache to that web application. As before, this was not too hard to setup, and it's pleasing to be able to use Aiven services for both the database and the cache.

More things to look at

The Oscar documentation has a lot more
information about the project.

Django 4.2 is a Long Term Support (LTS) release, supported until at least 2026. For more information, check out:

the Django 4.2 documentation.
the Django 4.2 cache framework documentation and its sections on Caching and the per-site cache.

If you're just wanting to see the current state of Django, then check out the latest version of the Django documentation.

Also check out Aiven for Valkey,
and if you're not using Aiven services yet, go ahead and sign up now for your free trial at https://console.aiven.io/signup.

Table of contents

Summary of the previous article
The basics
Talking to Aiven for PostgreSQL®
Continue using the Oscar e-commerce example
Disable the Django Debug Toolbar
Create an Aiven for Valkey service
Install the Valkey CLI
About Oscar and Django versions
The relevant Python libraries
Tell Django we're going to be doing caching
Tell Django to use (this) Aiven for Valkey as its cache
Use variables rather than constants
Set the cache behavior
Show it in action
Change the TTL
What we've achieved
More things to look at

Set up Django to use Aiven for Valkey™

Summary of the previous article

The basics

Talking to Aiven for PostgreSQL®

Continue using the Oscar e-commerce example

Disable the Django Debug Toolbar

Create an Aiven for Valkey service

Install the Valkey CLI

About Oscar and Django versions

The relevant Python libraries

Tell Django we're going to be doing caching

Tell Django to use (this) Aiven for Valkey as its cache

Use variables rather than constants

Set the cache behavior

Show it in action

Change the TTL

What we've achieved

More things to look at

Talking to Aiven for PostgreSQL^®