grantjenks/python-diskcache

Python disk-backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python.

Summary

DiskCache is a pure-Python library that provides a disk- and file-backed cache built on SQLite and memory-mapped files, exposing a dict-like API similar to Redis or Memcached. It ships with a Django-compatible backend, persistent Deque and Index containers, and helpers for cross-process synchronization. The README positions it as a way to use spare disk capacity for caching without running a separate server process or compiling C extensions.

What it is useful for

The library targets Python applications that need more cache capacity than memory allows, or that want a persistent cache without external infrastructure. Concrete uses from the README include:

• Replacing or supplementing Memcached/Redis when memory is tight but disk is plentiful.
• Django projects: the README describes Django's built-in file-based cache as "essentially broken" because of random culling and linear-time scans; DjangoCache provides a drop-in alternative backend.
• Cross-process coordination via documented recipes: memoize_stampede for cache stampede prevention, Lock for cross-process locking, and throttle for cross-process throttling.
• Larger-than-RAM working sets with controllable eviction: LRU and LFU policies are supported, along with tag-based eviction.
• Sharding across directories using FanoutCache to spread load across multiple underlying caches.

Reported uses in the README's testimonials include cutting Elasticsearch query load by over 25% on a site serving over a million users per day and roughly tripling the speed of some Ansible runs. The README also cautions that micro-benchmarks are not a substitute for real measurements in your environment, and does not document a benchmark procedure that would replicate those results on your hardware, so treat the cited numbers as illustrative rather than guaranteed.

How engineers can use it

Installation is documented in the README's quickstart:

pip install diskcache

Basic usage from the same quickstart:

import diskcache as dc
cache = dc.Cache('tmp')
cache[b'key'] = b'value'
value = cache[b'key']

For Django, configure DjangoCache as a cache backend. For larger workloads, point multiple Cache instances at separate directories and combine them with FanoutCache. For persistent sequence or mapping behavior, use Deque and Index instead of Cache. Beyond the quickstart, the README directs readers to a separate tutorial and API reference, so anything not shown above (for example, configuring eviction policies, tag metadata, or memoize_stampede parameters) needs to be checked in the full DiskCache Tutorial and API Reference linked from the README.

The library is described as thread-safe and process-safe, and is tested on CPython 3.6-3.10 on Linux, macOS, and Windows. One caveat: the GitHub repository's license field shows NOASSERTION, while the README states the code is licensed under Apache License 2.0, so confirm the license with the project before redistribution.