How I over-engineered my private document management system

I gave this talk at Accenture’s internal architecture meetup on 25 January 2024, and never published it. I recently found the slides again and looked back at them. Going paperless has proven itself over the years, and so have the infrastructure and the Zero Trust parts. The over-engineering was worth it. Maybe it’s useful for someone else, so here is the talk, with notes where my setup has changed since.

The folders held the papers of my family and of my companies, like BauCloud GmbH.

My vision was to go completely paperless:

• Get rid of all non-essential paper. Keep just one folder in the flat, and keep only the relevant documents.
• Convenient integrations. Get notified about new documents in Google Chat, and forward invoices to DATEV Unternehmen Online.
• Access from everywhere. Have the documents at my fingertips.
• Multi-tenancy. Six tenants: family, two private, three companies.

The setup

I chose to run Paperless-ngx on my Synology NAS. Paperless-ngx is open source, actively developed, has a simple architecture and an ecosystem of apps. The NAS has four bays, 31 TB of capacity and 20 GB of RAM.

I didn’t do much research at the time. I had heard of Paperless-ngx and was looking for a good use case for my NAS. I needed six instances, so a hosted DMS would probably have got quite costly.

Paperless-ngx picks up new documents from a folder on the NAS, runs OCR on them and classifies them automatically. I search them in the web UI, or on my iPhone with Less Paper, a native iOS client. I also point AI agents at its REST API. I keep that simple: no MCP server for now.

Architecture

The catch: Paperless-ngx isn’t ready for multi-tenancy. It has users and permissions, but I want data isolation: the ability to export or move one tenant without touching any other. There are only two or three users, and their access overlaps almost completely. I also want to ask questions like “which letters did company X get from vendor A?” Permissions alone can’t answer that.

For me, multi-tenancy always means weighing the isolation I need against what the software supports.

The simplest and strictest isolation is at the infrastructure level: every tenant gets its own full set of infrastructure. That gives proper data isolation and lets you allocate costs, but it has the highest total cost of ownership. Users and permissions are a kind of shared everything, and I didn’t consider that viable at all. Paperless-ngx doesn’t support a shared application approach, so the next best thing is isolated infrastructure. From there, I looked for parts of the infrastructure the tenants could still share.

All tenants share the server, the NAS, and the auxiliary services, Tika and Gotenberg. That cut the memory use on the NAS, its biggest constraint, significantly.

Tika and Gotenberg are stateless, so sharing them is easy. The tenants also share one Valkey, a fork of Redis, as their task queue, and each tenant uses a database number of its own in it. Each tenant keeps its data in its own SQLite database, the Paperless-ngx default.

Networking

At first, each tenant ran on its own port, from my-nas.local:32121 up. To use the DMS from everywhere, I needed a proper look at networking. My requirements:

• Proper, memorable domains.
• Accessible from everywhere, but restricted to two users.
• HTTPS certificates, at least to avoid browser warnings.
• Zero ops: no self-signed certificates, no manual certificate rotation.

The question is how to make the Paperless-ngx instances reachable, and how to enable single sign-on (SSO) in a pragmatic way.

There are two or three typical approaches. At one extreme, you publish the HTTP services to the public internet, for example with DynDNS. At the other, you trust the network itself, so you can only use them at home. A variant adds a VPN, for example on the FritzBox router. That setup is tedious and doesn’t solve authentication.

I was looking for something better, and that’s how I came back to BeyondCorp, Google’s take on Zero Trust.

The main idea of Zero Trust is to never trust, always verify. NIST SP 800-207 puts it like this:

Zero trust provides a collection of concepts and ideas designed to minimize uncertainty in enforcing accurate, least privilege per-request access decisions in information systems and services in the face of a network viewed as compromised.

Tailscale is a software-defined mesh VPN that makes such a Zero Trust network easy. A central control server, run by Tailscale in the cloud, handles keys and policy, and identity comes from an existing login such as Google Workspace. The agents are open source, and Headscale is an open source control server. The building blocks I needed:

• Mesh VPN based on WireGuard.
• MagicDNS: automatic DNS names for all hosts.
• Automatic HTTPS for all MagicDNS names.
• Identity headers: HTTP request headers that say who the user is, for authentication.
• ACLs for authorisation.
• Sharing nodes with other tailnets.

In the talk I demoed a proof of concept with Caddy and Tailscale, and tailscale serve as the next step.

One Caddy served all tenants, with the caddy-tailscale plugin. Each site in the Caddyfile binds to its own tailnet node, so every tenant got its own name and certificate:

:443 {
	bind tailscale+tls/dms-tenant-a
	reverse_proxy webserver-tenant-a:8000
}

Each tenant is one Tailscale Service, and the NAS serves all of them:

tailscale serve --service=svc:dms-a --https=443 http://127.0.0.1:18001

The tenants’ URLs look like https://dms-a.example.ts.net, and grants in the tailnet policy say who may reach which service. Paperless-ngx takes the user from the identity header that Tailscale adds, so there’s no separate login:

PAPERLESS_ENABLE_HTTP_REMOTE_USER=true
PAPERLESS_ENABLE_HTTP_REMOTE_USER_API=true
PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_TAILSCALE_USER_LOGIN

Paperless-ngx trusts that header without checking where it came from. So the grants allow only the services on port 443, never the NAS’s own ports. A peer that could reach port 18001 directly could send any user name it likes.

Document management process

I don’t keep every document, only the relevant ones: certificates (Urkunden) and tax-relevant documents such as invoices and account statements.

Archive serial numbers (ASN) let me find the paper originals I keep. I took the idea from margau’s post paperless-ngx with QR codes as ASN: a QR code sticker carries the ASN, Paperless-ngx reads it when it consumes the scan, and the paper goes into a binder in ASN order.

How to print such labels? I could reuse something:

• Download the PDF from the blog post. It covers ASN001 to 189, but no more. And what if I want another prefix than “ASN”?
• Use an open source Python CLI to generate the PDF. But I don’t like Python, and I don’t want to run code for this.

Or I could build something, and use it as an opportunity to try and learn. I bet one doesn’t need to generate a PDF file in today’s world to get a perfectly aligned print-out for Avery L4731 labels.

And indeed, one doesn’t, if you use Google Chrome. The ASN QR code label generator is on this site, built with Tailwind CSS and Alpine.js, and its source is on GitHub.

In 2024, I scanned straight to a network folder on the NAS, and Paperless-ngx consumed the documents from there.

To scan many documents in one go, I used patch sheets: separator pages with a barcode. Paperless-ngx splits a scan at each one.

What I learned

These were my key learnings in 2024:

• I can and I will go paperless now.
• I got a better understanding of Zero Trust.
• Tailscale is great. I wonder why it’s not used more in enterprises.
• Caddy works well as a reverse proxy, and Caddy with Tailscale is very promising.
• The browser is enough to create print-outs like labels or reports.
• I played with Alpine.js again and created an open source project to generate labels.
• Tailwind CSS is always worth recommending.
• A webhook integration with Google Chat is easy.

Looking back, what started as an example of over-engineering has served me very well. I added a couple of new technologies and architectural approaches to my tool belt, and I’ve used them ever since. Caddy became my preferred reverse proxy, and Tailscale is excellent. I use both for Librario now.