This article is going to look at something a little different to my usual sitecore posts - a side project I have been chipping away at to give us a proper last resort backup of our whole website, northumbria.ac.uk.
It came about after one of those conversations where someone asks "what actually happens if the site gets compromised, or just falls over and wont come back?". We have database backups and all the usual stuff, but nothing that lets you actually SEE the site as it looked on a given day, or serve it back up if the worst happened.
So I set out to build one. The brief in my head was dead simple to start with - crawl the site, store the html of every page in a database, and do it again each day so we build up a history.
Note: I built most of this alongside an AI coding agent (Claude), which is definately becoming a bit of a theme for me. More on how that went at the end.
The simple idea (and why it wasnt that simple)
The first plan was: read the sitemap, grab each page, store the full html. Then a daily job re-reads the sitemap and only stores a new version of a page if its actually changed. Set a point in time, pull the latest version of each page at that point, and serve the lot.
Straight forward enough on paper.
The first thing that bites you is that html on its own doesnt render. A stored page links out to all the css, js, fonts and images on the live site - so if production is down (i.e. the exact situation were trying to cover for) all of those are dead too and you get a broken, unstyled mess.
So you have to capture the assets as well, and rewrite the pages to point at your stored copies.
Where to put it all
I went back and forth on storage. The pages themselves are tiny once you gzip them (we are getting about 8x), and you want to be able to query and diff them, so those live in Azure SQL.
The assets are the bulk - and the bit you never need to query, you just fetch one by its key. I did wonder about shoving them in SQL too as blobs, but that gets expensive fast.
So assets go into Azure Blob storage instead, deduplicated by a SHA-256 hash of their content. i.e. the same logo that appears on every single page is only ever stored once. SQL just holds the hash and a path. Works out roughly ten times cheaper per GB than keeping them in the database.
Whats actually changed? Hashing is harder than it looks
This was the bit that caught me out, and Im glad I tested it properly before trusting it.
The whole model relies on detecting when a page has genuinely changed. My first instinct was to just hash the html - if the hash is different, store a new version.
Except every single page came back as "changed" on the very next crawl, even ones I knew I hadnt touched.
After some head scratching I dumped two captures of the same page and diffed them. They were byte identical... except for one line - a calendar.js bundle tag that Sitecore was emitting on some requests and not others, completely at random.
So the fix is to normalise the html before you hash it - strip out the per request noise (cache buster query strings, csrf tokens, the odd nonce, and that pesky calendar.js tag) - then hash whats left. The stored page stays exactly as captured, its only the hash input that gets cleaned up.
Crucial, that one. Otherwise the daily job would store a fresh copy of all 13,000-odd pages every night for no reason.
The sitemap lies to you
I had assumed the sitemap was a complete list of the site. It isnt.
I noticed it when /international/ - a fairly important section landing page - came back as not found in the backup. Turns out it simply isnt in sitemap-en.xml at all, even though loads of the pages underneath it are.
So the crawler had to grow up from a simple sitemap reader into a proper spider - it seeds from the sitemap AND every ancestor path, then follows internal links to discover everything else. I also had it prioritise by depth, so the homepage and top level sections get done first.
Note: spidering a real site throws up some lovely rubbish. I found it merrily following links like businessenquiry@www.northumbria.ac.uk/... (an email mis-authored as a link) and unresolved sitecore {localLink:13805} tokens, both of which were creating junk duplicate pages. A couple of small guards to strip the url userinfo and reject anything with braces sorted that, and I cleared out the ~500 junk rows it had already collected.
Serving it back
The serving side is a small read only app. You give it a path and optionally a point in time, it finds the latest version of that page at or before that moment, decompresses the html, rewrites all the asset urls to point at the stored blobs, and serves it. Assets come back from a /_asset/{hash} endpoint.
Theres also a little audit UI on top - search the pages, view any historic version, and diff two versions against each other. Which is genuinely useful well beyond the disaster scenario, as a "what did this page actually say last month" tool. Bit like having our own private wayback machine.
What if we actually had to use it?
The interesting question is whether you could really cut over to this if production was compromised. A few things matter here.
First of all, you do NOT want to serve the compromised version back to people. The point in time feature is exactly right for this - you pin the whole site to a last known good moment (i.e. just before the attack) and it will never serve anything captured after it. I wired that up as a config switch, plus an "archived backup" banner, so it can be flipped on the day without a redeploy.
Then theres scale. The everyday setup is sized for the odd audit, not production traffic. But the content is basically ideal for caching - the assets are immutable, the pages barely change - so sticking a CDN / front door in front of it gets you alot of the way. For the really paranoid option I also built a static export that bakes a chosen snapshot out to flat files for an azure storage static website, with no database in the request path at all.
I wrote all of that up as a proper DR runbook too, so its not me trying to remember it whilst the building is on fire.
Summary
So in this article we looked at building a point in time backup of an entire website - crawling it with a link following spider, storing compressed html in SQL and deduplicated assets in blob storage, only versioning pages that genuinely change (once youve dealt with the random html), and serving any point in time back through a small app, right up to a full disaster recovery cutover.
It started life as a simple "just store the html" idea and grew a fair few arms and legs along the way, as these things tend to...
And on the AI angle - having an agent do the actual typing whilst I steered was genuinely effective for a project like this. Lots of moving parts but nothing especially novel or invloved. The bits that needed a human were the judgement calls - the hashing problem, spotting the sitemap gap - not the code.
Hopefully this is useful to anyone thinking about doing something similar. Im sure there are better ways to do half of it, but it works, and thats good enough for a last resort.