From Scrapbook to Storyteller: Our Very Own Primer

We didn’t mean to build a storyteller. We meant to build better search capabilities over our personal information. But stories are what surfaced, threads we hadn't realize we’d left for ourselves, links between moments separated by years, and reminders that small details matter.

Overview ~ Literary déjà vu ~ How it works ~ First impressions ~ Examples

Overview

Back in February 2024, we shared an update on our Scrapbook project, a tool we've used for many years to organize our lives. As we have described in different posts on the subject, Scrapbook is like our memory chest: searchable, organized, and expandable. In the last year, something interesting has happened. It's learned to talk back.

What we've done is woven a large language model (LLM) into the heart of Scrapbook. It doesn't roam the internet unless we ask it to. It answers questions using our corpus of data. It lives entirely inside our world of words and pictures answering questions, surfacing connections, and connecting threads between moments that even we'd forgotten.

Our data is broken into tens of thousands of entries, with each entry including text, digital assets, and links to other entries. The text of an entry can be anywhere from a few words to thousands of words. In short, there there is fair amount of material for an LLM to work with.

Literary Déjà Vu

The more we use our LLM-enabled Scrapbook, the more it reminds us of Neal Stephenson's The Diamond Age and its Young Lady's Illustrated Primer, a fictional, adaptive book which could teach, guide, and grow alongside its reader.

Like the Primer, our Scrapbook is becoming more than a static repository:

• It adapts — tailoring its responses to the quirks, references, and rhythms of our own material.

• It guides — when we search for a trip, a recipe, or an old photo, it brings back more than just the thing itself; it recalls the surrounding story.

• It connects dots — sometimes between entries years apart, revealing patterns we never set out to find.

The Primer in The Diamond Age was a thought experiment about how a tool could shape its reader’s growth through tailored narrative. Our version isn't teaching martial arts to Victorian ingénues, but it is helping us understand and navigate our own history in a way that's private, persistent, and woven into the things that matter to us.

We didn't set out to build a piece of science fiction. But step by step, feature by feature, Scrapbook has edged closer to being a kind of personal storyteller. Not just a record of where we've been, but a guide to help us decide where to go next.

How does it work

There is a lot going on, but here is the high level overview:

• Our solution creates a ChatGPT-like frontend experience over your own Cosmos NoSQL documents using RAG (Retrieval Augmented Generation).

• We use Azure OpenAI Service to access GPT models, and Azure AI Search for data indexing and retrieval.

• Our assistant is hosted in a Blazor application that has allows for adding and curating data.

First impressions

How's it working? In the first few months (mid 2025) of working with it, we found largely what you find with LLMs: they can be brilliant in the best of times and hallucinatory in the worst of times. Though, when it hallucinates on a body of data we know well, it's almost kind of funny and we can spot it easier.

Then we got to coding and updated the APIs we were using (migration from the Assistants API to Responses API), got access to GPT 5 models, rebuilt vector store for use in AI file search tool, and rewrote our system prompts. After this code push (early 2026), the Scrapbook Assistant was brilliant consistently and stable, with way less hallucinations. Phew! In many cases, it behaves almost like a magician in what it surfaces (correctly).

Given that LLMs work with words, our words matter. How we curate our data matters. No surprise, right? When we create a Scrapbook entry for an important event it really behooves us to be clear about who did and said what if we want to that be surfaced correctly later.

Writing clearly and consistently is key. In fact, after asking questions of our assistant, we often realize we need to go back and do editorial work. After the text changes, and rerunning the indexes for the LLM, the results improve. Another nonsurprise: It seems all new tools we aim at our Scrapbook corpus of data end up shining a light on how important curation is.

To date, our Scrapbook Assistant is great with words and less so with dates and ordering. For example, if we ask it to tell us about hikes (a Scrapbook data category) that we really liked recently, it doesn’t do what you might expect. It returns hikes we rated highly but may include some from recent times and some from way back. We don’t understand enough yet to know how to control this. We are very much in the early phases of this, and our guess is that this too will be something that gets improved.

Examples

There's a certain pleasure in discovering what you experienced but forgot. We now have conversations with our Scrapbook Assistant and collaborate with it.

Give me a rundown of glaciers we have visited.

We get back not just the date, but the glacier in context of what trip we were on, when, and a summary, size depending on model selected.

Tell me about the foods that Ottavia likes to eat?

We get back notes and anecdotes from dinners together we had together. It makes planning future meals together much easier. Our friend has a lot of food dislikes, so it's always good to check.

Explain our tax liability for the year 2024.

Try this looking in a filing cabinet!

What is the booking info for the last time we stayed in Berlin?

Out comes everything you need to understand length, price, contact info, and even what rooms you stayed in and liked. This works because all hotel and accommodations are recorded.

Give me some ideas for a Travelmarx-style blog post for the hike we did recently to Misma?

Hikes and in general large-effort activities are part of Scrapbook. We jot some notes down, and the activity is part of a larger unfolding of events that the assistant can tap into and tell a story. Also, all old blogs (hundreds and hundreds) are archived as entries in Scrapbook, so there is a lot to pull from to understand Travelmarx-style. Result is here: Le Vie di Misma - Trails 511 and 510.

What are some ideas from notes and previous food entries for a gluten-free friend visiting for a few days?

Indispensable guidance.

What is the recipe for a garbanzo bean dish we had in Seville?

No more searching around for recipes to share. And with the Scrapbook Assistant, follow up with “translate into Italian”.

Give me a one paragraph summary of our Japan 2025 trip including places visited and length of time at each stop.

These kinds of quick summaries of past events are super useful when someone asks. We can just ask the assistant without much time spent. We can customize the response for a friend who is interested in tours we took. In the response are links back to each individual item.