Showing posts with label personal information management (pim). Show all posts
Showing posts with label personal information management (pim). Show all posts

Monday, June 15, 2026

From Scrapbook to Zettelkasten: Discovering the Slip Box


Zettelkasten mock up with Scrapbook themes. Sawangwongse Yawnghwe work at the Venice Biennale 2026
Left: Zettelkasten mock up with Scrapbook themes.
Right: Sawangwongse Yawnghwe work at the Venice Biennale 2026.

File Under: A Conversation in Venice


At the Venice Biennale 2026, we found ourselves standing in front of a group of paintings by Sawangwongse Yawnghwe. The paintings looked a little like diagrams and a little like maps, although they were not the sort of maps that would reliably get you from one place to another. Words, names, arrows, and relationships organically spread across the surface. They seemed to be conveying a complicated political history all at once.

We started talking with another visitor, a German art teacher named Philipp. He seemed also interested in the graphical nature of the art. One topic led to another and we ended up talking about note taking and saving information. Philipp mentioned Notion, the digital workspace used for organizing notes and projects. He also mentioned the German sociologist Niklas Luhmann and the philosopher Hans Blumenberg. Both were known for maintaining elaborate collections of notes using a system called Zettelkasten.

It sounded slightly formidable, as German words can when you see them for the first time. But the literal translation is unassuming: a box of slips of paper. A slip box.

We made a note of the word, along with notes about the conversation, and continued through the exhibition.

Later, we entered the exchange with Philipp into Scrapbook and asked what else in our collection might be related to the idea of the Zettelkasten.

That was when things became interesting.

File Under: The Slip Box


A Zettelkasten is a collection of individual notes, traditionally written on small pieces of paper or index cards and kept in a box. But it is not simply a filing cabinet in miniature.

The important part is linking between notes. A note does not sit alone under a broad heading and wait patiently to be retrieved. It points to other notes. One thought leads to another. A new note can branch off from an older one, connect two subjects that had previously seemed unrelated, or become the start of a trail that grows over time.

Niklas Luhmann’s Zettelkasten became famous partly because of its scale. Over decades, he accumulated tens of thousands of notes and used them as a working partner in his writing. The system did not merely help him remember things. It helped him encounter his own ideas again in new combinations.

This is the part that caught our attention.

Most organizational systems promise to help you put things in the correct place. A Zettelkasten seems to accept that the more interesting question is what might happen when two things placed years apart unexpectedly find each other.

There is an appealing modesty to the physical object. No elaborate machinery is required. You need paper, a box, a way of numbering notes, and enough patience (stubbornness?) to create links between them. Intelligence is not hidden inside the box. Rather, it accumulates gradually through the act of connecting one thought to another.

Our Scrapbook is not a Zettelkasten in any strict sense. It contains photographs, journal entries, maps, recipes, travel notes, and various fragments that seemed worth saving at the time. It has categories, links, metadata, and search. It has also become much larger and stranger than anything that could fit comfortably into a wooden card catalogue.

What was satisfying with learning about Zettelkasten was that the underlying idea felt familiar. We had been circling around it for years, although not always aware of it.

File Under: Communication Partner


Luhmann is the right center of gravity for this story, not simply because he kept a very large number of index cards. Other people have managed that, especially if they have ever attempted to organize a lifetime of recipes, jokes, or book notes.

What makes his system interesting is the way he described his relationship with it. A mature Zettelkasten was not merely a storage box or an external memory. It became a kind of communication partner.

At first, a collection gives back only what you put into it. You write a note, place it somewhere sensible, and retrieve it later. But as the notes multiply and the links between them accumulate, the system begins to develop enough internal complexity to surprise you. A new entry leads to an old one you had forgotten. Two thoughts written years apart suddenly sit next to each other and suggest a third. The system returns something that was already yours, but not in a form you had anticipated.

This is not magic. The slip box has not become sentient. It is still paper, numbering, cross-references, and the accumulated work of the person tending it. But, in a way it feels like it is communicating with us.

We have written elsewhere about Scrapbook starting to "talk back" after we added a large language model (LLM) to it. That phrase now seems less novel than we first thought. Luhmann had already described a similar experience with his paper system. The tools were different, but the surprise was recognizable: a collection could become more than a passive repository. It could participate in the development of an idea.

File Under: Recursion


There is a pleasing recursion to how this all played out.

First, we were standing in front of Yawnghwe’s paintings at the Biennale, looking at diagrams that attempted to capture complicated networks of political history.

Then we began talking with Philipp. The conversation wandered from the paintings to Notion, then to Luhmann, Blumenberg, and the Zettelkasten.

We entered the new term into Scrapbook.

Later, we asked Scrapbook where we had encountered related ideas before.

The assistant (looking only at our data) reached backward through the collection and returned with a small intellectual family tree: the cabinet of curiosities, Vannevar Bush’s memex, the Microsoft Research project MyLifeBits, and our own earlier attempts to describe Scrapbook. All of these are detailed entries in Scrapbook.

We had not filed those older notes under Zettelkasten because we had not yet learned the word. But there they were related, waiting patiently in digital drawers.

The relationship is not a clean historical progression. The Zettelkasten predates several of the digital projects. It is better to think of these as overlapping attempts to solve related problems:

Cabinet of curiosities
A personal collection of objects and fragments chosen because they matter to the collector.

Zettelkasten
A network of linked notes designed not only to preserve ideas but also to generate new ones.

Memex
Vannevar Bush’s imagined personal archive, navigated through associative trails rather than rigid hierarchies.

MyLifeBits
A digital attempt to capture and connect the documents, images, and other traces of a life.

Scrapbook
Our homegrown combination of archive, journal, database, photo collection, linked notes, and now LLM-assisted retrieval.

The amusing part is that the Scrapbook assistant surfaced this connection by behaving in a way that seemed suspiciously close to the point of a Zettelkasten. It returned a set of old notes in a new combination and helped us see an argument that had been taking shape without us noticing it.

We discovered the Zettelkasten twice: first through a conversation in Venice and then through Scrapbook’s ability to connect that conversation to older fragments of our own thinking.  

File Under: Tension


Our previous Scrapbook posts reveal two slightly different impulses.

The first is intentional curation. In our earlier post, Seven Laws of Organization and Disorganization, Scrapbook appears as our small rebellion against the chaos of camera rolls, social-media feeds, scattered chats, and cloud-based systems that save everything while making it surprisingly difficult to find anything. Each Scrapbook entry is a deliberate act of remembering.

The second impulse is productive surprise. In our earlier post about Scrapbook as a digital cabinet of curiosities and vademecum, the collection is not valuable only because it is orderly. It is valuable because an old photograph, a stray note, or an observation made during a trip can lead somewhere unexpected.

A Zettelkasten sits between these two impulses.

It requires curation. Notes must be written, linked, numbered, and tended. But it cannot be too tidy. Its usefulness depends on a certain controlled disorder. The point is not merely to place each note in the correct drawer. The point is to create enough trails that an older thought can return in a new context.

Scrapbook has always lived somewhere in this tension. We built it to impose a little order on our lives. However, after many years, we are starting to learn its greatest value may be its ability to return some of the disorder to us in useful forms.

The reward for carefully organizing your life for years is that your filing system could eventually start making suggestions.

File Under: Same Old Problem


We should be clear: we are not claiming to have invented anything. We are not Vannevar Bush, Niklas Luhmann, or Hans Blumenberg. We are two people who save too many digital artifacts and occasionally manage to find them again when we need them.

What struck us about discovering the Zettelkasten was not that we had unknowingly built one. We had not, at least, not in any strict sense. Scrapbook is too sprawling for that: part archive, part database, part journal, part photo album, part cabinet of curiosities, and now part talkative assistant.

What struck us was that we had been dancing around the same problem for years.

How do you collect the fragments of a life without turning them into an inert pile? How do you leave enough trails that one thought can lead back to another? How do you create a system orderly enough to be useful but loose enough to surprise you?

People have been worrying about these questions for a long time, using whatever tools happened to be available to them. Paper slips in boxes. Index cards. Cabinets. Microfilm. Hyperlinks. Databases. Search engines. Now large language models. Technology changes, but the itch remains much the same.

Perhaps the fact that we are so interested in this says something about us. We like collections. We like categories, except when we don't. We like the feeling that a forgotten meal, a half-remembered walk, or an idea scribbled down years ago has not entirely disappeared.

There may also be faint anxiety underneath it all: the suspicion that a life can become inaccessible even to the people who lived it.

This feels especially relevant now because many of us store large portions of our lives inside systems we did not design. Our photographs live in camera rolls and cloud libraries. Our observations sit inside old chats, social-media feeds, and apps we may stop using in three years. These platforms are often very good at saving things. They are also very good at deciding, on our behalf, which things should return.

A phone offers a memory from five years ago because its algorithm has noticed a date, a face, or a location. A social platform resurfaces a post because it expects a reaction. Sometimes the result is delightful. Sometimes it is baffling. But in either case, the editorial decision is not entirely ours. The system channels the story.

Scrapbook is our small attempt to preserve a little more agency. It is not perfect. It still contains unclear notes, missing links, duplicate entries, and photographs we will probably never identify. Its assistant sometimes makes connections that are illuminating and sometimes makes connections that are simply wrong.

But at least it is rummaging through our mess, a mess we chose and can still edit.

And in this case, it did something useful. We encountered the idea of the Zettelkasten in Venice, added it to Scrapbook, and asked a question. Scrapbook reached backward and found a trail we had not quite seen before: cabinets of curiosities, the memex, MyLifeBits, our own old notes about organization, and the system itself.

The question is no longer whether we can save everything. We more or less can. The harder question is whether we will still have a say in how our own stories are assembled, connected, and returned to us. A Zettelkasten is one answer. Scrapbook is our imperfect answer. At the very least, we would rather do some of the remembering ourselves.

Friday, November 21, 2025

Seven Laws of Organization (and Disorganization)


An image show a mess of information behind a phone
A gateway to disorganization.

Intro


We live in an age obsessed with order. Apps promise to streamline our lives, productivity gurus preach the gospel of minimalism, and yet our desks, inboxes, and camera rolls tell a different story. Maybe the truth is that organization is less a state of being and more a fleeting illusion?

Inspired by Carlo Cipolla’s Basic Laws of Human Stupidity, we wondered: what if we codified the everyday paradoxes of organization into their own set of “laws”? Not scientific laws in the strict sense, but humorous principles that capture the universal frustrations of trying to impose order on chaos.

Call them the Seven Laws of Organization (and Disorganization).

The Laws


1. The Illusion of Order Law

You are never as organized as you think you are. Systems are just chaos with labels, and labels are only as good as your memory of them.

2. The Search Paradox

When you’re looking for something, it hides. When you stop looking, it leaps out. Keys, glasses, wallets, phone, and an object you had in your hand 2 minutes ago all obey this cruel rhythm.

3. The Observer Effect of Clutter

The moment someone watches you search, the item vanishes into another dimension. A coworker hovering or a travel companion waiting impatiently for you to find your boarding pass only guarantees failure or at least a frantic search.

4. The Black Hole Principle

Anything you save for “later” is gone forever. Your camera roll swallows photos whole; that perfect shot of last month’s dinner is now somewhere between screenshots you don’t remember taking. Social media is worse: scroll long enough and you’ll find everything except the post you’re looking for. These are not archives — they are disappearing acts. 

5. The Law of Misplaced Priorities

The more important the item, the less likely you are to find it. Tax documents, passports, or vaccination records disappear, while trivial receipts remain eternally accessible.

6. The Entropy Law

Any organized space will inevitably collapse into chaos unless actively maintained. A tidy desk, a neatly packed suitcase, or a freshly sorted inbox all succumb to disorder in record time.

7. The Filing Cabinet Irony

The more carefully you file something, the less likely you are to remember where you put it. “Safe places” are too safe—they hide things even from you. And the verb file here doesn’t just mean in the classic sense.

Taken together, these laws explain why we keep losing things and why we keep insisting we won’t next time.

Intentionality


Instead of being discouraged by these laws, we should be amused. They remind us that disorganization is not a personal failing—it’s a universal condition. Just as Cipolla argued that stupidity is woven into the fabric of humanity, so too is clutter woven into the fabric of daily life.

Social media and cloud-based apps have promised us perfect organization with their automated albums, searchable memories, curated timelines. But in reality, we’ve outsourced our sense of order to algorithms we don’t control. We pump endless photos, notes, and thoughts into these platforms, only to retrieve them when “they” decide it’s time. How many times has your phone surfaced a random set of images, sometimes eerily appropriate for the moment or sometimes baffling, and you thought, “I couldn’t find those again if I tried”? We are at risk of losing any semblance of personal organization that isn’t algorithmically sanctioned. We don’t organize anymore; we scroll and hope.

In contrast, our Scrapbook project is a humble attempt to reclaim some of that lost intentionality. What started as a way to organize travel memories—photos, maps, notes—has evolved into a curated archive that tries to resist the algorithmic churn. It’s not perfect, but it’s ours. Each entry is a deliberate act of remembering, a small stand against the entropy of digital life. In a world where platforms decide what we see and when, the Scrapbook is our quiet rebellion: a place where we choose what matters.

Precedents and Paradoxes


The Seven Laws of Organization (and Disorganization) don’t exist in a vacuum. They join a long tradition of humorous, philosophical, and paradoxical takes on the human struggle to impose order. Consider:
  • Murphy’s Law and its many offspring: “Anything that can go wrong, will go wrong” has inspired countless organizational corollaries—like “The file you need is always the one you didn’t back up,” or “The one document you forgot is the one they’ll ask for.”
  • Parkinson’s Law: “Work expands to fill the time available.” A classic insight into why our to-do lists never shrink, no matter how much time we block off.
  • Office humor and cartoons: From Dilbert to desk memes, the futility of filing systems and the tyranny of inboxes have long been fertile ground for satire.
  • Disorganization Puns & Wordplay: Disorganization has inspired a rich lexicon of clever terms that blend satire, science, and style:
    • Procrastifiling: The act of organizing as a way to avoid doing actual work—filing instead of finishing.
    • Folderol: Originally meaning trivial fuss, now repurposed to describe excessive, often pointless labeling and categorizing.
    • Entropy Chic: The aesthetic of curated clutter—chaos that’s styled to look intentional and lived-in. 
    • Cluttercore: A maximalist design trend that celebrates visible mess as cozy and authentic.
    • Shelf-aware: A pun on “self-aware,” used to describe someone who knows their organizational systems are more performative than practical.
    • Inboxhaustion: The mental fatigue caused by managing an overflowing inbox that never seems to shrink.
    • Ctrl+Alt+Del Decor: A minimalist style that looks like someone wiped their life clean with a digital reboot.

While making us chuckle, these puns also reveal how deeply disorganization is woven into modern life. They’re coping mechanisms, cultural critiques, and linguistic clutter all rolled into one.

So yes, these laws are part of a lineage that is ever evolving. Perhaps we need to treat disorganization as a kind of philosophy, elevating it from mere frustration to something almost noble. It’s not just about losing things—it’s about understanding why we lose them, and what that says about how we live.

AI Slop


Every media revolution breeds not only brilliance but also rubbish. As a recent Scientific American article argues, the rise of AI is no different: it produces dazzling art and useful tools, but also a flood of “slop”—content that overwhelms rather than enlightens. (The article is The Slop Cycle—How Every Media Revolution Breeds Rubbish and Art and is by Deni Ellis Béchard.)

We feel this slop acutely in this moment in the realm of AI Chat conversations. We have dozens of chats we start with different assistants: a travel plan here, a biology question there, a stray idea in yet another thread. Each one promises clarity, yet together they can form a labyrinth. They produce anxiety: did I read something insightful in that chat, and if so, will I ever find it again? Or has it already slipped away? Should I save my chats somewhere?

This is not just clutter in the traditional sense; it’s a sort of chat hell, a new frontier of disorganization born from the very tools meant to help us. AI assistants are designed to ostensibly, yet they often scatter info across ecosystems, tabs, and threads.

Of course, this will improve. Apps and services will get better and heaven help us if ecosystems even talk to each other, so you are not locked into just one. But for now, AI has introduced a fresh layer of entropy—an invisible pile of chats, forgotten insights, and more organizational anxiety.

Perhaps the Seven Laws of Organization gain an eighth (half) sibling: The Law of AI Slop.

Even as technology evolves, disorganization evolves with it. Can we say that the more we outsource our order to machines, the more we risk drowning in their version of chaos? We would rather drown in our own induced chaos.

Closing For Now


Perhaps the lesson is not to fight these laws, but to embrace them. Laugh at the paradoxes, accept the entropy, and recognize that the search for order is itself a kind of comedy.

There are also quiet alternatives, like building your own system. It doesn’t have to be perfect or universal—it can be a homegrown method, a series of rituals, or a project like our Scrapbook. These efforts may not defy the laws entirely, but they carve out small pockets of intentionality in a world increasingly shaped by algorithms.

After all, the most organized among us are not those who conquer clutter, but those who learn to live with it—and occasionally, to laugh at it, or better yet, to shape it into something meaningful.

Saturday, February 3, 2024

Update on Our Scrapbook Project



History


Scrapbook is what we call our software/service platform that is part personal information management system, part asset management system, part diary and journal, and part digital keepsake. Here are some existing blog entries describing our Scrapbook project:

Since we started writing about Scrapbook, technologies have come and gone. Technologies we mention in the 2017 post for example have taken a back seat to the new technology of the day AI agents and large language models. That said, throughout the years the basic theme we’ve tried to illustrate and encourage is that you should take ownership of your own data regardless of the technology.

Growth


Here are some facts and figures about Scrapbook at it stands at the beginning of 2024.

In our main collection “MyJournal”, we have over 16,600 entries distributed over 30 categories. Each entry is a JSON document stored in Azure Cosmos and associated assets (images, videos, documents, etc.) stored in Azure Blob Storage. (We use other Azure services like registry, key vault, app registration, and more.)
 
  
Left: Growth of two Scrapbook collections. Right: Category distribution for MyJournal collection.


There are over 21,000 links between entries. A link can be something like A is related to B or A is in B.

We estimate that we use Scrapbook around 30-60 minutes a day, which means adding new entries, editing existing entries, and looking up information. The amount of development time is another thing!

Costs to run our Scrapbook depend on many factors such as amount of storage, services used, and regions selected to name just a few. For example, if you are setting up a prototype of our Scrapbook with minimal services and redundancy, you might be looking at 20 - 50 USD or so a month as a ballpark figure.

Recent Activity


This is what we’ve been up to since the last post. (Utterances are in quotes. Underlined words are a category type or a synonym of a category. We use # and @ as in other platforms for hashtags and mentions.)

Containerization

We moved from Azure App Service to Azure App Containers. There were several assumptions made when using App Service that did not work when going to containers. For example, for our container version of Scrapbook, we needed to re-think our Data Protection Layer and how we were handling/storing session state. Here’s what we followed:

Automated ingest

We found ourselves doing too much manual work to capture data. So we created hooks to pull in different types of data.

For example, our Travelmarx blog RSS feed (for one post) can be pulled in and automatically converted from HTML to markdown with all the photos pulled in to create a new Scrapbook entry. We have a similar hook we use for Spotify API and our Lostvibe music collection. The goal here is to use Scrapbook as a backup of other content creation points and to automate entry creation.

Natural language improvements

We rewrote from scratch our Natural Language Query Engine (see this post) to use Cognitive Language Understanding (CLU). We migrated from Azure LUIS (deprecated) to CLU.

The query engine now supports
  • Sorting, for example:
    • “Show my wines from France sorted by rating”
    • “Tell me about books sorted by modified date”
    • “What are hikes sorted by length ascending”
  • Compound logic (logical operators in queries
    • “Show me albums with the keyword cowboy and keyword blue”
  • Numerical comparisons (nice for rating fields)
    • “Show me hikes longer than 20 km”
  • Qualitative comparisons (high, low, great, bad)
    • “Show our favorite hikes great than 20 km”
  • Inferred queries - “show hashtag XYZ”, “show @person”
    • “Show Greece last year”
    • “Tell me about @Roberto”
  • Negations
    • “Show me items without assets”
    • “Show me activities except for yoga”
    • “Show me food except pasta”
    • “Show me books without ratings”
    • “Show me dining out without lunch”
We know that our NL query engine is a bridge to the time when we can implement Large Language Models (LLM), which at this point seem like our future.

First class assets

We promoted assets to be their own document records in Cosmos.

When we started years ago, Item records were the only document type and everything was packed into them. First, we pulled out relationship data (we call them edges) into their own records. Now, we did the same with assets.

It’s kind of like a normalization you’d do in traditional databases.

Asset document records will help us capture richer data around assets as we move forward and not clutter Item records with asset-related data. Think of AI work, OCR, tagging, etc. Updates can be done to each Asset record without updating Item records. While partial updates exist in Cosmos, updating an Item to update Asset info is clumsy. Plus, separate Asset records allow more easily sharing of assets between items.

Moving to Blazor

The current version of Scrapbook that we run is based on ASP.NET MVC / Web Forms.

As described here, “...ASP.NET Web Forms framework is based on a page-centric architecture. Each HTTP request for a location in the app is a separate page with which ASP.NET responds.”

On the other hand, “Blazor is a client-side web UI framework similar in nature to JavaScript front-end frameworks like Angular or React. Blazor handles user interactions and renders the necessary UI updates. Blazor isn't based on a request-reply model. User interactions are handled as events that aren't in the context of any particular HTTP request.”

We invested a lot of time in our ASP.NET Core MVC architecture with its server-side razor pages, JavaScript, and Bootstrap. This has proven to be solid for us. And usuable from desktop or mobile. But we are at a point where we can’t make changes as quickly as we’d like and the code is complicated, especially the JavaScript we’ve developed.

So, we’ve been working hard on a Blazor implementation of Scrapbook. There is a big difference between these two approaches. As it goes with new technology, you start rearchitecting and rethinking a lot of what you have done not so well 😊.

Interesting Scenarios


We covered scenarios in the post: Scrapbook Platform – A Personal Information System – Ten User Scenarios. Here we’ll mention just a few others we find particularly useful.

Organizing and documenting trips

We are now using Scrapbook to stub out upcoming trips. If work turns out to just be planning, that’s fine. We keep it in Scrapbook. Our philosophy is of it is something we spent time to prepare, something we learned about, we save it. Chances are it could be useful later.

Years later, it’s easy and gratifying to use a simple query to pull up everything related to the trip. For example, with our Natural Language query capability, we can say “Show hashtag #Turkey2023” and pull up all the related items for the trip.

show hashtag turkey2023
Query "show hashtag Turkey2023". Behind each of these image tiles is as much data about the item as we put in. Who, what, where, when and how.

Staying on the theme of Turkey, we could also query:
  • “Show museums in Istanbul”
  • “Show me hotels in Pamukkale, Turkey”
  • “What were our favorite restaurants in within 10 km of Ephesus”
We often get asked by friends for travel recommendations. Our platform makes it easier to pull out this information.


Capturing recurring themes

It’s often the case that the same subject comes up in different contexts (a museum exhibition, a talk) and it captures our attention. For example, in September 2023 we went to a museum in Spoleto where we saw Beverly Pepper sculptures. The name seemed familiar, and it was because we had saved an article 10 years ago in Scrapbook that was about her. It was very gratifying to make the connection between the new entry and the old entry.

Another example is an idea that we return to a lot in different contexts and that is “memory house”. We can links items together dealing with the subject and create a hashtag to quickly access them.

show hashtag memoryhouse - redacted Show exhibits with Hayez - redacted
Left: Query "show hashtag memoryhouse". Right: "show exhibits with hayez".


Other examples:
  • “Show me articles with democracy”
  • “Show me books about naples”
  • “Show me items with Pythagoras”.

Collection of ideas

We’ve started thinking that our platform is really a “research platform”. The things it contains could be about physical things, but just as easily about ideas or anything really.

Take our category “Books”. It includes books we’ve read and own and ones we haven’t read or don’t own. For example, when friends recommend books to us, and we make notes on them, and may read them, but always save any information as a “book” entry. Also, we often read a lot about a book and understand its major themes but never actually read it. We make a note of all that information and save it as a book entry. This is the sense of research platform that we are talking about.

Another example is our “Botanical” category. It contains plants we own, we owned and don’t anymore, and ones we may have seen in a garden somewhere. The idea is to give all these plants an “entry” in Scrapbook.

show books - redacted - card view  show books in 2005
Left: Query "show books" in card view. Right: Query "show books in 2005" in list view.


show botanical - image view
Query "show botanical" in image view.


Other example queries:
  • “Show books read in 2005”
  • “Show plants in the family Crassulaceae”

Museum visits are more interesting

We like going to museums. We often find that we are standing in front of a work (say a painting) and wondering if we’ve seen it before. Often, we think yes, we have. But how to know that? We take out our phones and search for the artist's name in our “exhibition” category. If we find that we have something about the artist, it makes the museum experience richer for us. A connection is made. It’s like having a personal reference guide and context always on hand.


show assets with lorenzo lotto show assets with Picasso
Left: Query "show assets with lorenzo lotto". Right:  Query "Show assets with picasso".

Other related queries:
  • “Show me exhibits with Picasso sorted by date in ascending order
  • “Tell me about museums in Milan, Italy”
  • “What are exhibits that we really liked in 2022”.
  • “Show exhibits with ‘video installation’"

Thursday, January 13, 2022

A Natural Language Query Parser for an Information Management System Called Scrapbook



Overview


Scrapbook Platform - NL Query Engine Diagram
An architectural diagram showing NL processing in an information management system.


This is a diagram of our NL QueryEngine service. It illustrates how we process natural language queries (questions posed in plain English) that are passed into the QueryEngine API from our Scrapbook web application or bot service applications (MS Teams, Alexa). The QueryEngine service generates the appropriate actions and SQL language queries to return results requested from the user’s Scrapbook collection.

The diagram reads from left to right. At left, our API accepts an NL query object which includes the user’s request as natural language text (NL Query), an application key and other parameters. The NL query text is routed to the LUIS cognitive service app for evaluation against our ML trained language model to extract the relevant ‘intents’ and ‘entities’ that we’ve prepared our model to recognize and which we use to identify an action and to construct an appropriate SQL database query.

Depending on the inferred ‘intent’, NL Query processing is routed via a specific processing pipeline (labeled above as Intent Processors). Each NL query follows a similar process flow – Starting from the Request Router, ML language analysis, routing, request parsing, query generation, query execution, language generation (summarizing the query as interpreted and the results found), and finally back to the Request Router which assembles and returns a response. Query State allows for contextual or ‘follow-on’ queries.

Various Azure cloud services including LUIS and Cosmos DB are utilized in the processing pipeline (as seen in the upper part of the diagram.)



What are we writing about and why are we writing about it?

This post is about our work on a Natural Language (NL) parser for an Information Management Platform we developed called Scrapbook. We've covered Scrapbook in several previous posts (2017 introduction, 2019 our memex, 2021 user scenarios) and we continue in this post with a focus on how we deal with NL queries. By “NL query”, we mean a question or request posed in ordinary plain language as you would to another person. Users interact with the Scrapbook platform through any of various bot channels including MS Teams and Alexa, or via our Scrapbook web application. For example: “Tell me about walks we did in Greece last summer with Mary.”


How did we end up using NL in Scrapbook?

There were two major impetuses that drove our implementation of NL, the first of which emerged organically in our implementation of a bot service as a means to access the Scrapbook platform via chat experiences such as Alexa, MS Teams, Skype, and Messenger. These experiences all intrinsically imply some level of NL interaction, the most typical implementation of which is an interrogative or a “waterfall” style dialog. An example:

"What can I help you with?" => "I'm looking for books."
"In what date range?" => "In 2015."
"Title word?" => "science"

We felt this would be cumbersome as the Scrapbook data model allows for many dimensions, and the data itself spans a broad range of possible information domains. Forcing the user through a chain of questions to probe these dimensions is unwieldy and at best, tedious for the user. We sought a more conversational and fluid ‘natural language’ user experience: "Show me books I read in 2015 with science in the title".

In order to achieve this, we architected a query processing ‘engine’ that allows us to handle free-form user queries against the depth and breadth of Scrapbook’s collections and data models. We’ll explain later how this works.

The second major phase of NL query processing development was driven by our realization that the work we were doing with bots could be more broadly applied to Scrapbook searching in general, regardless of the app, and in particular, from our web application. This created an opportunity to completely separate the NL processing logic from the bot code within which it was originally developed, and to generalize it as a web service that can be called from any Scrapbook user experience whether via a bot channel, web app, mobile, or other.

We understood that as our data model and data itself became richer and more complex, it would be increasingly challenging and expensive to build and maintain forms-based query interfaces within the application. A further challenge was ensuring that the search experience remained intuitive and efficient to use. Forms-based query interfaces may be implemented as single or multiple search boxes with dropdowns and other standard controls to refine the search. These controls remain available in our Scrapbook web application. However, we found that the NL processing and query generation capability that we were implementing in the bot service had already begun to exceed that of our forms-based queries in the app, and was at once more powerful, faster and easier to use.

So, we added an NL query option in the Scrapbook web application that called our newly generalized NL Query Engine, now exposed as a web service. Natural Language querying became almost immediately the go-to search experience in Scrapbook and has continued to evolve in both capability and robustness.



The NL Query Engine, how does it work?

Referring back to the diagram above, the first step in the NL Query Engine processing chain is ML (Machine Learning) Language Analysis, which is called from the Request Router and leverages Microsoft’s LUIS (Language Understanding Intelligence Service) in Azure.

The LUIS service accepts a training model, which is essentially a structured document containing a large number of labeled utterances (sample NL phrases) that we anticipate receiving and that we want our application to recognize and handle intelligently. To enhance ML training and to boost recognition performance, we associate a combination of machine learned, built-in, static and dynamic features, as well as sentence patterns. Once the ML training has been completed and verified, a Scrapbook ML ‘app’ is published on LUIS as a service that we call from our NL Query Engine to analyze Scrapbook user input. We perform this training and publishing process iteratively, both to introduce new functionality as well as to improve recognition performance. The Scrapbook LUIS ML app returns a JSON object which contains the intent and a set of entity features inferred from the input phrase. Depending on the returned LUIS intent and query context, the Request Router directs program flow to the appropriate ‘Intent Processor.’

The business logic within each Intent Processor may differ, but the query processing flow is always the same, passing next to the Request Parser, then to the Query Generator, Query Execution, and finally back to the Request Router.

The Request Parser accepts a LUIS object containing the intent and entities identified in the user’s NL query, a Collection Definition object enumerating the specific categories, subcategories and synonyms associated with the Scrapbook collection being queried, and optionally, a prior Request Object that we use to facilitate contextual parsing. The Request Parser extracts and builds what we call a Request Object. The Request Object holds all of the query terms and parameters that are needed to generate the actual database query. These include category, subcategory, date-time, location, field, text string, negation, sorting, and various selection parameters.

The query, “Show me wines from France in 2018 with a rating of at least 3” for example, is routed after ML Language Analysis to the ‘Find Intent ‘processor from which the Request Parser extracts the following elements and parameters:
  • Category: ‘drink’
  • Subcategory: ‘wine’
  • Geolocation: ‘({location: france}, {type: countryRegion})’
  • DateRange: ‘({1/1/2018 12:00:00 AM}, {1/1/2019 12:00:00 AM})’
  • QueryTerm: ‘with’
  • Field: ‘rating’
  • Text: ‘at least 3’

The next step in the pipeline is the Query Generator that then transforms the parsed query elements and parameters from the Request Object into a set of SQL language query fragments held in a Query Object. Continuing with our example:
  • Category: AND ( STRINGEQUALS(c.category, "Drink", true) )
  • Subcategory: AND ( (RegexMatch(c.bodyObj["type"], "^(|.*,)(|\\s)wine(,.*|)$", "i")) )
  • GeoLocation: AND ST_WITHIN(c.geoLocation, {"type":"MultiPolygon","coordinates":[[[[…]]]] } )
  • DateRange: AND ( c.itemDate >= "2018-01-01T00:00:00" AND c.itemDate < "2019-01-01T00:00:00" )
  • SearchString: AND StringToNumber(c.bodyObj["rating"]) >= 3

We assemble the SQL fragments into one or more SQL query requests that are run against the appropriate Scrapbook collection hosted in an Azure Cosmos database. Other intent processing pipelines may perform other actions.

We generate natural language fragments along the way that ‘restate’ the query as it was interpreted by our processing, and once the query has been executed, we summarize the results returned from the database whether successful or not. “Here are the 6 items found in category Drink of type Wine within France between 2018 and 2019 with a rating >= 3.”

The Intent Processor returns control to the Request Router along with the new Request and Response objects.

Finally, the Request Router saves the Request and Response objects to state memory as context for possible follow-on queries, and returns the results to the calling application as an NL Response object.



So to recap, how do we search for information in Scrapbook?

Users have two options:

  • Natural Language requests (default) – submitted from our Scrapbook web application, a web bot, or other channels including Alexa and MS Teams – are translated as described above by our NL Query Engine into SQL queries, which are run against the active Scrapbook collection stored in Cosmos DB. Search results are presented according to the application or channel. For an Alexa Spot device for example, the results are spoken. Requests such as to switch collections, change views, select a specific result, explore relationships or to ask for help are also supported via natural language.
  • Form-based searches – as is common in many web or console applications – are translated into LINQ queries which are run against the active Scrapbook collection stored in Cosmos DB (LINQ is a programming model abstraction for querying data. The LINQ syntax is translated behind the scenes into SQL by the Cosmos DB API.)
Some example user queries and resulting SQL expressions are shown below.

Background

This section describes some of the terms and components we use in the Scrapbook Query Engine service.


What is natural language processing (NLP)?
  • NLP is what computers need to do to interpret human language, how to process natural language into a meaningful action or result.
  • For example, in Scrapbook we can ask "Show me hikes we did in 2016 near Cortina d'Ampezzo". NLP is the interpretation and processing of this sentence to return a set of results that satisfy the user’s question.
  • Equally important, Scrapbook generates a natural language response describing what it did and what it found, in language easily understood by the user whether displayed or spoken.

What is Language Understanding (LUIS)?
  • Azure LUIS is a cloud-based conversational AI service that applies custom machine-learning intelligence to a user's conversational, natural language text (called an ‘utterance’) to predict and score an overall intent, and to extract and label relevant, detailed information within the text.
  • An utterance is textual or spoken input from the user, the user’s question or query.
    • In Scrapbook for example, an utterance might be "Show me album covers from the 1980s with the keyword ‘hair’."
  • We create an ‘application’ in LUIS by defining a model. Within the model we identity the features we want to recognize. There are two principal feature categories - intent and entity.
  • An intent may be a task or action that the user wants to perform.
    • For example, in Scrapbook, our intents are "find", "drilldown", "map", "count", “related”, “collections”, “sort”, "help", "debug", "select", among others.
    • In the query "What are lunches we’ve had nearby that we’ve rated at least 4?", the intent is "find".
    • Our Scrapbook ML model currently distinguishes 18 intents.
  • Entities are specific features that we train our LUIS application recognize within the utterance, akin to the parts of a sentence.
    • In the query "Show me wines we had from Italy last year with the variety primitivo", the entities we recognize are "datetime (last year)", "geography (Italy)", "category (drink)", "subcategory (wine)", "field (variety)", and "text (primitivo)".
    • Our ML model entities include category, subcategory, text, number, dimension, ordinal, parameter, query type (with, by, …), nearby, location, geolocation, datetime, and query object (when not a category).
    • We currently recognize 19 entities and roles.
  • Our Scrapbook ML model in LUIS uses a combination of built-in entities such as DateTime, machine-learned entities such as location or text, fixed lists, and dynamic lists. Category and SubCategory are examples of dynamic-list entities that we pass into the model via the API with the utterance. We do this because each Scrapbook collection has its own category definitions. This strategy allows us to achieve excellent category entity recognition across multiple collections. For examples of different collections, see the post 2021 user scenarios.
  • We refine our LUIS ML model by adding or modifying training utterances, patterns and lists in a language understanding (.lu) format file which once uploaded to our LUIS Conversation App in Azure, is used to train a new instance of our model. Once the updated model passes acceptance testing, we publish it into production.

What is Azure Cosmos DB?
  • Cosmos DB is a managed NoSQL database service that stores documents in collections that can be queried using standard SQL query language syntax. Each "item" in a Scrapbook collection is stored as a Cosmos DB document.
  • Cosmos DB is a schema-free database, which means we structure or model the data as we need for the domain of the collection it represents (see the data model),
  • Structure Query Language (SQL) queries are how we query our collections in Cosmos DB

What is the Scrapbook data model?
  • Platform / Collection / Item
    • Scrapbook collections are maintained in distinct Cosmos DB database collections.
    • A Scrapbook item is stored as a Cosmos DB document within a collection.
    • The Scrapbook platform supports one or more collections.
    • A Scrapbook collection has one or more items. For reference, our collections have thousands of items.
  • Item / Category / Subcategory (or type) & Fields
    • Scrapbook items have mandatory id, datetime, and category fields. All other fields are optional depending on the category definition. Items are organized principally by category.
    • Each category may have one or more subcategories and synonyms to enhance collection organization, flexibility and querability. For example, Category ‘activity’ might have ‘run’ as a subcategory, and ‘jogging’ as a synonym for ‘run’.
    • Each category has an associated set of fields that further define it. The category ‘book’ for example, would have an author field, which might not be used in other categories.

User Query Examples

In this section, we show an NL query and the final SQL that is run against Cosmos DB. Other examples with visual results are shown in our post 2021 user scenarios. These queries were run against our MyJournal collection which has categories that support these queries. Geolocation related polygons are shortened with ellipsis (…) here.
 

"Show me hikes"

SELECT  c.id, c.itemDate FROM c  

WHERE c.type = "scrapbookItem"  

AND  ( STRINGEQUALS(c.category, "Activity", true) )   

AND  ( ( RegexMatch(c.bodyObj["type"], "^(|.*,)(|\\s)hike(,.*|)$", "i") ) )  

ORDER BY c.itemDate DESC 


 

"Tell me about trips we took last year in Lombardy, Italy"

SELECT  c.id, c.itemDate FROM c  

WHERE c.type = "scrapbookItem"  

AND  ( STRINGEQUALS(c.category, "Travel", true) )   

AND  (c.itemDate >= "2020-01-01T00:00:00" AND c.itemDate < "2021-01-01T00:00:00")   

AND  ST_WITHIN(c.geoLocation, {"type":"Polygon","coordinates":[[[9.2592,44.6784], …, [9.2592,44.6784]]]})  

ORDER BY c.itemDate DESC 



"Show me wines rated greater than 3 from France"
  • followed by “What about red wines?”
  • followed by “What about from Napa Valley?”

SELECT  c.id, c.itemDate FROM c  

WHERE c.type = "scrapbookItem"  

AND  ( STRINGEQUALS(c.category, "Drink", true) )   

AND  ( ( RegexMatch(c.bodyObj["type"], "^(|.*,)(|\\s)wine(,.*|)$", "i") ) )   

AND  StringToNumber(c.bodyObj["rating"]) > 3   

AND  ST_WITHIN(c.geoLocation, {"type":"Polygon","coordinates":[[[2.65916,42.34262] …, [2.65916,42.34262]]]})  

ORDER BY c.itemDate DESC 

 

SELECT  c.id, c.itemDate FROM c  

WHERE c.type = "scrapbookItem"  

AND  ( STRINGEQUALS(c.category, "Drink", true) )   

AND  ( ( RegexMatch(c.bodyObj["type"], "^(|.*,)(|\\s)red(,.*|)$", "i") ) )   

AND  StringToNumber(c.bodyObj["rating"]) > 3   

AND  ST_WITHIN(c.geoLocation, {"type":"Polygon","coordinates":[[[2.65916,42.34262] …, 

[2.65916,42.34262]]]})  

ORDER BY c.itemDate DESC 

 

SELECT  c.id, c.itemDate FROM c  

WHERE c.type = "scrapbookItem"  

AND  ( STRINGEQUALS(c.category, "Drink", true) )   

AND  ( ( RegexMatch(c.bodyObj["type"], "^(|.*,)(|\\s)red(,.*|)$", "i") ) )  AND  StringToNumber(c.bodyObj["rating"]) > 3   

AND  ST_WITHIN(c.geoLocation, {"type":"Polygon","coordinates":[[[-122.29542286330422,38.26322398466052],[-122.29542286330422,38.25549854951917],[-122.28230540818015,38.25549854951917],[-122.28230540818015,38.26322398466052],[-122.29542286330422,38.26322398466052]]]})  

ORDER BY c.itemDate DESC 



"Show me items within 100 meters except museums"

SELECT  c.id, c.itemDate FROM c  

WHERE c.type = "scrapbookItem"   

AND NOT  ( ( RegexMatch(c.bodyObj["type"], "^(|.*,)(|\\s)museum(,.*|)$", "i")  

AND IS_DEFINED(c.bodyObj["type"])) )   

AND  ( ST_DISTANCE(c.geoLocation, {'type': 'Point', 'coordinates':[current lon, lat]}) < 100 )  

ORDER BY c.itemDate DESC  



"How many books did I read this year"
  • followed by "Show me a list"

SELECT VALUE COUNT(1) FROM c  

WHERE c.type = "scrapbookItem"  

AND  ( STRINGEQUALS(c.category, "Book", true) )   

AND  (c.itemDate >= "2022-01-01T00:00:00" AND c.itemDate < "2023-01-01T00:00:00")   

 

SELECT  c.id, c.itemDate FROM c  

WHERE c.type = "scrapbookItem"  

AND  ( STRINGEQUALS(c.category, "Book", true) )   

AND  (c.itemDate >= "2022-01-01T00:00:00" AND c.itemDate < "2023-01-01T00:00:00")  

ORDER BY c.itemDate ASC  



"Show me hikes last year except with Roberto"
  • followed by "Show me a map"

SELECT  c.id, c.itemDate FROM c  

WHERE c.type = "scrapbookItem"  

AND  ( STRINGEQUALS(c.category, "Activity", true) )   

AND  ( ( RegexMatch(c.bodyObj["type"], "^(|.*,)(|\\s)hike(,.*|)$", "i") ) )   

AND  (c.itemDate >= "2021-01-01T00:00:00" AND c.itemDate < "2022-01-01T00:00:00")   

AND NOT (CONTAINS(c.bodyObj["who"], "roberto", true) AND IS_DEFINED(c.bodyObj["who"]))  )  

ORDER BY c.itemDate DESC  


“Show me a map” results in a ‘map’ intent which we interpret as a command to display a map of results.


"Find me books of type reference"

Here's the SQL from the NL query:

SELECT  c.id, c.itemDate FROM c  

WHERE c.type = "scrapbookItem"  

AND  ( STRINGEQUALS(c.category, "Book", true) )   

AND  ( ( RegexMatch(c.bodyObj["type"], "^(|.*,)(|\\s)reference(,.*|)$", "i") ) )  

ORDER BY c.itemDate DESC  


In the Scrapbook web application, we can also search via form-based web controls which generate a LINQ query that in turn translates into the following similar SQL:

SELECT VALUE root FROM root  

WHERE (((true AND CONTAINS(LOWER(root["bodyObj"]["type"]), "reference"))  

AND (LOWER(root["category"]) = "book"))  

AND (root["type"] = "scrapbookItem"))  

ORDER BY root["itemDate"] DESC