I've undertaken a few ambitious tech projects in my time. But perhaps none was as grueling as the month-long process of taking a home inventory using the excellent open source project Homebox.

My home office had reached a point of non-functionality. Trying to find anything I needed among my sprawling collection of tech things was proving essentially impossible. When it became quicker to buy cables fresh than find the ones I already had, I knew that my current system was broken, wasteful and needed changing.

The process of creating a home inventory using Homebox is a laborious one, but has ultimately proved highly efficient. While I initially bought a laser printer and used QR codes to record my inventory, I later switched over to using NFC tags.

As crazy as the system is, it undeniably works. I can now use Homebox to look for anything I need, find its box number and retrieve it all in about 10 seconds.

However, with so many advances in AI over the past year, I couldn't help but think that it would be pretty cool if I could find some way to connect a large language model to my inventory. That way I could sort of avoid the Homebox UI entirely and simply use natural language in a chatbot interface to interact with an agent.

Distlaimer, I haven't quite yet reached that objective. The tools required to get models to interact with APIs are in their relative infancy and more presciently, my Homebox instance is secured behind a Cloudflare network and I don't want to go through the hassle of figuring out how to safely provide remote API access to it. But here is a proof of concept that shows that it can work.

The limitation here is that this is a static system, and therefore, if you're regularly updating your inventory, you would have to go through the trouble of deleting the previous context data and uploading a new file to simulate overriding the data. But here is at least a starter system.

Export Inventory, Embed In Vector Database (RAG)

The first thing you'll want to do is export your inventory from Homebox. This will download the data as a CSV.

Context-enhanced agents rely on RAG in order to serve up relevant data to the user. I decided to give the vector database a bit of a head start by going through the CSV export and manually deleting all the columns that did not contain data and I thought might confuse the agent.

For the same reason, I also did a little bit of reformatting of the column header names in order to make what they were more obvious to the AI:

RAG still being a fairly early stage technology, you may have to play around with the download format in order to find a file type that embeds well into your vector store.Although it doesn't make much sense for a data file, you could try using PDF or HTML if the embedding in CSV doesn't work nicely.

Next, I created an agent in LibreChat in order to be able to interact in a chat interface with my inventory store. I chose Sonnet 3.5 as the underlying model, but it would probably make more sense to choose a less powerful LLM for this relatively rudimentary task.

The type of agent that can be created in systems like LibreChat involves taking a stock large language model, adding a system prompt, usually described as instructions, and then connecting them to their specific contextual knowledge source.

The system prompt doesn't have to be amazing, here's one that I wrote that works reasonably well.

Now it's time to connect the agent to the knowledge. Use the Upload for File Search button in LibreChat in order to allow access and make sure that you tick the Enable File Search functionality so that the agent has permission to access the data.

When you can see the file name display, the process has been successful and your agent can now access the inventory:

Make sure to click on save to make sure that the knowledge has been added:

Now our chatbot is ready for use and we can ask you questions about our inventory:

You can see that it's checking the files when asked a question:

It was able to identify all the infrared controllers in my tech cabinet and isolate the one that I was looking for, number 3!

You can ask follow-up questions to retrieve details about the thing you're looking for: