The work since then has been about something harder: making a system that works well enough that you actually trust it with real questions about real documents.

Ten build sessions. 17 Docker containers. A vector store holding over 5,600 documents spanning more than a decade of financial and personal paperwork. Here is an honest account of the problems encountered, the false starts, and what ended up working.

The Problem With "Working"

After the initial RAG pipeline was running, it looked impressive. You could ask a question and get an answer. But live testing against real documents revealed a pattern anyone who has built a production RAG system will recognise: it worked when you already knew what to expect, and failed precisely when you needed it most.

Three specific failure modes drove most of the work that followed.

Documents that existed in the system were not being retrieved. The retrieval logic was too narrow. Keyword extraction only identified formal proper nouns, missing common domain terms that appeared in conversational queries. Search was capped at a handful of candidates per term, and both the keyword and semantic tracks used a flat merge that scored every keyword hit equally regardless of specificity or frequency.

The system was confusing document dates with the current date. The system prompt included today's date for context. When a user asked "what date is the offer valid until", the model blended today's date with the date field it found in the document — and reported the wrong answer with confidence. There was no explicit instruction preventing date substitution.

Rich structured metadata was completely invisible to retrieval. The AI was extracting valuable structured information from every document — document type, sender category, financial keywords, summaries — and storing it in a separate structured database. But none of that information was being placed into the vector index. Queries that depended on that structured data returned nothing, even when the information clearly existed.

Each failure exposed a fundamental design assumption that needed correcting. The most important of these — the invisible metadata problem — also had the largest downstream impact.

The Invisible Data Problem

Figure 1: Before and after the metadata fix — AI-extracted structured fields are now appended to document content before chunking and embedding, making them findable at query time

The document processing pipeline fetched each document's text content, split it into overlapping chunks, generated vector embeddings for each chunk, and stored them in the vector database. This is standard RAG practice. The problem was what it was not doing.

The document management system stored AI-extracted metadata in eight custom fields per document — property type, sender category, document keywords, structured summary, and more. The processing pipeline fetched this data as part of its document retrieval call — and then silently discarded it, indexing only the raw OCR text.

The fix was architecturally simple but operationally significant: append all non-empty custom fields as a clearly labelled metadata block at the end of each document's content before chunking. Every vector chunk generated from that document then carries the metadata context. A query for a specific financial figure or document category reaches the metadata block directly.

The same field mapping must be maintained in two separate places — the webhook-triggered document processor and the bulk backfill script — because they follow different code paths. A new field added in one place but not the other creates a silent gap in index coverage.

The critical lesson: a document AI system is only as good as what you actually place in the index. Rich structured information sitting in a separate database is functionally invisible to any retrieval system that does not know to look there. This single fix resolved more user-facing query failures than any subsequent retrieval algorithm improvement.

Backfill at Scale: When Parallelism Exposes Hidden Assumptions

Figure 2: Self-healing backfill architecture — parallel throughput phase catches most documents; the coverage check and sequential retry phase guarantees completeness

Fixing the invisible metadata problem required rebuilding the entire vector index — all 5,600 documents needed to be reprocessed through the corrected pipeline. A parallelised approach using 16 concurrent workers processed documents much faster than a sequential approach — and immediately exposed a capacity problem that sequential processing had been hiding.

Under sustained concurrent load, the document management backend began timing out on content fetch requests. The retry mechanism exhausted its attempts for 181 documents, which were logged as failures. A post-backfill coverage check revealed the scale of the gap: nearly 400 documents were absent from the vector index compared to the source system.

A second issue compounded the problem. The backfill orchestration script was supposed to increase the AI model server's parallelism setting before running. This was silently failing — the script was editing the base systemd service configuration file, but the parallelism setting actually lives in a drop-in override file in a different directory. The change was applied to the wrong file, reported success, and the service restarted unchanged.

The corrected orchestrator addresses both issues. The parallel phase handles throughput. After it completes, the orchestrator computes the difference between all document IDs in the source system and all document IDs present in the vector index, then sequentially retries every gap until coverage is complete. It also explicitly verifies that the model server configuration change took effect before starting.

The broader pattern: parallelism surfaces capacity assumptions that serial processing hides. A system that reliably processes one document per second may fail at sixteen simultaneously — and the failure mode is often silent data gaps rather than visible errors.

Upgrading Retrieval: From Text Matching to Hybrid Search

Retrieval did not get fixed in one step — it went through four distinct generations over the course of this project, each one addressing the failure modes the previous version left behind. The diagram below shows the end state: the complete query pipeline as it stands today.

Figure 3: The complete Herald query pipeline — LLM analysis, hybrid BM25 + dense search, cross-encoder reranking, and guardrailed generation

Before getting into how each component works, it is worth understanding the journey that led here. The system started with basic substring keyword matching and progressively replaced each weak link — first with real BM25 scoring, then with semantic embeddings, then with a cross-encoder reranker that scores query-document pairs directly. Each generation fixed a specific class of failure. The diagram below maps that evolution.

Figure 4: Four generations of retrieval — what each approach got right, what it left broken, and how many chunks ultimately reached the LLM at each stage

The central insight that drove most of these changes: dense semantic embeddings and sparse BM25 vectors are not competing approaches — they catch fundamentally different types of queries. Neither is sufficient on its own. The diagram below shows exactly what each method handles well and where it fails, which is why the architecture uses both together.

Figure 5: Dense vs sparse BM25 vectors — what each method catches and misses, and why hybrid search is necessary for real document retrieval

The Problem With the Original Keyword Track

The original keyword retrieval was not true keyword search — it was substring scanning with no understanding of term importance. Every occurrence of a common term across thousands of documents scored equally, whether it appeared once in a passing reference or fifty times in a directly relevant document. There was no concept of inverse document frequency — how rare a term is across the corpus — which is what makes keyword search precise.

The consequence: queries for common terms retrieved too much, while queries for specific reference numbers or financial figures retrieved too little.

BM25 Sparse Vectors

The replacement uses true BM25 scoring — the same algorithm that powers traditional search engines. Each document is represented as a sparse vector where each dimension corresponds to a unique token in the vocabulary, weighted by term frequency and inverse document frequency. Common words receive low weights; rare, distinctive terms receive high weights. This is generated at index time via a CPU-only library that runs in under a millisecond per query — fast enough to run on every request without impact.

The BM25 sparse vectors are stored alongside the dense semantic embeddings in the same vector database collection, and both searches run simultaneously as a native database operation. This is architecturally cleaner than running separate searches and merging results in application code.

Dense Semantic Embeddings

The dense embedding model encodes documents as high-dimensional floating-point vectors in a semantic space, where documents with similar meaning are positioned close together regardless of whether they share the same words. This means a search for "invoice" finds documents containing "bill" or "statement", and a natural language question about payment due dates finds payment schedule sections even without exact wording matches.

Dense search handles paraphrasing and conceptual similarity well, but fails on exact terms — a specific policy reference number has no semantic neighbours, so dense search will not find it reliably.

Reciprocal Rank Fusion and Cross-Encoder Reranking

The two search legs are combined via Reciprocal Rank Fusion — a method that merges rank positions from multiple result lists without requiring score normalisation across different scales. Documents appearing in both the sparse and dense results receive a significant scoring boost. The output is approximately 20 candidate chunks.

These candidates then pass to a cross-encoder reranker. Unlike the bi-encoder approach used for retrieval — where query and document are encoded separately and compared by cosine distance — a cross-encoder reads the query and each candidate document together in a single forward pass through a transformer model. This produces a much more accurate relevance score that directly reflects the relationship between what was asked and what the document contains.

The reranker was upgraded midway through the project from a 6-layer to a 12-layer model, improving precision meaningfully at acceptable CPU latency. An adaptive bypass was added: when multiple metadata filters are already applied to the search, the candidate pool is tight enough that reranking adds no value, and the step is skipped entirely — reducing response time from approximately 2 seconds to near-instant for filtered queries.

Only the top 5 reranked chunks reach the LLM. Fewer, better-selected chunks produce more accurate answers — the LLM focuses on directly relevant information rather than sifting through tangentially related content from other documents.

One final dimension of retrieval quality that is easy to overlook: how documents are chunked and what goes into each chunk matters just as much as which algorithm searches them. The diagram below captures this — including why the metadata embedding fix described earlier had such a disproportionate impact compared to algorithm changes.

Figure 6: How chunking strategy and what goes into each chunk determines retrieval quality — the metadata embedding fix, and why dense embeddings alone are insufficient

The Rules-Based Trap: Entity Resolution

Figure 7: Rules-based entity resolution vs LLM-native query analysis — everything the rules-based approach required, and what replaced it entirely

A core capability the system needs is entity resolution: mapping what the user says to the structured entities that actually exist in the document management system. When a user asks about a specific document type or property, the system needs to understand which tag, correspondent, or document type in the catalogue they mean — and apply that as a precise filter on retrieval.

Version 1: Rules-Based Matching

The first implementation used a fuzzy matching engine with manually tuned confidence thresholds, stop word lists, minimum token length rules, acronym detection, a configuration file for keyword aliases, and a learned mappings file that persisted to disk. This system worked for the cases it was designed for, and failed for every edge case it had not seen. Short words triggered substring collisions with unintended entries. Phrases were learned as complete mappings rather than decomposed into components. Every new failure mode required a code change, a rebuild, and a redeployment.

The system accumulated approximately 200 lines of heuristics, two persistent files, and a configuration format — and still required constant maintenance as new query patterns emerged in production.

Version 2: LLM-Native Analysis

The replacement is a single call to the local LLM before any retrieval takes place. The model receives the user's raw query alongside the complete live catalogue — every tag name, document type, and correspondent name — fetched directly from the document management system at startup and refreshed on demand via an admin endpoint without requiring a container restart.

The model returns a structured analysis: a rewritten version of the query with context resolved, the exact matching values from the catalogue if applicable, any date range implied by the query, whether the query is exhaustive or looking for specific information, and a clarification question if the query is genuinely ambiguous.

A safety layer validates every returned entity value against the live catalogue before applying it as a filter. If the model returns a value that does not exist in the system, that filter is silently discarded and retrieval proceeds without it, rather than producing an empty result set. The LLM handles synonyms, abbreviations, partial names, and edge cases naturally. No code changes have been required for new query patterns since this version was deployed.

The Exhaustive Query Problem

Vector search is a ranking operation — it returns the most relevant K documents from a much larger set. This is the right behaviour for targeted questions. It is the wrong behaviour for exhaustive questions: "find everything matching these criteria."

When a user asks to find all documents for a particular property, or list every invoice from a specific supplier, a top-K retrieval system returns the highest-scoring results and stops. If there are 66 matching documents, 61 of them are invisible. This was discovered in testing when a query designed to retrieve all documents for a property returned 5 results, while a direct database count showed 66.

The solution routes exhaustive queries — detected by the LLM analysis step — to a completely separate code path. Instead of vector search, this path uses paginated scrolling through the vector database with payload filters applied. Every page of results is collected, document IDs are deduplicated, and the full set is returned with no ceiling. The property matching uses a five-tier filter cascade to handle OCR inconsistencies, partial addresses, and documents that reference a property without being explicitly tagged to it.

Automatic Data Extraction: Hardening the Pipeline

Figure 8: Webhook idempotency handling and flood protection — the state machine for managing multiple events per document and protecting against bulk operation overload

Running alongside the retrieval improvements was a separate service that automatically classifies incoming documents and extracts structured financial data from them. Every document that arrives triggers a two-stage AI pipeline: classify the document type, then extract the relevant structured fields. Extracted records enter a human review queue before being committed to the database — the AI does the extraction work, the human approves, corrects, or discards.

The Webhook Storm Problem

The document management system notifies downstream services of document changes via webhooks. The AI metadata enrichment process writes each extracted field one at a time — each write triggers a new webhook event. A single document being enriched generates between five and eight separate notification events in rapid succession.

A naive extraction handler processes each event independently, meaning each document triggers up to eight extraction attempts. The handler needed an explicit state machine: approved records are always skipped; discarded records are reset and re-extracted; pending records with high extraction confidence are treated as duplicates; pending records with low confidence are re-extracted because later events carry more complete metadata.

A second problem emerged during bulk operations. Tagging fifty documents simultaneously triggered approximately 300 webhook events in seconds. Without concurrency control, the system attempted 300 simultaneous AI calls, saturating the model server and leaving the event loop unable to respond to any other requests. A concurrency limiter restricts extractions to match the model server's actual capacity, with tasks queuing in application memory rather than flooding the model.

LLM Output Defensiveness

Every LLM integration in a production system requires explicit handling for the failure modes the model exhibits in normal use. The extraction pipeline encountered two recurring patterns: the model occasionally returns a list containing the extracted object rather than the object directly, causing downstream code to fail; and the model sometimes truncates its output at the token limit mid-string, producing unparseable output. Both require explicit recovery logic. A failed parse writes a placeholder record flagged for manual attention rather than silently dropping the document.

The classifier prompt was also hardened after observing production misclassifications. Quotes and estimates were being classified as completed transactions because both involve financial figures. The distinction between an unexecuted financial proposal and a completed payment was made explicit in the prompt, with worked examples of each category.

A SQL Aggregation Bug

The financial dashboard query joined issues and transactions on their shared parent property in a single query. A property with four open issues and three recorded transactions produces twelve rows — a cartesian product. Every transaction was being counted four times, and the dashboard displayed financial totals that were multiples of the correct figures. Pre-aggregating transactions in a subquery before joining resolved this. A standard SQL pattern, but one that is easy to miss when building incrementally and adding joins over time.

What the Stack Looks Like Now

Seventeen containerised services. The query path for a user question:

•        LLM query analysis resolves entity references against the live catalogue, rewrites the query for clarity, identifies whether the query is exhaustive or specific, and extracts any date range or identifier mentioned

•        Exhaustive queries route to a paginated scroll with payload filters — no ranking ceiling, complete result sets

•        Specific queries run hybrid BM25 sparse and dense semantic search simultaneously, with server-side rank fusion producing approximately 20 candidates

•        Cross-encoder reranking scores every candidate against the query — skipped entirely when metadata filters have already narrowed the pool sufficiently

•        The top 5 reranked chunks reach the LLM with a guardrailed prompt that explicitly prohibits date substitution, prohibits volunteering unrequested information, and includes direct links to source documents

•        The response is streamed to the browser and rendered from raw text to formatted markdown on completion

The document ingest path:

•        Documents arrive in the document management system and are enriched by a vision-capable AI model for OCR, title generation, tagging, and custom field extraction

•        Each document change fires webhook events to three parallel services: vector indexing with embedded metadata, deadline and calendar event extraction, and financial document classification

•        Extracted financial records enter a human review queue; approved records populate a structured property database with financial year filtering and export

Honest Observations

RAG quality is predominantly a data quality problem. The largest single improvement in query reliability came from embedding structured metadata into the vector index — not from upgrading the retrieval algorithm. The algorithm can only find what you put in front of it.

LLMs require explicit failure mode handling. Every LLM integration in this system required targeted prompt engineering against specific failure patterns observed in production: date blending, type misclassification, malformed output structure, hallucinated entity names, truncated output. These are not edge cases. They appear in normal daily use.

Rules-based logic accrues maintenance debt rapidly. The fuzzy entity resolution system required a code change for every new query pattern it had not seen. The LLM-native replacement has required zero code changes for new patterns since deployment. The maintenance cost difference compounds over time.

Human review is the right default for structured data extraction. Committing AI-extracted financial data directly to a database creates errors that compound silently. A review queue with good edit tooling is minimal additional friction and eliminates an entire category of data quality risk.

Parallelism reveals capacity assumptions that sequential processing hides. The backfill coverage gap only appeared when running sixteen concurrent workers. At one document per second, the backend handled the load without issue. Parallel failure modes tend to be silent data gaps, not visible errors.

The home lab remains the best learning environment: real hardware, real data, real failure modes, no safety net. The engineering problems encountered here are exactly the problems that appear at production scale. Building against real constraints is the only way to encounter them.

What's Next

Email ingestion via a self-hosted automation engine — whitelisted senders only, email bodies treated as text documents, attachments routed through the existing pipeline, calendar events extracted automatically from email content. The existing RAG pipeline, extraction service, and calendar integration require no changes — email content flows through them automatically once it is present in the document system.

The core system is stable. The next phase is expanding what it ingests.

herald-ai.hashnode.dev

#AIInfrastructure #RAG #LLM #DocumentAI #Docker #VectorSearch #PrivateAI #HomeLab #MachineLearning #BM25 #RetrievalAugmentedGeneration

A hands-on story about building a local AI document intelligence stack, the promise of NVIDIA NIM, and what the RTX 50 series Blackwell architecture actually means for AI practitioners today.

I spent today trying to run NVIDIA NIM containers on my home lab machine — an Acer Predator laptop with an RTX 5090 (24GB GDDR7), 192GB ECC RAM, running a full Paperless document management stack on Docker Desktop with WSL2 Ubuntu for Ollama underneath.

The short version: NIMs don't run on an RTX 5090 yet. And the reasons why are genuinely interesting.

But let me start at the beginning.

What I Was Actually Building

I run Paperless-ngx at home for document management — utility bills, insurance documents, bank statements, the usual. On top of that I've built a small AI stack that uses an LLM to OCR my scanned documents, extract metadata, auto-title everything, and feed it all into a Qdrant vector store for RAG queries later.

The stack currently looks like this:

• Paperless-ngx — document ingestion and management

• paperless-gpt — LLM-based OCR and document intelligence

• Ollama (WSL2 Ubuntu) — serving the LLMs, talking to paperless-gpt via Windows port forwarding

• Qdrant — vector database for RAG

• Custom RAG processor — webhook receiver that embeds documents on ingest

It works well. But I wanted to compare NVIDIA NIM as a serving backend against Ollama — partly to understand the performance difference, partly because understanding the NVIDIA enterprise AI stack is genuinely useful for anyone working in this space professionally.

So I set up a second docker-compose-nim.yml — a completely separate NIM experiment stack that would run a second instance of paperless-gpt pointing at NIM instead of Ollama, differentiated by separate document tags so the two instances would never conflict.

Clean design. Good learning objective. Immediately hit interesting problems.

NIM vs Ollama — The Architecture Difference That Matters

Before getting into what went wrong, it's worth understanding what NIM actually is, because it's meaningfully different from Ollama in ways that matter.

Ollama is flexible. You run a server, pull any compatible model, swap between them with a single command. It uses llama.cpp under the hood, which handles quantisation and CPU offloading — so on my 192GB RAM machine, I can run a 70B model with most layers on GPU and overflow to RAM. The ceiling is effectively much higher than my 24GB VRAM alone would suggest.

NIM is purpose-built. One container = one model. Each NIM image is compiled specifically for that model architecture and optimised for target GPU hardware using TensorRT-LLM or vLLM. There's no model swapping, no CPU offloading. The model must fit entirely in VRAM. What you trade away in flexibility you're supposed to gain in inference performance — better batching, paged attention, lower latency at scale.

For enterprise deployment on dedicated GPU infrastructure, this is the right trade-off. For a learning/dev environment on a laptop? The rigidity starts creating problems immediately.

Problem 1: GPU Display Memory

First attempt — NIM failed with:

Free GPUs: <None>
Non-free GPUs: NVIDIA GeForce RTX 5090 Laptop GPU [current utilization: 17%]

NIM requires a free GPU — one with essentially no VRAM consumed. My 5090 was in "NVIDIA GPU only" mode in the Acer Predator settings, meaning it was driving the display AND handling compute on the same GPU. The display output was consuming ~4GB of VRAM. That was enough for NIM to refuse to start.

Fix: Switch the laptop to Optimus mode. In Optimus, the Intel iGPU handles display output and the 5090 is reserved purely for compute. After a reboot, NIM saw a free GPU and moved on.

This is a real architectural difference worth knowing: NIM is designed for dedicated GPU serving environments. It assumes the GPU isn't doing anything else. On a workstation or data centre node, this is always true. On a laptop where the GPU also drives the screen, you need to engineer around it.

Problem 2: The RTX 5090 Is Too New

This is the more interesting failure, and the one with broader implications.

After switching to Optimus, NIM started — selected a profile (vllm-bf16-tp1), downloaded ~22GB of model files and runtime components, started the vLLM process, began loading the model... and then crashed with:

NVIDIA GeForce RTX 5090 Laptop GPU with CUDA capability sm_120 is not compatible
with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75
sm_80 sm_86 sm_90.
RuntimeError: CUDA error: no kernel image is available for execution on the device

The RTX 5090 is Blackwell architecture — CUDA compute capability sm_120. Current stable PyTorch supports up to sm_90 (Ada Lovelace, i.e. RTX 4090 generation). NIM containers are built on stable PyTorch. Therefore: no NIM on RTX 5090.

This isn't specific to the model I was trying. It affects every current NIM container — because the PyTorch incompatibility is at the container base level, not the model level.

The irony? Ollama works fine on the same GPU. llama.cpp — the engine Ollama uses — added Blackwell CUDA support through a different compilation path, independently of PyTorch, and did so faster. So the older, more flexible tool runs on the newest hardware, while the enterprise-grade optimised serving platform can't.

What I Actually Learned About the NIM Stack

Even though I couldn't get inference running, going through this process taught me things I wouldn't have understood from reading documentation:

1. NIM's first-run process is substantial. It's not just pulling a Docker image. On first start, NIM performs GPU profile detection, downloads model weights separately from the container image (~15-22GB), and for supported GPU architectures, compiles TensorRT engines specific to your hardware. The downloaded artefacts are cached so subsequent starts are fast — but that first run is a genuine installation process, not just a container start.

2. Profile selection is intelligent. NIM inspects your GPU and selects the best inference profile — TensorRT-LLM for supported architectures (pre-compiled engine, maximum performance), vLLM as fallback for others. In my case it selected vllm-bf16-tp1 because there's no prebuilt TensorRT engine for Blackwell yet. This auto-selection is genuinely useful in enterprise contexts where you're deploying across heterogeneous GPU fleets.

3. The GPU exclusivity model is a deliberate design choice. NIM's requirement for a dedicated, unshared GPU isn't a bug — it's the architecture. In production you'd provision dedicated GPU nodes for inference. The constraint only feels strange in a dev/learning environment where the same GPU is doing multiple things.

4. Per-model EULA acceptance is part of the governance model. Having NGC Catalog permissions in your API key isn't sufficient to pull NIM containers. Each model requires explicit terms acceptance on the NGC website. This is NVIDIA's model governance approach — fine-grained control over which models your account can access. Worth knowing before you try to automate NIM deployments.

5. NIM separates container from model artefacts. The Docker image contains the runtime framework. Model weights are downloaded at first start and cached separately. This means NVIDIA can update the runtime independently of the model, and you can switch GPU profiles without re-downloading weights. It's a cleaner architecture than bundling everything into a single image.

The Honest Comparison

After today, here's where I'd actually characterise the two approaches:

Ollama strengths:

• Works on newer hardware faster (llama.cpp moves quicker than PyTorch stable)

• CPU offloading — on a machine with 192GB RAM, I can run models much larger than VRAM alone

• Model flexibility — swap models with an env var change

• Lower friction for home/dev environments

NIM strengths (when it works):

• TensorRT-LLM optimisation — meaningfully faster inference on supported hardware

• Paged attention — more efficient KV cache management at long context lengths

• Production-grade batching and concurrency

• OpenAI-compatible API — drop-in replacement for application integration

• Enterprise governance model built in

The key insight: Ollama and NIM aren't really competing for the same use case. Ollama is the right choice for development, experimentation, and home labs. NIM is designed for production serving on dedicated, enterprise-grade infrastructure. Trying to run NIM on a cutting-edge consumer laptop is a useful experiment precisely because it reveals these assumptions.

What's Actually Running on My RTX 5090 Right Now

My Paperless stack is producing good results with Ollama and ministral-3:14b as the OCR model. I know from testing that 14B parameters is the floor for reliable OCR quality on my document types — smaller models consistently produce errors that make the whole pipeline unreliable.

The 5090 with 192GB system RAM means I could push to 70B models via Ollama's GPU+RAM offloading — something NIM's hard GPU-only constraint would prevent. That's an interesting architectural point: sometimes the "less optimised" serving infrastructure gives you more headroom in practice.

The RAG layer is next — a chat interface over my document store. And voice after that.

The Broader Point for Anyone Building on New Hardware

If you're running RTX 50 series (Blackwell) hardware and trying to use PyTorch-based frameworks: you're in a gap period. The hardware shipped, drivers work, llama.cpp works, CUDA toolkit 12.8 supports sm_120 — but stable PyTorch doesn't, and everything built on stable PyTorch (vLLM, standard NIM containers) inherits that limitation.

The path forward is either:

• PyTorch nightly builds with cu128 support (less stable, more work)

• Building custom containers from NVIDIA's PyTorch nightly base images

• Waiting for stable PyTorch + NIM to catch up (probably a few months)

This is normal early-adoption friction. It happened with every previous GPU architecture. But it's worth knowing before you invest time trying to run enterprise inference frameworks on day-one consumer hardware.

Takeaways

This wasn't a day where everything worked. It was a day where I understood why things didn't work, which is usually more valuable.

I came away understanding NIM's architecture at a level I couldn't have reached from documentation alone — the profile selection system, the separation of container and model artefacts, the GPU exclusivity model, the governance layer, and exactly where the Blackwell compatibility gap sits in the stack.

The home lab is the best kind of learning environment: real hardware, real constraints, real failure modes. No sandbox, no pre-configured environment, no safety net. Just docker logs and patience.

Next up: RAG chat interface and voice. More soon.

Building this on: Acer Predator | RTX 5090 Laptop 24GB | 192GB ECC RAM | Docker Desktop | WSL2 | Paperless-ngx | Ollama | Qdrant

Tags: #NVIDIA #NIM #Ollama #LLM #RAG #AIInfrastructure #MachineLearning #HomeLab #Docker #RTX5090 #Blackwell