Smarter AI With Smarter RAG and Reasoning

Retrieval-augmented generation (RAG) is not new now, but as AI and LLMs become mainstream, businesses are exploring how they can help further improve their products and users’ experience. RAG has made it possible, using context-aware generation.

But is RAG enough? And if you think I’m not making any sense here, let me explain using a case study I did for one of my clients.

Case Study

The client is a large real estate company with more than 20 years in the business, building huge projects worldwide. In the process of doing their business, they’ve accumulated a huge amount of knowledge in the form of physical and digital data, including project contracts, regulatory documents, architectural drawings, vendor profiles, compliance reports, project planning, case studies, and more.

The Problem

The huge amount of data (in physical and digital media) was scattered across printouts, folders, shared drives, email threads, and legacy systems. Searching through this unorganized mess was a huge challenge, and it needed a fix. Everyone agreed that we should leverage AI and LLMs to help.

We started a classic RAG architecture with a simple process:

• Chunk documents using intelligent parsers
• Embed them using the Vertex AI embedding API
• Store the vectors in Vertex AI Matching Engine
• Retrieve relevant chunks at query time and pass them into Gemini via Vertex AI

A typical RAG system that helps answer questions about their data, it was working flawlessly for straightforward, document-level questions. For example, if anyone asks, “Who handled HVAC in the 2020 Jeddah project?” Gemini could generate a fluent, factual answer based on the retrieved context.

But soon we hit a new class of questions — the kind that required reasoning across documents, entities, and relationships:

• Which vendors have worked on at least three delayed projects across different cities?
• Who’s collaborated with the same consultant on more than two projects?
• Show me all vendors involved in projects that exceeded budget and had timeline overruns.

Our RAG setup, as solid as it was, simply couldn’t answer these questions, and we had to go back on the drawing board to find a solution.

RAG works chunk by chunk. It retrieves what looks relevant. But it doesn’t connect the data to other data. It can’t chain logic, so it can’t see that a vendor linked to a project is also linked to other projects, projects with some similarities.

That’s when we realized we needed reasoning. Finding the right chunk is good, but connecting the dots is smarter and what we needed.

The Reasoning Layer

To bridge the gap, we added a reasoning layer using Neo4j, a graph database built for representing relationships. This changed everything. With Neo4j, we created a graph of the real estate domain:

• Nodes: Vendors, projects, cities, consultants, contracts, delayed events
• Relationships: WORKED_ON, LOCATED_IN, DELAYED_BY, SIGNED_WITH, OVERSEEN_BY

This structure allowed us to move from flat retrieval to relational thinking. We could now answer:

• Who did what, where, and when
• What roles individuals played
• How different vendors were connected via shared consultants or project timelines
• Which relationships formed patterns of risk or success

Introducing the reasoning layer was a great help. Here are some of the outcomes we saw within weeks of launch:

• 80-percent reduction in time-to-answer for complex questions
• 70 percent of internal queries handled without human escalation
• 90-percent accuracy in multi-hop responses when compared to SMEs
• More trust in AI-generated answers thanks to traceable logic
• System adopted by three other departments (finance, compliance, engineering)

The users have described it as finally getting a version of AI that understands our business.

Under the Hood

During our document processing pipeline, alongside chunking and embedding, we also performed entity extraction and relationship tagging.

Here’s an example from a contract document:

Vendor: ABC Ltd
Project: Jeddah Mall
Location: Jeddah
Year: 2020
Delay: 3 months
Consultant: Delta Group

We stored these as graph relationships:

MERGE (v:Vendor {name: "ABC Ltd"})
MERGE (p:Project {name: "Jeddah Mall", year: 2020})
MERGE (c:Consultant {name: "Delta Group"})
MERGE (l:Location {city: "Jeddah"})
MERGE (v)-[:WORKED_ON]->(p)
MERGE (p)-[:LOCATED_IN]->(l)
MERGE (p)-[:DELAYED_BY]->(:Delay {duration: "3 months"})
MERGE (p)-[:OVERSEEN_BY]->(c)

At query time, when a user asked a simple query that needed a relationship lookup (“Which vendors had delays on multiple projects in different cities?” for example), we queried Neo4j:

MATCH (v:Vendor)-[:WORKED_ON]->(p:Project)-[:DELAYED_BY]->(d:Delay),
      (p)-[:LOCATED_IN]->(l:Location)
WITH v.name AS vendor, count(DISTINCT p) AS projects, 
      collect(DISTINCT l.city) AS cities
WHERE projects >= 3 AND size(cities) > 1
RETURN vendor, projects, cities

This gave us a factually grounded context that we passed into Gemini to generate a fluent response.

More Examples and Use Cases

Multi-Project Risk Analysis

List vendors involved in three or more delayed projects:

• Queried graph to count delayed relationships
• Used results as Gemini context
• Business users gained visibility into repeat risk patterns

Consultant-Centric Project Mapping

Show all projects Delta Group consulted on:

• Traversed graph with Consultant → Projects
• Enriched output with location, timeline, and outcomes

Cross-Entity Insights

Find vendors who worked on over-budget and late projects:

• Combined DELAYED_BY and OVER_BUDGET edges
• Helped procurement teams avoid risk clusters

Compliance Correlation

Which contractors were cited in projects with compliance issues?

• Matched vendor relationships to flagged compliance events
• Linked to regulatory filing metadata

Each of these use cases required connecting entities and filtering relationships across dimensions, projects, time, people, and outcomes, which was impossible to do with pure RAG but becomes effortless with graph.

Lesson Learned

This project has taught us a few things that are worth sharing:

• Embeddings aren’t enough. They help find similar text, but fail at logical connections.
• Graph databases add structure to what RAG can only guess.
• LangChain + Neo4j + Gemini = a production-grade reasoning system.
• Grounded prompts win. Always provide real data context to your LLMs.
• Smarter AI isn’t just bigger models — it’s smarter retrieval and reasoning.

Should You Add a Reasoning Layer?

The short answer is no, unless your business and solution involves any of the following:

• Relies on relationships (between people, projects, events)
• Needs traceable, explainable AI outputs
• Has domain-specific knowledge that spans documents and systems

Adding a reasoning layer will make your AI smarter, more reliable, and more trustable. It’s a force multiplier that elevates your stack from helpful to insightful.

Final Thoughts

While building, we explored so many tools, techniques, and technologies we can leverage to become faster and efficient. Here’s what we ended up with:

• Google Cloud Storage for document intake
• Vertex AI for embeddings and LLM generation
• Matching Engine for vector search
• Firestore for metadata filtering
• Neo4j for relationship modeling and multi-hop reasoning
• LangChain for orchestration

If you’re building a knowledge system that needs to go beyond just retrieval, start thinking about reasoning. The real goal isn’t just finding data — it’s helping your business understand it.

Want to learn more or see it in action? Reach out to me on LinkedIn. I’m happy to discuss!

---

Smarter AI With Smarter RAG and Reasoning was originally published in Neo4j Developer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.