Why do AI chatbot projects fail security approval?
AI chatbot projects usually stall because the sponsor cannot show where internal data will go. The support or knowledge team wants to put internal documents behind an AI assistant. Legal wants to know whether those documents ever leave the company.
For most companies, a SaaS chatbot hosted in Japan, with encryption in transit and a commitment not to train on customer data, is enough to get approval. We covered the questions to ask a SaaS vendor in [5 security questions to answer before approving an AI chatbot](/en/blog/ai-chatbot-security-questions-enterprise).
This article is for the remaining cases: internal policy, client contracts or regulators that do not allow data onto the internet at all. These are common in finance, healthcare, government and among the SIers who build systems for them.
What has to run inside the network for "no data leaves"?
A RAG chatbot (Retrieval-Augmented Generation: it finds relevant passages, then has an AI write the answer from them) only keeps data inside when all five components below run inside your network.
Some products marketed as "on-premise" keep components 1 to 3 inside but still call an external LLM API to write the answer. The user's question and the matching passages then travel over the internet. Ask about each component separately instead of asking "is it on-premise?".
How do you verify that a system sends nothing out?
A promise in a brochure is not verification. Before acceptance, ask the vendor to complete three steps:
1. List every outbound connection in the design. Each one needs a reason, a destination and the type of data it carries. The shorter the list, the easier the approval.
2. Scan the source code and configuration. Find every environment variable, API key and external service address. The scan result must match the list from step 1.
3. Run the system with all outbound traffic blocked at the firewall. If it still answers normally, nothing depends on the internet behind your back.
VAON ran step 2 on one of its own products: an on-premise invoice and contract management system built in 2026. A scan of the full codebase for external service credentials found 6 environment variables, all for sending email over SMTP (measured Aug 24, 2026). That system is not a chatbot, but the method is the same. For a RAG chatbot, the step 1 list should ideally be empty, or contain only your internal mail server.
What do you give up by running the LLM on your own hardware?
The biggest trade-off of an on-premise RAG chatbot is the language model. You cannot use large commercial LLMs through an API, so you run an open-weight model on your own servers.
Quantization (storing model weights at 8-bit or 4-bit precision) shrinks a model enough to run on a CPU. The cost is slower responses, and small models tend to struggle with multi-step questions and specialist Japanese documents. Getting closer to commercial LLM quality usually means GPUs, and hardware cost rises with it.
VAON does not quote a fixed "X% cheaper" figure for on-premise. Total cost depends on user count, question volume, the model you can run and the hardware you already own, so a generic number is likely wrong for your case. Instead, VAON proposes a small PoC (proof of concept) that measures three things on your real data:
1. Answer accuracy on a set of 50 to 100 real questions, with answers prepared by your business staff.
2. Response time under concurrent use, on the exact server configuration you plan to buy.
3. Hardware and operating cost estimated over 3 years, next to an equivalent SaaS quote.
With these three numbers, management can compare both options on the same basis.
Where should document permissions live in a RAG chatbot?
Permissions in a RAG chatbot belong in the retrieval step, not in the prompt sent to the LLM. If the system retrieves a confidential document and then tells the LLM "do not mention this", the confidential text is already in the model's input and can leak into the answer.
The safer approach is to tag each passage with access attributes (department, role, project) when it enters the vector database. When someone asks a question, the system searches only the passages that person may see. Restricted passages never reach the LLM.
The same principle applies to both options. ONEBOT, VAON's SaaS AI chatbot, has role-based access control (RBAC) that limits which documents each user group can get answers from. For an on-premise chatbot, settle permissions against your real org structure from day one, because reworking them after thousands of documents are loaded is far more expensive.
When is an on-premise RAG chatbot the wrong choice?
An on-premise RAG chatbot is not the default. SaaS is the better fit in these four cases:
1. Your rules allow data online if encrypted and stored in Japan. SaaS is then cheaper, faster and usually gives better answers.
2. Nobody can operate the servers. An internal system needs monitoring, backups and updates. Without staff, the operational risk can outweigh the data risk.
3. You need to answer customers on LINE. LINE runs over the internet, so it cannot coexist with a "nothing goes online" rule.
4. You need results within weeks. A SaaS such as ONEBOT rolls out in 2 to 4 weeks. An internal system needs an assessment, a PoC and hardware procurement first.
Running on-premise also does not make you compliant with Japan's Act on the Protection of Personal Information (APPI) by itself. The architecture narrows where data can travel, but which data goes in, who can see it and how long it is kept still need their own legal review.
Which option should agencies and SIers offer their clients?
Agencies and SIers should pick the option that fits the end client's rules, not the one that is easiest to sell. The end client will ask the questions in sections 2 and 3, and a proposal that cannot answer them gets dropped.
VAON supports partners in two ways. For clients who accept SaaS, VAON runs an OEM program so partners can offer ONEBOT under their own brand. For clients who require no internet traffic, VAON builds on-premise RAG chatbots as custom development projects under its AI and automation services, starting with an infrastructure assessment and a PoC.