Putting AI Inside Business Software: Six Rules We Learned Building It into Our Own

Most AI features in business software are a text generator bolted onto a form. The difference that matters is grounding: whether the assistant composes from what the record actually contains, or free-composes something plausible. A follow-up drafted from what was really quoted, what was really promised and what the customer last said is a useful head start. A follow-up invented from nothing is a liability, because you now have to proofread it against your own memory, which is more work than writing it.

So the prompt is assembled by the system from the record's own fields, and the screen shows what the draft was built from. If the data is not in the record, the assistant says so instead of filling the gap. This is also the cheapest way to keep model costs and error rates down, and it is the same principle behind retrieval-augmented systems: the model supplies language, the record supplies the facts.

Never create a permission called something like use AI that stands on its own. If keeping a drafted circular needs the permission to write circulars, and keeping a drafted report-card remark needs the permission to enter marks, then those are the permissions the assistant borrows. A separate ai.apply permission would let a role act outside its own boundary simply by routing the action through the assistant.

The part that gets missed is that the borrowing happens at the ask, not only at the file. In the school software we are building, we shipped the accept path checking the target permission while the generate path checked only that the user could use AI. The result: a class teacher who may never send a circular could have a fluent, school-wide circular written for her, sitting in her hands, ready to copy into a messaging group. It was the same error as the one we had just closed, one step earlier, and invisible because the real gate looked correct.

The fix has three parts. One function maps the kind of draft to the permission it needs. Both the generate gate and the accept step call that one function, because two hand-written copies had already started to drift inside a single release. And acceptance has no author bypass: the person who created a draft has no authority from having created it. A teacher who lost the right to enter marks on Friday must not be able to file Friday's remark on Monday.

When an assistant drafts something about a person, send the facts that matter and not the identity: scores, dates, a pronoun, never a name or an identifier. That is de-identifying by construction. But a privacy promise enforced only by the code that builds the prompt has a half-life, because that is exactly the code someone edits in a hurry to add a little more context, and nothing will fail when they do.

So the second lock belongs in the database. The function that records a generation already knows which person it concerns, so it can look the name up and refuse to record the draft if the name appears anywhere in what was sent. Because the draft is only handed back after it is recorded, a leaked name means nobody gets to use what came back.

The trap is how you match. A naive substring test looks stricter and is much worse: a pupil called Ram blocks the ordinary word programme, a pupil called Ann blocks cannot, a pupil called Ali blocks quality. A guard that cries wolf on a screen where the user can plainly see there is no name gets worked around within a week, and then protects nothing. What works is the full name matched case-insensitively, plus any name part of four letters or more matched on word boundaries, using only alphabetic parts so the pattern is safe to build.

An AI feature can be switched off by several different things: the vendor connection, a setting the school or company controls, the person's own role. Three different people fix those: the vendor, a settings manager, the role's owner. A single yes-or-no gate collapses them into one message, and the one message is usually wrong. Our first version did exactly that, and for a class teacher whose own role was the problem it said her school had switched the feature off. That was false, and it would have sent her to the principal about a setting that was already correct.

Return an answer with a reason and a next step: allowed or not, the reason code, a human sentence, and who can change it. It costs a few lines and it is the difference between a feature people trust and one they report as broken.

Assistants and analytics get asked for things that sound reasonable and do harm: a score per teacher, a ranking of staff, a performance number for each person. A class average is a property of the class, not of the teacher; whoever is handed the weakest set will always show a lower one. A product that turns that into a per-person score is doing harm with arithmetic, and it will be used in appraisals, because a number on a screen always is.

So we build what is useful and refuse the derived artefact. Name the person, so someone can find who to talk to about a class. Build no aggregate over the person: nothing sortable or rankable by them and no best-or-worst styling. Show the spread and not just the middle, because two classes averaging the same can be one tightly clustered and one split into strong and struggling groups, and they need opposite responses. And write the refusal into the product copy itself, not only a code comment, because that is what stops someone asking for it again next quarter.

The last rule ties the others together. The assistant produces a draft; a human reads it and accepts it; acceptance is the moment the authority check runs and the audit record is written. Nothing is sent, filed or posted by the assistant on its own authority. It is slower by one click and it removes the largest category of AI incidents, which are confident, fluent, wrong actions taken without a person in the loop.

None of this needs a bigger model. It needs the unglamorous parts built first: grounding, one permission map used at both gates, a database check behind the prompt builder, explanatory refusals and a human accept step. If you are scoping an AI feature, how to add AI to your SaaS without burning cash covers choosing the first workflow, and what AI features cost covers the budget. Choosing between a ready-made assistant and a custom one is in chatbot versus custom AI. If you want these rules applied to your system, our AI integration service starts with a written scope, and you can send us the brief.