Switchboard: taking a customer change from request to verified execution
An AI agent investigates customer change requests and prepares findings for review. The application controls independent approval, execution, and delivery verification.
Investigating a customer change
Changing a production webhook endpoint means deciding where customer events will be sent. An incorrect change could interrupt delivery or send those events to an unauthorized destination. Before an engineer updates the configuration, they need to establish what the customer requested, whether it’s permitted, and who must approve it.
Switchboard handles that process for a fictional integration SaaS company. An AI agent investigates the request and prepares findings for review; the application controls proposal creation, independent approval, execution, and verification. I built it to extend my experience with customer-facing AI products into internal tools, where someone needs to review an agent’s findings before changing a customer’s configuration.
In the main scenario, Acme asks to replace its production webhook endpoint. The investigator reads the ticket, checks the requester against authorized customer contacts, and compares the proposed destination with registered endpoints. It also encounters two policy versions: an older rule permits self-approval, while its replacement requires approval from a technical lead assigned to the customer who did not propose the change.
The application supplies the employee’s identity, and the agent’s tools check customer access before returning records. An employee assigned to Acme cannot retrieve Globex’s configuration through the investigator. When the model recommends preparing a proposal, a separate LangGraph stage checks the proposed change against current business records before saving it for approval.
Checking the agent’s conclusions
I tested the investigator against six scenarios, including an unauthorized requester, an unregistered destination, and instructions embedded in a ticket that tried to override policy. After prompt revisions, all six produced the expected candidate-or-blocked outcome, though one attempt exhausted its token budget and needed a retry. The cross-customer case also passed three repeated checks.
But a correct outcome didn’t mean the explanation was accurate. In the valid-request scenario, the agent correctly recommended preparing a proposal while misstating the recovery policy. The policy allowed restoration of the old endpoint when the approved plan permitted it and no intervening change made restoration unsafe. The summary turned that conditional permission into a blanket prohibition.
I added a policy-faithfulness evaluator and tested it against four examples covering conditional permission, an unconditional allowance, a blanket ban, and the investigator’s limited authority. The evaluator initially objected to equivalent wording about escalation. In a later run, it flagged a summary for mentioning independent approval without also mentioning that the approver had to be assigned to the customer. The summary hadn’t claimed that independent approval was the only requirement.
After I refined the instructions to account for those distinctions, the follow-up evaluation matched all four expected results. That corrected the observed mistakes, but four examples aren’t enough to estimate accuracy across unfamiliar requests.
Policy review now runs automatically before a report is published. The evaluator compares the findings with the workspace’s policy documents and cites excerpts for questionable claims. The application checks that those excerpts occur in the cited sources, allows one revision of the findings, and evaluates them again. If validation fails, it saves neither a completed investigation nor a proposal. The employee sees “The investigation report could not be validated. Try again.” and can start another investigation.
The evaluator can quote a policy accurately and misinterpret it, so application code independently enforces authorization. In the report, missing evidence, confirmed violations, and requirements that apply later receive different statuses. A reviewer can open source records alongside the findings and inspect the evidence captured during the investigation, even if the underlying configuration has since changed.
This screenshot uses a prepared development fixture. The public demo calls the model when a visitor starts an investigation.
Executing an approved change
By the time a proposal reaches execution, the customer may have revised its instructions, an employee may have lost access, or another engineer may have updated the endpoint. Switchboard checks current authority, approval, the permitted change window, and configuration again before applying the change.
As I added approval and execution, I questioned the separate databases originally used for investigation scenarios. These operations needed a consistent view of the same business records. Bringing that state together also allowed the configuration update and its execution receipt to commit together.
Suppose a proposal expects configuration version 7, but another change has advanced it to version 8. Execution must reject the stale proposal. Now suppose Switchboard performs the approved update, advances the version, and loses the response before the browser receives it. A retry should return the result of that successful execution.
The saved receipt distinguishes those cases. After checking the caller’s access, an execution retry returns the existing receipt without repeating the update. A separate verification step checks the active endpoint and records the outcome of a synthetic delivery event. Confirmed delivery closes the ticket; failure or uncertainty requires manual intervention. The application doesn’t perform automatic rollback.
I tested persistence separately through local integration tests against the Worker-to-D1 storage service. A mismatched configuration version left the endpoint unchanged and created no execution receipt. After a Worker restart, retrying a completed execution returned its original receipt without advancing the configuration version again.
Trying the workflow
For visitors, I replaced a separate demo-case selector with an inbox containing requests at different stages. Someone can investigate a new request, review a pending proposal, or execute an approved change through the same interface.
The inbox shows requests assigned to the selected fictional employee and the stage each has reached.
I also reconsidered requiring Microsoft sign-in for a portfolio demonstration. Visitors receive isolated workspaces and can switch between fictional employees to experience their different responsibilities. Each persona’s permissions apply within that workspace; Microsoft Entra authentication is supported separately.
The company, customer records, and delivery events are synthetic. Try Switchboard to follow a valid change through approval, or investigate an unsafe destination and inspect why it cannot proceed.
Development
Built with AI coding tools, reviewing behavior, module interfaces, and test scenarios before implementation.
Have a product priority that needs an owner?
I join B2B SaaS teams on contract to ship the features, internal tools, and integrations already on your roadmap.