Case Study
From AI-Built App to Production Pilot
Technical Lead & Senior Consultant, Quisitive · July to October 2026
Three workstreams for one manufacturer: shop-floor work orders, a multi-agent operations platform, and AI purchase-order intake.
At a glance
- Situation
- The client's team had built most of a shop-floor app with AI coding agents. Before anyone could run production on it, it needed a reliable ERP integration, security work, and release controls.
- My role
- Tech lead. I did the technical assessment and the architecture, designed and built the ERP integration, and led delivery.
- Key decision
- Keep the app's workflows and replace the spreadsheet and SQLite backend underneath, instead of rebuilding from scratch.
- Outcome
- The work order app went to the client for acceptance, validated and ready for a controlled production pilot. I designed, built, and tested its retry-safe ERP write-back. The client asked me to stay an extra month.
- Status
- When my engagement ended in October 2026, order intake was still in UAT. None of the three workstreams was in broad production use yet.
A manufacturer running on paper, and an app it had already built.
The shop floor ran on paper travelers, the printed work orders that follow a job from station to station. Nothing that happened on the floor reached the ERP until someone typed it in later. The client wanted tablets, QR codes, and live quantity and labor data flowing into the ERP as the system of record.
This client was already deep into AI-assisted development. Their team used AI coding agents every day and moved fast. Quisitive was brought in across several workstreams, and I was the tech lead on the work order application, responsible for architecture, integrations, and MVP delivery.
Less than three weeks in, a client demo changed the plan. Their own AI-built shop-floor app already covered much of the MVP we had been hired to build. My job became working out what that meant, recommending a better path, and then leading the work to get their software ready for production.
Rewire, not rebuild.
I led the technical reassessment and wrote the executive assessment. My recommendation was to keep what the floor team already used and replace what couldn't go to production.
Original plan
- Build a new shop-floor application from scratch
- Re-create features the floor team already relied on
- Ask users to learn a new interface
- Spend early budget catching up to what already existed
Rewire, not rebuild
- Keep the app's features, workflows, and interface
- Replace the spreadsheet and SQLite backend with live ERP reads and writes
- Put engineering time into integration, security, and deployment
- Treat the client's AI-built work as a head start
That changed what the project was. It went from building software to productionizing it. Engineering time moved to where the actual risk was: the ERP integration, security, deployment, and the controls around code that agents had written. The assessment also made security a condition for going to production, not a follow-up item.
The most valuable call on this project came before the integration work started. The client already had most of what it needed. The job was making it dependable.
Making AI-built software production-ready.
The client's AI-assisted approach had done its job. It put working software in front of the floor team quickly. Getting that software to production took more engineering discipline around source control, deployment, security, and maintainability, the kind of work that's easy to defer when a team is moving fast. I helped them put it in place without slowing them down.
Source control and release discipline
I wrote a playbook for moving the repositories under a GitHub organization with proper access groups, and connected GitHub to Azure DevOps Boards so every branch, commit, and pull request traced back to a work item. I introduced feature branching for the client's developers, tagged the existing state as v1.0.0 so there was a known baseline, and started dated releases. I also added Copilot instruction files to the repositories so the coding agents followed the same conventions as the people reviewing their work.
Deployment and approvals
The production approval gate was set up so strictly that releases stalled. Every approver had to sign off, and nobody could approve their own deployment. I proposed and validated a fix, then added a way to deploy a release or hotfix branch to production that still went through the gate.
Security and configuration
The assessment set a standard for configuration and secrets: Azure App Configuration, Key Vault, and managed identity. I also made the ERP environment configurable, so a development build could never write to production ERP data.
ERP integration and data integrity
Shop-floor counts could not be lost or counted twice. A tablet can drop off Wi-Fi halfway through an update, and a request can time out after the ERP has already accepted it. I designed, built, and tested the write-back path around both cases.
Step 1
Tablet event
A count or labor entry from the floor, tagged with a client-generated event ID.
Step 2
Outbox
Recorded first. A background drainer retries failures likely to clear on their own and parks the rest for review.
A dropped connection doesn't lose the count.
Step 3
Integration API
Behind a feature flag that was off by default. Passes the ERP's exact error message back to callers.
Step 4
ERP write-back
Insert-only endpoint that uses the event ID as its idempotency key, so a retry is recognized as the same event.
Duplicates stop here.
Failures had to be visible and fixable. Parked events show up in an admin view, where someone can requeue them once the cause is fixed. The outbox turns cumulative totals into deltas and records corrections as signed adjustments instead of edits. The audit trail is append-only, and role-based access fails closed. At my request, the client's ERP team built the insert-only endpoint, and I tested the failure paths with their ERP manager, who could read the outbox's error messages and tell exactly what had gone wrong. Reads were already working, so I left them alone.
The first real write was rejected with a permissions error. I had Claude go through the logs. It ran its own tests and traced the failure to a missing access grant for the write-back API. That gave me a specific request to take to the client's ERP manager instead of a vague report that something was broken. I also changed the integration API to pass the ERP's exact error message back to callers, so the next failure would explain itself. Writes started posting in the sandbox once access was granted.
Testing and onsite UAT
I spent four days onsite running UAT and training with the floor team, testing nesting, scheduling, inspection, outside processing, inventory, and file imports with the people who would use them every day.
When a bug came in, I fixed it there. With GitHub Copilot and the Claude Agent SDK working across three repositories at once, I'd reproduce the problem, find the cause, fix it, and deploy it to UAT. The pilot team checked each fix there before we promoted it to the pilot's production environment. Five bugs went from report to verified fix, typically in about 30 minutes. We also built three requested enhancements that the client validated the same day.
The easiest bug to explain was about shared data. Settings and inventory that the whole floor shared were being saved in each person's browser. One person could update them, and nobody else would ever see the change. I moved that data into the shared system of record, and we confirmed the fix across browsers the same day.
Partway through UAT, an ERP partner's upgrade removed the custom APIs from the test environment. I worked out what had happened, got the right people involved, and moved the team to work that didn't depend on the ERP so we didn't lose the week.
Three workstreams, three different roles.
The work order app, covered above, was the main engagement. There I also worked alongside the team's developers and QA, reviewing their pull requests, helping clear blockers, and testing the app with them. The other two workstreams were shorter, and my role in each was different.
Shop-floor work orders
Tech lead
July to early September 2026
Went to the client for acceptance as operational and validated, ready for a controlled production pilot.
Multi-agent operations platform
Interim tech lead
Two weeks, early September 2026
Working UAT and production environments after a production-readiness sprint. Ready for controlled pilot use, not a broad rollout.
AI purchase-order intake
Developer, part time
Late September to October 2026
In UAT when I left, working toward a controlled pilot and go-live.
Multi-agent operations platform
Interim tech lead · Two weeks, early September 2026
I served as interim technical lead for the platform during a two-week production-readiness period. It uses a master orchestration agent to coordinate departmental agents for inventory, production, shipping, customer communications, and analytics. It answers natural-language questions over ERP, machine, and Microsoft 365 data, with citations and human review, and runs on .NET 10, Azure Functions, and the Microsoft Agent Framework on Azure AI Foundry.
Most of the work was getting production to behave. I traced the agent failures to identity and access configuration between Azure services, fixed it, and captured it in Bicep so the next deployment couldn't quietly undo it. I also spotted model rate limiting, rebalanced quota across environments, and owned the sprint demo, which ran on production data.
I also owned the evaluation of how to bring Claude into the platform alongside its GPT models. We compared an external agent behind Azure API Management, Foundry's bring-your-own-model gateway, direct API calls, and a client-owned Foundry instance, and deferred the last option to a later phase.
AI purchase-order intake
Developer, part time · Late September to October 2026
After I rolled off, the client brought me back part time to help get this system ready for production. Customers email in purchase orders. The system extracts the lines, validates them against ERP master data and pricing, and sends exceptions to a human review queue. Approved orders post to the ERP as sales orders.
My work went into the parts the review team relies on: recording which reviewer made each edit and decision, making closed orders read-only on the server, showing the ERP submission result in validation history, and letting reviewers open the original email and PDF. I also sorted out access to the review endpoint in Azure, and set up local Azure authentication so the team could develop on Windows and macOS.
AI across the whole lifecycle, with a person accountable for the result.
I used AI at almost every stage of this project. Writing code was only one of them.
- Discovery: Copilot turned the client's spec into structured workshop questions, and I published the findings the same day.
- Understanding unfamiliar code: AI helped me explain the client's non-.NET codebase to a team of .NET developers.
- Planning: it drafted about 40 candidate user stories for the next phase, which I reviewed and cut down by hand.
- Implementation: agents did a lot of the typing, guided by instruction files and reviewed through pull requests.
- Troubleshooting: Claude read logs and ran tests to narrow down the ERP permissions failure.
- Review and validation: AI test passes and code review caught issues I hadn't, described below.
- Documentation: AI-assisted assessments and comparison reports, plus Copilot-drafted PR descriptions so reviewers had context.
Estimating with and without AI
The client wanted its next version production-ready by October 1. The two codebases had diverged far enough that converging them was never going to be a simple merge. I sorted every feature into port, skip, decide, or drop, grouped the work into tiers, and recommended productionizing before adding anything new.
I estimated every tier two ways. A traditional approach came to 41.5 to 63.5 engineering days. An AI-assisted approach came to 24.25 to 36.5, about 42% less. Those were planning estimates, not measured results. Pricing both gave leadership a concrete basis for deciding how much to lean on AI-assisted delivery, and the committed scope fit two engineers, with a fallback plan for one.
AI as a second reviewer
An AI test pass I ran with Opus 5.5, while testing something else, turned up four endpoints failing with server errors. The cause was a missing ERP permission that only showed up under certain queries. Rather than wait for the next surprise, I proposed a script that calls every ERP endpoint the system uses, so the ERP itself reports exactly which permissions are missing.
Copilot's code review caught a quieter bug. Decision history mixed local and server timestamps for ordering, which would drift over time. AI found both problems. Deciding what they meant and what to change was still my job.
I coached teammates the same way. Use plan mode, and read the plan before the agent touches anything. Don't take a code comment as truth, and don't take an AI's summary of the code as truth either. A second model is a useful first pass, but the code itself, the tests, and someone who understands the system are what decide whether a conclusion holds up.
Twenty-four years of shipping software tells me what to look for. AI lets me look a lot faster. I hand agents the investigation, the first drafts, and plenty of the implementation. The architecture calls, the accountability for what ships, and the conversations with the client stay with me. That's what made the speed worth something.
What fifteen weeks across three workstreams produced.
Pilot-ready
Work order app at handoff, validated for a controlled production pilot
Retry-safe
ERP write-back I designed, built, and tested
~30 min
From a bug reported on the floor to a fix verified in UAT
+1 month
Extension the client asked for
When my engagement ended in October 2026, each workstream was ready for a controlled pilot or still in UAT. None was in broad production use. Across the three, I opened 25+ pull requests in five repositories and wrote the architecture, assessment, and planning documents the client used to make its decisions.
The clearest signal came from the client. They asked for me to stay an extra month, and after I rolled off, they brought me back to help with order intake.
What I took away: a team working with AI agents can get to working software very fast. Knowing what production needs that a prototype doesn't, and making those calls early enough that the speed doesn't turn into rework, still takes experience.
Technologies across the three workstreams
Let's have a conversation.
If something here caught your attention, I'd like to hear about it. I'm always up for a conversation about building software that holds up, or about how teams are actually using AI agents.