Data Governance · Fertilizer & Clean Ammonia (Enterprise)
AI Data Quality Engine for Enterprise Material Master
Production Azure platform that cleans hundreds of thousands of legacy SAP material records - AI shortening, taxonomy classification and governed abbreviations behind a private, zero secret architecture.
- Role
- AI Engineer & Cloud Architect
- Owned the agent layer, FastAPI backend, private Azure landing zone and the Architecture Review Board submission.
- Industry
- Fertilizer & Clean Ammonia (Enterprise)
- Timeline
- May 2026 - Jul 2026
Business impact
- records in scope
- 100,000s
- secrets in code
- Zero
- network posture
- Private only
- security gates
- 5 blocking checks
Overview
Built a data quality engine that turns decades of inconsistent, hand typed SAP/ERP material descriptions into standardized, classified, governed master data, with human review queues for anything the model is not confident about.
The client's ERP held hundreds of thousands of industrial material records described differently by every plant and user, with no enforced taxonomy and duplicates inflating inventory. The engine shortens descriptions to SAP compliant short texts, classifies against a managed taxonomy, standardizes terminology and routes low confidence output to data stewards. Deployed as a fully VNet integrated, private only Azure footprint with Managed Identity everywhere.
Technical architecture
- Network Edge
- Azure Front Door with WAF enforcing a corporate IP allow list
- Frontend
- Static Web App behind an inbound private endpoint, reachable only from inside the network
- Gateway
- API Management in Internal VNet mode with rate limiting and JWT validation
- Backend
- FastAPI on an internal Container Apps environment using System Assigned Managed Identity
- AI
- Azure OpenAI GPT 4.1 through the OpenAI Agents SDK, accessed by Managed Identity with RBAC
- Data
- PostgreSQL Flexible Server with public access disabled, plus private endpoint Blob and Key Vault
- CI/CD
- Four stages with a blocking five check security gate and approval gated deploy
Key features
- AI description shortening into clean, standardized, SAP field compliant short texts
- Taxonomy classification against a managed, stewarded taxonomy
- Governed abbreviation dictionary replacing ad hoc plant level shorthand
- Integrity flagging for duplicates, near duplicates and missing attributes
- Human in the loop review queues feeding corrections back into the engine
- Steward and admin console for taxonomies, dictionaries, queues and access
- Role hierarchy with new SSO users created inactive until an admin activates them
- Enterprise SSO via backend handled Entra ID authorization code flow
Challenges solved
- Legacy free text with no rules - combined LLM normalization with a governed dictionary and taxonomy
- Achieving a private only posture: no public data plane, plus network and identity double gating on the frontend
- Zero secrets in code - Managed Identity with RBAC for every resource including Azure OpenAI and PostgreSQL
- Building a shift left security gate on Azure native tooling that blocks the build on critical findings
- Treating confidence as a routing decision so stewards only see what the model is unsure about
Tech stack
- Python
- FastAPI
- SQLModel
- OpenAI Agents SDK
- Azure OpenAI (GPT 4.1)
- PostgreSQL 16
- React 18
- TypeScript
- Azure Container Apps
- Azure Static Web Apps
- Azure API Management
- Azure Front Door + WAF
- Managed Identity
- Microsoft Entra ID
- Azure DevOps
- Docker