Financial Intelligence · Fertilizer & Clean Ammonia (Enterprise)
CFO Digest: Weekly Market Intelligence Pipeline
Automated multi-stage AI + data pipeline, scrape to email, running Serper into a Fabric warehouse, agent narratives, evaluation agent, weekly Power BI PDF.
- Role
- AI Engineer
- Built both pipelines end to end including the agent chain, seven narrative agents, data model and documentation set.
- Industry
- Fertilizer & Clean Ammonia (Enterprise)
- Timeline
- Delivered - production weekly job
Business impact
- digest tables
- 10
- production pipelines
- 2
- agents in production
- 7 + 3 role chain
- watch list
- ~130 companies
- cadence
- Weekly, unattended
Overview
Automated multi-stage AI + data pipeline, end to end from scrape to email. The Serper API pulls date-scoped market and financial data into Fabric warehouse tables; downstream agents load and threshold-filter records, draft stakeholder narratives, then hand off to an evaluation agent.
Runs weekly to generate a Power BI PDF report emailed per stakeholder group, orchestrated end to end via an Azure AI Foundry agent.
Technical architecture
- Orchestration
- Copilot Studio agent rendering an adaptive card, sequencing topics and emailing the digest
- Data Agent
- Read only Fabric Data Agent translating requests to T SQL against a caller supplied schema contract
- Pipeline Runtime
- Fabric Notebooks with per pipeline module libraries packaged as Fabric Environments
- AI Runtime
- Azure AI Foundry Responses client calling pre created agents by name and version
- Deterministic Pre Stage
- Story de duplication and materiality scoring in pure Python before any model spend
- Storage
- Raw and curated Delta tables written by Spark merge and consumed by Power BI
Key features
- Ten digest tables produced automatically on a weekly gated schedule
- Triage, Narrative and Eval agent chain running per story on the open web pipeline
- Cross language story de duplication collapsing every rendering of a wire story into one canonical document
- Explainable 0-1 materiality score deciding both what gets narrated and the published rank order
- Seven pre created Foundry agents invoked by name and version, so rolling forward is a version bump
- Whole table narratives produced in one batch call rather than row by row
- Strict separation - every figure comes from SQL, the web supplies only the causal reason
- Pydantic schema validation of every agent response before it reaches the Delta merge
- Window replace merge so re runs never duplicate a reporting week
- Copilot Studio orchestrator with an adaptive card for window and topic selection
Challenges solved
- Syndication looked like volume - one wire story republished across editions produced duplicate curated rows
- Agent scores saturated at 0.90-0.97, making score based ranking effectively random; replaced with a deterministic score
- Cutting the candidate set deterministically before the agent chain so tokens are only spent on material stories
- Keeping figures trustworthy by architecture - SQL supplies every value, the agent supplies only the sentence
- Working around search platform limits where a broken query plan is indistinguishable from a quiet news week
Tech stack
- Microsoft Fabric
- Azure AI Foundry Agent Service
- Azure OpenAI (GPT 4.1)
- Microsoft Copilot Studio
- Fabric Data Agent
- Python
- PySpark
- pyodbc
- Delta Lake
- Structured Outputs
- Pydantic
- Power BI