Case Study 1: AI-Powered Chat
This case presents the evolution of an enterprise AI product at The World Bank, from a focused conversational AI MVP to a secure, governed workspace connecting employees with institutional knowledge and AI-powered workflows. The product evolved through user testing, continuous feedback, and the introduction of new AI capabilities and internal knowledge sources.
The diagram provides context for the following Case Study within the Enterprise-wide AI Platform
PRODUCT STRATEGY
Product Context, Business Problem
As AI rapidly changed the way knowledge-intensive work could be performed, The World Bank faced a broader organizational challenge: how to make AI a trusted and useful part of everyday work while preserving the security, integrity, and governance of institutional knowledge.
The organization holds a vast body of specialized knowledge across projects, reports, documents, and other internal sources. Employees spend significant time searching, interpreting, synthesizing, and transforming this information as part of their daily workflows.
Generative AI created an opportunity to fundamentally improve how employees interact with this knowledge, from finding relevant information to analysing documents, generating content, and supporting complex knowledge workflows.
However, adopting AI at enterprise scale required more than providing access to a general-purpose model. The organization needed an AI environment designed around its own data, users, workflows, security requirements, and governance principles.
The organization holds a vast body of specialized knowledge across projects, reports, documents, and other internal sources. Employees spend significant time searching, interpreting, synthesizing, and transforming this information as part of their daily workflows.
Generative AI created an opportunity to fundamentally improve how employees interact with this knowledge, from finding relevant information to analysing documents, generating content, and supporting complex knowledge workflows.
However, adopting AI at enterprise scale required more than providing access to a general-purpose model. The organization needed an AI environment designed around its own data, users, workflows, security requirements, and governance principles.
Product Vision
The initial product vision was to create a trusted conversational layer over World Bank knowledge, allowing employees to interact with institutional information through natural language rather than navigating fragmented systems.
The product was designed to become a foundation for enterprise AI adoption, connecting employees with institutional knowledge, AI capabilities, and reusable AI applications within a secure and governed environment.
The vision evolved around 4 main principles:
1. AI Enabled Knowledge Access. Transform how employees discover and interact with institutional knowledge by providing natural-language access to relevant information across trusted internal sources.
2. AI Assisted Workflows. Move beyond information retrieval toward supporting real workflows, analysing documents, synthesizing information, drafting content, translating, and generating new outputs.
3. Trusted, Governed AI. Create a trusted environment for employees to use AI within the organisation security and governance boundaries, with controlled data access, traceability, transparency, and responsible AI principles.
4. AI Platform. Use the product not only as an application, but as a foundation for broader AI adoption and capability building across the organisation, connecting new data sources, models, workflows, and reusable AI applications.
The product was designed to become a foundation for enterprise AI adoption, connecting employees with institutional knowledge, AI capabilities, and reusable AI applications within a secure and governed environment.
The vision evolved around 4 main principles:
1. AI Enabled Knowledge Access. Transform how employees discover and interact with institutional knowledge by providing natural-language access to relevant information across trusted internal sources.
2. AI Assisted Workflows. Move beyond information retrieval toward supporting real workflows, analysing documents, synthesizing information, drafting content, translating, and generating new outputs.
3. Trusted, Governed AI. Create a trusted environment for employees to use AI within the organisation security and governance boundaries, with controlled data access, traceability, transparency, and responsible AI principles.
4. AI Platform. Use the product not only as an application, but as a foundation for broader AI adoption and capability building across the organisation, connecting new data sources, models, workflows, and reusable AI applications.
AI Use Case Discovery
The discovery process focused on finding the intersection between generative AI capabilities and existing employee needs and workflows. Several areas showed strong potential for AI assistance:
- finding information across institutional knowledge;
- understanding and summarizing reports and documents;
- extracting insights from large amounts of content;
- drafting and transforming work-related content;
- translation and multilingual work;
- working with multiple documents and sources.
Why a Custom AI Product?
The decision to build an internal capability was driven by the need for control, contextualization, and scalability. Rather than relying solely on standalone external AI tools, the product could be tailored to the organization's:
Data: internal knowledge and repositories.
Security: authenticated World Bank employees, security, organizational policies.
Workflows: tailored institutional experience.
Unified Platform: expand to a broader enterprise AI workspace.
Data: internal knowledge and repositories.
Security: authenticated World Bank employees, security, organizational policies.
Workflows: tailored institutional experience.
Unified Platform: expand to a broader enterprise AI workspace.
Enterprise Readiness
AI adoption is not only a technology implementation. It requires the organisation, its data, technology, and people to be ready.
Data: Data scientists and technical teams assessed internal repositories and selected the most relevant sources with appropriate metadata, tagging, and access controls for the MVP. Other sources were prepared for later integration.
Employees: AI Knowledge Hub supported adoption by providing learning resources, AI guidance, information on approved tools, risks, and responsible AI.
Enterprise Requirements: Security, data, technology, and governance teams worked with the product team to ensure the AI solution met organisational requirements and policies.
Data: Data scientists and technical teams assessed internal repositories and selected the most relevant sources with appropriate metadata, tagging, and access controls for the MVP. Other sources were prepared for later integration.
Employees: AI Knowledge Hub supported adoption by providing learning resources, AI guidance, information on approved tools, risks, and responsible AI.
Enterprise Requirements: Security, data, technology, and governance teams worked with the product team to ensure the AI solution met organisational requirements and policies.
PRODUCT FOUNDATION
High-Level AI Product Architecture
The MVP provides the starting point for the product, combining conversational AI with selected internal data sources and RAG-based retrieval. As the product evolved, the architecture was extended to support additional data sources and AI capabilities.
Click to Zoom
AI Architecture, RAG
The product used a Retrieval-Augmented Generation (RAG) approach to connect the LLM with relevant World Bank knowledge.
The system:
received the user's query,
retrieved relevant information from selected internal World Bank sources,
passed the retrieved context to the LLM,
generated a response grounded in that context,
returned the answer together with source references.
This approach allowed the product to work with organization-specific knowledge while keeping responses connected to the underlying sources.
The system:
received the user's query,
retrieved relevant information from selected internal World Bank sources,
passed the retrieved context to the LLM,
generated a response grounded in that context,
returned the answer together with source references.
This approach allowed the product to work with organization-specific knowledge while keeping responses connected to the underlying sources.
AI Governance, Responsible AI
Key elements:
Controlled access: access to the product and internal information is linked to authenticated World Bank employee identities and access permissions.
Data protection: internal data is handled within the organisation controlled environment and through approved external service integrations.
Grounded responses: AI responses is connected to retrieved source information to improve transparency and reduce unsupported answers.
Source traceability: users can access source references to understand where information came from.
Controlled access: access to the product and internal information is linked to authenticated World Bank employee identities and access permissions.
Data protection: internal data is handled within the organisation controlled environment and through approved external service integrations.
Grounded responses: AI responses is connected to retrieved source information to improve transparency and reduce unsupported answers.
Source traceability: users can access source references to understand where information came from.
AI Validation & Testing
AI products require validation beyond traditional usability testing. The product needed to be evaluated not only on whether the interface worked, but also on whether the AI produced relevant, grounded, and trustworthy responses.
Response quality: relevance and usefulness of generated answers.
Retrieval quality: whether the system surfaced appropriate internal information.
Source references: whether responses could be traced back to the retrieved sources.
User feedback: understanding where AI responses or interactions didn't meet user expectations.
Security: ensuring users could only retrieve information available to them.
Validation is treated as an ongoing part of product development rather than a one-time activity.
Response quality: relevance and usefulness of generated answers.
Retrieval quality: whether the system surfaced appropriate internal information.
Source references: whether responses could be traced back to the retrieved sources.
User feedback: understanding where AI responses or interactions didn't meet user expectations.
Security: ensuring users could only retrieve information available to them.
Validation is treated as an ongoing part of product development rather than a one-time activity.
MVP launch
The MVP launch helped validate workflows, observe user behaviour, identify issues, and gather feedback before expanding the product's capabilities.
Main focus on:
Main focus on:
- A conversational AI interface
- A limited set of internal knowledge sources
- RAG-based retrieval
- Source-grounded responses
- Secure employee access
- Controlled rollout to selected internal group
MVP Evaluation
Feedback on the MVP was collected through MS Viva Engage, with UX researchers reaching out to users directly. In parallel, moderated usability sessions were conducted. Users were given simple tasks, and their first interactions with the product were observed. This helped capture both user feedback and real user behaviour, providing the basis for key learnings and the next product iteration.
PRODUCT ITERATIONS
MVP Key Findings
MVP feedback revealed that employees expected AI to support much more than simple question answering. They wanted to use it across broader knowledge workflows, finding information, working with documents, generating content, translating, and continuing work across conversations. This pointed to an opportunity to evolve the product from a conversational assistant into a broader AI productivity layer.
MVP Key Learnings
MVP feedback revealed several areas for further exploration. Users needed more guidance with structuring prompts, expected to upload and work with files and images. They also wanted more control over the knowledge used in responses, access to different AI models, faster way to provide feedback, and support for multilingual translation.
These findings shaped Product Iteration 1, which introduced improvements across prompting and onboarding, file input, conversation history panel, knowledge filtering, in-app feedback, sources panel, model selection, and translation.
These findings shaped Product Iteration 1, which introduced improvements across prompting and onboarding, file input, conversation history panel, knowledge filtering, in-app feedback, sources panel, model selection, and translation.
Product Iteration 1
From Conversational AI to AI-assisted Workflow
Key Product Changes
Better AI interaction:
- Guided onboarding
- Prompt Library
- Improved input and feedback patterns
- File and image upload
- Knowledge / dataset filtering
- In-chat translation
- History panel functionality updates
- LLM options
- Web Search
Key Learnings
As usage of the product grew, feedback revealed new needs that went beyond the single-conversation experience. Users needed to work with multiple documents and organize information around ongoing tasks, access relevant internal documents without uploading them manually, and discover reusable AI tools built for specific needs. They also expected more flexible ways to interact with the product, including mobile access and voice input, while the growing number of capabilities required a simpler and clearer interface.
These findings shaped the 2nd Product Iteration, which introduced Spaces for multi-document work, in-chat document search, an AI Agents Library, mobile access and further improvements.
These findings shaped the 2nd Product Iteration, which introduced Spaces for multi-document work, in-chat document search, an AI Agents Library, mobile access and further improvements.
Product Iteration 2
From AI-assisted Workflow to an Enterprise AI Workspace
Key Changes
AI Spaces, Multi-Document Work
- Grouped documents, conversations, and context
- In-chat search across internal documents
- Outputs based on multiple sources
- AI Agents / AI Apps Library
- Domain-specific AI tools
- Improved discoverability and filtering
- Voice input and transcription
- Mobile experience
- Simplified entry points and quick actions
Outcome
The 2nd Product Iteration expanded the product from a single-conversation experience into a structured AI workspace. Users could organise multiple documents and conversations into dedicated Spaces, search internal documents directly from chat, and work with information from multiple sources. In parallel, a curated AI Agents / Apps library surfaced reusable internal AI solutions across departments, creating a unified place to discover and use domain-specific internal AI capabilities.
A connected employee identity layer added a more personalized experience based on the user organisational context.
A connected employee identity layer added a more personalized experience based on the user organisational context.
PRODUCT KEY METRICS
As an internal enterprise product, AI Chat’s value was reflected not through direct revenue, but through employee adoption, successful task completion, knowledge access, and everyday use.
Enterprise Impact
Primary impact focus: how effectively AI is helping employees complete real work tasks.
Enterprise Reach
20,000+ employees across 189 countries.
Meaningful Task Adoption ~55%
Around 55% of active users completed at least one meaningful work task with AI Chat during the measurement period.
Task Success ~45%
Around 45% of users reported that AI Chat helped them complete or move forward with a work task, while reducing time spent on it.
Repeat Adoption ~60%
Around 60% of users returned to AI Chat and used it again.
Trust ~70%
Around 70% of users rated AI Chat responses as useful and sufficiently trustworthy.
Enterprise Reach
20,000+ employees across 189 countries.
Meaningful Task Adoption ~55%
Around 55% of active users completed at least one meaningful work task with AI Chat during the measurement period.
Task Success ~45%
Around 45% of users reported that AI Chat helped them complete or move forward with a work task, while reducing time spent on it.
Repeat Adoption ~60%
Around 60% of users returned to AI Chat and used it again.
Trust ~70%
Around 70% of users rated AI Chat responses as useful and sufficiently trustworthy.
MY ROLE
Responsibilities & Skills
I worked closely with Product Management, engineering, and business stakeholders across the product lifecycle. Beyond interface design, I contributed to discovery, problem framing, and product direction to connect user needs, business goals, and AI capabilities.
Design Direction
Shaped product experience and interaction patterns based on user needs, business goals, AI capabilities, and technical constraints.
Discovery, Research
Explored workflows and AI use cases through user feedback, usability testing, prototyping, and iterative validation.
Product Design
Designed end-to-end user flows, interaction patterns, information architecture, and key features across successive product iterations.
Cross-functional Collaboration
Worked closely with engineering, data, architecture, and business teams to translate concepts into feasible product experiences.
Areas of expertise: Product Thinking · AI Product Discovery · AI Product Design · Enterprise AI Platforms · Technical AI Literacy · AI Governance & Adoption
See detailed list of skills
Design Direction
Shaped product experience and interaction patterns based on user needs, business goals, AI capabilities, and technical constraints.
Discovery, Research
Explored workflows and AI use cases through user feedback, usability testing, prototyping, and iterative validation.
Product Design
Designed end-to-end user flows, interaction patterns, information architecture, and key features across successive product iterations.
Cross-functional Collaboration
Worked closely with engineering, data, architecture, and business teams to translate concepts into feasible product experiences.
Areas of expertise: Product Thinking · AI Product Discovery · AI Product Design · Enterprise AI Platforms · Technical AI Literacy · AI Governance & Adoption
See detailed list of skills
PRODUCT DELIVERY
From Idea to Release
The product evolved through an iterative Agile approach, combining structured planning, cross-functional collaboration, and continuous feedback.
Plan:
Business Context · Product Vision · User Needs · Product Goals · Priorities · Roadmap
Explore:
Requirements · User Research · User Journeys · User Personas · User Flows · Prototypes · MVP Definition
Deliver:
Agile/Scrum · Azure DevOps · User Stories · Figma · Design Handoff · Backlog Refinement · Daily Stand-ups · Sprint Reviews
Learn and Iterate:
User feedback · Testing · Analytics · Data-driven Decisions · Continuous improvement
Plan:
Business Context · Product Vision · User Needs · Product Goals · Priorities · Roadmap
Explore:
Requirements · User Research · User Journeys · User Personas · User Flows · Prototypes · MVP Definition
Deliver:
Agile/Scrum · Azure DevOps · User Stories · Figma · Design Handoff · Backlog Refinement · Daily Stand-ups · Sprint Reviews
Learn and Iterate:
User feedback · Testing · Analytics · Data-driven Decisions · Continuous improvement