Google Gemini (AI / OCR)
PPP uses AI models for three features:
- Medical sticker OCR — extracting UR number, date of birth, and gender from photographed patient stickers in the surgical logbook
- Operating list OCR — extracting surgical case details from photographs of hospital theatre lists (#3107)
- Reflection quality scoring — grading the quality of a CPD reflection and suggesting improvements
- PDP generation — turning a user's CPD history into a draft Personal Development Plan
Historical note — "Gemini" is a misnomer today
PPP originally used Google Gemini for all three features. The main PPP backend image and text services have since been migrated to Azure OpenAI (gpt-4o-mini as the primary model, gpt-4o as the fallback) for availability and regional-data-residency reasons. The environment variable is still called GEMINI_API_KEY for continuity, and the WhatsApp ops bot (api/whatsapp-bot/) still uses the real Gemini API. This page covers both paths so readers don't get confused when they see Gemini references in secret lineage and Azure OpenAI references in the service code.
1. Where it's wired
Main PPP backend (Azure OpenAI today)
| File | Purpose |
|---|---|
api/backend-ppp/src/services/imageService/index.ts | Azure OpenAI vision client, medical sticker + operating list OCR |
api/backend-ppp/src/services/reflectionQualityService/index.ts | Reflection quality scoring; AI text provider is Azure OpenAI |
api/backend-ppp/src/services/pdpGenerationService/index.ts | PDP generation; AzureOpenAITextProvider as the default |
api/backend-ppp/src/routers/procedures/analyzeImage/ | tRPC procedure called by the mobile/web client |
WhatsApp ops bot (real Gemini API)
| File | Purpose |
|---|---|
api/whatsapp-bot/src/services/gemini-parser.ts | Classifies incoming WhatsApp messages into GitHub issues |
The WhatsApp bot is a separate Azure Function App. It ingests support messages, asks Gemini to classify them, and then creates a GitHub issue with the right labels. It is intentionally decoupled from the main PPP backend. The Pulumi infrastructure project (api/infra-whatsapp-bot/) was removed in #5124 — the bot's Azure resources were never deployed.
2. Medical sticker OCR
What it does
A clinician on the ward photographs a patient's sticker with the PPP mobile app. The image is uploaded to blob storage via a presigned URL, the tRPC client calls analyzeImage.medicalSticker, and the backend:
- Validates the MIME type is one of
image/jpeg | png | webp | heic | heif - Loads the editable prompt from the
system_promptstable (key:medical-sticker-ocr) with a hard-coded fallback - Sends the image + prompt to Azure OpenAI (
gpt-4o-miniprimary,gpt-4ofallback) - Parses the JSON response into
{ urn, dob, gender, confidence } - Returns the extracted fields to the client, which pre-fills the logbook form
The frontend displays each extracted field with an indicator of whether it was AI-populated so the clinician can review before saving.
What it does not do
- No patient identifying data is retained on disk beyond the blob already uploaded to PPP storage. The response is streamed back to the client and discarded.
- No cross-referencing — the OCR does not call any external PAS or EHR system. The extracted UR number is stored as a plain string in the logbook; it is not resolved to anything.
- No free-text output — the prompt strictly enforces a JSON shape, and the server rejects responses that do not parse.
Prompt management
The prompt lives in system_prompts and can be edited via the Super Admin UI. The DEFAULT_MEDICAL_STICKER_PROMPT constant in imageService/index.ts is a safety fallback for when the DB-backed prompt is unavailable. If you edit the DB prompt, make sure it still enforces the JSON shape — the parser depends on it.
3. Operating list OCR (#3107)
Similar shape to the sticker OCR, but the prompt extracts an array of surgical cases from a photographed theatre list. Each case has patient name, UR number, procedure, surgeon, role, time slot, hospital, and notes. The list also surfaces listDate and hospitalName at the top level.
Prompt key: operating-list-ocr (DB-backed, with DEFAULT_OPERATING_LIST_PROMPT as fallback). Same Azure OpenAI model chain. The feature is still behind a rollout flag in UAT at the time of writing.
4. Reflection quality scoring and PDP generation
Both features use an AiTextProvider abstraction in api/backend-ppp/src/services/pdpGenerationService/index.ts:
const aiProvider = provider ?? new AzureOpenAITextProvider();The provider interface is deliberately narrow so tests can inject mocks without spinning up a real model. In production the default is Azure OpenAI; in tests the provider is always mocked.
- Reflection quality scoring — takes a user's reflection text and returns a score + structured feedback. Used to help trainees improve their CPD reflections over time.
- PDP generation — takes a user's recent CPD activities and returns a draft Personal Development Plan structured by the RCS PDP template. The user reviews and edits the draft before saving.
5. Configuration
Main PPP backend env vars (Azure OpenAI)
| Variable | Purpose | Required |
|---|---|---|
AZURE_OPENAI_API_KEY | Azure OpenAI API key from the configured resource | Yes |
AZURE_OPENAI_ENDPOINT | Azure OpenAI resource endpoint (https://<name>.openai.azure.com) | Yes |
AZURE_OPENAI_API_VERSION | Preview API version (defaults to 2024-08-01-preview) | No |
The client code rejects placeholder values (REPLACE_ME, your-api-key-here, empty string, etc.) at request time and returns a tRPC PRECONDITION_FAILED error so a misconfigured deploy fails loudly instead of silently.
Legacy / WhatsApp bot env var (real Gemini)
| Variable | Purpose | Required for |
|---|---|---|
GEMINI_API_KEY | Google AI Studio API key for generativelanguage.googleapis.com | WhatsApp bot |
This variable is still listed in docs/site/development/backend/environment.md as "required for OCR image analysis (Gemini)", which dates from before the Azure OpenAI migration. It remains injected into the PPP Container App by Pulumi for backward compatibility and for any code path that falls back to the direct Gemini API, but the current main image and text services do not consume it.
Secret lineage
Both AZURE_OPENAI_API_KEY and GEMINI_API_KEY flow through the same pipeline as every other external credential:
- Source of truth: Azure Key Vault (
AZURE-OPENAI-API-KEY,GEMINI-API-KEY) - Exported into Doppler for developer access (
ppp/uat,ppp/prod) - Read by Pulumi (
api/infra-azure/src/index.ts) viarequireEnvSecretat deploy time - Injected into the Container App env at provisioning time
See api/infra-azure/SECRET_LINEAGE.md for the full table.
6. Resilience
Image analysis calls go through the shared visionCircuitBreaker:
- Circuit breaker — trips after 5 consecutive failures, resets after 60 seconds
- Fallback model —
gpt-4o-miniis tried first; on failure the service retries withgpt-4obefore giving up - Timeout — handled by the Azure OpenAI SDK's built-in client
When the breaker is open, analyzeImage returns a tRPC SERVICE_UNAVAILABLE error with a user-friendly message ("OCR is temporarily unavailable, please enter the fields manually"). The signup and logbook flows all degrade gracefully — the AI path is an accelerator, never a requirement.
7. Failure modes
| Failure | User impact | Operator action |
|---|---|---|
| Azure OpenAI rate limits | OCR calls 429; circuit breaker opens after 5 failures | Review request volume; request a quota increase |
AZURE_OPENAI_API_KEY missing or placeholder | analyzeImage returns PRECONDITION_FAILED immediately | Re-apply Pulumi; check Doppler binding |
| Model returns non-JSON | Handler logs and returns a generic extraction-failed error | Review the system prompt for regression |
| Primary model unavailable | Service retries with the fallback model transparently | None — self-heals |
| Image too large (>20MB) | Pre-check fails with a clear error; client shows message | None — enforced by MAX_IMAGE_FILE_SIZE_BYTES |
| Unsupported MIME | Pre-check fails; client shows "unsupported image format" | None — frontend already filters to supported types |
8. Cost and billing
Azure OpenAI
PPP is billed for Azure OpenAI on a per-token basis with separate rates for input and output tokens. For the image path:
- gpt-4o-mini — the primary model, chosen specifically because it is roughly an order of magnitude cheaper than gpt-4o per million tokens while still producing accurate results for the structured OCR prompts PPP uses.
- gpt-4o — fallback only. Only used when gpt-4o-mini fails, which should be rare.
Cost observability for Azure OpenAI is part of the same Azure cost tooling as the rest of the infrastructure. There is no dedicated Grafana dashboard yet — cost is tracked in the monthly Azure invoice review.
Sticker OCR is the hot path: one sticker per surgical case logged, typically 0.5–2 kB of output. Operating list OCR is more expensive (larger images, longer structured JSON outputs) but lower volume (one list covers 5–20 cases).
Real Gemini (WhatsApp bot)
The WhatsApp bot uses gemini-1.5-flash at low volume (a few hundred classifications per month). Cost is bounded by the fact that each message is ≤1 kB and the model is the cheapest tier. Billed directly to the Google Cloud project tied to GEMINI_API_KEY.
9. Security and privacy
- No prompt logging — prompts and model responses are not written to application logs. Diagnostic logs contain the service name, user ID, action, and error code only.
- Placeholder detection — the image service refuses to call Azure OpenAI if the API key looks like a placeholder (
REPLACE_ME,your-api-key-here, etc.), preventing accidental calls from dev machines that never got their.envset up. - Input validation — image MIME types are validated both in the tRPC Zod schema and at the service layer. The prompt-injection surface is small because the user supplies an image, not free text.
- Retention — raw images are stored in PPP blob storage under the existing retention policy. Extracted fields are stored in the users / logbook tables as plain strings. Nothing is retained inside Azure OpenAI by PPP.
10. Testing
cd api/backend-ppp
pnpm test -- imageService
pnpm test -- pdpGenerationService
pnpm test -- reflectionQualityServiceThe test suite mocks the Azure OpenAI client using handlers in api/backend-ppp/src/__fixtures__/handlers/ (still named gemini.ts for historical reasons — see handlers directory for the current set) and covers:
- Successful OCR extraction (all fields populated)
- Partial extraction (some fields missing)
- Unreadable image (all fields null)
- Model timeout and retry
- Circuit breaker behaviour
- Unsupported MIME rejection
- Prompt loading from DB with fallback to hard-coded default
Contract tests in api/backend-ppp/src/__fixtures__/external-services.contracts.test.ts exercise the full external-service integration path with recorded fixtures.
11. Related
- Integrations overview — the five external services PPP depends on
- Surgical Logbook feature spec — where sticker OCR surfaces in the product
docs/site/development/backend/environment.md— canonical env var list (contains a known stale "Gemini" label forGEMINI_API_KEY)api/infra-azure/SECRET_LINEAGE.md— how AI credentials flow from Key Vault into Pulumi into the Container App