Azure OpenAI Service

Azure OpenAI Service provides access to OpenAI's large language models — including GPT-4, GPT-3.5-turbo, DALL-E, and Embeddings models — through Microsoft's Azure cloud infrastructure. Unlike using the OpenAI API directly, Azure OpenAI gives you enterprise features: private networking via VNet/Private Link, Microsoft's compliance certifications (ISO 27001, SOC 2, GDPR), content filtering, customer data privacy (your data is not used to train OpenAI's base models), and integration with Azure RBAC and monitoring.

Azure OpenAI requires an approved subscription — access is gated and must be requested via a form. Once approved, you create an Azure OpenAI resource, then create a deployment within it. A deployment binds a model (e.g. gpt-4) to a deployment name (e.g. my-gpt4) and a token quota (tokens-per-minute limit). You call the API using your deployment name, not the base model name. Multiple deployments with different models or quotas can coexist in one resource.

The chat completions API (/openai/deployments/{deployment}/chat/completions) accepts a list of messages with roles (system, user, assistant) and returns the model's response. The system message sets the model's behaviour and persona. The user message contains the prompt or question. Previous turns of the conversation are included in the messages array so the model maintains context across a multi-turn dialogue. Temperature (0–2) controls response randomness.

Prompt engineering is the practice of crafting effective input messages to get the best results from a language model. Key techniques include: System prompt — give the model a clear role and constraints; Few-shot examples — show the model 2–5 examples of the desired input-output format; Chain-of-thought — ask the model to think step by step before giving the final answer; Constraints — explicitly instruct the model on format, length, and what to avoid. Good prompts reduce hallucinations and improve consistency.

Embeddings convert text into a dense numerical vector that captures semantic meaning. Two pieces of text with similar meaning will have vectors that are close together in vector space (measured by cosine similarity). The Azure OpenAI embeddings endpoint (/embeddings) uses the text-embedding-ada-002 model to generate 1,536-dimension vectors. You use embeddings to build semantic search (find documents similar to a query), RAG (Retrieval-Augmented Generation), and recommendation systems.

Azure OpenAI includes a multi-layer content filtering system that runs on both prompts (input) and completions (output). It detects and blocks content across four categories: hate, sexual, violence, and self-harm. Each category has three severity levels (low, medium, high) and you can configure the filtering thresholds per category. Prompt injection attempts (jailbreaks that try to override the system prompt) are also filtered. Content filtering is on by default and cannot be disabled for most models without a use-case justification.

RAG grounds language model responses in your own data rather than relying solely on the model's training knowledge. The pattern: (1) index your documents into a vector store (e.g. Azure AI Search), (2) when a user asks a question, embed the question and search the vector store for relevant document chunks, (3) include those chunks in the system prompt as context, (4) the model answers based on the provided context. RAG reduces hallucinations and enables LLMs to answer questions about private, up-to-date data without fine-tuning.

Two approaches extend model capabilities with custom data. Fine-tuning trains a new model version on your examples, baking the knowledge into the model weights — best for teaching the model specific styles, formats, or behaviours that are hard to describe in a prompt. RAG retrieves relevant information at inference time and provides it in the context — better for large knowledge bases that change frequently, since you update the vector index rather than retraining the model. For most enterprise use cases, RAG is preferred due to lower cost and faster iteration.

Azure AI Studio (being rebranded as Azure AI Foundry) is a unified portal for building generative AI applications on Azure. It provides a playground for testing model completions interactively, tools for evaluating model quality against ground-truth datasets, a prompt flow designer for building and deploying RAG pipelines as REST APIs, and integration with GitHub and VS Code for developer workflows. Azure AI Studio simplifies the journey from model deployment to production AI application.

Azure OpenAI is priced per 1,000 tokens processed — where a token is approximately 4 characters or 0.75 words. Both input (prompt) and output (completion) tokens are billed. GPT-4 costs significantly more per token than GPT-3.5-turbo. Each deployment has a tokens-per-minute (TPM) quota that limits throughput to prevent abuse. If your application exceeds the quota, the API returns 429 TooManyRequests. You can request quota increases or implement client-side exponential backoff and retry logic.

Microsoft applies its Responsible AI principles specifically to generative AI. Groundedness — responses should be based on verifiable information (RAG helps). Transparency — users should know they are interacting with AI. Harm prevention — content filters block harmful outputs. Privacy — your data in Azure OpenAI is not used to train OpenAI's global models. Applications built on Azure OpenAI must include a use-case risk assessment and implement mitigations for identified harms as required by Microsoft's Acceptable Use Policy.

Test your understanding of Microsoft Azure Fundamentals (AZ-900) concepts from this lesson.

In this lesson you learned: Azure OpenAI Service provides enterprise-grade access to GPT-4 and other models with private networking and compliance features, the chat completions API powers multi-turn conversations using system/user/assistant message roles, and embeddings and RAG ground model responses in your private data to reduce hallucinations. Next up we explore Azure Arc for managing hybrid and on-premises resources through the Azure control plane.