Elumenotion

Is MCP Really Necessary? Why "APIs Are Hard" Is Overstated and MCP Is Oversold

Mon, 12 Jan 2026 00:00:00 GMT

Is MCP Really Necessary? Why “APIs Are Hard” Is Overstated and MCP Is Oversold

The sales pitch around Model Context Protocol (MCP) usually starts with a familiar complaint: APIs are too hard, too messy, too fragmented. The official MCP documentation leans into this story, emphasizing “standardization” through unified interfaces and protocols, promising “efficiency” via optimized context management, and highlighting “ease of use” with simple, intuitive APIs and a low barrier to entry. MCP Docs

OpenAI’s Agents SDK doubles down with the “USB‑C for AI” analogy. In that framing, “MCP is an open protocol that standardizes how applications provide context to LLMs. Think of MCP like a USB‑C port for AI applications. Just as USB‑C provides a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools.” OpenAI Agents SDK (MCP)

The implied premise is straightforward: direct API integration is so painful that you should prefer going through MCP whenever possible.

APIs as City Streets, MCP as Highways

A useful way to think about this is roads.

APIs are like roads. When MCP marketing talks about “APIs are hard,” it’s implicitly comparing itself to the worst dirt roads and pothole‑ridden alleys, then claiming it’s the only way to drive safely.

But most modern REST APIs aren’t like that at all. They’re more like well‑paved city streets: clear lane markings, sensible signs, predictable intersections. You can drive on them just fine, especially if you have a good map.

Legacy or undocumented systems are the exceptions. Those are the occasional dirt roads at the edge of town. They’re rougher, and you might need a map and some careful observation (developer tools plus an LLM) to figure out where they go, but you can still drive on them.

MCP, in this metaphor, is more like a standardized highway interchange system. It becomes valuable when you’re coordinating a lot of traffic between many different cities, and you want consistent on‑ramps, off‑ramps, and signage across a whole region.

The key point is that you don’t need a highway to drive across town. For many teams, the paved city streets they already have (their existing REST APIs, plus a bit of LLM help) are more than enough. Highways and interchanges are useful infrastructure at scale, but they don’t suddenly make ordinary streets “too hard” to use.

As a general claim, that doesn’t hold up.

Are APIs Really So Hard That You Need MCP?

If you listened only to MCP marketing, you might come away believing that working directly with APIs is a slog of glue code, inconsistent conventions, and integration hell. In reality, a large fraction of modern APIs are already straightforward to use (especially when you have a capable language model helping you).

Consider a very ordinary task: integrating a single, well‑documented REST endpoint. I recently asked an LLM agent to read the Brave Images Search API documentation and generate an OpenAPI schema for the /res/v1/images/search endpoint. Brave Images Search API Docs The interaction was almost comically simple. I pointed it to the documentation page and asked it to create an OpenAPI schema in JSON format. I told it not to use components. Then I asked it to simplify the request and response schemas, removing any properties that weren’t necessary to find and display images.

After a couple of iterations, I had a minimal schema. The request consisted of a query string, a count, and a safesearch flag. The response was an array of results, each with a URL, a thumbnail, and a title. That was enough to build a working, agent‑friendly tool that could find images and show them in a UI. There was no elaborate glue code, no “integration hell,” and no special framework. The API itself was clean: a single base URL, a single endpoint, one auth header, straightforward parameters, and JSON responses. The language model did the grunt work of turning prose documentation into a machine‑readable schema; I merely nudged it to match my preferences.

This Brave example is not a cherry‑picked miracle. A large fraction of modern APIs already look like this. They follow reasonable REST conventions, use simple authentication schemes such as API keys or bearer tokens, define clear parameters, and provide concrete examples. They speak JSON, often with an OpenAPI specification available or trivial to infer from the docs.

If your starting point is a halfway decent REST API plus an LLM, the idea that you need MCP just to make integration tractable becomes very weak. MCP may still be useful in some contexts (large organizations with many systems and strong governance concerns, for example) but the baseline claim that “APIs are too hard” is simply not true in many real‑world cases. MCP proponents often compare themselves to the worst APIs and the worst integration practices, then claim victory. When you compare MCP to competent use of existing APIs combined with LLM‑assisted schema generation, the value gap shrinks dramatically.

What Happens When There Are No API Docs (Do You Need MCP Then?)

At this point, someone usually raises a fair objection. The Brave example worked because the documentation was good. What about internal or legacy systems with no documentation at all? Isn’t that where MCP really shines?

Even then, you often don’t need formal API docs (or MCP).

Modern browser developer tools give you direct access to the real API traffic your application is already sending. For many internal features, that traffic is more reliable than whatever half‑written wiki page might exist. You can picture the process as a flow from a human using a feature in the browser to an agent calling a tailored API schema.

Here is the flow, visualized:

Flow: from using a feature in the browser to an agent using a tailored API schema.

You begin by opening the application in a browser and navigating to the feature you care about, perhaps a “Client Lookup” screen in a support tool. You open the browser’s developer tools and switch to the Network tab so you can observe HTTP traffic. Then you use the feature exactly as a normal user would: you type a client name, press search, and click through the results.

As you do this, you watch the requests that appear in the Network tab. Somewhere in that stream of calls, you will see a request that looks suspiciously like the client search you just performed, something like GET /api/clients?query=alice or POST /api/clients/search. Once you’ve identified the relevant request, you right‑click it. Modern browsers let you copy the request as cURL or in a similar format, capturing the exact HTTP call your application is making, including headers, URL, method, and parameters.

Next, you click the same entry and look at the Response pane, where you can see the JSON payload the server returned. You copy that response as well. At this point, you have two critical artifacts: a concrete request and a concrete response.

From these, either you or an LLM can extract the URL and HTTP method, identify the query string or body parameters, and infer the structure of the JSON response, including the key fields and how they nest. You can feed that request and response into an LLM and ask it to propose a clear, domain‑specific operation name. Instead of a generic searchClients, you might ask it to suggest something like searchClientsForSupportAgent. You can tell it to keep only the fields your agent actually needs (perhaps an ID, a name, an email address, and a status) and to ignore everything else. You can also ask it to suggest sensible defaults or constraints, such as a maximum number of results or a default sort order.

Once the model understands the shape of the request and response and the intent behind the operation, you ask it to emit a minimal OpenAPI schema that reflects this operation. The result is a small, tailored spec that describes exactly the “Client Lookup” behavior you care about, with a name and surface area that make sense for your agent. From the agent’s perspective, it now has a clean, well‑named, tightly scoped API for client search, even though no official documentation ever existed.

The important point is that even in the “no docs” scenario, the combination of browser developer tools, captured traffic, and an LLM makes describing an API almost trivial. You don’t need MCP. You don’t need a vendor‑maintained spec. You don’t need to wait for anyone to wrap the system. If the basic value premise is that APIs are hard to understand and describe, this workflow directly contradicts it. For many real applications, the friction is low enough that an MCP layer is a nice‑to‑have infrastructure choice, not a fundamental requirement.

What Are the Downsides of MCP for Cost and Quality?

So far, we’ve focused on why MCP is often unnecessary. There are also real downsides, especially if your goal is to lower cost and improve quality with tool‑based agents.

These downsides become clearer when you compare MCP’s default usage patterns to the kind of architecture described in “How to Lower Cost and Improve Quality with Tool‑Based Agents,” which relies on narrow, task‑specific tools, early filtering, and specialized worker agents. How to Lower Cost and Improve Quality with Tool‑Based Agents

Does MCP Encourage Overly Broad Tool Surfaces?

MCP makes it easy to expose a large catalog of tools from a server. That convenience comes with a risk: it becomes equally easy to expose too much.

In practice, tools from different MCP servers tend to use generic operation names like search, list, get, and create. Their descriptions are often vague or overlapping, promising to “search for items,” “retrieve records,” or “list resources” without clearly stating which domain they belong to or how they differ from one another. Their parameters are technically valid but semantically unclear. To a language model, this is not a neat toolbox; it is a junk drawer.

In a tool‑based architecture focused on cost and quality, the tools look very different. They are narrowly defined, named in domain‑specific terms, and designed as clear contracts for the agent. That tight surface area is a key lever for both cost and reliability. When MCP lowers the friction to exposing big, generic tool surfaces, it quietly increases token usage, confusion, and the risk of misuse. The more overlapping operations you show the model, the more ways it has to be slightly wrong.

Does MCP Make It Too Easy to Send Raw Data into the Model?

MCP’s default pattern is simple: the model calls a tool, and the tool returns a raw response directly to the model. There is no built‑in concept of filtering or compressing the data before it hits the main context. MCP Docs

In the tool‑based agents pattern, the architecture is deliberately different. Noisy work (like crawling and reading web pages) happens in specialized sub‑agents. Each sub‑agent sees a small, focused context, often just one page and the user’s query. Those sub‑agents can respond with “NOT FOUND” when nothing relevant appears, discarding irrelevant content early. Only filtered, relevant snippets ever reach the main agent. How to Lower Cost and Improve Quality with Tool‑Based Agents

This early filtering and context isolation is what keeps token usage low and answer quality high. With MCP, it is easy to fall into a lazier pattern: call a search tool, get back a large result set, and pass it all to the model; call a “read” tool, get back full documents, and dump them into context; let the main agent do all the filtering and extraction. The downside is straightforward: more tokens, more noise, worse answers. MCP does not force you into this pattern, but its convenience at the protocol layer makes it very tempting.

Does MCP Push Complexity into One Big Agent Instead of Specialized Workers?

The default story around MCP and hosted tools is that you have a model, it discovers tools, it calls them directly, and it interprets their responses. The OpenAI Agents SDK documentation describes how “the model lists the remote server’s tools and invokes them without an extra callback to your Python process,” and how “each MCP server supports tool filters so that you can expose only the functions that your agent needs.” OpenAI Agents SDK (MCP)

There is no structural encouragement in that story to split responsibilities between an orchestrator and specialized workers, or to use different models for different tasks.

In a cost‑optimized tool‑based design, you typically do the opposite. You use an orchestrator model for high‑level reasoning and planning, and cheaper, smaller models for extraction, filtering, and classification. You isolate contexts so each worker sees only what it needs. This pattern can significantly reduce cost while improving reliability. How to Lower Cost and Improve Quality with Tool‑Based Agents

MCP doesn’t forbid this architecture, but it doesn’t promote it either. Without deliberate design, teams tend to default to a single large model doing orchestration, extraction, filtering, and synthesis, all in one big context, with direct access to a large set of tools. That increases cost, because you are paying a premium model to do cheap work, and it makes behavior more brittle, because all the complexity is concentrated in one place.

Can MCP Give a False Sense That Tool Design Is “Solved”?

Because MCP standardizes how tools are exposed, discovered, and called, it is easy to assume that “tool design” is largely taken care of. You stand up an MCP server, define some tools, and now your agents can use them. Problem solved (or so it seems).

In reality, all the hard work remains. You still have to decide which tools belong in a given agent’s world, name them in domain‑specific, unambiguous ways, trim their surface area to what the agent actually needs, set defaults and constraints, and write prompts and descriptions that explain when and why to use each tool. MCP does not remove any of that effort. At best, it makes some plumbing more convenient.

The danger is that teams think, “we have MCP, so our tools are agent‑ready,” and under‑invest in the design work that actually drives cost and quality. The result is a system that looks standardized on paper but behaves unpredictably in practice.

Does MCP Really Deliver Composable Tools?

Another major claim around MCP is composability. Because tools all speak the same protocol, the story goes, agents can discover them dynamically and chain them together. The OpenAI Agents SDK documentation explains how hosted MCP tools let the model list a remote server’s tools and invoke them without an extra callback to your Python process, and how each MCP server supports tool filters so that you can expose only the functions your agent needs. OpenAI Agents SDK (MCP) The narrative that emerges is one of a plug‑and‑play ecosystem: tools that “just work together” because they share a protocol.

This sounds great. It is also mostly wishful thinking.

MCP standardizes how tools are called. It does nothing to standardize what they mean. In practice, tools use generic operation names, vague descriptions, and parameters that are technically valid but semantically unclear. To a language model, this is not composability; it is ambiguity.

The fact that everything speaks MCP does not tell the model which search finds images, which search queries CRM contacts, which search digs through logs, or how to chain a list, a get, and an update across different domains. The hard part is not calling a function over a protocol. The hard part is semantic alignment: giving the model a set of tools that are named, scoped, and documented in a way that matches the agent’s job.

A thin, bespoke API description layer gives you exactly that control. Instead of exposing a generic search operation, you rename it to something unambiguous in your agent’s world, such as searchBraveImagesForDisplay, getCrmContactsForEmailCampaign, or lookupProductBySkuForCheckout. These names carry domain intent in a way that “search” never will. You trim the surface area of each tool to what the agent actually needs, set defaults so the model doesn’t have to decide on every detail, and embed assistant‑specific guidance right into the descriptions. You might tell the model to use a particular tool only when the user explicitly asks for images to show in the UI, or to fetch customer records with another tool before drafting an email.

MCP does not prevent you from doing any of this, but it does not do it for you either. The claim that MCP gives you composable tools because they share a protocol ignores the real bottleneck, which is semantics and user experience for the model. Protocol‑level composability is the easy part. Semantic composability is the hard part, and MCP does not solve it.

Does Tool Discovery in MCP Mean the Model Understands the Tools?

A third pillar of the MCP story is discovery. The MCP docs talk about “standardization” and “ease of use,” presenting MCP as a way to “simplify development” and “optimize context management.” MCP Docs The OpenAI Agents SDK shows how hosted MCP tools allow the model to list and call tools from remote servers without custom listing and dispatch logic. OpenAI Agents SDK (MCP)

Again, this sounds nice: the model can “discover” tools and just start using them. But discovery is not understanding.

Discovery gives the model a list of tools, their signatures, and short descriptions. It does not give the model a sense of which tools are appropriate for this agent’s responsibilities, a strategy for choosing between overlapping tools, a plan for sequencing tools in multi‑step workflows, any real notion of business rules or safety constraints, or a concept of when not to call a tool even if it looks superficially relevant.

Without strong system prompts and curated tool surfaces, models with a large discovered toolset behave exactly as you would expect. They overuse some tools and ignore others. They call tools redundantly. They misread vague descriptions and pick the wrong operation.

All of the hard work remains in tool selection, in deciding which tools even belong in this agent’s world; in naming and scoping, in making tools match the domain and the agent’s role; in prompt design, in telling the model when and why to use each tool; and in policy and safety, in enforcing constraints outside the model. MCP does not remove any of that effort. At best, it makes some plumbing more convenient. The idea that discovery plus a protocol is enough to make tools “just work” with agents is optimistic marketing.

So When Is MCP Actually Worth Using?

None of this means MCP has no value. There are real benefits in certain contexts.

MCP can standardize how tools are exposed and called across teams and products, and can centralize some aspects of connectivity, especially when you use hosted MCP tools. MCP Docs OpenAI Agents SDK (MCP) It can provide large organizations with a common substrate for governance, logging, and access control. These are meaningful advantages (but they are infrastructure and governance wins, not magic bullets for “APIs are hard” or “agents can’t use tools”).

If we are honest about the value proposition, it looks different from the USB‑C analogy. USB‑C solved a brutal mess of incompatible connectors. Modern HTTP APIs with JSON and a capable language model are nowhere near that level of chaos.

If your primary goals are to lower cost and improve quality, patterns like orchestrator‑and‑worker agents, narrow, domain‑specific tools, early filtering and context isolation, and using different models for different tasks matter far more than whether you use MCP. Those are the patterns described in “How to Lower Cost and Improve Quality with Tool‑Based Agents,” and they apply whether your tools are exposed via MCP, direct HTTP, or something else entirely. How to Lower Cost and Improve Quality with Tool‑Based Agents

For many teams, the simplest and most effective path remains to use the APIs they already have, to capture real traffic when documentation is missing, to let an LLM help generate a thin, bespoke OpenAPI spec, and to curate a small set of tools with clear prompts that explain how those tools relate to the agent’s job. MCP can be a useful piece of infrastructure, especially at scale. But as a universal answer to “APIs are hard” or “agents can’t use tools,” its case is overstated.

How to Lower Cost and Improve Quality with Tool-Based Agents

Sun, 11 Jan 2026 00:00:00 GMT

A Deep Dive into the Search + ReadWeb Design (vs. MCP)

This article shows you how to build agent systems that are cheaper to run and produce better answers. I'll use a concrete example - the Search assistant in GuideAnts Notebooks - to illustrate the pattern, but the principles apply to any agent system that orchestrates tools.

The core ideas are: abstract complexity behind precise tool definitions (this is where OpenAPI descriptions shine), isolate noisy work in specialized sub-agents, and match model capability to task complexity. I'll also explain why this approach delivers results that a typical Model Context Protocol (MCP) setup cannot.

The Concrete Flow: Search → crawl → ReadWeb → GetContentFromUrl

The screenshot below illustrates a single user turn where the Search assistant orchestrates multiple tools and subagents.

The process begins when the Search assistant receives the user's query and calls the crawl tool. The crawl tool returns a lightweight set of page titles and URLs relevant to the query - metadata only, with no full page content yet.

For each candidate URL, Search invokes the ReadWeb assistant in an isolated context. ReadWeb uses the GetContentFromUrl tool to fetch and parse the page, then decides whether the page actually contains content matching the user's query. If no relevant content is found, it simply returns NOT FOUND. However, if it finds relevant material, it extracts and returns only the specific snippets or sections of interest, rather than the entire page.

Finally, the Search assistant aggregates the results. Pages that returned NOT FOUND are effectively discarded, so only the extracted, query-relevant content is brought back into the main context for answer generation. As you can see in the trace (AGENT Search → TOOL: crawl, followed by repeated AGENT Read Web → TOOL: GetContentFromUrl calls), the bulk of tokens are consumed in the main Search turn, while the ReadWeb interactions remain smaller and cheaper.

Why This Design Works: Precise Tools, Isolated Contexts, Early Filtering

Three mechanisms drive the token savings and quality improvements in this pattern.

Precise Tool Definitions. Both crawl and GetContentFromUrl have simple, focused OpenAPI descriptions that hide complexity from the LLM. The LLM never sees how crawling works, how HTML is parsed, or how relevance is scored internally. It only sees a clear contract: "give me URLs for this query" or "extract relevant content from this URL, or say NOT FOUND." This keeps prompt size small and reduces the chance of incorrect tool usage.

Isolated Contexts. Each page is handled in a separate ReadWeb context. The sub-agent sees only the query and a single URL; it returns either NOT FOUND or a compact snippet. We never load multiple full web pages into the main Search context. This isolation prevents the main agent from being overwhelmed by irrelevant content and keeps each sub-call cheap.

Early Filtering. Because ReadWeb can respond with NOT FOUND, irrelevant pages are discarded before they ever reach the main context. The Search assistant only sees pages that actually contain matching content - and specifically, only the parts that matter. This avoids the common failure mode of stuffing the context with everything a crawler finds and hoping the LLM picks the right pieces.

The result: fewer tokens, less noise, and an LLM that can focus on reasoning and synthesis rather than sifting through garbage.

Model Specialization: The Cost Multiplier

Beyond token savings, this design enables a powerful cost optimization: running different models for different roles.

The Search assistant handles the complex work - reasoning about the user's intent, synthesizing information from multiple sources, and writing a coherent answer. This requires a capable (and expensive) model.

The ReadWeb assistant has a much simpler job: fetch a page, decide if it's relevant, extract the good parts. This is mostly pattern matching and extraction, not deep reasoning. It can run on a mini model that costs approximately 10x less per token.

Because the architecture isolates these roles, you can make this tradeoff cleanly. Most of the "web reading" cost is paid at mini-model rates. The expensive model is reserved for the synthesis work that actually requires it. In a typical search query that touches 5-10 pages, this can reduce total cost by 50% or more compared to running everything on a single large model.

How This Differs from MCP

The Model Context Protocol (MCP) standardizes access - it gives you a consistent way for agents to discover and call tools across different servers. This design optimizes interfaces and execution.

MCP typically exposes tools at the server level and returns raw tool responses directly to the LLM. The client or LLM is responsible for filtering and interpreting the data. This is fine for simple tools, but for complex operations like web search, it means the LLM sees everything the tool returns - relevant or not.

In contrast, this Search design uses assistant-specific tool abstraction. Search sees a high-level crawl tool and a ReadWeb assistant (not a raw HTTP client). ReadWeb sees a specialized GetContentFromUrl tool (not a generic web fetcher). Response filtering and compression happen at the tool or assistant level - NOT FOUND vs. "here is the exact snippet" - before anything reaches the main context.

You can think of it this way: MCP standardizes how agents talk to services. This pattern optimizes what agents see and how much they pay to see it.

The two approaches aren't mutually exclusive. You can use MCP for tool discovery and transport while still applying these abstraction and specialization patterns within your agent architecture.

Takeaways for Any Agent System

These patterns apply far beyond search. Any time you're building an agent that orchestrates tools, consider:

Specialized Roles. Split complex workflows into an Orchestrator that manages high-level reasoning and Specialized Workers that handle noisy or expensive sub-tasks. The Orchestrator should never see raw, unfiltered data from the outside world.

Precise Tool Definitions. Don't expose raw APIs to your agents. Create task-specific abstractions with clear contracts. OpenAPI descriptions are ideal for this - they let you define exactly what parameters the LLM can use and what shape of response it will receive, hiding everything else.

Model Specialization. Match model capability to task complexity. Use mini models for extraction, filtering, and formatting. Reserve your strongest models for synthesis and reasoning.

Filter at the Source. Don't dump raw tool output into your main context and hope the LLM figures it out. Filter early, compress aggressively, and let irrelevant results die in the sub-agent that found them.

The goal is simple: give your main agent only the signals that matter, at the lowest possible cost.

Faster, Cheaper, Better: Why Guides and Assistants Beat Single-Model AI

Sat, 10 Jan 2026 00:00:00 GMT

Faster, Cheaper, Better: Why Guides and Assistants Beat Single-Model AI

When you ask an AI to do something complex (like creating a podcast, designing a cover image, and producing a video) it's tempting to throw the entire request at a single model in a single conversation thread. But there's a smarter, faster, and cheaper way: break the work into guides and assistants, each with its own role, tools, and even its own model.

In this article, we'll walk through a real example - "Today in History Podcast Video with Python" - to show how modular AI architecture, combined with parallel execution, can deliver high-quality results quickly while keeping costs under control.

What Are Guides and Assistants?

Before diving into the example, let's define the core concepts:

Guide → The project manager. Understands the overall goal, breaks it into steps, and delegates tasks to specialized assistants.
Assistants → Specialists. Each is narrowly focused: searching the web, reading pages, generating media, or executing code.
Crew → The team inside a guide. A guide can include multiple assistants, each with its own tools and model configuration.

Think of it like a newsroom: The editor-in-chief (guide) assigns reporters (assistants) to cover different angles of a story. Designers and technicians (other assistants) prepare visuals and audio, all coordinated into one final product.

Types of Assistants

Global Assistants are pre-configured by the platform and available to all teams. They include:

Search (Web Search and Scrape)
Code Executor
Read Web
Media Creator
Diagrams
And many more specialized tools

Custom Assistants are created by your team, tailored to your specific needs, with custom tools, files, and instructions. They're only available to your team and can be reused across multiple guides.

The "Today in History" Project: A Real-World Example

Let's examine how a guide orchestrated multiple assistants to complete a complex multimedia project. This example was built using GuideAnts Notebooks, an AI-powered workspace that makes it easy to create guides, configure assistants, and build modular workflows like the one shown here.

User request:

"Create a podcast about today in history and a cover image, then combine the two to make a video with sound using Python."

Guide's plan:

Search for historical facts for January 8
Read and extract relevant details from web pages
Create a podcast script and record narration
Design a cover image
Combine audio and image into a video using Python

Result: Here's the completed video that was generated by the guide and its crew:

How the Crew Worked Together

The guide orchestrated four specialized assistants to complete this task:

Search Assistant (Model: GPT-4.1) - Found "Today in History" events from multiple sources
Read Web Assistant (Model: GPT-4.1 Mini) - Loaded pages and extracted key facts
- This step consumed 35,528 tokens across multiple page reads, so using a cheaper model saved significant costs
Media Creator Assistant - Produced podcast audio narration and cover image
Code Executor Assistant - Combined the audio and image into a final .mp4 video file

Cost Breakdown

From the conversation activity report:

Total Tokens: 86,298
Total Cost: $0.2128
Assistant Steps: 4
Tool Calls: 14

The heavy web reading tasks (which consumed over 50,000 tokens) were handled by GPT-4.1 Mini, keeping costs low without sacrificing quality for extraction tasks. The guide's orchestration and creative synthesis used GPT-4.1, ensuring high-quality decision-making where it mattered most.

Quality Through Context Isolation

One of the most significant advantages of the guide + assistants architecture is how it handles conversation context.

The Single-Thread Problem

In a single model, single-thread approach:

Every step shares the same conversation context
Large inputs (like full web pages) can bloat the context window, pushing important instructions out
Raw data from early steps can distract or contaminate later outputs
The model must juggle multiple task types simultaneously, reducing focus

The Invocation Tree Advantage

In the guide + assistants invocation tree:

Each assistant runs in its own local context, isolated from other assistants
The Read Web assistant sees only the page content and extraction instructions - it's not burdened with the podcast script or Python code steps
The guide receives clean, distilled summaries instead of noisy raw data
Each assistant can fully focus on its specialized task without irrelevant context noise

Result: Higher accuracy per step, and the guide's reasoning stays sharp because it maintains a lean context focused on orchestration and synthesis.

Context Isolation Comparison

Aspect	Single Model, Linear Context	Guide + Assistants Invocation Tree
Focus per step	Mixed, all steps share context	Laser-focused per assistant
Context size	Can bloat quickly	Lean guide context, isolated assistant contexts
Model fit	One-size-fits-all	Task-specific models
Noise risk	High (irrelevant tokens persist)	Low (clean outputs passed back)
Quality consistency	May drift over long context	Maintained per assistant role

Cost Efficiency Through Model Specialization

Perhaps the most practical benefit of this architecture is the ability to optimize costs by matching models to tasks.

Strategic Model Assignment

Token-heavy tasks (like reading multiple web pages) use GPT-4.1 Mini - cheaper, faster, and well-suited for extraction
High-reasoning tasks (like synthesizing the podcast script) use GPT-5.2 - more capable for complex decision-making
Creative generation tasks can use models tuned specifically for media output

Beneficial Impacts

In the January 8 project, the Read Web assistant processed over 50,000 tokens across multiple web pages. If this had been done with GPT-4.1 instead of GPT-4.1 Mini, the cost would have been significantly higher. By using the right model for the right job:

Cost savings: The cheaper model handled high-volume, low-complexity extraction
Quality preservation: The premium model was reserved for tasks requiring deeper reasoning
Performance: Lightweight models process faster on large payloads

The work performed by crew members is done using a specific model for each assistant, outside of the main context window. This is crucial because token-heavy operations (like reading web pages) can consume enormous amounts of tokens, and using an expensive model for these tasks would quickly become cost-prohibitive.

Speed Through Parallel Operations

One of the most powerful features of this architecture is parallel execution. When assistants run independently with their own models and contexts, tasks that don't depend on each other can execute simultaneously.

How Parallelism Works

Assistants run at the same time when tasks don't depend on each other
Search can query multiple sources while Read Web processes several pages in parallel
Media Creator can start image generation while audio is being recorded
Code Executor can prepare the video script while assets are being finalized

Impact on Speed

Sequential execution: Time = sum of all task durations
Parallel execution: Time ≈ longest single task duration

For the January 8 project:

Multiple web pages were read simultaneously
Audio and image generation overlapped
The final video assembly happened immediately after assets were ready

This turns what would be a multi-minute sequential process into seconds or a minute of wall-clock time.

Parallelism + Invocation Tree = Optimal Architecture

When combined:

Invocation tree provides isolation, model specialization, and cost control
Parallel execution provides speed and scalability

This architecture is essentially AI multiprocessing - the guide acts like a scheduler, assistants act like worker threads, and each has its own optimized environment. The system's excellent support for parallel operations is what makes the modular guide/assistant architecture practically viable for large, multi-step AI projects without sacrificing quality or blowing through budgets.

The Big Picture Benefits

Let's summarize the key advantages of this modular approach:

Role Separation - Guides orchestrate, assistants execute. Clear boundaries prevent confusion and overlap.
Context Isolation - Each assistant focuses only on what's relevant, avoiding context bloat and contamination.
Model Optimization - Assign the right model to the right task, maximizing quality per dollar.
Cost Control - Save money on high-volume steps without losing quality on critical reasoning tasks.
Parallel Execution - Dramatically reduce total run time by running independent tasks simultaneously.
Transparency - Activity reports show exactly who did what, with token and cost breakdowns for optimization.
Reusability - The same assistants can be plugged into other guides, building a library of specialized capabilities.
Scalability - Add or swap assistants without redesigning the whole workflow.
Maintainability - Update an assistant's instructions or tools, and all guides using it benefit instantly.
Quality Consistency - Each assistant maintains its specialized focus, preventing quality drift over long contexts.

Conclusion

The "Today in History Podcast Video with Python" project demonstrates how guides and assistants - combined with parallel execution - create fast, cost-efficient, and high-quality AI workflows.

Instead of forcing one model to handle everything in a bloated context, this architecture:

Breaks work into specialized roles
Matches models to tasks
Runs tasks in parallel
Keeps the guide's context lean for clear reasoning

This is faster, cheaper and better.

Getting Started

If you're building AI-powered workflows, start thinking like a project manager:

Define guides to orchestrate complex workflows
Create assistants for specialized tasks (or use global assistants where appropriate)
Pick the right model for each job - match the model's capabilities to the task's requirements
Run them in parallel when possible to maximize speed
Monitor activity reports to optimize token usage and costs

The modular guide/assistant architecture turns AI workflows into well-structured, reusable systems. Just like well-designed software projects, this approach gives you the building blocks to create powerful, efficient, and maintainable AI applications.

Ready to build your own? The example in this article was created using GuideAnts Notebooks, which provides the platform and tools to design guides, configure assistants with custom models, and orchestrate complex workflows. You can start building similar modular AI workflows today - Get Started now.

Why "Guides" - from prompts to workflows

Thu, 08 Jan 2026 00:00:00 GMT

Why "Guides" - from prompts to workflows

Most teams don't need "a chatbot." They need guidance to find and use information to perform work using natural text, voice and vision. This distinction matters because it shifts the conversation from "let's add AI" to "let's make this workflow reliable."

This post introduces the core mental model we'll use throughout the series. Guides are coordinators for getting work done. Assistants are specialized helpers the guide can delegate to. Tools are capabilities like APIs, retrieval, code execution, and integrations. Knowledge and files are sources of truth the workflow can reference. And publishing or embedding turns internal guidance into product experiences.

Why "prompting" isn't an architecture

Prompts are great for exploration, but they're fragile as a foundation for applications. When you rely on prompts alone, you run into several problems.

First, there's the issue of inconsistent outputs. A "pretty good" answer isn't good enough for a workflow step that ships to customers. You need reliability, not occasional brilliance. Second, when results are wrong, it's hard to tell which step failed because the process is hidden inside a single prompt. Third, there's no modularity. The moment you add "and also do X," you create a mega-prompt that's difficult to test and maintain. Finally, prompts are hard to operationalize. You need controls around tool access, authentication, limits, retention, and monitoring.

Production applications require guided work, not vibes.

The "Guide" mental model

A Guide is an agent designed like a product feature. It has a purpose, boundaries, and a repeatable operating procedure. This is the core shift in thinking.

When you define a guide, you should be able to answer several questions. What job is it responsible for? What inputs does it require? What tools is it allowed to use? What outputs does it produce, including both format and quality bar? When does it delegate, and to whom? How does it handle failure? How do you measure whether it's working?

Think of a guide as the orchestrator in a system, not the whole system. It's the thing that knows what needs to happen and coordinates the pieces to make it happen.

From one agent to many: crews and specialized assistants

In real applications, one agent doing everything is a reliability trap. The more responsibilities you pile onto a single agent, the more likely it is to fail in subtle ways.

Instead, use a crew. The guide stays focused on planning, delegation, and synthesis. Specialized assistants do narrow work like research, code review, security analysis, formatting, and data extraction.

This is the same reason modern software is built from services and modules instead of one giant function. Separation of concerns isn't just good architecture; it's how you build systems that can be tested, improved, and trusted.

Tools: when "AI" becomes "software"

Tools are what turn chat into application behavior. Without tools, an AI can only describe what should happen. With tools, it can actually do things.

Tools let you query internal data, create and validate artifacts, run deterministic checks, integrate with third-party systems, and enforce policy about what's allowed and what's not. If your AI can't call a tool, it can't reliably complete real tasks. It can only describe them.

Extend your AI beyond your team

Guides are used interactively inside notebooks, but publishing is a key value proposition. Treat a guide, along with its crew, tools, and knowledge, as a reusable component you can deploy across many surfaces.

Build and test guides internally, then publish them anywhere. Once defined, these guides can be rendered anywhere via standard web components. For example, the guideants-chat widget allows reuse. But the core idea is broader: your guide becomes a portable capability you can embed into new or existing systems.

You can embed AI chat widgets directly into your website, customer portal, or internal tools. You can create public-facing assistants for support, documentation, or interactive experiences. You can integrate with your app using client-side tool execution for dynamic, context-aware conversations. And you can maintain control with authentication options, usage limits, and real-time analytics.

A concrete reference stack

Throughout this series, we'll reference two complementary ways to build guided agent workflows.

GuideAnts is the product and workflow layer. It's where you author guides, crews, tools, and knowledge, then publish and embed them. You can learn more at GuideAnts.

AntRunner is the code and implementation layer in .NET. It provides tool-based assistants, streaming, external tool calls, and OAuth token forwarding. It's open source and available at AntRunner on GitHub.

You can use either independently, but together they illustrate the full spectrum from designing agent workflows to implementing them.

What "good" looks like: a guide design checklist

Use this checklist when you turn a prompt into a guide.

Clear scope: what the guide does and does not do.
Repeatable procedure: the steps the guide follows every time.
Structured output: a stable format like headings, tables, or JSON schema, whatever your app needs.
Tool boundaries: only the tools required, no "tool buffet."
Delegation rules: when to call specialized assistants and what each assistant is responsible for.
Test prompts: five to ten golden test cases that represent real user requests.
Observability hooks: what you'll monitor, including cost per run, tool call counts, error patterns, and user success rate.

We'll expand each of these in upcoming posts.

What's next

In the next post we'll build a reusable assistant using Search as an example. You'll see how to define clear boundaries, structure tool calls, and create something that can be tested independently and composed into larger workflows.

If you're building an application and want a reliable AI feature, start here. Don't ask "what prompt should I use?" Ask: what work am I trying to guide and standardize?

ToolBasedAgents

Mon, 06 Jan 2025 00:00:00 GMT

Tool-based Agent Pattern

I ended my last article, Retrieval Augmented Generation is an Anti-pattern by saying, ”Cheaper and better models with large context windows allow us to use the LLM to manage generation and greatly simplify the architecture by giving it the entire context and allowing it to manage retrieval and any other tools it needs to fulfill its purpose. The use of knowledge sources becomes just like any other thing the agent knows how to do.”

This article is about the pattern I use in Go AntArmy!, AntRunner, and ask@antarmy.ai. I call it the Tool-based Agent Pattern.

Attribution

One aspect of calling something a pattern is that it is commonly used. The OpenAI Assistants API is an example, as is AntRunner. It is documented elsewhere as Agentic RAG, but I maintain that “Agentic RAG” is not in fact a RAG pattern because it is not a given that an agents tools involve retrieval.

If you disagree, ask yourself if a tool like Code Interpreter (a python sandbox) is a form of retrieval. The agent’s LLM makes all the decisions. There is no ‘augmentation’. The agents writes code based on the model’s training. It sin’t RAG because there is no ‘R’etrieval and no ‘A’ugmentation, only ‘G’eneration. Furthermore, nothing in the name suggests there are, generically, tools at the center of the pattern.

If an agent can write notes via a tool, is it not doing retrieval, adding a new tool to the same API that can read notes doesn’t change the architecture pattern, can your design become RAG by including a read operation?

Thanks to Mahmoud Hassan | LinkedIn for asking about this in a comment on the last article!

Terminology aside, this is a common approach you should understand because: it works, it is easy to understand, it is versatile, and it is architecturally sound as you can extend it without making fundamental changes to the pattern.

Software Architecture Methodology

This article will explain the pattern using a software architecture methodology known as Structured Analysis and Decomposition.

Structured Analysis and Decomposition

Structured Analysis and Decomposition (SA/SD) is a systematic approach used in software engineering to analyze business requirements and develop specifications for converting these requirements into functional systems. This methodology emerged in the 1970s and is characterized by its focus on breaking down complex systems into smaller, manageable components. The primary goals of SA/SD are to improve clarity, enhance communication among stakeholders, and ensure that the system meets user needs.

You can read more about Structured Analysis on Wikipedia.

Data Flow Diagrams (DFDs) in Structured Analysis

Data Flow Diagrams (DFDs) are a crucial tool in the structured analysis phase of SA/SD. They visually represent the flow of data within a system, highlighting how information is exchanged between processes and external entities. Here’s how DFDs are utilized in structured analysis:

Modeling Data Flow: DFDs depict the movement of data through processes, showing inputs, outputs, and data stores without detailing the timing or sequence of data exchanges.
Identifying Processes: Each process in a DFD represents a specific function that transforms input data into output data, facilitating a clear understanding of system operations.
Clarifying Requirements: By illustrating data flows, DFDs help stakeholders visualize system functionality, ensuring that all necessary processes are included.
Facilitating Communication: DFDs provide a common framework for developers and stakeholders, improving understanding and collaboration throughout the project lifecycle.

Elements of Data Flow Diagrams

Processes: Represented by circles or ovals, indicating where data is processed.
Data Stores: Shown as open-ended rectangles, representing where data is stored.
External Entities: Illustrated as squares, indicating outside actors that interact with the system.
Data Flows: Arrows that show the direction of data movement between processes, stores, and entities.

The goal is communication. You don’t need to know a lot of technical details and there is no code.

Context diagram

Drawing DFD’s is like a game where if you follow the rules honestly, you can be reasonably sure the design is correct and complete.

You start at the top…

And work your way down…

With a few simple rules:

Inputs and outputs must match: In the diagrams above each has the same inputs and outputs such that we can replace the first drawing’s single process with the second drawing
No Miracles Allowed: The data in the flows must have a source. When you are in a decomposition layer and it requires data that isn’t plausibly coming from somewhere, there is a hole in the design or the layers above are structurally deficient
No black holes: The data produced by a process must go somewhere. If not, there is a hole in the requirements or the layers above are structurally deficient
No avoidably vague process names: Processes with names like Process data are a sign that the design is not finished or completely thought through.
Iterate until done: Keep working until you have broken the problem down to sufficient detail, fixing mistakes as they are found.

Requirements and Reasons Why

Architecture always depends on requirements. Without them, how do you judge anything? In turn, each requirement needs justification.

Cohesively functional, stable, and measurable

First and foremost, what we build needs to work and we need to be able to prove it works. This is not easy with traditional software where operations are generally deterministic which means that given the same input we will get the same output.

An agent is inherently nondeterministic which means that we really don’t know what will happen. The natural language processing has randomness built-in and is a giant statistical model. In fact, when the agents work together in a network where one agent can choose to ask other agents, the resulting system is indeterministic (see ‘Notes from Carl Hewitt on the Actor Model’. I won’t go off on a tangent, but this is the specific reason why it is helpful to apply the actor model of computing. If you don’t get what I mean when I say an agent is actor-like, don’t worry about it but if you are interested in distributed systems designs, I’d encourage you to expand your horizons in this direction.

We need ways to test and measure the performance of an agent because there is a likelihood that a given operation will yield unexpected, possibly undesirable, results.

Defining functional agents

The inputs to an agent are practically unbounded, but we can narrow the scope of what the agent can do based on its instructions which include the tools the agent can use. This is necessary because a single agent is limited by the capabilities of today’s models. In an agentic system we will use multiple specialized agents with an array of available tools in each.

I find it helpful to think of an AI agent in terms of cohesion and modularity. Cohesion and modularity are important concepts in software design and architecture:
- Cohesion refers to how closely related and focused the responsibilities of a single module or component are. A module with high cohesion performs a specific task or a set of related tasks, making it easier to understand, maintain, and reuse. High cohesion is desirable as it leads to systems that are easier to manage and less prone to bugs.
- Modularity is the degree to which a system's components can be separated and recombined. A modular system is composed of distinct modules that can be developed, tested, and updated independently. This separation helps in organizing code, improving maintainability, and facilitating collaboration among multiple developers.

We want high cohesion within agents and strong modularity across the system.

The importance of stability

Depending on the model there is a point where the performance will degrade if the instructions are too complex. Furthermore, we might want to evaluate other models or alter the parameters.

The instructions with tools, the model, and the model parameters are the Agent Configuration. We can measure the performance of the agent using real or synthetic data and iterate and (hopefully) improve by including better instructions, using different models, and altering the model parameters. When we start to build and improve an agent or system, we need to manage the Agent Configuration as a versioned artifact. For now, at the architectural level this means the Agent Configuration is a unit with its own management lifecycle.

Measurability requires cohesion and stability

Measurability requires us to say up front what we are measuring; what the agent does and what its limits are. The measurement allows us to iterate for better results and gives us a way to consider how the agent fits into a system and is a tool we can use to identify additional requirements to compensate for its deficiencies. Perfection is impossible but we can design working systems with imperfect pieces if we build in ways to compensate for faults.

Doing this kind of measurement and tuning resulted in the creation of AntRunner which makes it easy to package a assistants in easily version-able forms, deploy them, and run them with test scripts.

Cohesive AntArmy Ants

Go AntArmy! comes with a number of predefined agents.

Each Ant has a clear purpose, and the tools support the purpose. They work together on a thread directed by the user. We can add new ants without altering the existing, tested, and stable ants.

Modular and reusable

Agents should be modular and reusable. AntArmy is a chat interface and an interactive UI. AntRunner supports scripts, applications like jobs, and individual services. These facts are evidence of the pattern’s modularity and support for reuse.

Pre-defined orchestrations

This diagram shows a job which extracts meaning from transcripts. The steps in this orchestration are explicit. The individual agents use the same pattern as the AntArmy Ants, but there is no UI or human in the loop.

SpeakerNameIdentifier has tools to read a transcript from a database given its ID, and write records that identify the speaker names and roles from the context of the transcript
SpeakerNameIdentifierEvaluator has instructions to evaluate the identified speakers and a tool to identify speakers missing in the database
SpeakerInsightsExtractor extracts meaning from passages in the transcript and a tool to insert the results into the database

Self-directed orchestrations, aka Agent Swarms

Consider that a ‘Tool’ can be anything. The assistants API supports file retrieval, code interpreter, and functions. This means that a tool can be another agent which, in turn, can also use other agents as tools.

AntRunner contains a sample notebook which illustrates this using a WebSearchAgentAnt which has a tool for web search and another tool which is an agent named WebPageAgentAnt which reads a web page and returns the content that answers the question, if any. The notebook is a cut-down and faster version of ask@antarmy.ai which uses a similar design with more reasoning and quality checks.

I will write more about these agents in future articles.

Tool-based Agent Pattern

My apologies for the long preamble, the rest of the article is about the actual pattern, starting at the top.

For clarity, these diagrams show the data elements in the flows, but in truth the bigger system with many agents would show them at a higher-level because the threads and thread runs are shared by all the agents in the system. We can create a thread by using one agent, and then continue it with others.

Tool-based Agent

1. Assemble Thread Run

Agents work against a thread which represents the conversation. Invoking an agent either creates a new thread or continues an existing thread.

Inference (using the LLM) is inherently stateless and its input contains everything the model needs the produce a response including:

The input message
Previous messages, i.e. the thread
The model parameters, e.g. temperature
The agent’s specific instructions
Any additional context required, e.g. today’s date, the user’s location
Tool definitions

1.1 Add Instructions & Model Parameters

At the beginning of the flow, we have the input message and optionally a thread identifier. Given that the input message could be an entire previous conversation, the persisted thread and thread run are optional. Implicit and not shown is the agent’s identity or name. The process begins by combining the input message such as:

“What is my name”

With the agent instructions, such as:

“You use a tool to get the user’s profile”

And the model parameters such as:

top_p = 0.7  
Temperature = 0.5

Note that names and number of model parameters depend on the specific model.

1.2 Add Context Messages

Frequently an agent will need extra information such as the current date and time or a user’s preference. If your agent has a code sandbox, you could use it to get the current date and time, but that uses tokens and time. A better alternative is to add messages with whatever information is required before starting the run.

AntArmy Chat includes tools that an agent can use to save preferences such as the current OneNote notebook. The agent instructions specify which are added.

1.3 Add Tool Definitions

Generically, a tool definition is something in the prompt which we can translate to running an operation and getting a result. In the case of the Open AI Assistants API we configure file retrieval and the python sandbox in the assistant definition and can also specify other tools using JSON. The Cat Facts AntRunner sample. Incudes this a tool named GetCatFacts.

{  
  "name": "GetCatFacts",  
  "description": "Retrieve and query facts",  
  "parameters": {  
    "type": "object",  
    "properties": {},  
    "required": []  
  },  
  "strict": false  
}

Notice that the JSON doesn’t have any information about how GetCatFacts works. The LLM’s instructions do not include information about this because the LLM is only a model - it can’t run code or use the internet. The LLM will ‘call’ the tool by asking for it via generation, in this case GetCatFacts(), the application using the model, in this case the AntRunner library, to give it the result of using the tool somehow. Accordingly, implementations of this pattern can vary widely without altering the pattern.

We will get into the mechanics of tool calling later, but for now, note that here we are translating a complete definition of the tool including such information as the server URL and auth, into a simplified subset.

1.4 Get or Create Thread

Finally, a thread is created or retrieved if there is history, and everything is packaged to send to the LLM to run the thread.

2. Run Thread

The assembled “thread” already has a few pieces of information: the system’s settings (Agent Config), the latest messages in the thread, and an ID to keep everything organized. First, we create a new “Thread Run.” Think of this as opening a new session dedicated to handling the conversation with an agent. It can be a different agent than those used previously.

Next the LLM generates an actual response for the conversation. It looks at the conversation’s history, applies the agent’s behavior rules, and then crafts the next output message. If the system decides it needs outside help - like calling a search API or some other external tool - it creates a "tool call" to describe what it wants the external service to do. In the next step, the system handles those tool calls. If the new information from the tools change what needs to be said next, the process loops back to generating output again, this time using the fresh tool results. This cycle continues until the conversation produces a final response or no longer needs outside services.

The Thread Run stores the state and all the messages, tool calls, and tool call results, for the run.

Example Thread Run

Below is an example of how a Thread Run might unfold when an agent that has two tools (“GetLocation” and “GetWeather”) receives the user’s message: “What is the current weather?”

Each row represents a step in the overall process (Create Thread Run, Generate Output, Handle Run Step), showing what goes in and what comes out.

Step #	Step Name	Key Inputs	Process / Action	Outputs
1	Create Thread Run	- Agent Config (with tools GetLocation, GetWeather) - Thread Messages (User: “What is the current weather?”) - Thread ID (e.g. 1234)	- System initializes a new Thread Run context. - Generates a Thread Run ID (e.g. run_001).	- Thread ID (1234) - Thread Run ID (run_001) - (Agent Config & Thread Messages passed along)
2	Generate Output	- Agent Config, Thread Messages (incl. “What is the current weather?”), Thread ID (1234), Thread Run ID (run_001)	- Agent analyzes the user request. - Determines it needs a location before it can fetch the weather. - Decides to call GetLocation.	- Output Messages: No new direct message yet. - Tool call: “GetLocation” - Thread Run ID
3	Handle Run Step	- Tool call: “GetLocation” - Thread Run ID (run_001)	- System invokes the “GetLocation” tool. - GetLocation responds with a location (e.g., “New York, NY”).	- Tool call result: “New York, NY” - Thread Run ID (run_001)
4	Generate Output	- Agent Config, Thread Messages, Thread ID (1234), Thread Run ID (run_001) - Tool call result: “New York, NY”	- Agent sees the location is “New York, NY.” - Decides it now needs the weather for that city. - Creates a new tool call for “GetWeather(‘New York, NY’).”	- Output Messages: No new direct message yet. - Tool call: “GetWeather(‘New York, NY’)” - Thread Run ID
5	Handle Run Step	- Tool call: “GetWeather(‘New York, NY’)” - Thread Run ID (run_001)	- System invokes GetWeather with “New York, NY.” - Tool returns weather data (e.g., “Currently sunny, 25°C”).	- Tool call result: “Currently sunny, 25°C” - Thread Run ID (run_001)
6	Generate Output	- Agent Config, Thread Messages, Thread ID (1234), Thread Run ID (run_001) - Tool call result: “Currently sunny, 25°C”	- Agent now has location and weather info. - Crafts the final message: “It’s currently sunny and 25°C in New York.”	- Output Message (to user): “It’s currently sunny and 25°C in New York.” - No further tool calls required.
7	End of Thread Run	- N/A	- System sees no more steps are needed. - Thread Run concludes.	- Final output delivered to user. - Thread Run complete.

3. Use Tool

Finally, we get to using tools. Remember: a tool definition is something in the prompt which we can translate to running an operation and getting a result.

In the case of the Cat Facts AntRunner sample the LLM’s definition was:

{  
  "name": "GetCatFacts",  
  "description": "Retrieve and query facts",  
  "parameters": {  
    "type": "object",  
    "properties": {},  
    "required": []  
  },  
  "strict": false  
}

3.1 Get Tool Definition

When the LLM asks for a tool call of GetCatFacts, the system must provide the tool call result somehow. Mechanically this could be a web service, a local function, or a step in a low-code solution. In this case AntRunner uses an Open API schema that describes a simple web service.

{  
  "openapi": "3.0.0",  
  "info": {  
    "title": "Cat Facts API",  
    "description": "API to retrieve and manage cat facts.",  
    "version": "1.0.0",  
    "contact": {  
      "name": "alexwohlbruck",  
      "url": "https://github.com/alexwohlbruck/cat-facts"  
    }  
  },  
  "servers": [  
    {  
      "url": "https://cat-fact.herokuapp.com",  
      "description": "Base URL for all endpoints"  
    }  
  ],  
  "paths": {  
    "/facts": {  
      "get": {  
        "operationId": "GetCatFacts",  
        "summary": "Retrieve and query facts",  
        "description": "Get a list of animal facts."  
      }  
    }  
  }  
}

AntRunner creates the tool definition that the LLM uses from the full specification and uses the full specification to do call the real tool. In this case a web service, but local function calling is also supported as shown in this notebook

In Conclusion

The Tool-based Agent Pattern is a practical way to design agent systems by utilizing modular agents, each with specific tools. This method enhances flexibility and reusability, making it easier to adapt to changing requirements.

Cohesive and modular agents are essential components of this pattern. Cohesion refers to how closely related the responsibilities of a single agent are, ensuring that each agent performs a specific task effectively. For example, consider an agent designed to handle customer inquiries. This agent could be responsible for answering questions about product availability, while another agent could focus on processing orders. By separating these functions, each agent can be optimized for its specific role, leading to better overall performance.

The applicability of the Tool-based Agent Pattern extends across a wide range of scenarios. For instance, pre-defined orchestrations allow for structured workflows where agents perform specific tasks in a coordinated manner. Interactive UIs enable users to engage with multiple agents seamlessly, enhancing the overall experience. Additionally, job orchestration automates complex processes, allowing agents to work together without human intervention. This versatility makes the Tool-based Agent Pattern a valuable framework for developing efficient and responsive systems in various domains.

Next up…

A key reason why the Ants work well is because a lot of time went into creating and tuning the instructions and tool descriptions. At the heart of the AntRunner library there is a focus on making the most of tools to enable:

Quality output
Agent swarms
Security authorization
Local functions
Output filtering to conserve tokens, improve performance, and lower cost
More

Thanks for reading this very long article!

If you like this kind of long-form content, please share and link.

Up with humans!
--Doug Ware

AboutAnts

Tue, 31 Dec 2024 00:00:00 GMT

AntArmy and AntRunner

Two weeks ago, I started the process of getting AntArmy released to the first group of testers and now the uses table has several new entries! I am grateful for the help and feedback, and I am excited to make it to this milestone. If you would like to help test and give feedback, please send me a message!

I recently released Antunner, a .NET library, PowerShell module and Azure Functions tools we use to make, test, deploy and improve the AI assistants we call Ants. I've been busy adding features and fixing bugs. Over the next weeks and months I'll be using these tools to teach and demonstrate an effective way to build AI apps.

AntRunner is very much targeted at the .NET, Microsoft Azure, and M365 communities. If it sounds useful to you, I hope you'll ty out the demo notebooks.

Ants - More than just a fun name

A key philosophy of building and using ants is that they are small, specialized, and independent and that they work together. I took this name from notes I took long ago about the Actor Model of computing because ants, conceptually, are actors.

I created the section that follows using two ants in collaboration. The first ant searched the web and then autonomously sent messages to five other ants to read the pages identified by search to extract applicable content for the first ant who wrote the final response. See the 8-Web-Search-Ant-As-Tool notebook for details

Ensuing that you include links to pages supporting key statements are included throught your output, explain how the quote 'one ant is no ants' applies to the actor model of computing. Do not simply povide a list of links at the end

Start of Ant Output (images are from the pages the ants found...)

The phrase "one ant is no ants" is often used to emphasize the collective and collaborative nature of ants in a colony. This concept is directly applicable to the Actor Model of computing, which relies on the interactions and cooperation among multiple actors to achieve concurrent and distributed computations effectively.

Explanation:

Collective Strength

In an ant colony, the strength and effectiveness of the colony arise from the interactions and cooperation among individual ants. An individual ant is limited in what it can achieve alone; however, together as a colony, ants can perform complex tasks and solve problems efficiently. This idea is encapsulated in the saying "one ant is no ant."

Image Source: towardsai.net

Actor Model Collaboration

Similarly, in the Actor Model of computing, the phrase "one actor is no actor" underscores that the power and effectiveness of the model come from the interactions and collaboration among multiple actors. An individual actor can perform a specific computation or task, but the true strength of the Actor Model is realized when multiple actors work together, exchange messages, and coordinate their actions. This is essential for building scalable and fault-tolerant applications.

From the Brian Storti's article:

One ant is no ant. And one actor is no actor. They come in systems. In the actor model everything is an actor and they need to have addresses so one actor can send a message to another.

This highlights the importance of having multiple actors that interact with each other to create a functional concurrent system, much like how ants work together in a colony.

Image Source: vecteezy.com

For more details, you can explore the full content here.

Summary:

"One ant is no ant" underscores the collective strength of ants in a colony, while "one actor is no actor" emphasizes the collaborative nature of the Actor Model in concurrent and distributed computing. The power of the Actor Model lies in the interactions among multiple actors, just as the effectiveness of an ant colony lies in the cooperation among individual ants.

By understanding and applying this analogy, developers can better grasp the necessity of designing systems where multiple actors work together to manage state and perform computations effectively, ensuring scalability and fault-tolerance.

### End of Ant output... the rest is human including the typos

Ants versus Copilots

One of the biggest requirements I used to drive this architecture was that each AI application, i.e. each ant, needs to be self standing and reusable. The AntRunner functions and AntRunner library make it easy to turn ants into web services which in turn supports several flavors of copilot extensibility and anything else that can use a web service. So, your copilot can look to an army of ants to do all sorts of useful things that were not practically possible eighteen months ago!

--Doug Ware, September 26, 2024

P.S. It is hard to get noticed with all the noise about AI. If you like the project and are the type of person who uses github, contributions are very much appreciated.

AntRunnerSwarms

Tue, 31 Dec 2024 00:00:00 GMT

'Reasoning Tokens' and swarm-based agent applications

Today is an exciting time to be a product builder and AI applications consultant. Two of the biggest obstacles to building useful systems with generative AI have fallen away in the last month or two: available capacity of state of the art models, and high costs.

As of today (Sept 30, 2024) the default quotas are documented on the Azure OpenAI Service quotas and limits page.

Massive capacity increases of available Tokens Per Minute (TPM) per deployment

Azure OpenAI Service imposes quotas and rate limits to manage resources effectively. These limits are defined in terms of Tokens Per Minute (TPM) and Requests Per Minute (RPM).

Default TPM Limits:
- GPT-4o: 450,000 TPM (Default), 30 million TPM (Enterprise agreement)
- GPT-4o-mini: 2 million TPM (Default), 50 million TPM (Enterprise agreement)
- GPT-4 Turbo: 450,000 TPM (Default), 2 million TPM (Enterprise agreement)

... and steep price declines

In terms of practical architecture and design, going from a quota of 80k per minute to 2000k changes things dramatically.

'Reasoning Tokens', swarms, and high TPM requirements

Recently OpenAI released o1-preview. The o1 models prioritize enhanced reasoning and extended thinking time for complex tasks.

OpenAI hasn't disclosed how the system works and are reportedly banning people who get too nosey. API billing for the model includes 'reasoning tokens'. Reasoning tokens are a key aspect of the o1 models, representing the model's internal thought process as it breaks down the prompt, considers various approaches, and formulates a response. These tokens are not visible in the API response but are still billed and counted as output tokens.

Thinking for yourself

While I don't know how OpenAI's chain of thought system works, I can guess that it is based on the same extension mechanism used for image generation, file search, code interpreter and plugins which, in the assistants API is called tools.

Tools allow a language model to know about and 'call' other programs. The language model has no ability to talk to other systems (+1 for humanity), the program (such as a chatbot) using the language model has to do the work of invoking the tool (somehow) with the input provided by the language model, get the results (however is appropriate) and continue the generation by including the tool call output in the next generation operation.

My work with AI ecosystems over the past couple of years has included a mix of python, javascript/typescript, and .NET. A big reason I used c# and .NET for AntRunner instead of one of the others is the strength of .NET for asynchronous operations and parallel tasks. That is to say simply: it's easy to do a lot of things at once, instead of one thing at a time.

Doing a lot of things at once is really important if you have a lot of work to do that can be done in parallel.

Over the coming days, weeks, and months I plan to write a lot about building agentic AI applications. I'll be using AntRunner which is an open source library and associated tools for building AI applications. AntRunner allows an ant to use tools which might be other ants. In those cases, a swarm of ants work together in parallel to create the answer.

This chart shows one such run which took just over 60 seconds. During that time a total of 41 ants search the web, loaded pages and extracted relevant content, but it also shows a massive spike in usage which the quota for the deployments easily permit.

The prompt

Use python to get today's date and then use it to find things to do in Atlanta for each of the next four weekends with pictures and correct links from FindAnswers.

Don't forget - include only valid links!

Sample output

You can view the output of a sample run here

Cost breakdown

Using mostly gpt-4o-mini keeps costs down. The single call to gpt-4o cost almost as much as 40 calls to gpt-4o-mini!

This same ant swarm with only gpt-4o produces similar, slightly better results but costs around $1.20 instead of 12 cents. I can easily reduce it to 9 cents by replacing code interpreter with a local function.

--Doug Ware September 30, 2024

P.S. Need to build AI applications but could use a guide? I'd love to hear from you!

RagIsAnAntipattern

Tue, 31 Dec 2024 00:00:00 GMT

Retrieval Augmented Generation is an Anti-pattern

In 2025, don’t start your AI journey solving problems from 2022

It’s hard for me to believe that I started my AI/ML journey in earnest over 2.5 years ago. At the beginning my interest was in computer vision, but ChatGPT came out at the end of 2022 and turned me toward generative AI. Since then, I’ve been working successfully as a consultant and building commercial and open-source products including Go AntArmy!, AntRunner, and ask@antarmy.ai.

Educating myself and keeping up with the changes is a lot of work, but building things for my own business and for others helps give me a point of view based on first-hand experience. If you are an experienced AI software builder, you may disagree with what I am about to say. If you are new to the space, I think this is important advice: The most popular approach to chatbot and agent design, Retrieval Augmented Generation (RAG), is outmoded!

This article explains why I say this and what I think is a better alternative.

Conventional wisdom

If you search LinkedIn for ’RAG Patterns’ you will find many posts and articles telling you why it is good and how to use it with numerous variations.

Conventional wisdom

It is a popular approach for building chatbots and agents. Go AntArmy!, AntRunner, does not use this pattern in any of its agents, but the Web Ants do something similar using an alternative pattern. If I ask them is this correct? "Retrieval Augmented Generation is an Anti-pattern“, they will tell me something like: The statement "Retrieval Augmented Generation is an Anti-pattern" is not widely supported by current literature or expert consensus. Instead, Retrieval-Augmented Generation (RAG) is generally viewed as a beneficial approach in the field of natural language processing (NLP) and artificial intelligence (AI).

AntArmy.AI is integrated with M365 and is a private solution for people to run in Azure, and so there is no public demo for it yet, but anyone can use the search ants by sending an email to ask@antarmy.ai.

What is an Anti-pattern?

Wikipedia says an anti-pattern: in software engineering, project management, and business processes is a common response to a recurring problem that is usually ineffective and risks being highly counterproductive.

To be an anti-pattern then is to be a pattern that seems reasonable and is popular, and where, by implication, there are better alternatives. Sometimes we make patterns to work around challenges that later disappear or are mitigated enough that the pattern no longer fits the real world. Retrieval Augmented Generation is an example.

What is RAG?

Retrieval-Augmented Generation (RAG) is an approach in natural language processing that combines a search index(s) with a generative model(s). In RAG, a specialized embeddings model first retrieves relevant documents or pieces of information from an index based on the user's query. This retrieval process ensures that the generated responses are grounded in factual data, enhancing their relevance and accuracy.

Once the relevant information is retrieved, a generative model, creates a response using information in the retrieved content. This method allows the application to use information that the model does not contain such as anything that happened after the model was trained.

RAG is used in applications such as question answering, conversational agents, and content generation, where accurate and contextually relevant information is crucial. By combining retrieval and generation, RAG models try to produce more informed and contextually appropriate responses, and address limitations with models that rely solely on pre-trained knowledge without real-time data access.

This diagram shows the basic pattern. I didn’t make it and I don’t know who did - I’ve seen it several times and image search yields more than 20 pages going back to 2023 in different sites including the OpenAI community site!

There are many variations on this pattern (which is a symptom of the problem), but the basic pattern is:

Retrieval happens in steps 2 and 3 using the original user query
Augmentation happens in step 4 when the original user query is augmented or rewritten to contain the (hopefully) relevant information that was retrieved
Generation of the response happens when the language model responds to the augmented query

There are some simple cases where this basic pattern works well, and it makes for great demos. In December of 2023 I wrote Building Good Chatbots Part One, No-Code with Microsoft Copilot Studio and Azure AI Studio. At the end of that article, I teased a following article ‘Castle on a hill’ that was to include a lot of the techniques I and others use extend the pattern, but the more I built this less comfortable I got about recommending the pattern, and by May 2024 felt that there was a better way.

If you are Monty Python fan, you’ll appreciate it when I say ‘that castle sank into the swamp’.

RAG Challenges

In the diagram retrieval is done in the box labeled Search Relevant Information with input named ‘Query’ coming from the computer icon which represents a chat application.

Searching Relevant Information

How does this Search Relevant Information process know which knowledge sources to use and how to invoke them? In simple systems there is only one, but this might be complicated. Typically, making a decision about which sources to use is simply not made and instead the process will query everything in parallel and use a ranking function to pick the winning information. If a decision is made up-front it is often done by a fast and inexpensive models, not by the LLM used for the final generation.

Why wouldn’t you use an LLM, perhaps the same one used for the final generation? This part of the pattern is an example of a solution to problems that were solved by time. The first is inference cost. For example, GPT-4o-mini is approximately 200 times cheaper than GPT-4 and prices continue to decrease while quality continues to improve. A second is scarcity. Getting enough capacity to use GPT-4 in a system typically required a commitment to spend tens of thousands of dollars per month! Today capacity is more abundant with metered rates often with no minimums.

Backing up a step, how does the chat application know that the user asked a question that should be handled by search? In a simple implementation the usual answer is that it just doesn’t worry about it and does a search anyway. If there are no results, the chat application deals with it somehow in the augmentation step.

Knowledge Sources

A popular first project is the chatbot using a set of documents. If you use Copilot Studio or the Azure AI Foundry Chat Playground, you can get started quickly using documents you already have and deploy a new chatbot in minutes.

SharePoint Agents are another of the many options.

With high quality documents under active content management and a narrow focus, you can create useful chat agents with good quality. In this case RAG is not an anti-pattern because it fits and doesn’t require a lot of extensions. But what if the content is not curated, covers a wide variety of topics, has hand-written text, pictures, or tables, or includes important metadata on a cover page, or as a document property?

What if people need to ask questions about the overall corpus like, ‘what was the most common complaint customers made?’ Document search depends on text that is in documents. Unless there is a document with text about common complaints, a chatbot using search will not be able to answer the question because the answer is not in the text.

Multiple knowledge sources

It is very important to note that you can’t eliminate the need for knowledge sources! If you want to create an AI agent like the Web Ants or ask@antarmy.ai, you have to provide the correct knowledge to the LLM for everything to work. The question is what patterns you use to build the agent.

A temptation is to try to create a single knowledge source that works for every scenario, but this is only practical when you have narrow requirements that everyone agrees on with high quality data, so basically never.

Go AntArmy! solves this challenge by using many agent ants with specific and narrow definitions, each with their own knowledge sources and tools. The pattern is very different from RAG, but many of the same pieces are required. However, they are organized differently.

Augmentation introduces fragility

Step 4 is the creation of an ‘enhanced prompt’. At this point the chat application takes the original question, adds the knowledge, and in the case of the Azure AI Foundry ‘Add your data’ extensions (among others), adds instructions concerning citations.

Depending on the implementation, this new prompt might be very different, or even at odds with the user or system prompt. I encountered this when I had a knowledge source that consisted of questions, answers, and links to source documents that contained the official answers. Using the ‘Add your data’ extensions, it proved very difficult to get output that included the specific links from the answers in the Q&A documents instead of citing the Q&A documents themselves.

We wanted it to use the links in the content, but the creators of that specific RAG implementation assumed that citations to the indexed documents were desirable. The resulting augmented prompt overrode the instructions about the links.

A big flaw comes from the fact that the pattern explicitly separates the LLM from the true context and relies on an intermediary which frequently uses less capable or overly specialized models or instructions.

This intermediary is another element that made sense previously due to the constraints imposed by the size of the context window. In 2022, gpt-3.5-turbo had a context window size of 4096 tokens compared to 128,000 tokens with gpt-4o. So, when implementing RAG with gpt-3.5-turbo, managing the size of the context widow was much harder as today we have almost 32 times as much space to work with.

This means we can:

Put more facts in the prompt
Provide more instructions for dealing with different knowledge sources
Have longer threads
Do more ‘thinking’

Much of the augmentation and context management done by the chat application with RAG can now be managed by the LLM which has a large enough context window to ‘see’ and participate in the entire flow.

Bigger picture RAG pattern

I looked at a lot of articles about RAG as I was putting this article together and I was impressed by the completeness of this one from Hevo Data - What is RAG Architecture? A Complete Guide | Hevo.

This diagram includes the ingestion pipeline to the vector database which is only one kind of knowledge source. Often there are easier, better, and more direct ways to provide the LLM with knowledge.

Agents with tools are easier to build and can do more things

Cheaper and better models with large context windows allow us to use the LLM to manage generation and greatly simplify the architecture by giving it the entire context and allowing it to manage retrieval and any other tools it needs to fulfill its purpose. The use of knowledge sources becomes just like any other thing the agent knows how to do.

Thanks for reading

This article is the preface for what I expect to be a long series articles about building AI agents with tools like the Go AntArmy! ants, using the AntRunner library, using the assistants api and Azure OpenAI. A big focus will be on how to define tools for existing APIs of your own and from vendors using examples like MSGraph and local functions.

A big focus of the architecture is the ability to use the resulting agents in other products, for examples, as Copilot Declarative Agents, and in fully custom systems or jobs such as ask@antarmy.ai.

Happy New Year!
--Doug Ware

AntRunnerAgentsSample

Tue, 31 Dec 2024 00:00:00 GMT

Here are some exciting things to do in Atlanta for each of the next four weekends, along with pictures and links to more information:

October 5-6, 2024

Taste of Tucker
- Date: October 5, 2024
- Description: Annual food festival along Main Street, Tucker, featuring food and drink from local restaurants and breweries, arts and crafts, music, and more.
- More Info
Festival on Ponce
- Date: October 5-6, 2024
- Description: Twice-annual arts festival in Olmsted Linear Park, with 125+ vendors of fine art, folk art, and outsider art, plus food and drink, and children's activities.
- More Info
Candler Park Fall Fest
- Date: October 5-6, 2024
- Description: Popular annual festival in Candler Park and neighborhood, with live music, art, food, rides, games, and more.
- More Info
Atlanta Horror Film Festival
- Date: October 4-6, 2024
- Location: Limelight Theater
- More Info
Atlanta Arab Festival
- Date: October 5-6, 2024
- Description: Annual festival of Arabic food and culture at the Alif Institute, with traditional dishes from Arabic countries, live performances, vendors, exhibits, workshops, games, and more.
- More Info

October 12-13, 2024

Atlanta Pride Festival and Parade
- Date: October 12-13, 2024
- Description: The Atlanta Pride Festival and Parade is a vibrant event that celebrates and supports the LGBTQ+ community.
- More Info
Taste of Acworth
- Date: October 12, 2024
- Location: Downtown Acworth
- Description: An affordable food festival featuring FREE admission, with small bites for just $1 to $5 each.
- More Info
Lilburn Daze
- Date: October 12, 2024
- Location: Lilburn City Park
- Description: A beloved fall arts & crafts festival with plenty of activities for kids, food, vendor booths, and live entertainment.
- More Info
Chalktoberfest
- Date: October 12-13, 2024
- Location: Marietta Square
- Description: A festival celebrating street art and craft beer, featuring live music and food, with a ticketed craft beer tasting.
- More Info
Old Fourth Ward Park Arts Festival
- Date: October 12-13, 2024
- Location: Old 4th Ward Park
- Description: A two-day arts festival with live acoustic music, about 150 vendor booths, food, and fall fun.
- More Info

October 19-20, 2024

A Taste of Chamblee
- Date: October 19, 2024
- Description: Annual food festival in downtown Chamblee, featuring food from neighborhood restaurants, craft vendors, children's activities, football screenings, live performances, wine tent, beer garden, and more.
- More Info
HarvestFest and Scarecrows In The Square
- Date: October 19, 2024
- Description: Annual fall festival in Marietta Square, with arts and crafts show, touch-a-truck, scarecrow displays, costume and pie-eating contests, and more.
- More Info
Cherokee Heights Arts Festival
- Date: October 19, 2024
- Description: Free annual arts festival in the Cherokee Heights neighborhood, with local artists and makers, local musicians, art activities, food, beer, and more.
- More Info
L5P Halloween Festival and Parade
- Date: October 19-20, 2024
- Description: Popular annual Halloween celebration in Little Five Points, with live music, parade, haunted house, ghost tours, and more.
- More Info
Johns Creek Arts Festival
- Date: October 19-20, 2024
- Description: Annual arts and crafts festival in Johns Creek featuring 140 artisans from across the country.
- More Info

October 26-27, 2024

Atlanta Halloween Half Marathon & 5K
- Date: October 26, 2024
- Location: Piedmont Park
- Description: A fun and spooky race where participants are encouraged to wear costumes.
- More Info
Little 5 Points Halloween Festival & Parade
- Date: October 26-27, 2024
- Description: A unique event with pumpkin decorating, street vendors, and live performances.
- More Info
Scarecrows in the Garden
- Date: October 26-31, 2024
- Location: Atlanta Botanical Garden
- Description: The garden will feature a variety of scarecrows built and decorated by community members and local businesses.
- More Info
Pumpkin Festival at Stone Mountain Park
- Date: October 26-27, 2024
- Description: Offers family-friendly activities including costume contests and trick-or-treat scavenger hunts.
- More Info
Howl-O-Ween at Zoo Atlanta
- Date: October 26-27, 2024
- Description: A family-friendly Halloween event at the zoo with trick-or-treating, costume contests, and animal encounters.
- More Info

These events provide a variety of activities and experiences perfect for enjoying the fall season in Atlanta!

AntRunnerTools

Thu, 12 Sep 2024 00:00:00 GMT

AntRunner

A full set of .NET tools for creating and using Open AI and Azure Open AI Assistants

Features

Create and manage Open AI Assistants (ants) from files stored in a file system and in Azure Blob Storage
Easy setup of file search with vector stores from files bundled with ant definitions
Easy setup of code interpreter and code interpreter files bundled with ant definitions
Download files from code interpreter outputs
Define and call REST API endpoints automatically with auth headers and oauth tokens
Define and call local .NET methods automatically as tool calls

AntRunner Projects and Tools

AntRunnerLib - The core library for creating and running ants, published on nuget.org
AntRunnerFunctions - Azure Functions for running ants including a REST API and Durable Functions
AntRunnerPowershell - A PowerShell module for managing and running ants

Notebooks

Get started with the sample notebooks

Example

var assistantRunOptions = new AssistantRunOptions() {
    AssistantName = "MsGraphUserProfile",
    Instructions = "What is my name?",
    OauthUserAccessToken = oAuthToken
};
var output = await AntRunnerLib.AssistantRunner.RunThread(assistantRunOptions, config);
Console.WriteLine(output.LastMessage)

Your name is Doug Ware.

AntArmyPreview

Tue, 30 Jul 2024 00:00:00 GMT

AntArmy - Together we are strong!

Your browser does not support the video tag

TranscriptionAndDiarization

Mon, 17 Jun 2024 00:00:00 GMT

Locally and Privately Extracting Meaning from Video with Transcription, Diarization, and LLMs

My previous entry was about using computer vision and OCR along with various versions of GPT-4 with vision to convert PDFs into text files containing descriptions of images and text-based diagrams. I’m excited by the ability to turn images into words, but I am also excited about turning sound into words!

… and sound into images, images into sound, etc.

Ultimately, I think user experiences should adapt to the person and be multimodal for the benefit of all kinds of people, but another reason for my interest is because there is a lot of useful information that isn’t written down. It must be seen and/or heard. How do we make that information usable?

Microsoft stirred up a lot of controversy these past weeks with Recall and it reminded me why I am so interested in private and local AI applications. That reason is simply that there are a lot of reasons people might hesitate to trust cloud providers with certain information.

Who is this entry for?

I am writing this for as general an audience as possible. The technical details are in the demonstration project and gihub. If you want to do this yourself, I broke it down into each step using jupyter notebooks and included one with the complete process to create a transcript with speakers from a video. Here, I will talk about it at a much higher level.

I am approaching this topic as a continuation of my previous ones about building and grounding chatbots and ingesting PDF content, but I also use these techniques often to create interview and meeting notes and to analyze content.

Teams Copilot does a great job at many of these tasks if you can use it, but the way it is licensed and the fact that it’s locked into Teams limit its utility for me. This works on any video or audio, I can use its output however I like, and it’s private and completely free.

The key thing though is that the transcript and other information I create when I do this is the end of the first step. The next step might be some kind of analysis, additional content generation, indexing, or work by an AI agent.

Local or privately hosted generative AI with open source models and libraries

Everything in this article ran on my desktop computer. It is similar in specs to the configuration shown below (core I9, 64GB memory, with NVIDIA Geforce RTX 4090 24GB).

Let me just start by saying that I completely get that that is an expensive setup and most people don’t have a desktop with a top-end consumer GPU and CPU. The good news is that everything I use here is possible on a lot of consumer hardware, but it will be slower, and you might have to break up the steps a bit. Part of my experiment was to see if I could do everything in a single process with my GPU.

Alternatively, you can do all of this in the cloud in notebooks, containers, or virtual machines using many different providers. The point is that the models and tools are all open source. All of this works on Windows, Mac, and Linux.

The process

This diagram shows the steps.

graph TD  
    A["Download a video from YouTube using pytube"] --> B["Extract audio from video using MoviePy"]  
    B --> C["Create transcript using whisper-large-v3"]  
    C --> D["Identify distinct speakers using pyannote.audio"]  
    D --> E["Combine Whisper transcript with pyannote speaker data"]  
    E --> F["Identify speaker names using phi-3-medium"]  
    F --> G["Create final output"]

Test video

The test video I am using is Developer’s Guide to Customizing Microsoft Copilot - Microsoft Build 2024. I chose it because it was a great presentation, because Jeremy has an accent, and because Barnam and Jeremy introduced themselves and said their names! It’s funny what you notice when you start looking for specific information, but often people forget to introduce themselves.

I use pytube to download videos from YouTube. There are some dodgy websites that will do it for you too.

Extract audio from video using MoviePy

I use moviepy to easily extract the audio track. Pytube and moviepy are venerable and popular, and ChatGPT has no trouble writing code using them.

Create transcript using whisper-large-v3

Whisper is a speech recognition model from OpenAI you can use as a service, but it has an open license which means that, unlike GPT-4 and DALL-E, you can download it and run it yourself. It is available on huggingface and has millions of downloads across its different versions and ranks among the very best automatic speech recognition options.

For comparison:

The YouTube transcript lacks spelling and punctuation and is also less accurate. Where YouTube failed badly, Whisper did a good job phonetically with Barnam’s name. The fact that it spelled it incorrectly isn’t surprising and is even expected.

To me ‘mistakes’ made by a model fall into two categories:

Things the model should know either through its training or grounding data
Things that the model does not know or have any way of knowing

The easiest way to identify the second category is to ask yourself if a normal person could do it. If we were talking, and I said his name, could you correctly spell Barnam Bora?

Who said what? Identify distinct speakers using pyannote.audio

If the audio is a single speaker, the whisper transcript would be enough to use with an LLM, but without knowing who said what, there are a lot of questions an LLM (or a person) would not be able to answer. Conceptually, this is the same sort of problem I wrote about in my article about ingesting PDFs. Concepts aside, it is a different problem and requires a different solution.

I use pyannote.audio which is also available as a more robust service to perform diarization. The diarization creates a Rich Transcription Time Marked (RTTM) file, a space-delimited text files containing one turn per line.

This video covers the topic in more depth:

For reference, the first five lines of the output look like this:

SPEAKER Developer’sGuidetoCustomizingMicrosoftCopilot-MicrososftBuild2024 1 10.240 3.324 <NA> <NA> SPEAKER_00 <NA> <NA>  
SPEAKER Developer’sGuidetoCustomizingMicrosoftCopilot-MicrososftBuild2024 1 14.257 19.389 <NA> <NA> SPEAKER_00 <NA> <NA>  
SPEAKER Developer’sGuidetoCustomizingMicrosoftCopilot-MicrososftBuild2024 1 34.102 0.017 <NA> <NA> SPEAKER_00 <NA> <NA>  
SPEAKER Developer’sGuidetoCustomizingMicrosoftCopilot-MicrososftBuild2024 1 34.118 0.456 <NA> <NA> SPEAKER_01 <NA> <NA>  
SPEAKER Developer’sGuidetoCustomizingMicrosoftCopilot-MicrososftBuild2024 1 34.574 0.388 <NA> <NA> SPEAKER_00 <NA> <NA>

Combine whisper transcript with pyannote speaker data

At this point we have a file with a transcript and time codes but not speaker information from whisper and another with time codes and speakers without a transcript from pyannote. The code is in this notebook. It also changes the time codes from seconds to something easier to read (hh:mm:ss). One wrinkle to this is that the time codes from the two models don’t line up perfectly, and so the final result often has some issues where the speaker changed in the middle of a line in the whisper transcript, although in this case it was fine.

The result looks like this:

[SPEAKER_00] : [(00:00:00.000, 00:00:23.660)] :  All right, everybody. Welcome, welcome, welcome, welcome. First breakout of Build. I'm Barnum Bora, and I lead the developer advocacy team for Microsoft 365 and Copilot Platform.  
[SPEAKER_00] : [(00:00:23.660, 00:00:24.340)] :  advocacy team for Microsoft 365 and Copilot platform.  
[SPEAKER_00] : [(00:00:27.920, 00:00:28.120)] :  But without further ado, before I come back and do more things,  
[SPEAKER_00] : [(00:00:31.980, 00:00:36.160)] :  I'm going to hand off to my good friend Jeremy Thake, and he's going to walk you through the first half of this session. Thanks, Barno. I appreciate it. And thank you for coming to Build.  
[SPEAKER_01] : [(00:00:37.960, 00:00:40.280)] :  So I'm Jeremy Thake. I'm a

Identify speaker names using phi-3-medium

Often at this point I have to do some manual steps to replace “SPEAKER_00” with a real name because I find that very often, perhaps most of the time, people don’t say their own names or refer to others by name. But still, if I am doing this process, I intend to use the transcript with a language model to get insight from the content.

In this case, Barnam and Jeremy demonstrated excellent presenter skills and the text contains their names. Thanks to this, I can use a language model to extract their names and update the transcript - an perfect example of using a language model to get insight from the content!

For this task I used microsoft/Phi-3-medium-4k-instruct-onnx-directml which is a version of Phi-3-medium-4k-instruct optimized for use with ONNX runtime which makes it super fast on my RTX-4090 GPU. Use it if you have a GPU and are using Windows.

My notebook contains code to pull out a few lines before and after a speaker’s first appearance and build a prompt:

 # Create the prompt with the current speaker and their context  
        prompt = f"""  
        The following text is a piece of a transcript. Examine the text and if the text identifies the speaker reply with ONLY the full name matching the exact spelling of the person listed as {speaker} and nothing else.   
        ----------------------------------------  
        {context_text}"""

The output of the function looks like this:

SPEAKER_00:  Barnum Bora  
SPEAKER_01:  Jeremy Thake

Once the file is updated, the final transcript looks like this:

[Barnum Bora] : [(00:00:00.000, 00:00:23.660)] :  All right, everybody. Welcome, welcome, welcome, welcome. First breakout of Build. I'm Barnum Bora, and I lead the developer advocacy team for Microsoft 365 and Copilot Platform.  
[Barnum Bora] : [(00:00:23.660, 00:00:24.340)] :  advocacy team for Microsoft 365 and Copilot platform.  
[Barnum Bora] : [(00:00:27.920, 00:00:28.120)] :  But without further ado, before I come back and do more things,  
[Barnum Bora] : [(00:00:31.980, 00:00:36.160)] :  I'm going to hand off to my good friend Jeremy Thake, and he's going to walk you through the first half of this session. Thanks, Barno. I appreciate it. And thank you for coming to Build.  
[Jeremy Thake] : [(00:00:37.960, 00:00:40.280)] :  So I'm Jeremy Thake. I'm a  
[Jeremy Thake] : [(00:00:40.280, 00:00:44.100)] :  principal program manager in the Copilot developer experience team. And a slight

Next steps

From here there is still room for improvement. The information extracted from the video contains only what was heard and none of what was seen. For example, one could further improve the pipeline by cutting the video into screen shots at transitions and enhance the content with both images as well as descriptions from a vision model. Often the names of the speakers are displayed as text in the video and looking for names in both places would presumably yield correct spellings for the speaker’s names.

Teams Copilot is exciting to people because it unlocks information and answers questions, but it is locked into Teams. You don’t have to settle for that.

Thanks for reading!
--Doug Ware
June 17, 2024

TranscriptionAndDiarization

Sat, 15 Jun 2024 00:00:00 GMT

This project demonstrates the use of local AI to transcribe, diarize, and enrich transcripts created from videos

It uses:

You will need to download and install microsoft/Phi-3-medium-4k-instruct-onnx-directml and update the model_path below. If you do not have GPU or are not using Windows, see the Phi-3 docs and set yourself up accordingly.

Pyannote.audio is gated on huggingface and requires an account and access key. See https://huggingface.co/pyannote/speaker-diarization-3.1 for instructions.

Notebooks

Name	Description
DownloadFromYoutube.ipynb	Use pytube to download a video
ExtractAudioFromVideo.ipynb	Extract mp3 from mp4 with moviepy
TranscribeVideoWithWhisperLarge.ipynb	Create a transcript with whisper-large-v3
DiarizeWithPyannote.ipynb	Diarize with Pyannote
Phi3-ONNX.ipynb	Use Phi-3 with ONNX to identify names and finalize transcript
phi3test-transformers.ipynb	A test for comparing transformers to ONNX (spoiler ONNX is waaaaay faster)
VideoToFullTranscriptWithWhisperPyannoteAndPhi3-medium.ipynb	Complete process in a single notebook

IngestingPdfContent

Thu, 13 Jun 2024 00:00:00 GMT

This project contains the companion code for GPT-4o versus Azure Document Intelligence and Azure Computer Vision OCR

Notebook Name	Description
PdfToTextPages.ipynb	Baseline, makes a text file for each page using pypdf to extract the text
PdfToPageImages.ipynb	Given a folder of PDF files, converts each page of each PDF file into JPEG images using a resolution of 300 dpi, and saves them in a structured directory format.
DocIntelligencePipeline.ipynb	C# Polyglot notebook with functions to convert an entire PDF to markdown and another that creates an OCR markdown file from each image created using PdfToPageImages.ipynb
turbo-2024-04-09.ipynb	Azure Open AI using GPT-4 with vision to create a markdown file for each image created using PdfToPageImages.ipynb
v4omni.ipynb	OpenAI using GPT-4o to create a markdown file for each image created using PdfToPageImages.ipynb
v4omni-image-plus-docIntelOcr.ipynb	OpenAI using GPT-4o to create a markdown file for each image created using PdfToPageImages.ipynb grounded with OCR text created using DocIntelligencePipeline.ipynb
visionWithOcr.ipynb	Azure Computer Vision GPT4-Vision OCR
visionWithOcrAndGrounding.ipynb	Azure Computer Vision GPT4-Vision OCR and grounding

LibreChatBicep

Sat, 08 Jun 2024 00:00:00 GMT

Setup LibreChat in Azure Container Apps with Bicep

Introduction

This post is by our summer intern, Louis de Fromont, who is helping out tailoring the excellent LibreChat open source AI chat platform to use in projects specifically hosted in Azure. This bicep template can have you up and running in short order!

Take it away Louis!

Using the script

This Bicep script is the first small project that I’ve been working on at Elumention and is used to deploy a fully functional LibreChat instance to Azure. If you’ve been looking for an example to model your own script, hopefully, this blog post should help by highlighting some of the specifics when working with LibreChat. If you’re just interested in getting the deployment up and running, you can take a look at the deployment instructions.

At a high level, this script creates a managed environment for containers hosting LibreChat, a MongoDB instance, and a Mongo Express web interface, with support for optional Azure Active Directory authentication. Additionally, it sets up a Cognitive Services account for OpenAI, creates storage accounts and file shares for configurations, and automates uploading the librechat.yaml file for configuration.
At the heart of the deployment are the three different container apps:

LibreChat container app:
- Image: librechat/librechat:v0.7.3-rc
MongoDB container app:
- Image: bitnami/mongodb:7.0.7
Mongo-Express container app:
   - Image: mongo-express
   - Provides an admin web UI to MongoDB.
All three container apps use public images from Docker Hub.
Additionally, two file shares are deployed:

MongoDB file share:
- Mounted for the MongoDB container app to persist its database contents.
LibreChat file share:
- Used to mount the librechat.yaml file for LibreChat.

As part of the deployment script, the librechat.yaml file is processed and uploaded to the deployed file share for use by the LibreChat container app. The processing is done by first loading the contents of the file in Bicep using loadTextContent('../librechat.yaml') then a series of replace() functions are called on the loaded text to replace the API key and instance name of the OpenAI group. The key and instance name are loaded dynamically from the OpenAI service created as part of this script. After replacing the loaded YAML file, the contents are uploaded to the LibreChat file share via a deployment script. I based the code to write this deployment script based on a Bicep file by Mattias Fjellström.

OpenAI Service Creation and Model Deployment:

To create the OpenAI service, a cognitive service account is first provisioned to manage the AI model deployments. Then, models are deployed based on the configuration defined in the models.json file. The deploymentName property for each model in the models.json file should match the corresponding deploymentName property in the librechat.yaml file. This external configuration is accomplished in the Bicep script using the loadJsonContent('./models.json') function and an iterative loop over every model. One note about this loop is I had to include the @batchSize() decorator above the loop to limit the deployments of the models to one at a time. This is to avoid an error I commonly got in Azure when deploying models in parallel stating that “another operation is being performed on the parent resource” (the parent resource being the OpenAI service).

A separate parameter in the script is available for configuring the location of the OpenAI service and its deployments. This is because not all regions have the same mode deployments available which is why you might want to deploy the OpenAI service in a separate region than the rest of the resource group. To ensure compatibility and availability of OpenAI models in your selected region, consult the Model Summary Table and Region Availability guide. This provides detailed information on which AI models are available in specific OpenAI service regions.

Configuring Optional Authentication for Mongo-Express:

To limit access to the Mongo-Express container app, an optional parameter called createMonogexpressAuthConfig is available. When set to true, an Azure AD app registration and authentication configuration is created, adding Azure AD as an identity provider for Mongo-Express. This is accomplished natively in the Bicep script using the Microsoft.Graph/applications@v1.0 resource type. However, as of the time of writing, it is still an experimental feature and has just recently entered a public preview. Instructions to configure Bicep to work with Microsoft graph resources can be found here.

Because this is an experimental feature, not all features are fully supported, for example, attempting to re-run the Bicep script after setting up the Azure AD app registration can result in an error from the Azure CLI because the app registration resource already exists and modifications to an active app registration can’t be made. This is another reason I chose to make this part of the script be enabled with the createMonogexpressAuthConfig parameter, with the idea being you set the parameter to true on the initial deployment, then keep it false on subsequent deployments to avoid having this part of the script cause the rest of the deployment to fail.

Louis de Fromont June 3, 2024

StringExtensions

Sat, 08 Jun 2024 00:00:00 GMT

DougWare.StringExtensionMethods

Source on Github

DougWare.StringExtensionMethods is a set of C# extension methods for the string class, providing functionality to extract substrings and lists of substrings between specified delimiters. These methods are particularly useful for text parsing tasks such as extracting content from HTML fragments or other structured text.

Installation

Install the NuGet package using the following command:

dotnet add package DougWare.StringExtensionMethods

Usage

Methods

`GetInnerText`

Retrieves the first substring between a specified preamble and postscript.

Signature:

public static string GetInnerText(this string input, string preamble, string postscript)

Parameters:

input (string): The input string.
preamble (string): The start substring.
postscript (string): The end substring.

Returns:

(string): The substring between the preamble and postscript, or an empty string if they are not found.

Example:

string input = "preambleHello, World!postscript";
string result = input.GetInnerText("preamble", "postscript");
// result: "Hello, World!"

`GetInnerText` with Case Sensitivity

Retrieves the first substring between a specified preamble and postscript with an option for case sensitivity.

Signature:

public static string GetInnerText(this string input, string preamble, string postscript, bool ignoreCase)

Parameters:

input (string): The input string.
preamble (string): The start substring.
postscript (string): The end substring.
ignoreCase (bool): Boolean flag to indicate if the search should be case insensitive.

Returns:

(string): The substring between the preamble and postscript, or an empty string if they are not found.

Example:

string input = "PREAMBLEHello, World!POSTSCRIPT";
string result = input.GetInnerText("preamble", "postscript", true);
// result: "Hello, World!"

`ToByteArrayUtf8`

Converts a string into a byte array using UTF-8 encoding.

Signature:

public static byte[] ToByteArrayUtf8(this string input)

Parameters:

input (string): The input string.

Returns:

(byte[]): The byte array representation of the input string.

Example:

string input = "Hello, World!";
byte[] result = input.ToByteArrayUtf8();
// result: [byte array of the string]

`GetInnerTextList`

Retrieves a list of substrings between specified preambles and postscripts.

Signature:

public static List<string> GetInnerTextList(this string input, string preamble, string postscript)

Parameters:

input (string): The input string.
preamble (string): The start substring.
postscript (string): The end substring.

Returns:

(List): A list of substrings between the preamble and postscript.

Example:

string input = "preambleHello, World!postscript preambleHello Again!postscript";
List<string> result = input.GetInnerTextList("preamble", "postscript");
// result: ["Hello, World!", "Hello Again!"]

`GetInnerTextList` with Case Sensitivity

Retrieves a list of substrings between specified preambles and postscripts with an option for case sensitivity.

Signature:

public static List<string> GetInnerTextList(this string input, string preamble, string postscript, bool ignoreCase)

Parameters:

input (string): The input string.
preamble (string): The start substring.
postscript (string): The end substring.
ignoreCase (bool): Boolean flag to indicate if the search should be case insensitive.

Returns:

(List): A list of substrings between the preamble and postscript.

Example:

string input = "PREAMBLEHello, World!POSTSCRIPT PREAMBLEHello Again!POSTSCRIPT";
List<string> result = input.GetInnerTextList("preamble", "postscript", true);
// result: ["Hello, World!", "Hello Again!"]

Unit Tests

GetInnerText

[TestMethod]
public void GetInnerText_ReturnsCorrectInnerText()
{
    var input = "preambleHello, World!postscript";
    var result = input.GetInnerText("preamble", "postscript");
    Assert.AreEqual("Hello, World!", result);
}

GetInnerTextList

[TestMethod]
public void GetInnerTextList_ReturnsCorrectList()
{
    var input = "preambleHello, World!postscript preambleHello Again!postscript";
    var result = input.GetInnerTextList("preamble", "postscript");
    var expected = new List<string> { "Hello, World!", "Hello Again!" };
    CollectionAssert.AreEqual(expected, result);
}

ToByteArrayUtf8

[TestMethod]
public void ToByteArrayUtf8_ReturnsCorrectByteArray()
{
    var input = "Hello, World!";
    var result = input.ToByteArrayUtf8();
    var expected = Encoding.UTF8.GetBytes(input);
    CollectionAssert.AreEqual(expected, result);
}

For more examples and detailed documentation, please refer to the source code and unit tests provided.

License

This project is licensed under the MIT License. See the LICENSE file for more details.

Feel free to contribute by submitting issues or pull requests to the GitHub repository. Happy coding!

DocBuilder

Sat, 08 Jun 2024 00:00:00 GMT

Getting Started

Prerequisites

DocFx

See Getting Started with DocFX | DocFX website
DocFX is an API documentation generator for .NET, which currently supports C#, VB, and F#. It generates API reference documentation from triple-slash comments in your source code. It also allows you to use Markdown files to create additional topics such as tutorials and how-tos, and to customize the generated reference documentation.
Chocolatey: choco install docfx -y
Homebrew (owned by the community): brew install docfx
GitHub: Download and unzip docfx.zip from GitHub DocFX releases, extract it to a local folder, and add it to PATH so you can run it anywhere.
NuGet: nuget install docfx.console. docfx.exe is under folder docfx.console/tools/

DocBuilder PowerShell module

Located in the same GitHub repo as this documentation, ey-org/daas-documentation-tools
Clone the repo
Run LoadModule.ps1 in your favorite (.NET Framework-based) PowerShell host to load the module
Note: This module depends on GDI+, which does not exist in all host types, even on Windows

The sample script used to build this site is here: daas-documentation-tools\DocBuilder\Documentation\BuildDocs.ps1

Building the Static Web Site

Basic Process

Notes

You can use the BuildOutput folder in the same way this repo does as the source for GitHub pages
You can split the basic processes into smaller steps if you want to edit the markdown generated from PPTX
- Generate the markdown to a different location
- Edit as desired and copy into StaticContent using the best process for your project
- Generate the static website

Converting PowerPoint to Markdown

About the DocBuilder PowerShell Module

First, ensure you have the module loaded, as described in the Getting Started section.

PowerPoint File Conversion

To convert PowerPoint files to Markdown, use the Convert-PowerPointToMarkdown cmdlet provided by the DocBuilder PowerShell module.

Example usage:

Convert-PowerPointToMarkdown -InputFile "path    o\your\presentation.pptx" -OutputDirectory "path    o\output"

Converting PowerShell Module to Markdown

Preparing Your Module

Ensure your PowerShell module is well-documented with appropriate comments.
Use the Export-ModuleMember cmdlet to explicitly export functions and aliases that you want to include in the documentation.

Generating Markdown

Use the Convert-PowerShellToMarkdown cmdlet from the DocBuilder PowerShell module.

Example usage:

Convert-PowerShellToMarkdown -ModuleName "YourModuleName" -OutputDirectory "path    o\output"

Generating A Static Website

Setting Up the Environment

Ensure you have all prerequisites installed, as mentioned in the Getting Started section.

Building the Website

Run the BuildDocs.ps1 script to generate the static website.
Example usage:
```
.\BuildDocs.ps1
```

Configuration

Customize the docfx.json file to suit your project's needs.

Example configuration settings:

{
  "metadata": [
    {
      "src": [
        {
          "files": [
            "src/**/*.csproj"
          ],
          "cwd": "../"
        }
      ],
      "dest": "api"
    }
  ],
  "build": {
    "content": [
      {
        "files": [
          "articles/**.md",
          "toc.yml"
        ]
      }
    ],
    "resource": [
      {
        "files": [
          "images/**"
        ]
      }
    ],
    "dest": "dist"
  }
}

Adding Topics

When you add new topic articles or wish to change the order of the topic list, you must edit StaticContent/topics/toc.yml manually.

Testing Locally

Using http-server

After executing BuildDocs.ps1, the BuildOutput folder will contain a complete static website.
One recommended choice for serving content is http-server from NPM.
Note that this option requires npm and its dependencies.

Hot Reloading

You don’t have to stop the web server to run the build script.
It is highly recommended to open developer tools and disable caching.

Using GitHub Pages

Setup

Updating the Site

Copy the contents of the BuildOutput folder to /docs and update the master branch with a Pull Request.
Wait a moment for the site to update.

LibreChatBicep

Sat, 08 Jun 2024 00:00:00 GMT

Source Files are located in the repo containing this site

Tools and Configuration Setup

Make sure you have installed Azure CLI and are signed in to your Azure account. If you already have Azure CLI installed, run the az bicep upgrade command to ensure you're on the latest version of Bicep.

librechat.yaml

The librechat.yaml file should be located in the parent directory of the main.bicep script. To properly configure the librechat.yaml as part of the Bicep deployment, create an OpenAI group named "openai", and set the apiKey to "openai-key" and the instanceName to "openai-instance-name".

endpoints:
  azureOpenAI:
    (...)
    groups:
    - group: "openai"
      apiKey: "openai-key"
      instanceName: "openai-instance-name"
      forcePrompt: false
      assistants: true
      models:
        gpt-4-turbo:
          deploymentName: gpt-4-turbo
          version: "2024-02-15-preview"
        gpt-3.5-turbo-16k:
          deploymentName: gpt-35-turbo-16k
          version: "2024-02-15-preview"

models.json

The models.json file should be located in the same directory of the main.bicep script. Configure the modelName, version, and capacity for each model based on the desired models available in Azure OpenAI Service. Ensure the deploymentName for each model matches the deploymentName parameter in your librechat.yaml file.

{
  "models": [
    {
      "deploymentName": "gpt-4-turbo",
      "modelName": "gpt-4",
      "version": "turbo-2024-04-09",
      "capacity": 75
    },
    {
      "deploymentName": "gpt-35-turbo-16k",
      "modelName": "gpt-35-turbo-16k",
      "version": "0613",
      "capacity": 150
    }
  ]
}

Deployment Instructions

To get a list of available location names for deployment, use the az account list-locations command and use the given name property when referencing that location.

To ensure compatibility and availability of OpenAI models in your selected region, consult the Model Summary Table and Region Availability guide. This provides detailed information on which AI models are available in specific OpenAI service regions.

Provisioning a New Resource Group

To provision a new resource group and deploy resources to it, use the following command. Replace your_region with your preferred Azure location, resource_group_name with your desired resource group name, and openai_service_region with the OpenAI service region that supports the models you need.

az deployment sub create --name librechat --location your_region --template-file .\rg.bicep --parameters resourcegroup=resource_group_name location=your_region openAiService_location=openai_service_region

If you accidentally deploy to the wrong region or want to deploy this script across multiple regions, you might get the following error:

{"code": "InvalidDeploymentLocation", "message": "Invalid deployment location 'westeurope'. The deployment 'librechat' already exists in location 'eastus'."}

To fix this, change the --name parameter in the command from librechat to something else, eg. librechat2.

Deploying to an Existing Resource Group

For deploying resources to an existing resource group, execute the command below. Replace resource_group_name with the name of your existing resource group and openai_service_region with the appropriate OpenAI service region.

az deployment group create --resource-group resource_group_name --template-file .\main.bicep --parameters openAiService_location=openai_service_region createMonogexpressAuthConfig=true

The createMonogexpressAuthConfig parameter will automatically setup a Microsoft identity provider for mongo-express. Remove this parameter or set it to false if you don't want this.

IngestingPdfContent

Mon, 20 May 2024 00:00:00 GMT

GPT-4o versus Azure Document Intelligence and Azure Computer Vision OCR

Introduction

It’s been about sixth months since my last post, Building Good Chatbots Part One, No-Code with Microsoft Copilot Studio and Azure AI Studio. At the end of that very long article, which I split in two for LinkedIn, I said I’d write another part called “Castle on a hill” and immediately got insanely busy with client work. This is not that article, but it is closely related as it is about techniques for processing PDF content to make it easier to use with generative AI systems.

PDF files can be a challenge: the format itself is old and very permissive, processing tables is hard, and they often contain images with important information. They also require a lot of space compared to text-based file formats which means they take longer to move around and read and can trigger file size limits in some ‘chat about your documents’ systems. For these and other reasons it is usually best to either extract the text or convert the file itself to a text-based format such as markdown.

If you are not familiar with markdown, it is a popular text-based format for documents and it looks like this:

On the left is the text, and on the right is the formatted output.

This post compares several generative AI options you can use to convert PDF’s to markdown:

Note: as of the time of this writing (May 18, 2024), turbo-2024-04-09 is not compatible with the Azure Computer Vision OCR extensions and GPT-4o is not available in Azure. To fully use the notebooks referenced in this post, you will need both Azure and Open AI subscriptions.

In the end you will understand how to combine Azure Document Intelligence OCR with GPT4o to transform binary PDF content like this:

Into pure text in markdown format which displays like this:

Without losing the information displayed in tables and images, for around 3 cents ($0.03) per page.

The test PDF file is here
Here are the results you can judge for yourself.

Output File	Notes
mtoExtractedText	Cheap, fast, and ugly. Significant information loss. The default for most systems including Azure AI Search
mtoGPT4Turbo	Cheap, fast, and pretty. Poor accuracy and precision with significant hallucinations
mtoGPT4Omni	Less cheap, not as fast, and pretty. Poor accuracy and precision with significant hallucinations
mtoFromDocIntelOnly	Less cheap, less fast, and less ugly, but dependable. Possibly the best option if workload doesn't require interpretation of visual content
mtoGPT4VisionPreviewAndAzureComputerVisionOCR	Expensive, slow, pretty, more reliable. Not recommended, see below
mtoGPT4TurboWithDocIntelOCR	Good OCR and interpretation of visual content, possibly fewer hallucinations
mtoGPT4OmniWithDocIntelOCR	Least expensive combination of OCR grounding and GPT-4* with vision. Possibly the best option if workload requires interpretation of visual content

Why this matters

In my previous post, I explained a little about [solving hallucinations with retrieval augmented generation] (https://www.elumenotion.com/Journal/CopilotStudioThoughts.html#solving-hallucinations-with-retrieval-augmented-generation-rag). To summarize, the easiest way to prevent hallucinations is to provide the information required to answer the user’s question to the LLM in a prompt that contains the question and instructions to answer the question based on the provided information.

This ‘one simple trick’ requires you to have the information somewhere, such as a search index, and to be able to find the right information for the question when the user asks.

PDFs and other document formats that contain images such as figures are challenging because simple text extraction loses all the visual information and because the document formatting itself conveys information about the relationships between whatever is on the page. Tables are an example of this. Correctly extracting tables from PDFs is a notoriously difficult task.

I will say flatly that, if you have complex PDF content and you upload it to any off-the-shelf chat-on-your-data product today, you are very likely to be disappointed by the resulting solution - especially if it is ready to answer questions within a few seconds. If it is fast, you can be sure it didn’t do more than extract the text.

Baseline: Azure AI Studio and Search with Hybrid Vector/Keyword search

As a baseline I am using a pdf of the entra identity multi-tenant-organizations documentation from Microsoft Learn. I selected it because I have been reading it all recently and am familiar with its contents and because it contains challenging tables, diagrams, and other content.

Azure AI Studio and Azure AI Search

I will not cover the details of setting it up here, but I created a new index in Azure AI Search by uploading entra-identity-multi-tenant-organizations.pdf in the Azure AI Studio Chat Playground with the vector hybrid options and embeddings.

The test index

When you create an index this way, AI Studio creates a job using a docker image which ingests the content and builds the index. The longest running step is typically “Crack, Chuck, and Embed” which cracks open the document, chunks it into pieces of text, and creates embeddings for the vector search index from each chunk.

When the job completes, you can test the index in Azure AI Search. The ** Search explorer ** tab [DTW] shows that 198 documents are in the index because the job created a chunk from each page of the original PDF file. This page is of interest it contains several elements: text, a table, and images.

Lost Information

The original page of the PDF looks like this:

And the result of the search looks like this:

As you can see, the text itself has odd spaces and the meaning conveyed by the table is completely lost. Assuming we can somehow infer the positioning of the check marks in the columns from the white space is wrong as there is not enough whitespace on the first row. An LLM is likely to make this mistake and give incorrect answers as a result.

There are additional challenges, but the final example I will give here is shown in the next image:

This content has a page break and an image. I already explained that the information in the image is lost, but even if it wasn’t the position of the page break ensures that the section will be split across chunks. It is unlikely that a search that finds the first chunk will naturally include the second page which might be very important. Were this a markdown document, the headings could be used to keep related text in the same chunk except for cases where the text under a heading is longer than the chunk size.

For completeness, a notebook, PdfToTextPages.ipynb, which chunks the test pdf to pages of text is located in this repo.

PDF to Markdown Options

Option One: Azure Document Intelligence Layout Model

First up is Azure Document Intelligence with the Layout Model. Of the notebooks related to this article, it is the only based on C# and .NET. The rest are python. My version is based on this sample by James Croft.

This service is part of the approach that I recommend today based on the work that is the subject of this article, but the first version is notable for the following reasons:

It can produce a single markdown file from a PDF with little more than a line of code
It is very fast, processing a 198 page PFD in less than a minute
It provided accurate results for table contents and long numbers
It is the least expensive single option at one cent per page

However, by itself it has several weaknesses:

The markdown formatting is inconsistent
It is OCR only and does not extract meaning from images and diagrams

The page with the table from the baseline looks like this:

The table content is correct, and the formatting is present, but the formatting is not very good.

PDF to Images

Each of the remaining options starts by turning the PDF into a series of image files for processing. A notebook to perform this task, PdfToPageImages.ipynb, is in this repository, but the repository also contains the images used by the notebooks.

Option Two: gpt-4-turbo-2024-04-09 (GPT 4 Turbo with Vision) -OR - GPT-4o (Omni)

GPT-4-Turbo, notebook, and GPT-4o notebook are multimodal models which can reason against images. We can send the image directly to the LLM with a prompt to create a page of markdown.

Sample Prompt

Give me the markdown text output from this page in a PDF using formatting to match the structure of the page as close as you can get.
Only output the markdown and nothing else. Do not explain the output, just return it.
Do not use a single # for a heading. All headings will start with ## or ###.
Convert tables to markdown tables.
Take great care to ensure the precision and accuracy of numbers especially longer series of digits - do not transcribe or reorder the digits!
Describe charts and images as best you can and, when possible use mermaidjs format.
DO NOT return in a codeblock. Just return the raw text in markdown format.
Remove any irrelevant text from the markdown, returning the cleaned up version of the content. Examples include any images or 'click here' or 'Listen to this article' or page numbers or logos.

As I progressed through the options I refined this prompt and there is a better version at the end, but this was my first version.

This key part: Describe charts and images as best you can and, when possible use mermaidjs format, adds an ability that Doc Intelligence alone lacks, which is the ability to describe images. There is no reason why the description must only be text, and so I asked the LLM to create diagrams.

Now this image (the original):

Became this markdown which includes descriptions of the visual content including some mermaid diagrams:

Unfortunately, it changed a few things in the process and the output is consistently untrustworthy. For example:

The table looks good, BUT it is factually incorrect and an example of a hallucination. In fact, every version of GPT-4, including the newest Omni, failed consistently to produce this table correctly.

A much more serious issue present in each of the GPT-4 models I tried was transcription of digits. Consider this source content:

The “VRS code:” is **886249**. The hallucination is easy to observe. On one run, GPT-4o turned it into it to 286492

On another run, GPT-4-Turbo-2024-04-09 transcribed it to 882469

I honestly don’t think I observed a single success on this test without using some form of real OCR such as Azure Document Intelligence or Azure Computer Vision.

Differences between GPT-4-Turbo-2024 and GPT-4o

I personally found GPT-4-Turbo-2024 to be a little better with my small sample with markdown and image descriptions. But each made serious mistakes with the OCR that would create false answers were the result used for grounding. That said, GPT-4o was consistently around 30% faster and costs 50% less.

Conclusion on Option Two

If precise and accurate results are crucial, solutions using only GPT-4-Turbo-2024 and GPT-4o without OCR are problematic.

Option Three: gpt-4-vision-preview with Azure Computer Vision OCR

From How to use the GPT-4 Turbo with Vision model - Azure OpenAI Service | Microsoft Learn:

The notebooks for this option are notebook with OCR and notebook with OCR and grounding.

Good results, but not recommended

This combination yields the best results of the tested options (so far) but there are three issues:

They are the most expensive at approximately $0.05-$0.07 per page
The combination is the slowest at approximately 40-50 per seconds per page
The reliance on proprietary Microsoft extensions

The extensions are detailed in the MS doc linked above. My objection is not so much that they are proprietary as that using them is a risk, and Microsoft has not done a good job keeping them up to date. At an API level, I can’t imagine why one can only use GPT-4-Vision-Preview and not the better, faster, and cheaper GPT-4-Turbo-2024 other than that MS hasn’t gotten around to changing the config, or perhaps that they don’t foresee this feature ever exiting preview and becoming generally available.

Option Four (WINNER!!!): Azure Document Intelligence OCR and GPT-4o

Fortunately, you don’t need to take any chances with a proprietary extension when you can easily do the same thing while exercising full control. The [final notebook, v4omni-image-plus-docIntelOcr.ipynb,] ](https://github.com/douglasware/ElumenotionSite/blob/master/Projects/PdfToMarkdownAndQaPairs/v4omni-image-plus-docIntelOcr.ipynb?short_path=78bb846) sends the text of OCR’d markdown of a single page created from an image and the following prompt to GPT-4o.

Sample Prompt

The provided image is a page of a document.

The following markdown formatted text is a true and accurate but incomplete conversion of the provided image. Information contained in figures and other illustrations is missing.

MARKDOWN START

MARKDOWN TEXT HERE

MARKDOWN END

Create new markdown that contains the EXACT original text and EXACT original tables WITHOUT alteration, while ADDING full text descriptions in place of each figure or illustration.

Following each text description, consider if it is possible to further describe the figure or illustration using valid mermaidjs syntax and do so taking care to avoid syntax errors.

Following these guidelines ensures valid and correctly rendered MermaidJS diagrams.

Node IDs: Use only alphanumeric characters and underscores (_). Avoid spaces and other special characters.
Labels with Special Characters: Avoid special characters in labels. Enclose labels with spaces or special characters in double quotes within brackets.

Example

graph TD;
OwnerTenantC["Owner tenant (C)"] --> MemberTenantA["Member tenant (A)"]
OwnerTenantC["Owner tenant (C)"] --> MemberTenantB["Member tenant (B)"]
MemberTenantA["Member tenant (A)"] --> MemberTenantB["Member tenant (B)"]

Correct Usage

Node ID: OwnerTenantC
Label: "Owner tenant (C)"

Avoid

Opening or closing params and other special characters in labels
Node IDs with spaces or special characters: Owner tenant (C)
Labels without quotes: [Owner tenant (C)]

Note that to use the notebook, you must first use the PdfToPageImages notebook to create images and then use the DocIntelligencePipeline notebook to convert the single page images from the PDFs into individual markdown files.

This prompt grounds the text from the high-quality OCR from Document Intelligence with the generative capabilities of GPT-4o to insert image descriptions and well formatted mermaid diagrams.

The process costs around 3 cents ($0.03) per page and took less than 10 seconds per page on average.

Sample Output

Interestingly, in some cases GPT-4o elected to add diagrams where previously there were none. This sequence diagram mermaid syntax is correct and but there is nothing like it on the source page!

A cursory examination might make you think the content is correct.

Unfortunately, it is a hallucination. In fact, although most of the mermaid diagrams are correct, they often have errors. I’m very interested in this behavior and plan to spend much more time on it. The model might do a much better job with mermaid alternatives such as PlantUml.

The Index

Earlier, I showed the Search Explorer in Azure AI Search using an index based on the PDF. Here is the index for the markdown file.

The index has fewer chunks and takes up less space while containing much of the information lost in the original translation. This is the content for the same page in the new index.

Final Thoughts

Over a year ago now, the amazing Ted Chiang wrote ChatGPT Is a Blurry JPEG of the Web. The multimodal capabilities of models like GPT-4o show that his interesting and thoughtful perspective is quite incorrect. These tools give us the ability to refine information and project it in new and interesting ways by combining text, sound, and vision to extract meaning. We did not have this ability before, and it is as fundamental to our future as the printing press was to our ancestors.

Thanks for reading! If you need help improving your chat bots, copilots, and other generative AI applications, drop me a line. We would love to help!
Doug Ware
May 18, 2024

mtoExtractedText

Mon, 20 May 2024 00:00:00 GMT

Tell us about y our PDF experience. Multit enant or ganizations documentation A multitenant organization is an organization that has more than one instance of Microsoft Entra ID. Describes ways that users can have a seamless experience accessing resources and collaborating across multiple tenants. About multit enant or ganizations ｅOVERVIE W Multitenant organization capabilities Compare multitenant capabilities Configur e a multit enant or ganization ｅOVERVIE W What is a multitenant organization? ｃHOW-T O GUIDE Microsoft 365 admin center PowerShell or Microsoft Graph API Configur e cross-t enant synchr onization ｅOVERVIE W What is cross-tenant synchronization? ｃHOW-T O GUIDE Microsoft Entra admin center PowerShell or Microsoft Graph API

Collaborat e in Micr osoft 365 ｐCONCEPT Identity provisioning for Microsoft 365 Microsoft 365 multitenant people search Plan for multitenant organizations in Microsoft 365

Multitenant organization capabilities in Microsoft Entra ID Article •04/23/2024 This article provides an overview of the multitenant organization scenario and the related capabilities in Microsoft Entra ID. A tenant is an instance of Microsoft Entra ID in which information about a single organization resides including organizational objects such as users, groups, and devices and also application registrations, such as Microsoft 365 and third-party applications. A tenant also contains access and compliance policies for resources, such as applications registered in the directory. The primary functions served by a tenant include identity authentication as well as resource access management. From a Microsoft Entra perspective, a tenant forms an identity and access management scope. For example, a tenant administrator makes an application available to some or all the users in the tenant and enforces access policies on that application for users in that tenant. In addition, a tenant contains organizational branding data that drives end-user experiences, such as the organizations email domains and ShareP oint URLs used by employees in that organization. From a Microsoft 365 perspective, a tenant forms the default collaboration and licensing boundary. For example, users in Microsoft T eams or Microsoft Outlook can easily find and collaborate with other users in their tenant, but don't have the ability to find or see users in other tenants. Tenants contain privileged organizational data and are securely isolated from other tenants. In addition, tenants can be configured to have data persisted and processed in a specific region or cloud, which enables organizations to use tenants as a mechanism to meet data residency and handling compliance requirements. A multit enant or ganization is an organization that has more than one instance of Microsoft Entra ID. Here are the primary reasons why an organization might have multiple tenants: Conglomerat es: Organizations with multiple subsidiaries or business units that operate independently.What is a tenant? What is a multitenant organization?

Mergers and acquisitions: Organizations that merge or acquire companies. Divestitur e activity: In a divestiture, one organization splits off part of its business to form a new organization or sell it to an existing organization. Multiple clouds: Organizations that have compliance or regulatory needs to exist in multiple cloud environments. Multiple geographical boundaries: Organizations that operate in multiple geographic locations with various residency regulations. Test or staging t enants: Organizations that need multiple tenants for testing or staging purposes before deploying more broadly to primary tenants. Department or employ ee-cr eated tenants: Organizations where departments or employees have created tenants for development, testing, or separate control. Your organization may have recently acquired a new company, merged with another company, or restructured based on newly formed business units. If you have disparate identity management systems, it might be challenging for users in different tenants to access resources and collaborate. The following diagram shows how users in other tenants might not be able to access applications across tenants in your organization. As your organization evolves, your IT team must adapt to meet the changing needs. This often includes integrating with an existing tenant or forming a new one. R egardless of how the identity infrastructure is managed, it's critical that users have a seamless experience accessing resources and collaborating. T oday, you may be using custom scripts or on-premises solutions to bring the tenants together to provide a seamless experience across tenants. To enable users across tenants to collaborate in Teams Connect shared channels , you can use Microsoft Entra B2B direct connect . B2B direct connect is a feature of External Identities that lets you set up a mutual trust relationship with another Microsoft Entra organization for seamless collaboration in T eams. When the trust is established, the B2B direct connect user has single sign-on access using credentials from their home tenant.Multitenant challenges  B2B direct connect

Here's the primary constraint with using B2B direct connect across multiple tenants: Currently, B2B direct connect works only with T eams Connect shared channels. For more information, see B2B direct connect overview . To enable users across tenants to collaborate, you can use Microsoft Entra B2B collaboration . B2B collaboration is a feature within External Identities that lets you invite guest users to collaborate with your organization. Once the external user has redeemed their invitation or completed sign-up, they're represented in your tenant as a user object. With B2B collaboration, you can securely share your company's applications and services with external users, while maintaining control over your own corporate data. Here are the primary constraints with using B2B collaboration across multiple tenants: Administrators must invite users using the B2B invitation process or build an onboarding experience using the B2B collaboration invitation manager . Administrators might have to synchronize users using custom scripts. Depending on automatic redemption settings, users might need to accept a consent prompt and follow a redemption process in each tenant. By default, users are of type external guest, which has different permissions than external member and might not be the desired user experience.  B2B collaboration 

For more information, see B2B collaboration overview . If you want users to have a more seamless collaboration experience across tenants, you can use cross-tenant synchronization . Cross-tenant synchronization is a one-way synchronization service in Microsoft Entra ID that automates creating, updating, and deleting B2B collaboration users across tenants in an organization. Cross-tenant synchronization builds on the B2B collaboration functionality and utilizes existing B2B cross-tenant access settings. Users are represented in the target tenant as a B2B collaboration user object. Here are the primary benefits with using cross-tenant synchronization: Automatically create B2B collaboration users within your organization and provide them access to the applications they need, without creating and maintaining custom scripts. Improve the user experience and ensure that users can access resources, without receiving an invitation email and having to accept a consent prompt in each tenant. Automatically update users and remove them when they leave the organization. Here are the primary constraints with using cross-tenant synchronization across multiple tenants: Doesn't enhance the current T eams or Microsoft 365 experiences. S ynchronized users will have the same cross-tenant T eams and Microsoft 365 experiences available to any other B2B collaboration user. Doesn't synchronize groups, devices, or contacts. For more information, see What is cross-tenant synchronization? .Cross-tenant synchronization 

Multitenant organization is a feature in Microsoft Entra ID and Microsoft 365 that enables you to form a tenant group within your organization. Each pair of tenants in the group is governed by cross-tenant access settings that you can use to configure B2B or cross-tenant synchronization. Here are the primary benefits of a multitenant organization: Differentiate in-organization and out-of-organization external users Improved collaborative experience in new Microsoft T eams Improved people search experience across tenants For more information, see What is a multitenant organization in Microsoft Entra ID? . Depending on the needs of your organization, you can use any combination of B2B direct connect, B2B collaboration, cross-tenant synchronization, and multitenant organization capabilities. B2B direct connect and B2B collaboration are independent capabilities, while cross-tenant synchronization and multitenant organization capabilities are independent of each other, though both rely on underlying B2B collaboration. The following table compares the capabilities of each feature. For more information about different external identity scenarios, see Comparing External Identities feature sets.Multitenant organization  Compare multitenant capabilities ﾉExpand table

B2B dir ect connect (Org-to-org external or internal)B2B collaboration (Org-to-org external or internal)Cross-t enant synchr onization (Org int ernal)Multit enant organization (Org int ernal) Purpose Users can access T eams Connect shared channels hosted in external tenants.Users can access apps/resources hosted in external tenants, usually with limited guest privileges. Depending on automatic redemption settings, users might need to accept a consent prompt in each tenant.Users can seamlessly access apps/resources across the same organization, even if they're hosted in different tenants.Users can more seamlessly collaborate across a multitenant organization in new T eams and people search. Value Enables external collaboration within T eams Connect shared channels only. More convenient for administrators because they don't have to manage B2B users.Enables external collaboration. More control and monitoring for administrators by managing the B2B collaboration users. Administrators can limit the access that these external users have to their apps/resources.Enables collaboration across organizational tenants. Administrators don't have to manually invite and synchronize users between tenants to ensure continuous access to apps/resources within the organization.Enables collaboration across organizational tenants. Administrators continue to have full configuration ability via cross- tenant access settings. Optional cross-tenant access templates allow pre- configuration of cross-tenant access settings. Primar y administrat or workflowConfigure cross-tenant access to provide external users inbound access to tenant the credentials for their home tenant.Add external users to resource tenant by using the B2B invitation process or build your own onboarding experience using the B2B collaborationConfigure the cross-tenant synchronization engine to synchronize users between multiple tenants as B2B collaboration users.Create a multitenant organization, add (invite) tenants, join a multitenant organization. Leverage existing B2B collaboration users or use cross- tenant synchronization to

B2B dir ect connect (Org-to-org external or internal)B2B collaboration (Org-to-org external or internal)Cross-t enant synchr onization (Org int ernal)Multit enant organization (Org int ernal) invitation manager .provision B2B collaboration users. Trust lev el Mid trust. B2B direct connect users are less easy to track, mandating a certain level of trust with the external organization.Low to mid trust. User objects can be tracked easily and managed with granular controls.High trust. All tenants are part of the same organization, and users are typically granted member access to all apps/resources.High trust. All tenants are part of the same organization, and users are typically granted member access to all apps/resources. Effect on usersUsers access the resource tenant using the credentials for their home tenant. User objects aren't created in the resource tenant.External users are added to a tenant as B2B collaboration users.Within the same organization, users are synchronized from their home tenant to the resource tenant as B2B collaboration users.Within the same multitenant organization, B2B collaboration users, particularly member users, benefit from enhanced, seamless collaboration across Microsoft 365. User type B2B direct connect user

N/AB2B collaboration user
External member
External guest (default)B2B collaboration user
External member (default)
External guestB2B collaboration user
External member (default)
External guest The following diagram shows how B2B direct connect, B2B collaboration, and cross- tenant synchronization capabilities could be used together.

To better understand multitenant organization scenario related Microsoft Entra capabilities, you can refer back to the following list of terms. Term Definition tenant An instance of Microsoft Entra ID. organization The top level of a business hierarchy. multitenant organizationAn organization that has more than one instance of Microsoft Entra ID, as well as a capability to group those instances in Microsoft Entra ID. creator tenant The tenant that created the multitenant organization. owner tenant A tenant with the owner role. Initially, the creator tenant. added tenant A tenant that was added by an owner tenant. joiner tenant A tenant that is joining the multitenant organization. join request A joiner or added tenant submits a join request to join the multitenant organization. pending tenant A tenant that was added by an owner but that hasn't yet joined. active tenant A tenant that created or joined the multitenant organization.  Terminology ﾉExpand table

Term Definition member tenant A tenant with the member role. Most joiner tenants start as members. multitenant organization tenantAn active tenant of the multitenant organization, not pending. cross-tenant synchronizationA one-way synchronization service in Microsoft Entra ID that automates creating, updating, and deleting B2B collaboration users across tenants in an organization. cross-tenant access settingsSettings to manage collaboration for specific Microsoft Entra organizations. cross-tenant access settings templateAn optional template to preconfigure cross-tenant access settings that are applied to any partner tenant newly joining the multitenant organization. organizational settingsCross-tenant access settings for specific Microsoft Entra organizations. configuration An application and underlying service principal in Microsoft Entra ID that includes the settings (such as target tenant, user scope, and attribute mappings) needed for cross-tenant synchronization. provisioning The process of automatically creating or synchronizing objects across a boundary. automatic redemption A B2B setting to automatically redeem invitations so newly created users don't receive an invitation email or have to accept a consent prompt when added to a target tenant. What is a multitenant organization in Microsoft Entra ID? What is cross-tenant synchronization?Next steps

What is a multitenant organization in Microsoft Entra ID? Article •04/24/2024 Multitenant organization is a feature in Microsoft Entra ID and Microsoft 365 that enables you to form a tenant group within your organization. Each pair of tenants in the group is governed by cross-tenant access settings that you can use to configure B2B or cross-tenant synchronization. Here are the primary goals of multitenant organization: Define a group of tenants belonging to your organization Collaborate across your tenants in new Microsoft T eams Enable search and discovery of user profiles across your tenants through Microsoft 365 people search Organizations that own multiple Microsoft Entra tenants and want to streamline intra- organization cross-tenant collaboration in Microsoft 365. The multitenant organization capability is built on the assumption of reciprocal provisioning of B2B member users across multitenant organization tenants. As such, the multitenant organization capability assumes the simultaneous use of Microsoft Entra cross-tenant synchronization or an alternative bulk provisioning engine for external identities . Here are the primary benefits of a multitenant organization: Differentiate in-organization and out-of-organization external users In Microsoft Entra ID, external users originating from within a multitenant organization can be differentiated from external users originating from outside the multitenant organization. This differentiation facilitates the application of different policies for in-organization and out-of-organization external users.Why use multitenant organization? Who should use it? Benefits

Improved collaborative experience in Microsoft T eams In new Microsoft T eams, multitenant organization users can expect an improved collaborative experience across tenants with chat, calling, and meeting start notifications from all connected tenants across the multitenant organization. Tenant switching is more seamless and faster. For more information, see Announcing more seamless collaboration in Microsoft T eams for multitenant organizations and Microsoft T eams: Advantages of the new architecture . Improved people search experience across tenants Across Microsoft 365 services, the multitenant organization people search experience is a collaboration feature that enables search and discovery of people across multiple tenants. Once enabled, users are able to search and discover synced user profiles in a tenant's global address list and view their corresponding people cards. For more information, see Microsoft 365 multitenant organization people search . The multitenant organization capability enables you to form a tenant group within your organization. The following list describes the basic lifecycle of a multitenant organization. Define a multitenant organization One tenant administrator defines a multitenant organization as a grouping of tenants. The grouping of tenants isn't reciprocal until each listed tenant takes action to join the multitenant organization. The objective is a reciprocal agreement between all listed tenants. Join a multitenant organization Tenant administrators of listed tenants take action to join the multitenant organization. After joining, the multitenant organization relationship is reciprocal between each and every tenant that joined the multitenant organization. Leave a multitenant organization Tenant administrators of listed tenants can leave a multitenant organization at any time. While a tenant administrator who defined the multitenant organization can add and remove listed tenants they don't control the other tenants. How does a multitenant organization work?

A multitenant organization is established as a collaboration of equals. Each tenant administrator stays in control of their tenant and their membership in the multitenant organization. Administrators staying in control of their resources is a guiding principle for multitenant organization collaboration. Cross-tenant access settings are required for each tenant-to- tenant relationship. T enant administrators explicitly configure, as needed, the following policies: Cross-tenant access partner configurations For more information, see Configure cross-tenant access settings for B2B collaboration and crossT enantAccessP olicyConfigurationP artner resource type . Cross-tenant access identity synchronization For more information, see Configure cross-tenant synchronization and crossT enantIdentityS yncPolicyP artner resource type . The following diagram shows three tenants A, B, and C that form a multitenant organization.Cross-tenant access setting s Multitenant organization example  ﾉExpand table

Tenant Descr iption A Administrators see a multitenant organization consisting of A, B, C. They also see cross-tenant access settings for B and C. B Administrators see a multitenant organization consisting of A, B, C. They also see cross-tenant access settings for A and C. C Administrators see a multitenant organization consisting of A, B, C. They also see cross-tenant access settings for A and B. To ease the setup of homogenous cross-tenant access settings applied to partner tenants in the multitenant organization, the administrator of each multitenant organization tenant can configure optional cross-tenant access settings templates dedicated to the multitenant organization. These templates can be used to preconfigure cross-tenant access settings that are applied to any partner tenant newly joining the multitenant organization. To facilitate the management of a multitenant organization, any given multitenant organization tenant has an associated role and state. Tenant roleDescr iption Owner One tenant creates the multitenant organization. The multitenant organization creating tenant receives the role of owner. The privilege of the owner tenant is to add tenants into a pending state as well as to remove tenants from the multitenant organization. Also, an owner tenant can change the role of other multitenant organization tenants. Member Following the addition of pending tenants to the multitenant organization, pending tenants need to join the multitenant organization to turn their state from pending to active. Joined tenants typically start in the member role. Any member tenant has the privilege to leave the multitenant organization.Templates for cross-tenant access setting s Tenant role and state ﾉExpand table ﾉExpand table

Tenant stateDescr iption Pending A pending tenant has yet to join a multitenant organization. While listed in an administrator’s view of the multitenant organization, a pending tenant isn't yet part of the multitenant organization, and as such is hidden from an end user’s view of a multitenant organization. Active Following the addition of pending tenants to the multitenant organization, pending tenants need to join the multitenant organization to turn their state from pending to active. Joined tenants typically start in the member role. Any member tenant has the privilege to leave the multitenant organization. The multitenant organization capability has been designed with the following constraints: Any given tenant can only create or join a single multitenant organization. Any multitenant organization must have at least one active owner tenant. Each active tenant must have cross-tenant access settings for all active tenants. Any active tenant may leave a multitenant organization by removing themselves from it. A multitenant organization is deleted when the only remaining active (owner) tenant leaves. Resour ce Limit Notes Maximum number of active tenants, including the owner tenant100 The owner tenant can add more than 100 pending tenants, but they won't be able to join the multitenant organization if the limit is exceeded. This limit is applied at the time a pending tenant joins a multitenant organization. This limit is specific to the number of tenants in a multitenant organization. It does not apply to cross-tenant synchronization by itself. T o increase this limit, submit a support request in the Microsoft Entra or Microsoft 365 admin center. In the Microsoft Graph APIs, the default limit of 100 tenants is only enforced at the time of joining. In Microsoft 365 admin center, the default limit is enforced at multitenant organization creation time and at time of joining.Constraints Limits ﾉExpand table

By defining a multitenant organization, as well as pivoting on the Microsoft Entra user property of userT ype, external identities are segmented as follows: External members originating from within a multitenant organization External guests originating from within a multitenant organization External members originating from outside of your organization External guests originating from outside of your organization This segmentation of external users, due to the definition of a multitenant organization, enables administrators to better differentiate in-organization from out-of-organization external users. External members originating from within a multitenant organization are called multitenant organization members. Multitenant collaboration capabilities in Microsoft 365 aim to provide a seamless collaboration experience across tenant boundaries when collaborating with multitenant organization member users. If you haven't previously used Microsoft Entra cross-tenant synchronization, and you intend to establish a collaborating user set topology where the same set of users is shared to all multitenant organization tenants, you might want to use the Microsoft 365 admin center share users functionality. If you're already using Microsoft Entra cross-tenant synchronization, for various multi-hub multi-spoke topologies , you don't need to use the Microsoft 365 admin center share users functionality. Instead, you might want to continue using your existing Microsoft Entra cross-tenant synchronization jobs. Here are the basic steps to get started using multitenant organization. For more information, see Plan for multitenant organizations in Microsoft 365 .External user segmentation Choosing between Microsoft 365 admin center and cross-tenant synchronization Get started Step 1: Plan your deployment

Create your multitenant organization using Microsoft 365 admin center , Microsoft Graph P owerShell , or Microsoft Graph API : First tenant, soon-to-be owner tenant, creates a multitenant organization. Owner tenant adds one or more joiner tenants. Join a multitenant organization using Microsoft 365 admin center or Microsoft Graph PowerShell , or Microsoft Graph API : Joiner tenants submit a join request to join the multitenant organization of owner tenant. To allow for asynchronous processing, wait up to 2 hour s. Your multitenant organization is formed. Depending on your use case, you may want to synchronize users using one of the following methods: Synchronize users in multitenant organizations in Microsoft 365 Configure cross-tenant synchronization Configure cross-tenant synchronization using P owerShell or Microsoft Graph API Your alternative bulk provisioning engine The multitenant organization capability requires Microsoft Entra ID P1 licenses. Only one Microsoft Entra ID P1 license is required per employee per multitenant organization. Also, you must have at least one Microsoft Entra ID P1 license per tenant. T o find the right license for your requirements, see Compare generally available features of Microsoft Entra ID . Plan for multitenant organizations in Microsoft 365 What is cross-tenant synchronization?Step 2: Create your multitenant organization Step 3: Join a multitenant organization Step 4: Synchronize users License requirements Next steps

What is cross-tenant synchronization? Article •01/03/2024 Cross-tenant s ynchr onization automates creating, updating, and deleting Microsoft Entra B2B collaboration users across tenants in an organization. It enables users to access applications and collaborate across tenants, while still allowing the organization to evolve. Here are the primary goals of cross-tenant synchronization: Seamless collaboration for a multitenant organization Automate lifecycle management of B2B collaboration users in a multitenant organization Automatically remove B2B accounts when a user leaves the organization Cross-tenant synchronization automates creating, updating, and deleting B2B collaboration users. Users created with cross-tenant synchronization are able to access both Microsoft applications (such as T eams and ShareP oint) and non-Microsoft applications (such as ServiceNow , Adobe , and many more), regardless of which tenant the apps are integrated with. These users continue to benefit from the security capabilities in Microsoft Entra ID, such as Microsoft Entra Conditional Access and cross- tenant access settings , and can be governed through features such as Microsoft Entra entitlement management . The following diagram shows how you can use cross-tenant synchronization to enable users to access applications across tenants in your organization.https://www.youtube-nocookie.com/embed/7B-PQwNfGBc Why use cross-tenant synchronization?

Organizations that own multiple Microsoft Entra tenants and want to streamline intra-organization cross-tenant application access. Cross-tenant synchronization is not currently suitable for use across organizational boundaries. With cross-tenant synchronization, you can do the following: Automatically create B2B collaboration users within your organization and provide them access to the applications they need, without creating and maintaining custom scripts. Improve the user experience and ensure that users can access resources, without receiving an invitation email and having to accept a consent prompt in each tenant. Automatically update users and remove them when they leave the organization. Users created by cross-tenant synchronization will have the same experience when accessing Microsoft T eams and other Microsoft 365 services as B2B collaboration users created through a manual invitation. If your organization uses shared channels, please see the known issues document for additional details. Over time, the member userT ype will be used by the various Microsoft 365 services to provide differentiated end user experiences for users in a multitenant organization.  Who should use? Benefits Teams and Microsoft 365

When you configure cross-tenant synchronization, you define a trust relationship between a source tenant and a target tenant. Cross-tenant synchronization has the following properties: Based on the Microsoft Entra provisioning engine. Is a push process from the source tenant, not a pull process from the target tenant. Supports pushing only internal members from the source tenant. It doesn't support syncing external users from the source tenant. Users in scope for synchronization are configured in the source tenant. Attribute mapping is configured in the source tenant. Extension attributes are supported. Target tenant administrators can stop a synchronization at any time. The following table shows the parts of cross-tenant synchronization and which tenant they're configured. Tenant Cross-t enant access settingsAutomatic r edemption Sync settings configurationUser s in scope Source tenant✔ ✔ ✔ Target tenant✔ ✔ The cross-tenant synchronization setting is an inbound only organizational setting to allow the administrator of a source tenant to synchronize users into a target tenant. This setting is a check box with the name Allow user s sync int o this t enant that is specified in the target tenant. This setting doesn't impact B2B invitations created through other processes such as manual invitation or Microsoft Entra entitlement management .Properties ﾉExpand table Cross-tenant synchronization setting

To configure this setting using Microsoft Graph, see the Update crossT enantIdentityS yncPolicyP artner API. For more information, see Configure cross- tenant synchronization . The automatic redemption setting is an inbound and outbound organizational trust setting to automatically redeem invitations so users don't have to accept the consent prompt the first time they access the resource/target tenant. This setting is a check box with the following name: Automatically r edeem invitations with the t enant The automatic redemption setting applies to cross-tenant synchronization, B2B collaboration, and B2B direct connect in the following situations:  Automatic redemption setting  Compare setting for different scenarios

When users are created in a target tenant using cross-tenant synchronization. When users are added to a resource tenant using B2B collaboration. When users access resources in a resource tenant using B2B direct connect. The following table shows how this setting compares when enabled for these scenarios: Item Cross-t enant synchr onizationB2B collaborationB2B dir ect connect Automatic redemption setting Required Optional Optional Users receive a B2B collaboration invitation emailNo No N/A Users must accept a consent promptNo No No Users receive a B2B collaboration notification emailNo Yes N/A This setting doesn't impact application consent experiences. For more information, see Consent experience for applications in Microsoft Entra ID . This setting isn't supported for organizations across different Microsoft cloud environments, such as Azure commercial and Azure Government. The automatic redemption setting will only suppress the consent prompt and invitation email if both the home/source tenant (outbound) and resource/target tenant (inbound) checks this setting. The following table shows the consent prompt behavior for source tenant users when the automatic redemption setting is checked for different cross-tenant access setting combinations.ﾉExpand table When is consent prompt suppressed?

Home/sour ce tenant Resour ce/tar get t enant Consent pr ompt behavior for sour ce tenant user s Outbound Inbound Suppressed Not suppressed Not suppressed Not suppressed Inbound Outbound Not suppressed Not suppressed Not suppressed Not suppressed To configure this setting using Microsoft Graph, see the Update crossT enantAccessP olicyConfigurationP artner API. For more information, see Configure cross-tenant synchronization . For cross-tenant synchronization, users don't receive an email or have to accept a consent prompt. If users want to see what tenants they belong to, they can open their My Account page and select Organizations . In the Microsoft Entra admin center, users can open their Portal settings , view their Directories + subscriptions , and switch directories. For more information, including privacy information, see Leave an organization as an external user . Here are the basic steps to get started using cross-tenant synchronization.ﾉExpand table How do users know what tenants they belong to? Get started

Cross-tenant synchronization provides a flexible solution to enable collaboration, but every organization is different. For example, you might have a central tenant, satellite tenants, or sort of a mesh of tenants. Cross-tenant synchronization supports any of these topologies. For more information, see Topologies for cross-tenant synchronization . In the target tenant where users are created, navigate to the Cross-t enant access settings page. Here you enable cross-tenant synchronization and the B2B automatic redemption settings by selecting the respective check boxes. For more information, see Configure cross-tenant synchronization . In any source tenant, navigate to the Cross-t enant access settings page and enable the B2B automatic redemption feature. Next, you use the Cross-t enant synchr onization page to set up a cross-tenant synchronization job and specify: Which users you want to synchronize What attributes you want to include Any transformations For anyone that has used Microsoft Entra ID to provision identities into a SaaS application , this experience will be familiar. Once you have synchronization configured,Step 1: Define how to structure the tenants in your organization Step 2: Enable cross-tenant synchronization in the target tenants Step 3: Enable cross-tenant synchronization in the source tenants

you can start testing with a few users and make sure they're created with all the attributes that you need. When testing is complete, you can quickly add additional users to synchronize and roll out across your organization. For more information, see Configure cross-tenant synchronization . In the source tenant: Using this feature requires Microsoft Entra ID P1 licenses. Each user who is synchronized with cross-tenant synchronization must have a P1 license in their home/source tenant. T o find the right license for your requirements, see Compare generally available features of Microsoft Entra ID . In the target tenant: Cross-tenant sync relies on the Microsoft Entra External ID billing model. T o understand the external identities licensing model, see MAU billing model for Microsoft Entra External ID . You will also need at least one Microsoft Entra ID P1 license in the target tenant to enable auto-redemption. Which clouds can cross-tenant synchronization be used in? Cross-tenant synchronization is supported within the commercial cloud and Azure Government. Cross-tenant synchronization isn't supported within the Microsoft Azure operated by 21Vianet cloud. Synchronization is only supported between two tenants in the same cloud. Cross-cloud (such as public cloud to Azure Government) isn't currently supported. Will cross-tenant synchronization manage existing B2B users?License requirements Frequently asked questions Clouds Existing B2B users

Yes. Cross-tenant synchronization uses an internal attribute called the alternativeSecurityIdentifier to uniquely match an internal user in the source tenant with an external / B2B user in the target tenant. Cross-tenant synchronization can update existing B2B users, ensuring that each user has only one account. Cross-tenant synchronization cannot match an internal user in the source tenant with an internal user in the target tenant (both type member and type guest). How often does cross-tenant synchronization run? The sync interval is currently fixed to start at 40-minute intervals. S ync duration varies based on the number of in-scope users. The initial sync cycle is likely to take significantly longer than the following incremental sync cycles. How do I control what is synchronized into the target tenant? In the source tenant, you can control which users are provisioned with the configuration or attribute-based filters. Y ou can also control what attributes on the user object are synchronized. For more information, see Scoping users or groups to be provisioned with scoping filters . If a user is removed from the scope of sync in a source tenant, will cross-tenant synchronization soft delete them in the target? Yes. If a user is removed from the scope of sync in a source tenant, cross-tenant synchronization will soft delete them in the target tenant. What object types can be synchronized? Microsoft Entra users can be synchronized between tenants. (Groups, devices, and contacts aren't currently supported.) What user types can be synchronized? Internal members can be synchronized from source tenants. Internal guests can't be synchronized from source tenants. Users can be synchronized to target tenants as external members (default) or external guests.Synchronization frequency Scope Object types

For more information about the UserT ype definitions, see Properties of a Microsoft Entra B2B collaboration user . I have existing B2B collaboration users. What will happen to them? Cross-tenant synchronization will match the user and make any necessary updates to the user, such as update the display name. By default, the UserT ype won't be updated from guest to member, but you can configure this in the attribute mappings. What user attributes can be synchronized? Cross-tenant synchronization will sync commonly used attributes on the user object in Microsoft Entra ID, including (but not limited to) displayName, userPrincipalName, and directory extension attributes. Cross-tenant synchronization supports provisioning the manager attribute. Both the user and their manager must be in scope for provisioning. For cross-tenant synchronization configurations created before January 2024 with the default schema / attribute mappings: The manager attribute will automatically be added to the mappings. This does not trigger an initial sync cycle. Manager updates will apply on the incremental cycle for users that are undergoing changes (e.g. manager change). The sync engine doesn’t automatically update all existing users that were provisioned previously. To update the manager for existing users that are in scope for provisioning, you can use on-demand provisioning for specific users or do a restart to provision the manager for all users. For cross-tenant synchronization configurations created before January 2024 with a custom schema / attribute mappings (e.g. you added an attribute to the mappings or changed the default mappings): You need to manually add the manager attribute to your attribute mappings. This will trigger a restart and update all users that are in scope for provisioning. This should be a direct mapping of the manager attribute in the source tenant to the manager in the target tenant. If the manager of a user is removed in the source tenant and no new manager is assigned in the source tenant, the manager attribute will not be updated in the target tenant. What attributes can't be synchronized?Attributes

Attributes including (but not limited to) photos, custom security attributes, and user attributes outside of the directory can't be synchronized by cross-tenant synchronization. Can I control where user attributes are sourced/managed? Cross-tenant synchronization doesn't offer direct control over source of authority. The user and its attributes are deemed authoritative at the source tenant. There are parallel sources of authority workstreams that will evolve source of authority controls for users down to the attribute level and a user object at the source may ultimately reflect multiple underlying sources. For the tenant-to-tenant process, this is still treated as the source tenant's values being authoritative for the sync process (even if pieces actually originate elsewhere) into the target tenant. Currently, there's no support for reversing the sync process's source of authority. Cross-tenant synchronization only supports source of authority at the object level. That means all attributes of a user must come from the same source, including credentials. It isn't possible to reverse the source of authority or federation direction of a synchronized object. What happens if attributes for a synced user are changed in the target tenant? Cross-tenant synchronization doesn't query for changes in the target. If no changes are made to the synced user in the source tenant, then user attribute changes made in the target tenant will persist. However, if changes are made to the user in the source tenant, then during the next synchronization cycle, the user in the target tenant will be updated to match the user in the source tenant. Can the target tenant manually block sign-in for a specific home/source tenant user that is synced? If no changes are made to the synced user in the source tenant, then the block sign-in setting in the target tenant will persist. If a change is detected for the user in the source tenant, cross-tenant synchronization will re-enable that user blocked from sign-in in the target tenant. Can I sync a mesh between multiple tenants? Cross-tenant synchronization is configured as a single-direction peer-to-peer sync, meaning sync is configured between one source and one target tenant. Multiple instances of cross-tenant synchronization can be configured to sync from a singleStructure

source to multiple targets and from multiple sources into a single target. But only one sync instance can exist between a source and a target. Cross-tenant synchronization only synchronizes users that are internal to the home/source tenant, ensuring that you can't end up with a loop where a user is written back to the same tenant. Multiple topologies are supported. For more information, see Topologies for cross- tenant synchronization . Can I use cross-tenant synchronization across organizations (outside my multitenant organization)? For privacy reasons, cross-tenant synchronization is intended for use within an organization. W e recommend using entitlement management for inviting B2B collaboration users across organizations. Can cross-tenant synchronization be used to migrate users from one tenant to another tenant? No. Cross-tenant synchronization isn't a migration tool because the source tenant is required for synchronized users to authenticate. In addition, tenant migrations would require migrating user data such as ShareP oint and OneDrive. Does cross-tenant synchronization resolve any present B2B collaboration limitations? Since cross-tenant synchronization is built on existing B2B collaboration technology, existing limitations apply. Examples include (but aren't limited to): App or serviceLimitations Power BI - Support for UserT ype Member in P ower BI is currently in preview. For more information, see Distribute P ower BI content to external guest users with Microsoft Entra B2B . Azure Virtual Desktop- External member and external guest aren't supported in Azure Virtual Desktop. How does cross-tenant synchronization relate to B2B direct connect ?B2B collaboration ﾉExpand table B2B direct connect

B2B direct connect is the underlying identity technology required for Teams Connect shared channels . We recommend B2B collaboration for all other cross-tenant application access scenarios, including both Microsoft and non-Microsoft applications. B2B direct connect and cross-tenant synchronization are designed to co-exist, and you can enable them both for broad coverage of cross-tenant scenarios. We're trying to determine the extent to which we'll need to utilize cross-tenant synchronization in our multitenant organization. Do you plan to extend support for B2B direct connect beyond T eams Connect? There's no plan to extend support for B2B direct connect beyond T eams Connect shared channels. Does cross-tenant synchronization enhance any cross-tenant Microsoft 365 app access user experiences? Cross-tenant synchronization utilizes a feature that improves the user experience by suppressing the first-time B2B consent prompt and redemption process in each tenant. Synchronized users will have the same cross-tenant Microsoft 365 experiences available to any other B2B collaboration user. Can cross-tenant synchronization enable people search scenarios where synchronized users appear in the global address list of the target tenant? Yes, but you must set the value for the showInAddr essList attribute of synchronized users to True, which is not set by default. If you want to create a unified address list, you'll need to set up a mesh peer-to-peer topology . For more information, see Step 9: R eview attribute mappings . Cross-tenant synchronization creates B2B collaboration users and doesn't create contacts. Does cross-tenant synchronization enhance any current T eams experiences? Synchronized users will have the same cross-tenant Microsoft 365 experiences available to any other B2B collaboration user.Microsoft 365 Teams

What federation options are supported for users in the target tenant back to the source tenant? For each internal user in the source tenant, cross-tenant synchronization creates a federated external user (commonly used in B2B) in the target. It supports syncing internal users. This includes internal users federated to other identity systems using domain federation (such as Active Directory Federation Services ). It doesn't support syncing external users. Does cross-tenant synchronization use S ystem for Cross-Domain Identity Management (SCIM)? No. Currently, Microsoft Entra ID supports a SCIM client, but not a SCIM server. For more information, see SCIM synchronization with Microsoft Entra ID . Does cross-tenant synchronization support deprovisioning users? Yes, when the below actions occur in the source tenant, the user will be soft deleted in the target tenant. Delete the user in the source tenant Unassign the user from the cross-tenant synchronization configuration Remove the user from a group that is assigned to the cross-tenant synchronization configuration An attribute on the user changes such that they do not meet the scoping filter conditions defined on the cross-tenant synchronization configuration anymore If the user is blocked from sign-in in the source tenant (accountEnabled = false) they will be blocked from sign-in in the target. This is not a deletion, but an updated to the accountEnabled property. Users are not soft deleted from the target tenant in this scenario:

Add a user to a group and assign it to the cross-tenant synchronization configuration in the source tenant.
Provision the user on-demand or through the incremental cycle.
Update the account enabled status to false on the user in the source tenant.
Provision the user on-demand or through the incremental cycle. The account enabled status is changed to false in the target tenant.
Remove the user from the group in the source tenant.Integration Deprovisioning

Does cross-tenant synchronization support restoring users? If the user in the source tenant is restored, reassigned to the app, meets the scoping condition again within 30 days of soft deletion, it will be restored in the target tenant. IT admins can also manually restore the user directly in the target tenant. How can I deprovision all the users that are currently in scope of cross-tenant synchronization? Unassign all users and / or groups from the cross-tenant synchronization configuration. This will trigger all the users that were unassigned, either directly or through group membership, to be deprovisioned in subsequent sync cycles. Please note that the target tenant will need to keep the inbound policy for sync enabled until deprovisioning is complete. If the scope is set to Sync all user s and gr oups , you will also need to change it to Sync only assigned user s and gr oups . The users will be automatically soft deleted by cross-tenant synchronization. The users will be automatically hard deleted after 30 days or you can choose to hard delete the users directly from the target tenant. Y ou can choose to hard delete the users directly in the target tenant or wait 30 days for the users to be automatically hard deleted. If the sync relationship is severed, are external users previously managed by cross- tenant synchronization deleted in the target tenant? No. No changes are made to the external users previously managed by cross- tenant synchronization if the relationship is severed (for example, if the cross- tenant synchronization policy is deleted). Topologies for cross-tenant synchronization Configure cross-tenant synchronizationNext steps

Multitenant organization identity provisioning for Microsoft 365 Article •04/24/2024 The multitenant organization capability is designed for organizations that own multiple Microsoft Entra tenants and want to streamline intra-organization cross-tenant collaboration in Microsoft 365. It's built on the premise of reciprocal provisioning of B2B member users across multitenant organization tenants. Teams external access and Teams shared channels excluded, Microsoft 365 people search is typically scoped to within local tenant boundaries. In multitenant organizations with increased need for cross-tenant coworker collaboration, it's recommended to reciprocally provision users from their home tenants into the resource tenants of collaborating coworkers. The new Microsoft T eams experience improves upon Microsoft 365 people search and Teams external access for a unified seamless collaboration experience. For this improved experience to light up, the multitenant organization representation in Microsoft Entra ID is required and collaborating users shall be provisioned as B2B members. For more information, see Announcing more seamless collaboration in Microsoft T eams for multitenant organizations . Collaboration in Microsoft 365 is built on the premise of reciprocal provisioning of B2B identities across multitenant organization tenants. For example, say Annie in tenant A, Bob and Barbara in tenant B, and Charlie in tenant C want to collaborate. Conceptually, these four users represent a collaborating user set of four internal identities across three tenants.Microsoft 365 people search New Microsoft Teams Collaborating user set

For people search to succeed, while scoped to local tenant boundaries, the entire collaborating user set must be represented within the scope of each multitenant organization tenant A, B, and C, in the form of either internal or B2B identities. Depending on your organization’s needs, the collaborating user set may contain a subset of collaborating employees, or eventually all employees. One of the simpler ways to achieve a collaborating user set in each multitenant organization tenant is for each tenant administrator to define their user contribution and synchronization them outbound. T enant administrators on the receiving end should accept the shared users inbound. Administrator A contributes or shares Annie Administrator B contributes or shares Bob and Barbara Administrator C contributes or shares Charles   Sharing your users

Microsoft 365 admin center facilitates orchestration of such a collaborating user set across multitenant organization tenants. For more information, see Synchronize users in multitenant organizations in Microsoft 365 . Alternatively, pair-wise configuration of inbound and outbound cross-tenant synchronization can be used to orchestrate such collating user set across multitenant organization tenants. For more information, see What is a cross-tenant synchronization . To ensure a seamless collaboration experience across the multitenant organization in new Microsoft T eams, B2B identities are provisioned as B2B users of Member userT ype. User synchr onization method Default userT ype pr oper ty Synchronize users in multitenant organizations in Microsoft 365Member Remains Guest, if the B2B identity already existed as Guest Cross-tenant synchronization in Microsoft Entra ID Member Remains Guest, if the B2B identity already existed as Guest  B2B member users ﾉExpand table

From a security perspective, you should review the default permissions granted to B2B member users. For more information, see Compare member and guest default permissions . To change the userT ype from Guest to Member (or vice versa), a source tenant administrator can amend the attribute mappings , or a target tenant administrator can change the userT ype if the property is not recurringly synchronized. To unshare users, you deprovision users by using the user deprovisioning capabilities available in Microsoft Entra cross-tenant synchronization. By default, when provisioning scope is reduced while a synchronization job is running, users fall out of scope and are soft deleted, unless T arget Object Actions for Delete is disabled. For more information, see Deprovisioning and Define who is in scope for provisioning . Plan for multitenant organizations in Microsoft 365 Set up a multitenant org in Microsoft 365Unsharing your users Next steps

Multitenant organization optional policy templates Article •04/23/2024 Administrators staying in control of their resources is a guiding principle for multitenant organization collaboration. Cross-tenant access settings are required for each tenant-to- tenant relationship. T enant administrators explicitly configure cross-tenant access partner configurations and identity synchronization settings for partner tenants inside the multitenant organization. To help apply homogenous cross-tenant access settings to partner tenants in the multitenant organization, the administrator of each tenant can configure optional cross- tenant access settings templates dedicated to the multitenant organization. This article describes how to use templates to preconfigure cross-tenant access settings that are applied to any partner tenant newly joining the multitenant organization. Within a multitenant organization, each pair of tenants must have bi-directional cross- tenant access settings , for both, partner configuration and identity synchronization. These settings provide the underlying policy framework for enabling trust and for sharing users and applications. When your tenant joins a new multitenant organization, or when a partner tenant joins your existing multitenant organization, cross-tenant access settings to other partner tenants in the enlarged multitenant organization, if they don't already exist, are automatically generated in an unconfigured state. In an unconfigured state, these cross- tenant access settings pass through the default settings . Default cross-tenant access settings apply to all external tenants for which you haven't created organization-specific customized settings. T ypically, these settings are configured to be nontrusting. For example, cross-tenant trusts for multifactor authentication and compliant device claims might be disabled and user and group sharing in B2B direct connect or B2B collaboration might be disallowed. In multitenant organizations, on the other hand, cross-tenant access settings are typically expected to be trusting. For example, cross-tenant trusts for multifactor authentication and compliant device claims might be enabled and user and group sharing in B2B direct connect or B2B collaboration might be allowed.Autogeneration of cross-tenant access setting s

While the autogeneration of cross-tenant access settings for multitenant organization partner tenants in and of itself doesn't change any authentication or authorization policy behavior, it allows your organization to easily customize the cross-tenant access settings for partner tenants in the multitenant organization on a per-tenant basis. As previously described, in multitenant organizations, cross-tenant access settings are typically expected to be trusting. For example, cross-tenant trusts for multifactor authentication and compliant device claims might be enabled and user and group sharing in B2B direct connect or B2B collaboration might be allowed. While autogeneration of cross-tenant access settings, per previous section, guarantees the existence of cross-tenant access settings for every multitenant organization partner tenant, further maintenance of the cross-tenant access settings for multitenant organization partner tenants is conducted individually, on a per-tenant basis. To reduce the workload for administrators at the time of multitenant organization formation, you can optionally use policy templates for preemptive configuration of cross-tenant access settings. These template settings are applied at the time of your tenant joins a multitenant organization to all external multitenant organization partner tenants as well as at the time of any partner tenant joins your existing multitenant organization to such new partner tenant. Enablement or configuration of the optional policy templates , at the time of a partner tenant joins a multitenant organization, preemptively amend the corresponding cross- tenant access settings , for both partner configuration and identity synchronization. As an example, consider the actions of the administrators for an anticipated multitenant organization with three tenants, A, B, and C. The administrators of all three tenants enable and configure their respective optional policy templates to enable cross-tenant trusts for multifactor authentication and compliant device claims and to allow user and group sharing in B2B direct connect and B2B collaboration. Administrator A creates the multitenant organization and adds tenants B and C as pending tenants to the multitenant organization. Administrator B joins the multitenant organization. Cross-tenant access settings in tenant A for partner tenant B are amended, according to tenant A policy template settings. Vice versa, cross-tenant access settings in tenant B for partner tenant A are amended, according to tenant B policy template settings.Policy templates at multitenant organization formation

Administrator C joins the multitenant organization. Cross-tenant access settings in tenants A (and B) for partner tenant C are amended, according to tenant A (and B) policy template settings. Similarly, cross-tenant access settings in tenant C for partner tenants A and B are amended, according to tenant C policy template settings. Following the formation of this multitenant organization of three tenants, the cross-tenant access settings of all tenant pairs in the multitenant organization have preemptively been configured. In summary, configuration of the optional policy templates enable you to homogeneously initialize cross-tenant access settings across your multitenant organization, while maintaining maximum flexibility to customize your cross-tenant access settings as needed on a per-tenant basis. To stop using the policy templates, you can reset them to their default state. For more information, see Configure multitenant organization templates . To provide administrators with further configurability, you can choose when cross- tenant access settings are to be amended according to the policy templates. For example, you can choose to apply the policy templates for the following tenants when a tenant joins a multitenant organization: Tenant Descr iption Only new partner tenants Tenants whose cross-tenant access settings are autogenerated Only existing partner tenants Tenants who already have cross-tenant access settings All partner tenants Both new partner tenants and existing partner tenants No partner tenants Policy templates are effectively disabled In this context, new partners refer to tenants for which you haven't yet configured cross- tenant access settings, while existing partners refer to tenants for which you have already configured cross-tenant access settings. This scoping is specified with the templateApplicationLevel property on the cross-tenant access partner configuration template and the templateApplicationLevel property on the cross-tenant access identity synchronization template .Policy template scoping and additional properties ﾉExpand table

Finally, in terms of interpretation of template property values, any template property value of null has no effect on the corresponding property value in the targeted cross- tenant access settings, while a defined template property value causes the corresponding property value in the targeted cross-tenant access settings to be amended in accordance with the template. The following table illustrates how template property values are being applied to corresponding cross-tenant access setting values. Templat e Value Initial P artner Settings V alue (Befor e joining multit enant or g)Final P artner Settings V alue (After joining multit enant or g) null

	B2B direct connect (Org-to-org external or internal)	B2B collaboration (Org-to-org external or internal)	Cross-tenant synchronization (Org internal)	Multitenant organization (Org internal)
Purpose	Users can access Teams Connect shared channels hosted in external tenants.	Users can access apps/resources hosted in external tenants, usually with limited guest privileges. Depending on automatic redemption settings, users might need to accept a consent prompt in each tenant.	Users can seamlessly access apps/resources across the same organization, even if they're hosted in different tenants.	Users can more seamlessly collaborate across a multitenant organization in new Teams and people search.
Value	Enables external collaboration within Teams Connect shared channels only. More convenient for administrators because they don't have to manage B2B external users.	Enables external collaboration. More control and monitoring for administrators by managing the B2B collaboration users. Administrators can limit the access that these external users have to their apps/resources.	Enables collaboration across organizational tenants. Administrators don't have to manually invite and synchronize users between tenants to ensure continuous access to apps/resources within the organization.	Enables collaboration across organizational tenants. Administrators continue to have full configuration ability via cross-tenant access settings. Optional cross-tenant access templates allow pre-configuration of cross-tenant access settings.
Primary administrator workflow	Configure cross-tenant access to provide external users inbound access to tenant the credentials for their home tenant.	Add external users to resource tenant by using the B2B invitation process or build your own onboarding experience using the B2B collaboration	Configure the cross-tenant synchronization engine to synchronize users between multiple tenants as B2B collaboration users.	Create a multitenant organization, add (invite) tenants, join a multitenant organization. Leverage existing B2B collaboration users or use cross-tenant synchronization to ensure continuous access to resources across tenants.

	B2B direct connect (Org-to-org external or internal)	B2B collaboration (Org-to-org external or internal)	Cross-tenant synchronization (Org internal)	Multitenant organization (Org internal)
		invitation manager.		provision B2B collaboration users.
Trust level	Mid trust. B2B direct connect users are less easy to track, mandating a certain level of trust with the external organization.	Low to mid trust. User objects can be tracked easily and managed with granular controls.	High trust. All tenants are part of the same organization, and users are typically granted member access to all apps/resources.	High trust. All tenants are part of the same organization, and users are typically granted member access to all apps/resources.
Effect on users	Users access the resource tenant using the credentials for their home tenant. User objects aren't created in the resource tenant.	External users are added to a tenant as B2B collaboration users.	Within the same organization, users are synchronized from their home tenant to the resource tenant as B2B collaboration users.	Within the same multitenant organization, B2B collaboration users, particularly member users, benefit from enhanced, seamless collaboration across Microsoft 365.
User type	B2B direct connect user - N/A	B2B collaboration user	B2B collaboration user	B2B collaboration user
		- External member (default) - External guest (default)	- External member (default) - External guest	- External member (default) - External guest

Term	Definition
tenant	An instance of Microsoft Entra ID.
organization	The top level of a business hierarchy.
multitenant organization	An organization that has more than one instance of Microsoft Entra ID, as well as a capability to group those instances in Microsoft Entra ID.
creator tenant	The tenant that created the multitenant organization.
owner tenant	A tenant with the owner role. Initially, the creator tenant.
added tenant	A tenant that was added by an owner tenant.
joiner tenant	A tenant that is joining the multitenant organization.
join request	A joiner or added tenant submits a join request to join the multitenant organization.
pending tenant	A tenant that was added by an owner but that hasn't yet joined.
active tenant	A tenant that created or joined the multitenant organization.

Term	Definition
member tenant	A tenant with the member role. Most joiner tenants start as members.
multitenant organization tenant	An active tenant of the multitenant organization, not pending.
cross-tenant synchronization	A one-way synchronization service in Microsoft Entra ID that automates creating, updating, and deleting B2B collaboration users across tenants in an organization.
cross-tenant access settings	Settings to manage collaboration for specific Microsoft Entra organizations.
cross-tenant access settings template	An optional template to preconfigure cross-tenant access settings that are applied to any partner tenant newly joining the multitenant organization.
organizational settings	Cross-tenant access settings for specific Microsoft Entra organizations.
configuration	An application and underlying service principal in Microsoft Entra ID that includes the settings (such as target tenant, user scope, and attribute mappings) needed for cross-tenant synchronization.
provisioning	The process of automatically creating or synchronizing objects across a boundary.
automatic redemption	A B2B setting to automatically redeem invitations so newly created users don't receive an invitation email or have to accept a consent prompt when added to a target tenant.

Tenant	Description
A	Administrators see a multitenant organization consisting of A, B, C. They also see cross-tenant access settings for B and C.
B	Administrators see a multitenant organization consisting of A, B, C. They also see cross-tenant access settings for A and C.
C	Administrators see a multitenant organization consisting of A, B, C. They also see cross-tenant access settings for A and B.


Tenant role	Description
Owner	One tenant creates the multitenant organization. The multitenant organization creating tenant receives the role of owner. The privilege of the owner tenant is to add tenants into a pending state as well as to remove tenants from the multitenant organization. Also, an owner tenant can change the role of other multitenant organization tenants.
Member	Following the addition of pending tenants to the multitenant organization, pending tenants need to join the multitenant organization to turn their state from pending to active. Joined tenants typically start in the member role. Any member tenant has the privilege to leave the multitenant organization.


Tenant state	Description
Pending	A pending tenant has yet to join a multitenant organization. While listed in an administrator's view of the multitenant organization, a pending tenant isn't yet part of the multitenant organization, and as such is hidden from an end user's view of a multitenant organization.
Active	Following the addition of pending tenants to the multitenant organization, pending tenants need to join the multitenant organization to turn their state from pending to active. Joined tenants typically start in the member role. Any member tenant has the privilege to leave the multitenant organization.

Tenant	Cross-tenant access settings	Automatic redemption	Sync settings configuration	Users in scope
Source tenant		:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
Target tenant	:heavy_check_mark:	:heavy_check_mark:

Item	Cross-tenant synchronization	B2B collaboration	B2B direct connect
Automatic redemption setting	Required	Optional	Optional
Users receive a B2B collaboration invitation email	No	No	N/A
Users must accept a consent prompt	No	No	No
Users receive a B2B collaboration notification email	No	Yes	N/A

Home/source tenant	Resource/target tenant	Consent prompt behavior for source tenant users
Outbound	Inbound	Suppressed
Outbound	Inbound	Not suppressed
Outbound	Inbound	Not suppressed
Outbound	Inbound	Not suppressed
Inbound	Outbound	Not suppressed
Inbound	Outbound	Not suppressed
Inbound	Outbound	Not suppressed
Inbound	Outbound	Not suppressed


App or service	Limitations
Power BI	- Support for UserType Member in Power BI is currently in preview. For more information, see Distribute Power BI content to external guest users with Microsoft Entra B2B.
Azure Virtual Desktop	- External member and external guest aren't supported in Azure Virtual Desktop.

User synchronization method	Default userType property
Synchronize users in multitenant organizations in Microsoft 365	Member
	Remains Guest, if the B2B identity already existed as Guest
Cross-tenant synchronization in Microsoft Entra ID	Member
	Remains Guest, if the B2B identity already existed as Guest

Tenant	Description
Only new partner tenants	Tenants whose cross-tenant access settings are autogenerated
Only existing partner tenants	Tenants who already have cross-tenant access settings
All partner tenants	Both new partner tenants and existing partner tenants
No partner tenants	Policy templates are effectively disabled