Developing a voice-first, multi-domain AI assistant using watsonx Orchestrate and Apple Siri

Large language models rely on large training datasets, but those datasets have limits: they have a knowledge cutoff date and only include publicly available information. IBM watsonx Orchestrate addresses this issue by letting you bring your own private documents and inject real-time enterprise knowledge directly into the LLM.

In a busy workday, you're constantly jumping between tasks, asking questions about long custom documents, checking on your company’s stock trend, reviewing a GitHub repo, sending quick follow-up emails to your colleagues, or doing quick calculations on the fly.

Frequently, we use assistants like Apple’s Siri (or Amazon’s Alexa) to help us accomplish our tasks. However, most AI assistants still get in the way more than they help:

• They struggle to support multiple modes (voice, text, documents, APIs)
• They lose context when switching between different kinds of tasks
• You frequently have to tap, type, or deal with manual prompts
• They can’t follow or execute complex, multi-step workflows

Siri now integrates with IBM watsonx Orchestrate enabling developers to create a smooth, voice-driven assistant that can handle real work across multiple domains instantly and hands-free. Whether you're running between meetings or multitasking at your desk, this integration gives you a conversational way to get answers, automate steps, and stay focused on what actually matters.

In this tutorial, you will create a simple, voice-driven workflow where Siri captures the request, a watsonx Orchestrate agent retrieves the right enterprise knowledge using RAG, and the user gets instant answers or automated actions.

In this video, I demonstrate how a user interacts with Siri, which connects with watsonx Orchestrate to answer questions such as “What was IBM's total revenue in Q3 2025?” or “How many developers in IBM use Project Bob?”

Architecture: A voice-first, multi-domain AI agent ecosystem

This solution combines two powerful technologies:

1. Apple Shortcuts (and Siri) - For seamless iOS integration and voice activation
2. watsonx Orchestrate - For intelligent workflow orchestration

This is the sequence of steps followed in this sample use case:

• The user communicates with their voice with Siri using natural language.
• Siri triggers an Apple Shortcut that retrieves an IAM token for watsonx Orchestrate.
• The token is used to call the watsonx Orchestrate /chat/completions endpoint.
• watsonx Orchestrate performs retrieval, and reasoning, and response.
• The Apple Shortcut speaks the answer back through Siri.

Prerequisites

• IBM watsonx Orchestrate on IBM Cloud with IAM access. Sign up for a free trial using these steps in the watsonx Orchestrate docs.
• An Apple ID signed in with Siri enabled.
• MacBook admin access to modify and interact with Apple’s Siri AI tool.
• Apple Shortcuts using an iPhone, a MacBook, or an iPad.

Steps

To create your voice-first, multi-domain AI agent ecosystem, complete these steps:

1. Create a RAG agent using watsonx Orchestrate.
2. Get the API Key for watsonx Orchestrate.
3. Create an Apple Shortcut to connect Siri to watsonx Orchestrate.
4. Test your voice assistant.

Step 1. Create a RAG agent using watsonx Orchestrate

1. Log in to IBM watsonx Orchestrate. Create a no-code agent (for detailed step-by-step instruction on creating no-code agent, refer to this tutorial) with the following name and description:

   Name: Test RAG pipeline with Siri
   Description: This agent is built to connect Apple Siri with watsonx Orchestrate and your custom knowledge base.

   

2. In the Knowledge section, click Add source. Then click New knowledge, and then click Upload files.

3. Upload your own files. For example, upload the IBM-3Q25-Earnings-Prepared-Remarks report. Then, click Next.

4. Add your knowledge details and description.

   Name: Apple Siri RAG Pipelines
   Description: The knowledge base includes the IBM earning reports, to demonstrate simple Apple Siri and watsonx Orchestrate integration for RAG use cases.

   

5. Try to chat with the agents. For example, "What was IBM's total revenue in Q3 2025?"

   

Step 2. Get the API Key for watsonx Orchestrate

After you tested the agent in the watsonx Orchestrate UI, you can import it into Apple Siri using the watsonx Orchestrate API.

1. To get the watsonx Orchestrate API details, go to Settings and click Generate API key. Then, copy your service instance URL, which contains your tenant’s ID.

   

2. Open your agent in the browser and copy the Agent ID from the end of the service instance URL. Your final URL endpoint format will be like this: https://api.us-south.watson-orchestrate.cloud.ibm.com/instances/<TENANT-ID>/v1/orchestrate/<AGENT-ID>/chat/completions. Replace us-south with your region.

   

   Save this URL. You will need to paste it into the Apple Shortcut later in this tutorial.

Step 3. Create an Apple Shortcut to connect Siri to watsonx Orchestrate

A custom Apple Shortcut connects Siri to the watsonx Orchestrate API.

1. On your Apple device, add the Apple Shortcut “Ask watsonx” using this iCloud link, and then click Get Shortcut, then Open Shortcuts, and finally Add Shortcut.

2. Right-click Ask Watson-X, and then click Edit.

3. Expand the “Get Contents of https://iam.cloud.ibm.com/identity/token” drop-down, and replace the apikey value with the value of your watsonx Orchestrate API key from the previous step.

   

4. Take your service instance URL (which contains the Tenant ID) and append your Agent ID to form the complete chat completions endpoint. For example, in this URL https://api.us-south.watson-orchestrate.cloud.ibm.com/instances/<TENANT-ID>/v1/orchestrate/<AGENT-ID>/chat/completions. Replace and with your own values and replace us-south with your region, and use this final URL in the block after "Get content of" (see the following screen capture):

   

5. In the Settings app for your Apple device, go to Keyboard and enable the Dictation option (by default it's not enabled).

   

The first part of the Ask Watson-X shortcut should look something like this:

The second part should look something like this:

The shortcut performs eight actions in order:

1. Receive Trigger - Starts the shortcut through Siri or Quick Actions.
2. Dictate text – Listens and converts your speech into text.
3. POST Request (IAM Token) – Sends your API key to IBM’s IAM to get an access_token in realtime.
4. Extracts Token – Reads the access_token value from the IBM’s IAM response.
5. POST Request (watsonx Orchestrate Chat API) – Sends the dictated text to the watsonx Orchestrate /chat/completions endpoint using Authorization: Bearer <access_token>.
6. Extract Choices – Gets the choices array from the watsonx Orchestrate response.
7. Extract Message Content – Retrieves the assistant’s text reply.
8. Speak Response – Siri reads the final answer aloud.

Step 4. Test your voice assistant

Here are simple, practical test questions you can ask Siri to validate the agent using your PDFs and RAG agents that has been deployed. If you change or add more documents, you can customize these tests accordingly.

To test, say the following:

1. You: “Hey Siri, ask watsonx”

   Siri responds: “What’s the text?”

2. You: “How many developers in IBM use Project Bob?”

   Siri responds: “More than 8,000”

Extending the integration between Siri and watsonx Orchestrate with tools

You can extend the integration between Siri and watsonx Orchestrate further with these scenarios. The python code for these different tools is available on GitHub, in case you’d like to explore any of these tools.

• Web Search Expert (web_search_expert.py)

  • Purpose: Retrieve real-time information from the web.
  • Engine: DuckDuckGo, chosen for privacy-focused, reliable search results.
  • Capabilities: Current events, research lookups, and fact checking.
  • Typical Use Case: “What are the latest developments in AI technology?”

• Email Expert (email_expert.py)

  • Purpose: Automate email composition and sending.
  • Capabilities: Generates subject lines and message bodies and sends them through SMTP.
  • Integration: Works with Gmail using secure authentication.
  • Typical Use Case: “Send an email to john.doe@company.com confirming tomorrow’s meeting.”

• Financial Expert (financial_expert.py)

  • Purpose: Provide financial data insights and analysis.
  • Capabilities: Stock performance, market trends, and company financial summaries.
  • Data Source: Yahoo Finance with automatic mapping from company names to tickers.
  • Typical Use Case: “How is Tesla performing in the market today?”

• GitHub Expert (github_expert.py)

  • Purpose: Analyze GitHub repositories and extract meaningful insights.
  • Capabilities: Repository summaries, code structure evaluation, commit history analysis, and issue review.
  • Additional Features: Extracts function names across various programming languages.
  • Typical Use Case: “Give me an overview of the main repository.”

• Python Expert (python_expert.py)

  • Purpose: Execute Python code for calculations or quick evaluations.
  • Capabilities: Mathematical operations, custom logic execution, and lightweight data processing.
  • Safety: Runs in a controlled, read-only environment.
  • Typical Use Case: “Calculate compound interest on 10,000 at 5% for 10 years.”

To quickly import all agents, tools, connections, and knowledge bases, use this automated shell script in a terminal.

Summary and next steps

This tutorial links Apple Siri to watsonx Orchestrate to deliver a voice-first RAG assistant that pulls enterprise knowledge from your private documents and returns instant, hands-free answers or automated actions.

It combines Apple Shortcuts for voice capture, watsonx Orchestrate for knowledge routing and workflow orchestration, and a secure API/token flow so Siri can call the watsonx Orchestrate chat endpoint and speak back RAG-powered responses.

To extend this use case, you can integrate the watsonx orchestrate agents with other channels like WhatsApp, Slack and Facebook Messenger. See this GitHub repo for the detailed steps.

Acknowledgments

This tutorial was produced as part of the IBM Open Innovation Community initiative: Agentic AI (AI for Developers and Ecosystem).

The author deeply appreciates the support of Ela Dixit, Ahmed Azraq, and Michelle Corbin for the guidance on reviewing and contributing to this tutorial.