How to Build a Virtual Medical Scribe in Python Using Deepgram and OpenAI

• Clinical note-taking is a critical part of every doctor-patient interaction.
• However, it divides the doctor's attention between documenting and actually caring for the patient.
• This challenge can be solved by using artificial intelligence (AI).
• We can accurately transcribe doctor-patient conversations using a specialized Speech-to-Text (STT) model like Nova-3 Medical.
• Then, powerful large language models like OpenAI’s GPT-4.1 can transform those transcripts into fully structured clinical notes.
• This AI-driven workflow streamlines documentation and allows doctors to focus entirely on their patients.
• Want to skip the tutorial and head straight into the code? Check out this Github repository.

To ensure accurate and quality healthcare, doctors often take a medical scribe along to document their patient encounters.

In this article, we’re going to build an AI-powered medical scribe that doctors could use to make their care more efficient. Along the way, you’ll learn:

• How to use Deepgram’s Nova-3 Medical model for medical transcription
• How to build a scribe system using prerecorded audio
• How to work with real-time streaming audio
• How to generate clinical notes using OpenAI’s language models

Let’s dive in. 🚀

(Or, if you want to skip the tutorial and head straight into the code? Check out this Github repository.)

Much of software development is about replicating existing real-world processes. So, before we can build a Medical AI Scribe, we first need to understand the role of a human medical scribe.

Medical Scribes Improve Doctor-Patient Interaction

A medical scribe assists a doctor by taking notes during patient encounters. While the doctor speaks with the patient, the scribe documents important details, typically entering them into an Electronic Health Record (EHR) system using standardized formats, such as SOAP.

So, how do we replicate this with software?

First, we need a way to listen to and transcribe the conversation between the doctor and patient. That's where a Speech-to-Text (STT) model comes into play. Then, to structure the transcription into proper clinical documentation for an EHR, we need a large language model (LLM).

Since transcription is our first step, we need an STT model that can handle the complexity of medical conversations. We might consider using any standard STT model, such as Deepgram Nova-3, which is known for its speed and accuracy. 

However, general-purpose STT models often struggle with medical terminology, which can lead to misinterpretations or transcription errors.

That is why we turn to specialized medical transcription models. These models are trained specifically on medical conversations and vocabulary, making them much more reliable in clinical settings.

If regular Nova-3 isn't cutting it, you can try Nova-3 Medical. It builds on the strengths of the original Nova-3, like low latency and high speed, while offering even greater accuracy for medical transcription.

In this article, we will use Nova-3 Medical to power our AI scribe.

Once we have a clean transcription, the next step is turning it into structured clinical notes. This is where the LLM comes in. The good news is that most modern LLMs are flexible enough to handle this task with the right prompting.

In this article, we will use OpenAI’s models to generate clinical notes from transcribed text, producing outputs that are structured, readable, and ready for EHR documentation.

When it comes to capturing the conversation between the doctor and the patient, we have two main options: prerecorded audio or real-time streaming.

• The first is the prerecorded approach, where the doctor can simply record the conversation during the patient visit. After the session, the recorded audio is uploaded to the Medical AI Scribe, which handles transcription and generates the clinical notes.
• The second option is real-time streaming. In this setup, the transcription model streams the audio live as the doctor speaks with the patient. This technique allows the conversation to be transcribed in real time. By the end of the session, the transcription and the clinical documentation are already complete and ready for the EHR.

Deepgram’s Nova-3 Medical model supports both prerecorded and real-time streaming audio. 

This article will explore both methods and show you how to implement each.

(🧑‍💻 Find the complete code in this repository.)

Now that we’ve covered the fundamentals, let’s start building our AI scribe. We’ll first focus on creating the prerecorded AI Scribe and later move on to the real-time AI Scribe.

To build the prerecorded AI Scribe, we’ll need the following dependencies:

• deepgram-sdk: This provides access to the DeepGram API, allowing us to use the Nova-3-Medical model for transcription.
• openai: This library provides access to OpenAI’s models, which we can use to generate clinical notes.
• python-dotenv: This helps manage our environment variables securely.

First, install the necessary dependencies:

pip install deepgram-sdk python-dotenv openai

Next, grab your API keys from Deepgram and OpenAI and %%writefile (IPython/Jupyter only) them in a .env file:

DEEPGRAM_API_KEY= [Your Deepgram API Key goes here]
OPENAI_API_KEY= [Your OpenAI API Key goes here]

Now, let's jump into the code:

from openai import OpenAI
import sys
from deepgram import (
    DeepgramClient,
    PrerecordedOptions,
    FileSource,
)
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Initialize OpenAI client
client = OpenAI()

# SOAP instruction for the developer role
SOAP_INSTRUCTION = """
.....
"""

def transcribe_audio(audio_file):
    pass

def generate_note_and_save(transcript, output_file):
    pass

def main():
    if len(sys.argv) < 2:
        print("Usage: python main.py <audio_file_path>")
        sys.exit(1)

    audio_file = sys.argv[1]
    output_file = "generated_soap_note.txt"

    print("Starting transcription...")
    transcript = transcribe_audio(audio_file)
    print("Transcription completed.")

    print("Generating SOAP note...")
    generate_note_and_save(transcript, output_file)
    print("SOAP note generation completed.")

if __name__ == "__main__":
    main()

Explanation of the code:

We’ve made the necessary imports from DeepGram and OpenAI. Two key functions have yet to be implemented: transcribe_audio and generate_note_and_save. The transcribe_audio function will contain the transcription logic, while generate_note_and_save will handle the generation of clinical notes using OpenAI.

(🧑‍💻 Find the complete code in this repository.)

Let’s implement the transcribe_audio function. This function will interact with the DeepGram API via the DeepgramClient to transcribe the audio file. 

The PrerecordedOptions will allow us to configure transcription settings, such as which model to use and whether to diarize speakers.

Here’s the implementation of transcribe_audio:

def transcribe_audio(audio_file):
    # STEP 1: Create a Deepgram client using the API key
    deepgram = DeepgramClient()

    with open(audio_file, "rb") as file:
        buffer_data = file.read()

    payload: FileSource = {
        "buffer": buffer_data,
    }

    # STEP 2: Configure Deepgram options for audio analysis
    options = PrerecordedOptions(
        model="nova-3-medical",  # Specify the medical transcription model
        smart_format=True,       # Automatically add punctuation
        diarize=True             # Identify speakers in the conversation
    )

    # STEP 3: Call the transcribe_file method with the text payload and options
    response = deepgram.listen.rest.v("1").transcribe_file(payload, options)

    # STEP 4: Extract and return the transcript
    data = response["results"]["channels"][0]["alternatives"][0]["paragraphs"]["transcript"]

    return data

Key details about the transcription logic:

1. Deepgram Client: We initialize the Deepgram client using the API key to interact with the DeepGram service.
2. Audio File: The audio file is read in binary mode and passed as a payload for transcription.
3. Options: We specify the use of the Nova-3-Medical model, enable smart formatting to add punctuation, and diarize to identify different speakers (e.g., patient and clinician).
4. Transcription: The transcribe_file method sends the audio data to DeepGram, and the resulting transcription is extracted and returned.

Next, let’s implement the generate_note_and_save function. This function takes the transcript generated in the previous step and uses OpenAI's ResponseAPI to generate a clinical SOAP note.

Here’s how we implement it:

def generate_note_and_save(transcript, output_file):
    # Call the OpenAI API to generate the SOAP note
    response = client.responses.create(
        model="gpt-4.1",
        input=[
            {
                "role": "developer",
                "content": SOAP_INSTRUCTION
            },
            {
                "role": "user",
                "content": transcript
            }
        ]
    )

    # Save the generated SOAP note to a file
    with open(output_file, "w") as file:
        file.write(response.output_text)

    print(f"SOAP note saved to {output_file}")

SOAP instruction for clinical note generation:

The SOAP_INSTRUCTION string provides a template that the model follows to generate the clinical note. It specifies how to structure the note into Subjective, Objective, Assessment, and Plan sections.

SOAP_INSTRUCTION = """
You are a clinical documentation assistant. Generate a SOAP note based on the following conversation between a clinician and a patient. Follow this strict structure:

- Subjective:
  - Chief Complaint (CC): [One sentence summarizing why the patient is seeking care.]
  - History of Present Illness (HPI): [A detailed paragraph summarizing the progression and current state of the patient's illness.]
  - History: [Past medical history, surgical history, medications, allergies, family history, social history — as much as available.]
  - Review of Systems (ROS): [Organized by system (General, Cardiovascular, Respiratory, GI, Musculoskeletal, Neurological, Psychiatric). If not mentioned, state "Not reported."]

- Objective:
  - General appearance.
  - Vital signs (if available; if not, state "Not available via video.")
  - Physical examination findings (if any).
  - Relevant lab or diagnostic information (if available).

- Assessment:
  - [List the major active medical problems and a brief status assessment.]

- Plan:
  - [Outline management plan including treatments, follow-ups, further discussions needed, etc.]

Tone: Formal, clinical, objective.
Format: Use bullet points inside each section when listing items; otherwise, use clear paragraphs.
Note: Fill in missing information with "Not specified" or "Not available" rather than making assumptions.
"""

To see the final implementation, which integrates both transcription and clinical note generation, check out the main.py file in this Github repository.

Once the script is ready, you can run it by passing the prerecorded audio as a command argument:

python main.py <path_to_audio_file>

This should save the generated clinical notes in a file called generated_soap_note.txt.

To evaluate our pre-recorded scribe, we'll use a real-world video from YouTube:

We'll extract the audio from this video using pytubefix:

from pytubefix import YouTube
from pytubefix.cli import on_progress

url = "https://www.youtube.com/watch?v=SerstX6D_CU"


yt = YouTube(url, on_progress_callback=on_progress)
print(yt.title)

ys = yt.streams.get_audio_only()
ys.download()

Once the audio is downloaded, rename the file to sample_audio.m4a for easier reference.

Now, we can run our pre-recorded AI scribe to generate the clinical note:

python main.py sample_audio.m4a

This will produce a detailed SOAP note. Just as we prompted, the system took the full conversation transcript and used the OpenAI model to generate a detailed SOAP note, ready to be placed into an EHR.

Now that we’ve built our prerecorded AI scribe, let’s take things a step further by creating a Real-Time AI Scribe.

(🧑‍💻 Find the complete code in this repository.)

The real-time version will listen through your microphone, transcribe the conversation in real-time, and generate a clinical note when the session ends.

Unlike before, we won’t be using the Deepgram SDK. Instead, we'll connect directly to Deepgram’s live audio WebSocket endpoint, which accepts streaming audio data and returns real-time transcription results.

We'll use the websockets library to interact with the WebSocket endpoint and PyAudio to capture audio from the user's microphone.

pip install pyaudio websockets

Note: On some systems, installing pyaudio might require extra setup. If you run into issues, you may need to install portaudio separately.

Once installed, still make sure you have your Deepgram and OpenAI API keys saved inside your .env file.

Here’s the starting code layout we have for the real-time scribe:

import pyaudio
import asyncio
import json
import os
import sys
import websockets
from datetime import datetime
from openai import OpenAI
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Some global variables and settings
audio_queue = asyncio.Queue()
all_mic_data = []
all_transcripts = []

FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 16000
CHUNK = 8000

# SOAP developer prompt
SOAP_INSTRUCTION = """
...
"""

def mic_callback(input_data, frame_count, time_info, status_flag):
    pass

def get_speaker_transcripts(json_data):
    pass

async def sender(ws, audio_queue):
    pass

async def receiver(ws):
    pass

async def microphone(audio_queue):
    pass

async def run(key, **kwargs):
    pass

def generate_note_and_save(all_transcripts, output_file):
    pass

def main():
    key = os.getenv("DEEPGRAM_API_KEY")
    if not key:
        print("Please set the DEEPGRAM_API_KEY environment variable.")
        sys.exit(1)
    format = "text"
    model = 'nova-3-medical'

    asyncio.run(run(key, model=model))


if __name__ == "__main__":
    main()

Like before, we have a few functions that are not implemented or partially implemented, which we will need to complete step by step. 

Let’s walk through them.

First, let’s implement mic_callback and microphone. These two functions are responsible for:

• Capturing audio in real-time from your microphone
• Feeding that audio into an async queue for streaming

def mic_callback(input_data, frame_count, time_info, status_flag):
    audio_queue.put_nowait(input_data)
    return (input_data, pyaudio.paContinue)

async def microphone(audio_queue):
    audio = pyaudio.PyAudio()
    stream = audio.open(
        format=FORMAT,
        channels=CHANNELS,
        rate=RATE,
        input=True,
        frames_per_buffer=CHUNK,
        stream_callback=mic_callback,
    )

    stream.start_stream()

    print("🎙️  Microphone started. Press Ctrl+C once to stop.")

    try:
        while stream.is_active():
            await asyncio.sleep(0.1)
    except asyncio.CancelledError:
        pass
    finally:
        stream.stop_stream()
        stream.close()
        audio.terminate()

When the microphone is active, the script captures audio chunks and streams them into the queue asynchronously.

Now let’s move on to sender and receiver.

• sender sends captured audio chunks to Deepgram over WebSocket.
• receiver listens for transcription responses and stores them.

async def sender(ws, audio_queue):
    print("🟢 Ready to stream audio to Deepgram")
    try:
        while True:
            mic_data = await audio_queue.get()
            all_mic_data.append(mic_data)
            await ws.send(mic_data)
    except asyncio.CancelledError:
        return
    except websockets.exceptions.ConnectionClosedOK:
        await ws.send(json.dumps({"type": "CloseStream"}))
        print("🟢 Successfully closed Deepgram connection")
    except Exception as e:
        print(f"Error while sending: {str(e)}")
        raise

async def receiver(ws):
    first_message = True
    transcript = ""

    async for msg in ws:
        res = json.loads(msg)
        if first_message:
            print("🟢 Successfully receiving Deepgram messages")
            first_message = False
        try:
            if res.get("is_final"):
                transcript = get_speaker_transcripts(res)
                if transcript:
                    print(transcript)
                    all_transcripts.append(transcript)
        except KeyError:
            print(f"🔴 ERROR: Received unexpected API response! {msg}")

Deepgram provides speaker diarization, meaning it can label who said what. We need to parse that properly:

def get_speaker_transcripts(json_data):
    speaker_transcripts = {}
    channel = json_data.get("channel", {})
    alternatives = channel.get("alternatives", [])

    for alternative in alternatives:
        for word_info in alternative.get("words", []):
            speaker_id = word_info.get("speaker", "Unknown")
            punctuated_word = word_info.get("punctuated_word", word_info.get("word", ""))
            if speaker_id not in speaker_transcripts:
                speaker_transcripts[speaker_id] = []
            speaker_transcripts[speaker_id].append(punctuated_word)

    formatted_transcripts = []
    for speaker_id, words in speaker_transcripts.items():
        formatted_transcripts.append(f"Speaker {speaker_id}: {', '.join(words)}")

    return "\n".join(formatted_transcripts)

This function organizes words by speaker and punctuates them nicely.

Next, we need to open a WebSocket connection to Deepgram by providing our Deepgram API key as the authorization token:

async def run(key, **kwargs):
    deepgram_url = f'wss://api.deepgram.com/v1/listen?punctuate=true&diarize=true'
    deepgram_url += f"&model={kwargs['model']}"
    deepgram_url += "&encoding=linear16&sample_rate=16000"

    async with websockets.connect(
        deepgram_url, extra_headers={"Authorization": "Token {}".format(key)}
    ) as ws:
        print("🟢 Successfully opened Deepgram streaming connection")

        tasks = [
            asyncio.ensure_future(sender(ws, audio_queue)),
            asyncio.ensure_future(receiver(ws)),
            asyncio.ensure_future(microphone(audio_queue)),
        ]

        try:
            await asyncio.gather(*tasks)
        except asyncio.CancelledError:
            generate_note_and_save(all_transcripts, "generated_soap_note.txt")

If you stop the microphone (Ctrl+C), it automatically generates a SOAP note based on everything it has transcribed so far.

Finally, once we have all the transcripts, we use OpenAI to generate the clinical note:

def generate_note_and_save(all_transcripts, output_file):
    combined_transcript = "\n".join(all_transcripts)

    client = OpenAI()

    response = client.responses.create(
        model="gpt-4.1",
        input=[
            {"role": "developer", "content": SOAP_INSTRUCTION},
            {"role": "user", "content": combined_transcript},
        ],
    )

    with open(output_file, "w") as file:
        file.write(response.output_text)

    print(f"📝 SOAP note saved to {output_file}")

The code is the same as the code from the pre-recorded example.

With that, we have a scribe who can transcribe clinical encounters in real-time. To give it a shot, save it in a file and run it like you would run any other Python program.

Find the full code in the main.py file in this Github repository.

Next, let’s generate a SOAP note for a new video, but this time we'll use the real-time AI scribe instead of the pre-recorded one.

To run the real-time demo, first start the real-time scribe script by running python main.py

Then play the video. The script will listen to your microphone, capture the audio in real-time, and send it to Deepgram for live transcription. 

After the video finishes, press Ctrl+C once. The script will then stop streaming audio, compile the transcript, and generate the clinical SOAP note.

And there we have it! We’ve tested both AI scribe scripts and proven they can be applied in real clinical situations. 

(🧑‍💻 Find the complete code in this repository.)

Using Virtual Medical AI Scribe, for the first time, physicians can give their undivided attention to patients, knowing that every critical detail of the encounter is captured with precision and care. 

The AI scribes we built showcase the powerful synergy between medical speech-to-text (STT) models like Nova-3 Medical and the reasoning abilities of sophisticated LLMs.

They change raw conversations into structured, clinical-grade documentation reliably and effortlessly.

The improvement is more than just an efficiency boost. It is a glimpse into a future where technology becomes an invisible partner in medicine, helping doctors to heal, connect, and care at an even higher level.

Here are several helpful references and resources for learning more about building with Deepgram:

• Deepgram API Playground.
• Deepgram Docs: Speech-to-Text Getting Started.
• Deepgram Community and Discussion Forums.
• GitHub Community.