With the increasing pace of work, life, and society as a whole, the demand for technologies that help people save time has grown substantially. There are many apps and services out there that utilize machine learning to record, transcribe, and organize meeting notes. They are often associated with high costs or subscription models. However, the core components of an intelligent note-taking companion with enterprise speech-to-text capabilities are incredibly simple to build with little to no cost.

This tutorial aims to implement the basic core functionalities of an enterprise speech-to-text note-taking app and options to expand it further.

Basic Requirements

There are three main components to a speech-to-text note-taking app:

1. Recording function. The app should be able to record lectures, meetings, interviews, or podcasts and store the audio for processing. This can be done easily through existing libraries along with a functional microphone on the device on which the app will be run.
2. Speech to Text (STT) function. The app will then transcribe and store the recorded audio into an organized transcript. The transcript should contain accurate timestamps to indicate the temporal sequence of when sentences are spoken. Additionally, in the case of an interview, podcast, or any scenario that involves more than one speaker, the STT mechanism should also perform speaker diarization that labels each section of the speech with its respective speaker.
3. Intelligent processing. A meeting minute-like document will be generated containing bite-sized, but crucial information. This can be in the form of summarizations, segmenting the recorded audio into chapters, key points, and any potential action items.

This tutorial will provide a scalable implementation that can be extended and integrated into different platforms and apps that suit each individual’s wants and needs.

Workflow Overview

This is an overview of the app’s workflow using Python:

1. Python’s pyaudio and wave will be used to record audio on demand.
2. Deepgram’s enterprise-ready speech-to-text APIs will be used to transcribe the audio. The transcript will be automatically sectioned into paragraphs with appropriate timestamps and speaker labels if applicable.
3. We will use an LLM to generate a summary, table-of-contents with timestamps, as well as key points discussed in each section.
4. Implement any extra functionalities such as exporting the transcript and summarization document to a markdown file, generating hyperlinks between chapter titles and the actual transcript, etc.

We are going to use Python for this app, but most if not all the libraries and APIs used in this tutorial can be applied similarly in other popular languages such as JavaScript.

As mentioned previously, pyaudio and wave are the libraries responsible for recording the audio. Of course, any audio recording can be used, such as a pre-recorded lecture from your phone or an audio file from an online meeting. The function below will start recording from the device’s microphone when the “enter” key is pressed and stop recording when the key is pressed again. Obviously, the trigger condition can be changed arbitrarily.

import pyaudio  
import wave  
import threading  
import time  
  
  
def record_audio(filename, sample_rate=44100, chunk=1024):  
    """  
    Record audio from the microphone and save to a WAV file.    Recording starts when Enter is pressed and stops when Enter is pressed again.  
    Args:        
        filename: Output filename (should end with .wav)        
        sample_rate: Audio sample rate in Hz        
        chunk: Number of frames per buffer    
    """    
    p = pyaudio.PyAudio()  
  
    # Open stream  
    stream = p.open(format=pyaudio.paInt16,  
                    channels=1,  
                    rate=sample_rate,  
                    input=True,  
                    frames_per_buffer=chunk)  
  
    print("Press Enter to start recording...")  
    input()  # Wait for Enter key  
  
    print("Recording started. Press Enter again to stop...")  
  
    frames = []  
    recording = True  
  
    # Create a thread to listen for the Enter key to stop recording  
    def check_for_stop():  
        nonlocal recording  
        input()  # Wait for Enter key  
        recording = False  
        print("Recording stopped.")  
  
    stop_thread = threading.Thread(target=check_for_stop)  
    stop_thread.daemon = True  
    stop_thread.start()  
  
    # Record audio until Enter is pressed again  
    while recording:  
        data = stream.read(chunk)  
        frames.append(data)  
  
    # Stop and close the stream  
    stream.stop_stream()  
    stream.close()  
    p.terminate()  
  
    # Save the recorded data as a WAV file  
    wf = wave.open(filename, 'wb')  
    wf.setnchannels(channels)  
    wf.setsampwidth(p.get_sample_size(pyaudio.paInt16))  
    wf.setframerate(sample_rate)  
    wf.writeframes(b''.join(frames))  
    wf.close()  
  
    print(f"Audio saved to {filename}")

Due to the nature of the input() function being blocking, meaning that it will halt execution until the user provides input or presses the enter key, we cannot use a while loop and must perform the key press check in another thread. This is done with threading.thread, utilizing the check_for_stop function that will set the recording flag to false when the key press is detected in its own thread.

We then follow up with a while loop that reads the audio in chunks specified by the user until the recording flag is set to false.

The size of the chunk parameter acts as a trade-off between overhead and latency. The integer passed into the chunk parameter determines the number of samples that are processed at one time. With a lower chunk size, the recording function will be more responsive to stop commands but will face more CPU overhead due to the frequent processing. On the other hand, a larger chunk size will reduce overhead but increase the latency in detecting stop commands.

Finally, the wave library is used to save the audio data as a WAV file.

Now with the audio file in hand, we can move on to implementing the speech-to-text (STT) functionality.

Deepgram’s speech recognition API gifts any user signing up with 200 dollars of free credit without credit cards or any strings attached. Users can sign up here.

Every dollar of credit accounts for about four hours of audio processing.

Deepgram’s Python SDK can be installed with pip install deepgram-sdk. The audio processing pipeline is concise and straightforward.

def transcribe_audio(audio_file):  
    # Create a Deepgram client using the API key  
    deepgram = DeepgramClient(api_key=os.environ.get("DEEPGRAM_API_KEY"))  
  
    print("Reading audio file...")  
    with open(audio_file, "rb") as file:  
        buffer_data = file.read()  
  
    payload: FileSource = {  
        "buffer": buffer_data,  
    }  
  
    # Configure Deepgram options for audio analysis  
    options = PrerecordedOptions(  
        model="nova-3",  
        smart_format=True,  
        diarize=True,  
        paragraphs=True,  
    )  
  
    # Call the transcribe_file method with the text payload and options  
    print("Calling Deepgram...")  
    response = deepgram.listen.rest.v("1").transcribe_file(payload, options)  
  
    return response

Remember to store your Deepgram API key in the “DEEPGRAM_API_KEY” environment variable by executing the command export DEEPGRAM_API_KEY="YOUR API KEY".

The PrerecordedOptions class specifies certain helpful processing steps that will be performed along with the audio transcription. In our use case, we will set the smart_format, diarize, and paragraph options to True.

More options can be found in the speech-to-text documentation page under the section “Formatting.”

The response will be returned in JSON format. There is much information nested within dictionaries and lists, but we are only interested in a couple of entries; namely the actual transcript, speaker, and the timestamp each paragraph was spoken at. Their location within the JSON response looks something like this:

{
  "metadata": {
    // metadata...
  },
  "results": {
    "channels": [
      {
        "alternatives": [
          {
            "transcript": "",// the entire transcript in one paragraph
            "words":[ // list of individual words with information such as confidence, etc.
            ],
            "paragraphs": {
              "transcript": "",// the entire transcript in one paragraph,
              "paragraphs": [ // list of paragraphs with speaker information, THIS IS WHAT WE WANT
                {
                  "sentences": [
                    // list of each individual sentence
                  ],
                  "speaker": "", // speaker information
                  "start": "", // start timestamp of the paragraph
                },
              ]
            }
          }
        ]
      }
    ]
  }
}

Knowing the JSON structure, it’s not too difficult to extract the relevant information. We will format the timestamp within brackets like so: [mm:ss] along with the speaker information on the same line. Then the paragraph that follows will be written on an indented newline. Finally, the entire transcript will be written to a .txt file.

def export_transcript(response_data, output_file):  
    """Export transcript with formatting to a text file"""  
    with open(output_file, "w") as f:  
        f.write("=== TRANSCRIPT ===\n\n")  
  
        # Access the transcript data  
        results = response_data.get("results", {})  
        channels = results.get("channels", [])  
  
        if not channels or not channels[0].get("alternatives"):  
            f.write("No transcript data found.\n")  
            return  
  
        # Get the paragraphs data if available  
        paragraphs = channels[0]["alternatives"][0].get("paragraphs", {}).get("paragraphs", [])  
  
        if paragraphs:  
            # Format with paragraphs and speakers  
            for i, para in enumerate(paragraphs):  
                speaker = f"Speaker {para.get('speaker', '?')}"  
                start_time = format_timestamp(para.get("start", 0))  
  
                f.write(f"[{start_time}] {speaker}:\n")  
  
                for sentence in para.get("sentences", []):  
                    f.write(f"  {sentence.get('text', '')}\n")  
  
                f.write("\n")  
        else:  
            # Fallback to flat transcript if no paragraphs  
            transcript = channels[0]["alternatives"][0].get("transcript", "")  
            f.write(transcript)  
  
        f.write("\n=== END OF TRANSCRIPT ===\n")

We use the .get method here in case the response has missing keys or returns an empty transcript; the second parameter specifies what to return in case the specified key was not found.

There are many APIs and thousands of LLM models to choose from. In this tutorial, we will use Google’s Gemini API, which is completely free with generous limitations for an individual. For larger scale use cases, Ollama is a great local option that can be implemented with minimal code changes from Google’s Gemini API.

The LLM needs to complete three tasks: write a one-paragraph summary, segment the transcript into chapters, and identify any action items as well as any potential assignees. There is some work to be done with the prompt, especially the segmentation part of the speech.

We must strike an intricate balance to ensure that the model does not “overdo” by splitting every sentence into its own chapter with the slightest change of topic or “underdo” by including too few chapters for a content-rich transcript. We will also instruct the model to write a few concise and straightforward key points for each chapter, acting as a “TL;DR.”

After experimenting, here’s the prompt I came up with:

prompt = f"""  
You are an expert meeting analyst and transcriptionist with skills in topic segmentation and summarization.  
  
Below is a transcript of a conversation. Your task is to:  
  
1. Write a one-paragraph summary of the entire transcript that captures the main themes and important points.  
  
2. Create meaningful chapters by identifying major topic transitions where the conversation significantly shifts focus, not just minor subtopics. Titles that are specific enough to be informative but broad enough to encompass multiple related points. Avoid creating separate chapters for brief mentions or subtopics that are part of a larger discussion, and note the timestamp [MM:SS] where each major topic begins. For each chapter, produce a few key points that captures the essence of the chapter with the most crucial information. These should be very concise, straightforward, short sentences. At most three key points or less in each chapter.  
  
3. Identify any action items mentioned in the conversation - tasks that someone committed to do or was assigned. It is totally possible that the transcript has no action items or would not be appropriate to have action items.  
  
TRANSCRIPT:  
{transcript}  
"""

Depending on the model, the prompt may need to be adjusted for additional specificity and details.

However, merely prompting the model is not enough; we need the LLM to output its responses in a strict, specific format that can be processed by our code consistently and accurately every time. Luckily, Google’s Gemini API and most other LLM providers allow the model to generate in a “structured output” format that forces its outputs to follow a certain schema, such as JSON or XML.

Initially, direct prompting is used to instruct the LLM to output in a specific format. However, with the stochastic nature of these Transformer models, reliability is inconsistent at best. Today, most LLM providers utilize Finite State Machines (FSM) to ensure consistent structured output.

When generating structured output like JSON, the LLM’s typical token-by-token generation process is constrained by an FSM that tracks the “state” of the output being built. For example, when inside a JSON string, only certain characters are valid next tokens, and when after a property name, a colon must follow.

The FSM works by:

1. Tracking what “state” the generation is in (e.g., “inside object”, “after property name”, “inside array”)
2. For each state, defining which tokens are valid next steps
3. During token selection, applying logit biasing which adjusts the probability of every potential next token to make invalid tokens extremely unlikely or impossible.

This approach is different from regular prompting because it directly constrains the model’s token selection process at the probability distribution level, rather than just hoping the model follows instructions.

We’ll need a Gemini API key, which can be obtained here. We’ll first instantiate a client with our API key.

client = genai.Client(api_key=YOUR_API_KEY)

Then, we will define our desired JSON schema with classes, inheriting from pydantic's BaseModel class. Pydantic is a Python library that will automatically convert the LLM’s structured output to Python datatypes that are specified in the class.

class Chapter(BaseModel):  
    timestamp: str  
    title: str  
    key_points: List[str]  
  
class ActionItem(BaseModel):  
    task: str  
    assignee: Optional[str] = None  
  
class TranscriptSummary(BaseModel):  
    summary: str  
    chapters: List[Chapter]  
    action_items: Optional[List[ActionItem]] = None

We will pass the TranscriptSummary class to the LLM. Within the class, we are nesting the Chapter and ActionItem classes which contain their own properties such as timestamp, title, key points, task, and assignee.

Prompting the model is a simple one-liner:

response = client.models.generate_content(  
    model=model,  
    contents=prompt,  
    config={  
        "response_mime_type": "application/json",  
        "response_schema": TranscriptSummary,  
    },

Altogether, we have the analyze_transcript function. We’re using the gemini-2.5-flash-preview-04-17 model, which is one of the best models in its class with amazing efficiency. The model currently (as of May 14th, 2025) has a rate limit of 500 requests per day, 250,000 tokens per minute, and 10 requests per minute.

def read_transcript(file_path):  
    with open(file_path, 'r') as file:  
        return file.read()  
  
def analyze_transcript(transcript_path, model="gemini-2.5-flash-preview-04-17"):  
    transcript = read_transcript(transcript_path)  
  
    prompt = f"""PROMPT AS SHOWN ABOVE"""  
  
    # Call Gemini  
    client = genai.Client(api_key="AIzaSyDyuzHU80DbkbRW74eU7mRNNOqt6JQ2S1c")  
    response = client.models.generate_content(  
        model=model,  
        contents=prompt,  
        config={  
            "response_mime_type": "application/json",  
            "response_schema": TranscriptSummary,  
        },  
)  
  
    # Return the structured content  
    return response.parsed

For many developers seeking enterprise use cases, local models are a common choice due to the flexibility and security that they offer compared to other providers. Fortunately, the popular local LLM provider Ollama has a very similar syntax that performs structured output compared to Google’s Gemini.

from ollama import generate

def analyze_transcript(transcript_path, model):

    transcript = read_transcript(transcript_path)  
    
    prompt = f"""PROMPT AS SHOWN ABOVE""" 
    
    response = generate(
      prompt=prompt,
      model=model,
      format=TranscriptSummary.model_json_schema(),
    )

    return response.parsed

Putting it All Together

With each individual piece of the app done, we’ll put everything together into a basic command-line app.

We’ll first define a couple of command-line arguments:

• “file”: The file path to a pre-recorded audio file if available
• “dir”: The output directory for the generated transcript and summary files
• “model”: The LLM model to use for generating the chapters, summary, action items, and key points

def parse_args():  
    parser = argparse.ArgumentParser()  
    parser.add_argument("-f", "--file", type=str, help="File path to pre recorded audio file")  
    parser.add_argument("-d", "--dir", type=str, help="Output directory for transcript and summary files")  
    parser.add_argument("-m", "--model", type=str, help="LLM model to use for generating summaries")  
    args = parser.parse_args()  
    return args

The recording logic is up next. We’ll check if the user has provided a pre-recorded audio file. If so, we will use it; if not, we will run the record_audio function. We can also optionally copy the pre-recorded audio file to the same output directory for consistency.

args = parse_args()  
# Create output directory  
os.makedirs(args.dir, exist_ok=True)  
  
if args.file:  
    audio_file = args.file  
    # Copy the input file to the output directory  
    destination_file = os.path.join(args.dir, os.path.basename(audio_file))  
    if os.path.abspath(audio_file) != os.path.abspath(destination_file):  
        print(f"Copying audio file to output directory: {destination_file}")  
        import shutil  
        shutil.copy2(audio_file, destination_file)
        # record path to the audio file for transcription later  
        audio_file = destination_file  
else:  
    record_audio(filename=f"{args.dir}/audio_to_transcribe.wav", sample_rate=44100, channels=1, chunk=1024)  
    # record path to the audio file for transcription later  
    audio_file = f"{args.dir}/audio_to_transcribe.wav"

Then, we can just run the functions defined earlier in order to transcribe the audio, write it to a transcript file with proper formatting, and feed it into an LLM for the intelligent post processing.

output_file = f"{args.dir}/transcript.txt"  
  
response = transcribe_audio(audio_file=audio_file)  
print(f"Generating transcript file: {output_file}")  
export_transcript(response.to_dict(), output_file)  
print(f"Transcript exported successfully to {output_file}")  
print("Analyzing transcript...")  
transcript_notes = analyze_transcript(output_file, args.model)

The recording function will go first with an option for the user to provide a recorded audio file through a command-line argument.

Finally, we can organize the LLM’s output into a nicely formatted markdown file and write it to the same output directory as the transcript and the recorded audio.

def write_transcript_notes_to_md(notes, filename="transcript_notes.md"):  
    with open(filename, 'w') as file:  
        # Title and summary  
        file.write("# MEETING NOTES\n\n")  
  
        file.write("## Summary\n")  
        file.write(f"{notes.summary}\n\n")  
  
        # Chapters with timestamps and key points  
        file.write("## Chapters\n\n")  
        for chapter in notes.chapters:  
            file.write(f"### [{chapter.timestamp}] {chapter.title}\n\n")  
            for key_point in chapter.key_points:  
                file.write(f"- {key_point}\n")  
            file.write("\n")  
  
        # Action items with assignees if available  
        file.write("## Action Items\n\n")  
        if notes.action_items is not None:  
            for item in notes.action_items:  
                assignee = f" - {item.assignee}" if item.get('assignee') else ""  
                file.write(f"- {item.task}{assignee}\n")

# previous code remain the same
# ...
transcript_notes = analyze_transcript(output_file, args.model)  
write_transcript_notes_to_md(transcript_notes, filename=f”{args.dir}/transcript_notes.md”)

And that’s it! A fully functional intelligent note-taking app with its core functionality implemented.

Below is a sample result from the famous TED talk about procrastination. Since this is a TED talk, there are no action items.

For individuals, the app costs little to nothing, offers customizability, arguably containing much higher value than any subscription-based services that essentially accomplish the same thing with a nicer UI. For enterprise-scale applications, the implementation demonstrates the core functionality of the app with minimal lines of code, and it can be easily scaled or packaged for further expansion.

Conclusion and Extensions

There are many directions that can be taken with the app. A UI wrapper such as streamlit can be integrated smoothly, and the Deepgram API offers options in other languages with very similar usage syntax as Python.

We can also package the implementation into popular note-taking apps such as Obsidian and Notion.

Furthermore, internal hyperlinks in Obsidian can link to specific paragraphs identified by a caret symbol followed by a unique string of numbers and letters (i.e., ^abc123). We can utilize this to our advantage by appending a unique identifier after every paragraph in our transcript that corresponds to that paragraph’s timestamp.

Then, within our structured output format, we can add an additional field to the Chapter that stores these identifiers. In the prompt of the LLM, we’ll tell the model to populate the field with the identifier that corresponds to the timestamp that it selected for the start of a new chapter. This way, we can navigate to the exact spot on the transcript when we click on the chapter timestamp generated by our LLM.

There are endless possibilities for potential extensions; this tutorial is just the start.