add whisper.cpp integration for AI-167 mobile STT evaluation by akshhkaushik · Pull Request #90 · openMF/community-ai

akshhkaushik · 2026-03-05T08:09:21Z

What

Add whisper.cpp (C/C++ optimized Whisper) as the foundation for evaluating
multilingual STT support on mobile devices.

Jira Ticket: AI-167 - Evaluate multilingual support for smallest ai

implementation of whisper.cpp

Mobile focus: Runs on iOS, Android, Raspberry Pi
Performance: Lower latency & memory footprint critical for edge devices
Tooling: Built-in whisper-bench for measuring inference speed
Quantization: Native support for smaller model sizes
Edge optimization: Starter foundation for edge inference optimization

What's included

evaluation/whisper.cpp/: Git submodule (C/C++ Whisper implementation)
evaluation/SETUP.md: Step-by-step build & run instructions
evaluation/run_multilingual_eval.py: Minimal wrapper script
evaluation/RESULTS.md: Documentation hub for findings & reproduction

⚙️ Development & Code Accountability

AI Model Used: Claude Opus 4.5 (via GitHub Copilot)

What Claude was used for:

Analyzing whisper.cpp architecture and C/C++ codebase
Refining setup documentation for clarity and completeness
Researching AI-167 requirements and evaluation methodologies
Code review and quality assurance of logic flow

Full Accountability

All code in this PR remains under complete human(my) responsibility.

How to Reproduce

git clone https://github.com/aksh08022006/community-ai.git
cd community-ai/evaluation
git submodule update --init
cd whisper.cpp && cmake -B build && cmake --build build -j && cd ..
python [run_multilingual_eval.py](http://_vscodecontentref_/0)`




[AI-167]: https://mifosforge.jira.com/browse/AI-167?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

…king Add whisper.cpp as git submodule with setup documentation and minimal evaluation runner for AI-167 (multilingual support evaluation). Changes: - evaluation/whisper.cpp: C/C++ Whisper for on-device mobile inference - evaluation/SETUP.md: Build and setup instructions - evaluation/run_multilingual_eval.py: Thin wrapper to run benchmarks This enables evaluation of whisper-tiny, whisper-base, and whisper-small across 5 languages (en, hi, es, fr, de) with focus on mobile performance metrics: WER, CER, latency, model size, memory usage. Addresses: AI-167

Add docstrings and section headers explaining: - Why whisper.cpp was chosen over Python/HuggingFace (mobile focus) - Key design decisions (minimal code, validate state, use existing tools) - Collaborative refinement with Claude Opus 4.5 This makes the contribution more human and transparent about the thought process, not just presenting final code.

…dback Per Pronay Sarker's guidance, add comprehensive results documentation that includes: - Jira ticket reference (AI-167) - Repository link for mentor review and reproduction - Reproduction steps - Results template (WER/CER, latency, model size) - Findings section for analysis - Next steps for follow-up work This provides mentors with a single source of truth for: 1. How to access the work 2. How to reproduce it 3. Where findings will be documented

Copilot

Pull request overview

Adds whisper.cpp as a submodule and introduces initial tooling/docs to run a multilingual STT evaluation (AI-167) using whisper.cpp’s benchmarking utilities.

Changes:

Added whisper.cpp git submodule under evaluation/.
Added a Python runner script intended to orchestrate benchmarking.
Added setup + results documentation for reproducing and recording benchmarks.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
evaluation/whisper.cpp	Adds the whisper.cpp submodule pointer to a specific commit.
.gitmodules	Registers the whisper.cpp submodule path + URL.
evaluation/run_multilingual_eval.py	Introduces a wrapper script to run whisper.cpp benchmarking.
evaluation/SETUP.md	Documents build/run steps for the evaluation.
evaluation/RESULTS.md	Adds a results template and reproduction steps for AI-167.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Added a note about using AI model Claude Opus 4.6 for refinement.

Refine script with insights from Claude Opus 4.5 and related benchmarks.

Modified the benchmarking process to iterate over multiple models, running benchmarks for each model individually.

- Add llms.txt for Claude/AI tool compatibility - Add SKILL.md defining /benchmark-speech and /evaluate-language workflows - Add Deepgram STT provider wrapper with async streaming - Add WER/CER metrics calculation for accuracy measurement - Add language targets (English, Spanish, Hindi, Swahili, French, Portuguese) Total: 426 lines across 5 focused files

staru09 · 2026-03-11T04:21:58Z

please remove AI dev files (skills, llms.txt etc)

DavidH-1 · 2026-03-11T14:45:07Z

CLA check ok

Removed the section for the next PR regarding the evaluation runner.

akshhkaushik · 2026-03-11T17:42:03Z

please remove AI dev files (skills, llms.txt etc)

deleted skills, llms.txt

akshhkaushik · 2026-03-11T17:42:35Z

Please let me know if any another changes are required

staru09 · 2026-03-17T06:38:41Z

Just rename the results file to something like results_whisper
rest looks ok
please do the change and then we can merge this

akshhkaushik · 2026-03-17T15:11:05Z

Just rename the results file to something like results_whisper rest looks ok please do the change and then we can merge this

done

staru09 · 2026-03-27T13:24:20Z

@aksh08022006 is this needed or have you covered this in #99 ?

DavidH-1 · 2026-03-27T21:16:23Z

showing conflicts too

akshhkaushik · 2026-03-30T11:07:02Z

@aksh08022006 is this needed or have you covered this in #99 ?

i have covered them in #99

akshhkaushik · 2026-04-02T18:06:19Z

showing conflicts too

resolved the conflicts.

akshhkaushik added 3 commits March 5, 2026 12:37

akshhkaushik requested review from a team and Copilot March 5, 2026 08:09

Copilot AI reviewed Mar 5, 2026

View reviewed changes

akshhkaushik added 6 commits March 5, 2026 13:41

Refine RESULTS.md with AI model details

bd3ceee

Added a note about using AI model Claude Opus 4.6 for refinement.

Refine run_multilingual_eval.py with new insights

4b457f4

Refine script with insights from Claude Opus 4.5 and related benchmarks.

Iterate over models for individual benchmarking

dac919b

Modified the benchmarking process to iterate over multiple models, running benchmarks for each model individually.

Use sys.executable for Python command

1ddcec0

Merge branch 'openMF:dev' into dev

4a9b3e2

akshhkaushik added 3 commits March 11, 2026 23:09

Delete llms.txt

842eb7a

Delete SKILL.md

e8bdca8

Remove next PR section from foundation document

fb8e9c6

Removed the section for the next PR regarding the evaluation runner.

akshhkaushik added 3 commits March 17, 2026 12:51

Delete benchmarking_experiments/dataset/ai172_languages.py

ddee125

Delete PR_AI172_FOUNDATION.md

ebc52dd

Add results_whisper to evaluation directory

33deecd

Merge branch 'dev' into dev

fb05fa1

Uh oh!

Conversation

akshhkaushik commented Mar 5, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

implementation of whisper.cpp

What's included

⚙️ Development & Code Accountability

What Claude was used for:

Full Accountability

How to Reproduce

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

staru09 commented Mar 11, 2026

Uh oh!

DavidH-1 commented Mar 11, 2026

Uh oh!

akshhkaushik commented Mar 11, 2026

Uh oh!

akshhkaushik commented Mar 11, 2026

Uh oh!

staru09 commented Mar 17, 2026

Uh oh!

akshhkaushik commented Mar 17, 2026

Uh oh!

staru09 commented Mar 27, 2026

Uh oh!

DavidH-1 commented Mar 27, 2026

Uh oh!

akshhkaushik commented Mar 30, 2026

Uh oh!

akshhkaushik commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

akshhkaushik commented Mar 5, 2026 •

edited by atlassian Bot

Loading