#accessibility
#ai
Australia Flag

Australia

Engineering Kanvo: Building an ML-Powered Sign Language Recognition App

Kanvo sign language recognition app shown on a MacBook and iPads
Quotation mark
They know a startup's journey, put themselves in the founder's shoes, and are honest with what can and can't be done.
Gabriel Gichuhi
Founder & CEO ofKanvo logo
Clutch

From Vision to Impact

Built to Launch. Engineered to Grow.

What Was Done

We partnered with Kanvo to turn an ambitious accessibility concept into a working iPadOS product designed to help people with communication disabilities translate gestures into text.

Our team designed and engineered the product experience around machine-learning-powered gesture recognition, including camera-based gesture capture, recognition results, text-to-speech, user accounts, settings, and supporting communication flows. The core recognition module was successfully integrated into the application.

The engagement went beyond application development. We conducted technical R&D into sign-language recognition, developed an accessibility-focused UI/UX direction, established backend and AWS infrastructure, and designed and developed Kanvo’s landing page to support product validation and fundraising.

Delivered Result

Kanvo progressed from an early concept to a functional Alpha product with machine-learning gesture recognition running inside the iPadOS application and a complete core user flow ready for demonstration to investors and early adopters.

The recognition model was optimized during development, including additional model training to improve how the camera distinguishes hands across different backgrounds, while the product experience was refined around the needs of its accessibility-focused audience.

The foundation was designed for further evolution from character-level recognition toward word and gesture recognition and, ultimately, personalized on-device model training as the product progressed through its planned Beta and MVP stages.

Dataset images in the ASL alphabet dataset used as the foundation for Alpha ML development.

Core digital experiences delivered across the native iPadOS product and web presence.

Recognition evolution stages defined across Alpha, Beta, and MVP.

Authentication methods designed for account access.

Accessibility standard considered during UX/UI research and design.

Clutch rating reflecting client satisfaction with the collaboration.

Project details

Category

  • Accessibility
  • AI / Machine Learning
  • Australia
Hashtag icon

Clutch Review

“They know a startup’s journey, put themselves in the founder’s shoes, and are honest with what can and can’t be done.”
Gabriel Gichuhi Founder & CEO, Kanvo
Clutch

Project Timeline

Product Development:2023–2024
Laptop with summary icon

Technology Used

  • iPadOS: Native
  • Swift / SwiftUI
  • Apple Vision / AVFoundation
  • Core ML
  • Create ML
  • Backend API
  • AWS
  • Firebase
  • New Relic
Lamp with code icon

9 Team Members

  • Business Analyst / PM
  • Project Manager
  • UI/UX & Design
  • iOS Development
  • Backend Engineering
  • Web Development
  • Technical Leadership
Hand with rocket icon

About the Project

Kanvo is an accessibility-focused communication platform designed to help people with communication disabilities translate gestures into text using machine learning. The product was conceived around a native iPadOS experience, combining camera-based gesture capture, ML recognition, text-to-speech, and personalized communication flows within one accessible interface.

Kanvo website on a MacBook and the Kanvo app on an iPad

The Product

Kanvo is an ML-powered communication application that captures users’ gestures through the device camera and translates recognized signs into text, helping reduce communication barriers in everyday interactions.

Project Goal

To validate whether sign-language recognition could become a viable digital product and transform the initial concept into a functional Alpha application ready to demonstrate to early adopters and potential investors.

Responsibilities

Our team covered product discovery, technical R&D, UX/UI design, and development, including ML recognition research and integration, native iPadOS development, backend infrastructure, accessibility-focused design, authentication, and the supporting web landing experience.

Role in Project

We became Kanvo’s product development and technical partner, helping move the idea from early concepts and technical uncertainty to a working Alpha with gesture recognition integrated directly into the iPadOS application.

Workflow Plan

From Technical Validation to Working Alpha

Product & UX

  • Product Discovery
  • Requirements Analysis
  • Accessibility Research
  • User Flows
  • UX/UI Design
  • Inclusive Design
  • Alpha Scope Definition

ML & Engineering

  • Sign Language Recognition R&D
  • ML Dataset Research
  • Vision & AVFoundation
  • Core ML & Create ML
  • Native iPadOS Development
  • Backend Development
  • AWS Infrastructure

Validation & Delivery

  • ML Model Training
  • Gesture Recognition Integration
  • Recognition Optimization
  • TestFlight Builds
  • Product Iteration
  • Landing Page Development
  • Alpha Delivery
Engineering Kanvo

Turning Sign Language Recognition Into a Working Product

Kanvo was engineered as an accessibility-focused communication product that uses machine learning to recognize gestures captured through an iPad camera and translate them into text.

The challenge went beyond building a conventional mobile application. The core product depended on proving that sign language recognition could work within a practical user experience, requiring technical research, ML experimentation, camera processing, model training, and accessible UX design before the concept could become a functional Alpha product.

Engineering Challenge

Kanvo’s core functionality could not rely on an existing plug-and-play translation service. Our technical research found no open-source solution, paid API, or ready-made SDK capable of fully covering the product requirements.

  • Real-time camera-based gesture capture
  • Sign language recognition
  • ML model selection & training
  • Recognition across different backgrounds
  • Gesture-to-text communication
  • Accessible interaction design
  • Native iPadOS experience
  • Foundation for future recognition expansion

The key challenge was to transform an experimental ML capability into a communication flow simple enough to become part of an accessible everyday product.

Architecture & Engineering Solution

We developed a native Apple-based recognition approach combining camera processing with machine learning, while structuring the product so recognition capabilities could evolve through successive development stages.

  • Native iPadOS application
  • Vision / AVFoundation camera processing
  • Core ML recognition
  • Create ML model training
  • Image-based ASL dataset
  • Backend API & AWS infrastructure
  • Gesture recognition module
  • Text-to-speech flow
  • Authentication & user accounts
  • Recognition model optimization

This foundation resulted in gesture recognition running inside the Kanvo application and established a technical path toward word-level recognition and personalized on-device model training.

Platform Capabilities

Kanvo combines accessible communication flows, machine-learning gesture recognition, personalized gesture management, and supporting platform functionality within a native iPadOS experience designed around reducing communication barriers.

1/15

Gesture-Based Communication

Kanvo enables users to communicate by capturing gestures through the device camera and converting recognized signs into text.

Communication Capabilities
Camera-based gesture capture
Gesture-to-text recognition
Recognition results
Editable recognized text
Retake gesture flow
Recognition confirmation

Text-to-Speech

Recognized communication can be converted into audio, helping users communicate with people who do not understand sign language.

Speech Capabilities
Text-to-speech output
Audio playback
Confirm recognized message
Repeat communication flow
Editable text before playback

Accessible User Experience

The interface was designed around users with communication disabilities, making clarity and accessibility fundamental product requirements.

Accessibility Capabilities
Accessibility-first UX
High-contrast interface
Accessible typography
Simplified navigation
Large interaction areas
WCAG-informed design

Gesture Recognition

The core ML module processes camera input and identifies supported signs, turning gesture recognition into part of the communication experience.

Recognition Capabilities
ML-powered recognition
Camera-based input
Hand detection
Character recognition
Recognition result processing
Different-background optimization

ML Engineering

The recognition approach was developed after technical research found no ready-made API, SDK, or open-source solution capable of covering Kanvo’s requirements.

ML Capabilities
Core ML
Create ML
Vision / AVFoundation
Model training
Dataset research
Model optimization

Recognition Evolution

The ML architecture was researched around progressively expanding recognition capabilities rather than treating the initial model as the final solution.

Evolution Roadmap
Image-based recognition
Alphabet recognition
Video-based gesture research
Word recognition research
On-device training concept
Personalized recognition concept

Gesture Library

The product experience was designed to let users build and organize a personal library of gestures for recurring communication needs.

Library Capabilities
Personal gesture library
Gesture categories
Prepopulated categories
Add new phrase
Manual phrase entry
Gesture-based phrase creation

Unrecognized Gestures

When the system cannot recognize a gesture, the flow was designed to help users turn that interaction into reusable personal content.

Learning Flow
Unrecognized gesture state
Select from Library
Add new gesture
Category selection
Re-capture gesture
Confirm result

Personalized Recognition - MVP Vision

The product roadmap envisioned allowing users to expand the base recognition model with their own gestures directly on the device.

Planned Capabilities
User-specific gestures
On-device model expansion
Personal vocabulary
Recognition improvement
User-trained content

User Accounts

Kanvo supports account functionality designed to preserve users’ settings and provide a persistent product experience.

Account Capabilities
Email authentication
Apple Sign-In
Google Sign-In
Facebook Sign-In
Account information
Change email & password
Profile deletion

Backend & Infrastructure

The application is supported by backend and cloud infrastructure connecting accounts, application data, media, and operational services.

Platform Capabilities
Backend API
AWS infrastructure
Cloud storage
CDN delivery
Firebase
Application monitoring
Admin environment

Web & Product Presence

Alongside the iPadOS product, we designed and developed Kanvo’s web presence to explain the concept and support its next stage of growth.

Web Capabilities
Responsive landing page
Product presentation
Beta testing section
Accessibility-focused UI
Product CTAs
Product validation support

Web Development

Develop a fast, scalable web product with clean frontend architecture, reliable integrations, and launch-ready performance.

Web development illustrationExplore Service

iOS App Development

Build a native iOS app that feels fast, polished, and ready for growth. From product logic to App Store release, we help turn ideas into scalable mobile products.

iOS app development illustrationExplore Service

UI/UX Design

Our UI/UX design solutions are tailored to bring your brand vision to life, ensuring a smooth, engaging experience for your users. Explore our services to see how we can enhance your brand’s digital presence.

UI/UX design illustrationExplore Service

From Sign Language to Digital Communication

Building Kanvo meant turning a complex form of human communication into an accessible digital experience. The challenge was not only to recognize gestures with machine learning, but to make the entire interaction simple enough for people to use in real communication scenarios.

Technical Feasibility

Validate whether gesture recognition could support a practical communication experience without relying on a ready-made translation solution.

Accessible Interaction

Make gesture capture, recognition results, navigation, and communication flows clear and approachable for users with different accessibility needs.

From Recognition to Communication

Turn ML output into something immediately useful through readable text, editing, confirmation, and text-to-speech.

Recognition Recovery

Give users a clear way forward when recognition fails through retry, correction, and personal Library flows.

Turning ML Research Into Product

The Alpha was structured around proving the core recognition concept first, then integrating it into an accessible product experience that could become the foundation for more advanced recognition capabilities.

STEP I: Validate the Recognition Approach

We researched available recognition technologies, selected Apple’s Vision, AVFoundation, Core ML, and Create ML stack, and established an image-based approach for the first version of sign-language recognition.

STEP II: Build the Communication Experience

We designed and developed the native iPadOS flow around gesture capture and recognition, connecting ML output with editable text, text-to-speech, accounts, settings, and supporting communication interactions.

STEP III: Integrate & Optimize the Alpha

The recognition module was integrated directly into the application and iteratively improved through additional model training, including better hand distinction across different backgrounds. This resulted in a working Alpha suitable for product demonstration and further validation.

Understanding the Core Users

We designed Kanvo around communication between people who use sign language and people who may not understand it, with the product acting as a bridge between gestures, text, and spoken communication.

Elena, the sign language user personaElena, the sign language user personaElena

“I want to communicate naturally using gestures and have my message understood without depending on another person to translate.”

The Sign Language User

Uses sign language as a primary or preferred form of communication. They need a simple way to capture gestures, review the recognized message, correct it when necessary, and communicate the result to another person.

Marcus, the communication partner personaMarcus, the communication partner personaMarcus

“I want to understand what the person is communicating without needing to know sign language myself.”

The Communication Partner

Does not necessarily understand sign language but needs to communicate with someone who uses it. They need the translated message to be presented clearly through text or audio so the interaction can continue naturally.

From Problems to Product Logic

Define

Defining the Right Product Direction

Build Around Accessible Communication

We shaped Kanvo around a simple principle: machine learning should stay behind the experience. Users should be able to communicate through familiar gestures while the product handles capture, recognition, translation, and output with as little friction as possible.

Natural Input

Let users communicate through gestures instead of forcing them to adapt their communication to a conventional text-based interface.

Clear Output

Transform recognized gestures into readable text and spoken audio that can be understood by someone who does not use sign language.

Recognition Recovery

Design for cases where ML does not recognize a gesture, giving users a clear path to retry, edit, or work with their gesture Library.

Ideate

Turning Communication Barriers into Product Ideas

Rather than treating gesture recognition as an isolated AI feature, we designed supporting product flows around what happens before, during, and after recognition.

Making Everyday Communication More Accessible

Problem: Sign language users and people who do not understand sign language can face a direct communication barrier.Product logic: Capture gestures through the device camera, recognize supported signs, and transform the result into text and audio that the other person can understand.

Recognition Uncertainty

Problem: An ML model cannot be expected to understand every captured gesture correctly.Product logic: Give users control over the result through retaking, editing, confirmation, and flows for handling unrecognized gestures instead of turning recognition failure into a dead end.

Personal Communication Library

Problem: Users may repeatedly need gestures or phrases that the base recognition experience does not cover.Product logic: Introduce a personal Library where communication content can be organized into categories and new phrases can be added manually or through gesture capture.

Problems & Solutions

Bridging the Communication Gap

Kanvo turns gesture-based communication into an accessible digital interaction, helping sign language users communicate with people who may not understand sign language.

Problem

Everyday communication can become difficult when one person communicates through sign language and the other cannot understand it, often creating dependence on additional assistance.

Solution

We built a camera-based communication flow where users can perform supported gestures, have them processed by the ML recognition model, and receive the result as text with audio playback available through text-to-speech.

Impact

The Alpha demonstrated how Kanvo could bridge two different forms of communication within one interaction, turning recognized gestures into content another person can read or hear.

Everyday communication can become difficult when one person communicates through sign language and the other cannot understand it, often creating dependence on additional assistance.

1/3
Kanvo recognized message screen with text-to-speechKanvo recognized message screen with text-to-speechKanvo recognized message screen with text-to-speech
Kanvo screen for confirming or correcting a recognized messageKanvo screen for confirming or correcting a recognized messageKanvo screen for confirming or correcting a recognized message
Kanvo gesture Library categories screenKanvo gesture Library categories screenKanvo gesture Library categories screen

Wireframes

Visual Design

Typography & Colors

#F3F3F3
#505050
#4C8410
#6A2875
Product Decision

Make Complex ML Feel Simple
to Use

Technology Behind the Experience, Not in the Way

Gesture recognition is technically complex, but Kanvo users should not have to understand how the technology works. We designed the experience around familiar actions - capture a gesture, review the result, make corrections when needed, and communicate.
Recognition states, clear controls, accessible typography, and straightforward navigation keep the focus on the conversation rather than the machine-learning process.

Gesture Communication

Communicate Through Natural Gestures

Kanvo lets users begin communication in the way that feels natural to them - through gestures. The camera-based flow guides users through capturing a gesture and receiving a recognized result without exposing the technical complexity behind the interaction.
The experience keeps capture, recognition, and communication connected within one straightforward flow.

1/4
Kanvo website and gesture capture screens on iPadsKanvo website and gesture capture screens on iPadsKanvo website and gesture capture screens on iPads
Kanvo screens for reviewing and correcting recognized textKanvo screens for reviewing and correcting recognized textKanvo screens for reviewing and correcting recognized text
Kanvo gesture Library screensKanvo gesture Library screensKanvo gesture Library screens
Kanvo settings and profile screensKanvo settings and profile screensKanvo settings and profile screens
Recognition Results

Turn Recognition Into an Actionable Message

Recognition does not end when the ML model returns a result. The output needs to become something another person can actually understand.
Kanvo presents the recognized message as text and supports audio playback through text-to-speech. Users can review the result, make corrections when necessary, confirm the message, and continue with another interaction.

When Recognition Needs Help

Give Users a Way Forward

An unrecognized gesture should not become the end of the conversation. Kanvo includes recovery flows that let users retake the gesture or continue through the personal Library.
When recognition cannot identify a gesture, users can select or add communication content, organize it within a category, and capture the gesture again as part of the Library flow.

Personal Gesture Library

Organize Communication Around The User

The Library was designed as a structured space for communication content beyond the immediate recognition session.
Users can start with predefined categories or build their Library from scratch, browse stored content, and add new phrases either manually or through gesture capture. This gives the product a more personalized layer around the core recognition experience.

Kanvo logoKanvo logoKanvo logo

Give It a Try

Hand holding an iPad with the Kanvo appHand holding an iPad with the Kanvo appHand holding an iPad with the Kanvo app

FAQ About Kanvo

What Is Kanvo?

Plus icon

Kanvo is an accessibility-focused communication application designed to help people with communication disabilities communicate through gestures. The product uses machine learning to process gestures captured through the device camera and translate recognized signs into text, with text-to-speech supporting communication with people who may not understand sign language.

What Problem Does Kanvo Solve?

Plus icon

Kanvo addresses communication barriers between people who use sign language and people who may not understand it. The product was designed to turn gesture-based communication into text and audio within an accessible digital experience.

How Does Kanvo Recognize Sign Language?

Plus icon

Kanvo uses a machine-learning recognition pipeline built around Apple’s Vision and AVFoundation technologies for camera processing, Core ML for recognition, and Create ML for model creation and training.

Was A Third-Party Sign Language Recognition API Used?

Plus icon

No ready-made API, SDK, or open-source solution identified during technical research could meet Kanvo’s requirements. This led the team to develop a custom recognition approach using Apple’s computer vision and machine-learning technologies.

How Does Gesture-Based Communication Work In Kanvo?

Plus icon

Users capture gestures through the device camera and the recognition system processes them into a text result. The communication flow also supports reviewing the result, text editing, audio playback, confirmation, and starting another communication interaction.

What Happens If Kanvo Does Not Recognize A Gesture?

Plus icon

The product includes recovery flows for unsuccessful recognition. Users can retake a gesture or continue through the Library experience, where communication content can be selected, organized, or added through manual input and gesture capture.

What Is The Kanvo Gesture Library?

Plus icon

The Library is a structured space for managing communication content. Users can work with categorized gestures and phrases, start with predefined categories or build their Library from scratch, and add new phrases manually or through gesture capture.

How Was Accessibility Addressed In Kanvo?

Plus icon

Accessibility was considered throughout the UX/UI process, including typography, color contrast, interaction patterns, and navigation. The design research considered WCAG 2.1 AA and Australian digital accessibility requirements.

What Platform Was Kanvo Developed For?

Plus icon

The Kanvo Alpha was developed as a native iPadOS application, allowing the product to combine camera-based gesture capture, Apple’s machine-learning technologies, and an interface designed around accessible communication.

What Technologies Were Used To Build Kanvo?

Plus icon

Kanvo’s technical ecosystem included native iPadOS development, Vision and AVFoundation for camera processing, Core ML and Create ML for machine learning, backend services, AWS cloud infrastructure, Firebase, and supporting monitoring and product services.

What Was You Are Launched Responsible For?

Plus icon

You are launched supported Kanvo across product discovery, business analysis, technical research, UX/UI design, native iPadOS development, ML research and integration, backend and infrastructure work, and web development for the supporting landing page.

What Was Delivered During The Kanvo Alpha Stage?

Plus icon

The Alpha established Kanvo’s core gesture-recognition experience within a native iPadOS application, connecting camera-based gesture capture, ML recognition, recognition results, text-to-speech, and supporting account and product flows within one working experience.

How Was Kanvo’s Recognition Model Improved?

Plus icon

During development, additional model training was introduced to improve recognition performance, including the model’s ability to distinguish a user’s hand across different backgrounds.

How Was Kanvo Designed To Evolve Beyond The Alpha?

Plus icon

Kanvo’s technical foundation was designed to support progressive recognition development. The roadmap extended from Alpha-stage image-based character recognition toward video-based word and gesture recognition and, ultimately, personalized on-device model training.

dots
dots