Case study — mobile · computer vision · on-device ML
ISHARA
إشارة
Arabic Sign Language Recognition — a Flutter mobile app.
Ishara is a Flutter mobile app exploring Arabic Sign Language recognition with on-device machine learning. It works in two directions, and the recognition pipeline is designed to run locally — without depending on a network connection. The current model recognizes 31 Arabic sign classes, including the compound signs ال and لا.
The camera captures the hand, the pipeline predicts the corresponding letter, and the user builds it into a sentence.
Typed text is broken into letters, each shown as its sign image, with playback to move through the sequence.
Here's the same idea broken into its steps — a hand becomes a letter, or a letter becomes a hand.
Presented to the camera.
Captures the live frame.
The hand is located in frame.
Key points on the hand are extracted.
The model predicts a letter.
الحرف المتوقع.
Confirmed letters build a sentence.
The user types a word or phrase.
Text is broken into individual letters.
Each letter maps to a sign image.
The sequence can be reviewed and navigated.
The interface doesn't ask the user to blindly trust one prediction. It surfaces the prediction alongside correction options, so the user can confirm the intended letter before continuing.
The reverse interaction: text becomes letters, letters become signs, and the whole sequence can be reviewed at the user's pace.
Left: typed text ("منار") returns its first sign, ready to translate. Right: the sequence in review, with a progress bar, letter chips, and transport controls to step through it.
What happens between the camera seeing a hand and a letter appearing on screen.
The camera captures the frame containing the user's hand.
The hand is located within the frame.
Key points on the hand are identified.
Landmark points are prepared for the classifier.
An on-device model classifies the sign from 21 hand landmarks.
An Arabic letter is predicted.
Consecutive predictions are checked for consistency before a letter is accepted.
The result is shown to the user.
The landmark and prediction stages, as they appear in the real interface.
The recognition pipeline runs on the device rather than depending on a server round trip.
Only the components genuinely required to make both directions work — no backend, no database, no unnecessary layers.
Renders the interface and captures user input.
Coordinates camera input, recognition, and UI state.
Provides the live hand image.
Detects the hand and extracts landmark points.
Landmark data is prepared for the model.
Runs the on-device classification model.
Produces the predicted Arabic letter.
Accepts a letter only once consecutive predictions agree.
Displays the result back to the user.
Flutter Developer — the work spanned the interface and the recognition pipeline itself.
A prediction only becomes useful when it's communicated clearly.
Landmarks provide a structured representation of hand pose.
Local inference changes how feedback should feel.
A prediction should be correctable, not blindly trusted.
Language direction affects both layout and interaction, from the start.
Recognition, sentence building, and playback all need reliable UI state.
Raw predictions fluctuate frame to frame — only consistent signals should reach the UI.
A working Flutter prototype combining Arabic Sign Language interaction, computer vision, hand landmark extraction, TensorFlow Lite classification, text-to-sign interaction, sign playback, and offline / on-device recognition.
How to connect a real-time recognition pipeline to a mobile product experience — while keeping the interaction understandable and usable.