UMA Students Develop Conecta2 AI Lip-Reading App

AI Lip-Reading Tool Restores Voice to Non-Verbal Patients

Four Artificial Intelligence engineering students at the University of Málaga (UMA) have developed Conecta2, a mobile application designed to translate silent lip movements into spoken speech in real time.

As reported by Sur in English, the project leverages custom computer vision algorithms and machine learning models to analyze subtle labial and facial movements, converting them immediately into synthesized audio. The initiative addresses a critical barrier in clinical environments where patients are physically unable to vocalize words.

Technical Architecture and Visual Phoneme Recognition

The development of Conecta2 centers on real-time visual phoneme recognition. Unlike conventional voice-to-text systems that depend on acoustic input, this system operates entirely on visual data captured through a standard mobile device camera.

System Workflow

• Computer Vision Pipeline: Frames captured by the mobile camera are processed to isolate the lip and lower facial region, tracking geometric landmark points at high frame rates.
• Neural Network Processing: A deep learning model translates sequences of labial positions into probable phonemes and words, resolving visual ambiguities common in silent speech (such as identical lip positions for ‘p’, ‘b’, and ‘m’).
• Audio Synthesis Engine: Recognized visual sequences are instantly mapped to text and passed to a low-latency text-to-speech engine, outputting clear vocalized words with minimal delay.

Processing latency is a key metric in assistive communication tools. By prioritizing on-device processing capabilities, the application minimizes latency and functions reliably without demanding constant high-speed internet connectivity in hospital environments.

Clinical Applications and Target Demographics

Communication barriers in healthcare settings frequently elevate anxiety and complicate treatment for patients experiencing temporary or permanent speech deficits. Conecta2 targets several distinct clinical scenarios:

1. Intensive Care Unit (ICU) Patients: Individuals undergoing mechanical ventilation or intubation remain conscious but cannot produce acoustic sound. Conecta2 offers a non-invasive channel to communicate symptoms, basic needs, and emotional states.
2. Post-Surgical Recovery: Patients recovering from tracheostomies, laryngeal procedures, or facial trauma often experience prolonged periods of speech loss.
3. Neurological Conditions: Progressive disorders such as Amyotrophic Lateral Sclerosis (ALS), advanced Parkinson’s disease, or severe stroke impair vocal cord motor control while leaving facial mobility intact.

Clinical Context
Primary Speech Obstacle
Conecta2 Technical Function
Mechanical Ventilation
Airway blockage prevents vocal cord vibration
Translates silent mouth movements directly into audio
Tracheostomy
Airflow bypassed away from larynx
Generates speech output without requiring expiratory pressure
Post-Stroke Dysarthria
Impaired neuromuscular coordination of speech
Uses visual cues to reconstruct intended words

Málaga’s Expanding Role in Applied AI Innovation

The development of Conecta2 highlights the growing technical capacity of the University of Málaga’s School of Computer Engineering (E.T.S. de Ingeniería Informática). Málaga has steadily built a reputation as a major technological hub in southern Europe, pairing academic talent with regional infrastructure like Málaga TechPark (PTA).

Projects coming out of UMA reflect a clear shift toward applied artificial intelligence aimed at solving practical, real-world problems. The combination of university research talent and local incubator support creates a viable pipeline for turning academic course projects into functional biomedical tech tools.

Technical Challenges and Scalability Roadmap

While initial testing demonstrates high accuracy, transitioning lip-reading AI models into clinical deployment involves complex operational variables:

• Environmental Lighting and Angles: Hospital bedsides present uneven lighting conditions and non-standard camera angles, alongside potential obstructions from medical gear.
• Speaker Variability: Facial mechanics and movement amplitudes vary across individuals, requiring adaptive models that fine-tune to specific user speech patterns over time.
• Data Privacy and Compliance: Deploying camera-based software in medical facilities requires strict adherence to health privacy standards, ensuring video feeds are processed locally without unauthorized data retention.

The development team plans to expand training datasets across diverse accents and facial structures while pursuing clinical trial partnerships to evaluate app performance in active hospital settings.

It is deeply encouraging to see engineering students in Málaga directing advanced technology toward solving human hardship. As projects like Conecta2 move from university labs into testing phases, the potential to significantly improve patient care and quality of life continues to grow.