In a breakthrough development, researchers have introduced TinyVQA, a groundbreaking multimodal deep neural network designed to tackle visual question answering tasks on resource-constrained devices. Traditional machine learning models often demand high-performance hardware, rendering them impractical for deployment on devices with limited resources. However, the emergence of Tiny Machine Learning (tinyML) has opened new possibilities, yet integrating multiple data modalities into tinyML models has remained a significant challenge due to complexity, latency, and power consumption concerns.
TinyVQA represents a novel solution to this problem. Leveraging a supervised attention-based model, TinyVQA adeptly answers questions about images using both visual and language modalities. The model incorporates distilled knowledge from the supervised attention-based VQA model, training a memory-aware compact version suitable for deployment on tinyML hardware. Additionally, low bit-width quantization techniques are employed to further compress the model, ensuring efficient utilization of resources.
To validate its efficacy, the TinyVQA model underwent rigorous evaluation on the FloodNet dataset, commonly utilized for post-disaster damage assessment. Impressively, the compact model achieved an accuracy rate of 79.5%, underscoring its suitability for real-world applications. Furthermore, researchers successfully deployed the TinyVQA model on a Crazyflie 2.0 drone equipped with an AI deck and GAP8 microprocessor. The deployment showcased low latencies of just 56 milliseconds and a power consumption of 693 milliwatts, highlighting its adaptability for resource-constrained embedded systems.
This breakthrough holds significant implications for various industries, enabling enhanced capabilities in applications such as remote sensing, surveillance, and environmental monitoring. With TinyVQA paving the way for efficient and effective visual question answering on resource-limited devices, the future of machine learning deployment appears increasingly promising.
Download paper: https://arxiv.org/pdf/2404.03574.pdf
0 Comentarios