Vision-language-action model (VLA)
A machine learning model that takes camera images and a natural-language instruction as input and outputs the actions a robot should perform.
Photo: Igor Omilaev / Unsplash
The name comes from Google DeepMind’s RT-2 paper, posted in July 2023 (arXiv 2307.15818) and peer-reviewed at CoRL 2023, which called such models vision-language-action models. The recipe starts from a vision-language model, a large language model that can also read images, and trains it further on recordings of robot tasks. RT-2 wrote robot actions as text tokens, so the model outputs a movement the way it outputs a word. OpenVLA (arXiv 2406.09246, peer-reviewed at CoRL 2024) applied the idea with openly released weights.
The appeal is that knowledge from the web, such as what a sponge looks like, can carry over to physical tasks. The cost is size: running inference on a large model fast enough for smooth motion takes serious onboard computing, and both papers report results on test tasks set up by the researchers, not in ordinary homes.
Part of the IoT Glossary. Spotted an error or a missing term? Tell us.