The independent film company Adapt Entertainment has developed a technology based on automatic learning (‘machine learning’) and Artificial Intelligence (AI) to match actors’ mouth movements with different languages in voice overs.
This project is known as PLATO and, through the technology it uses, it is able to make its technology go unnoticed even if the scenes of a series or a movie have not been recorded in the language in which they are being viewed
As stated by the company on its website, with PLATO you can recreate the dubbing experience with the original cast of actors and the original version of the film but in non-original languages.
To do this, first the audio and video recordings are made with the original actors and also with the dubbing actors of other languages, if any. In these shoots, they stand in front of five cameras that record their facial expressions from different angles and identify each of their gestures, as well as the movements of the mouth, chin, jaw and cheeks.
Once the dubs have been recorded, the images are processed using Adapt’s AI-based technology in full motion. In this way, a scene from the original film or series can be synchronized with the new language -either the one recorded by the original actor of the production or by a different one-, adapting each of the actors’ gestures.
As a result, the words that the actors say will coincide with the position of their lips and the placement of the lower part of the face and it will not be so noticeable that it is a production subjected to a voice dubbing process.
For his part, the director of the film The Champion (The champion of Auschwitz), the first production that has integrated this technology, Maciej Barczewski has insisted that thanks to this method, viewers will be able to focus their attention on the images instead of the subtitles.
“This pioneering project that we are carrying out with Adapt Entertainment will allow us to bring our films to larger audiences in many other countries,” explained Barczewski, in a video published by the company.
last year met TrueSync de Flawlessa technology similar to ‘deepfakes’ to manipulate the faces of the actors and allow them to synchronize with the voice dubbed in another language.