The AI edit suite of the future
How will films be made in the future?
Will the classic edit suite even still exist?
Will it all happen in virtual reality?
Lately I have talked to a lot of people from the industry and we have a few theories. What the edit suite of the future has to do with the Emmy Award-winning series Westworld, you’ll see – of course – at the very end.
Prompt, prompt, prompt. Everybody talks about prompts. That is, the art of making the computer understand through text input what I want from it.
The so-called “prompt engineer” is a completely new line of work! But I don’t think it will last long. At least not in this form. Interaction with image and video editing AI will (hopefully soon) work through normal speech. Like a real conversation.
We already see that with our smart home assistants – by voice we switch on the light or the TV. Set alarms and timers. By voice we play music in different rooms or combined as a group, e.g. on the entire ground floor. Enter important appointments in the calendar and call the whole family to dinner via broadcast messages.
It stands to reason that I also want to talk to my edit suite.
Dictation, i.e. speech recognition – whether on a mobile phone, PC or Mac – makes (with reasonably clear pronunciation) almost no mistakes anymore and keeps learning. But really controlling computers by voice, (ahh) I’ve certainly been trying out every now and then for 20 years and so far I’ve always been disappointed… I’m giving voice input on the Mac another new chance right now – maybe I’ll make a video of its own on voice control soon. Feel free to write in the comments whether you want to see that. And since I’m on comments – it’s really great how many people really got involved with my last two videos:
about “SORA” and “Hollywood is finished”. Thank you so much for the lively discussion and the new food for thought.
So – back to the edit suite.
My vision is to edit films in VR, i.e. virtual reality, in the future. First I generate a scenery, a backdrop for my film.
That works, for example, with Luma AI.
Here you can generate 3D objects and sceneries as of recently. Either through classic prompt input, or I upload a video and a 3D scene is created from it.
Now, through my VR glasses, I stand in the middle of this living room and can walk around. Think this through with me.
Now I put my AI actors on the sofa and give them lines and directing instructions. Then I take care of the lighting. Like on a normal shoot. A lot of light comes in from outside anyway – but everything is possible later.
Now I place my cameras. A wide shot from diagonally above. A camera for the two-shot – with both people in it – on a slider. The slider is 3 meters long and the camera travels back and forth on it continuously. Two cameras shoot crosswise close-ups of the actors’ faces. That’s enough for this scenery for now.
You’ve surely seen the Meta Quest and Apple Vision Pro videos. There, windows are faded into the room. Such a window would then move with me the whole time as I walk through the room. On it you see my 1, 2, 3, 4 camera shots and the PGM – i.e. the currently active camera – in large.
The great thing – I’m only virtually in the scene with my glasses, meaning: I’m not in any camera operator’s way, cast no shadow and can get really close to my actors while they perform. Yes, I can even put myself into one of the characters, take their point of view. The director, editor or whatever this job will then be called (I always like anything with “operator”) can see and steer the emotions in the face very precisely. “Say that a little sadder and at the end a twitch of the corner of the mouth.” – for example. The actors know their lines, then I program my cameras.
The scene starts with the wide shot. We placed that diagonally above – maybe 3 seconds. Next, cut to the slider camera – 5 seconds. Then always show the person who is speaking in close-up. Every 45 seconds, for 5 seconds back to the slider camera and at the end of the scene a circular move around the actors.
Of course you’ll be able to change all of this afterward. Or switch actively while the scene is being played. I might NOT solve that with voice input, but by pressing the respective camera image.
A VR variant without cameras is of course also conceivable, then I can sit down on the sofa as a viewer or walk through the living room during the scene and listen – in explorer mode, so to speak.
In which film would you like to “play along”? Because that is inevitably the next development. We don’t just watch the film, we play along – we ARE one of the characters.
Steven Spielberg has just confirmed that Ready Player Two is finally being tackled, i.e. the sequel to Ready Player One. If you haven’t seen it or read Ernest Cline’s book – Ready Player One is THE showcase film/novel for the metaverse. The whole world meets in the virtual world called the Oasis, to play and also just to hang out. In the second part, “stepping into” characters is described in detail. The hero has to – among other things – re-enact scenes from films and gets points if he reproduces the lines from the original correctly.
“I drink and I know things!”
Let’s see whether this VR movie pop quiz scene finds room in the film adaptation. With the first part the book and the film were very different.
I have one more idea of what editing could look like in the future, before we look at what Westworld has to do with all of this.
So – imagine I have my AI actors and give them backstories. 2 years married, she studied this and that, he works as – for all I care – an insurance clerk. Blah, blah, blah. And now comes the thing – she has a dark secret that nobody suspects. Now I let the two play “their everyday life”. A week, a month, 5 years. Similar to the computer game “The Sims”. My AI actors move in a simulation. They re-enact “real” life. My instruction: little by little the man discovers clues to her secret. Until the confrontation comes. The big bang. The secret is revealed and the situation escalates. I, or rather the AI, watches the two and picks the best scenes out of 5 years of simulation. Voilà – my film is done. No script, no fixed dialogue.
Like in a laboratory, I let the story “grow”. And since the 5 years of simulation take only 5 minutes of our time, I can have 10, 20 such petri dishes of stories develop and take the most exciting one in the end.
I’ll just invent a name for that now: Evolutionary Editing. Short: Evo Editing.
Sounds good? You heard it here first! Evo Editing.
I love sitting at my edit suite and fiddling with films – nevertheless I’m happy about new ways to generate films and I throw myself at new technologies.
How the creators of Westworld imagined such an edit suite, I’ll show you now. For everyone who doesn’t know Westworld – shame, shame, shame! No, for everyone who hasn’t watched Westworld yet – I won’t spoil anything here. At its core, Westworld is about a huge Western-style amusement park. But we’ll jump right into the 4th and final season (sob!). One of the main characters has a job as a “storyteller” in this season. And this is what it looks like:
A miniature world in AR – i.e. augmented reality – that can be controlled and changed by voice. Before her eyes, sceneries and characters change. In another episode we see how she created various characters and can fall back on them. That is exactly how I imagine it too. You develop a character in as much detail as possible. Backstory, friends, hobbies, political views, mental and physical condition. All of that then yields a virtual person that I can even let interact with other virtual or real people. What makes up the art of the actor – putting yourself into a person – falls away here. The virtual character IS this person and “feels” and acts accordingly. There is no need to give it a motivation, as you do with “real” actors. The virtual person reacts in every situation exactly as is expected of that person. If the character doesn’t react the way I want, I simply have to change the backstory. Here we’re of course also in the middle of the discussion of when AI has consciousness. And if it has, does it automatically have rights? Will my characters fight not to be switched off – as in Westworld?
On this topic I’m very happy about your comments!
I probably won’t shoot my next video in VR yet, but again here in the studio. So you don’t miss it – hit subscribe firmly, don’t forget the bell – then we’ll see each other in the next episode of alphatester – the channel for innovation, tech and digital lifestyle. Ciao!