Filmmakers say AI will change the art — perhaps beyond recognition

The latest generative models make for great demos, but are they really about to change how people make movies and TV? Not in the short term, according to filmmaking and VFX experts. But in the long term, the changes could be literally beyond our imagining.

On a panel at SIGGRAPH in Denver, Nikola Todorovic (Wonder Dynamics), Freddy Chavez Olmos (Boxel Studio) and Michael Black (Meshcapade, Max Planck Institute) discussed the potential of generative AI and other systems to change — but not necessarily improve — the way media is created today. Their consensus was that while we can justly question the usefulness of these tools in the immediate future, the rate of innovation is such that we should be prepared for radical change at any time beyond that.

One of the first topics tackled was the impractical nature of today’s video generators.

Todorovic noted the “misperception of AI that it’s a one-click solution, that it’s going to get you a final VFX shot, and that’s really impossible. Maybe we’ll get there, but if you don’t have editability, that black box doesn’t give you much. What we’re seeing right now is the UX is still being discovered — these research companies are starting to learn the ways of 3D and filmmaking terms.”

Black pointed out that language fundamentally lacks the ability to describe some of the most important aspects of visual creation.

“I mean, things like yoga poses, ballet poses — there’s some classic things we have names for, that we can define, but most of the stuff we do, we don’t have names for,” he said. “And there’s good reason for that: It’s because humans actually have inside them a generative model of behavior. But I don’t have a generative model of images in my head. If I want to explain to you what I’m seeing, I can’t project it out of my eyeballs, and I’m not a good enough artist to draw it for you. So I have to use words, and we have many words to describe the visual world. But if I want to describe to you a particular motion, I don’t have to describe it in words — I just do it for you, and then your motor system sees me and is active in understanding that. And so we, I think it’s a biological reason, a neuro-scientific reason, that we don’t have words for all of our motion.”

That may seem a bit philosophical, but the result is that text-based prompt systems for imagery are fundamentally limited in how they can be controlled. Even the hundreds of terms of tech and art used every day on set and in post-production are inadequate.

Chavez Olmos pointed out that, being from Mexico, he had little opportunity to take part in the filmmaking world, because all the money and expertise was concentrated in LA. But he said that AI expertise (and the demand for it) is more widely distributed. “I had to leave Mexico because I had no opportunity there; I can see, now, having that same opportunity for people who don’t need to go overseas to do it.”

Black, however, is worried that sudden access to these processes may have unintended consequences in the short term.

“You can give somebody a powerful car, that doesn’t make them a Formula One driver, right? That’s a little bit like what we have now. People are talking about, everyone’s going to be making films. They’re going to be s—–, quite honestly,” he said. “The democratization thing is exactly what [Chavez Olmos] said, and the power is that maybe some new voice will have an opportunity that they wouldn’t otherwise. But the number of people making really good films is still going to be small, in my opinion.”

“The real revolution,” he continued, “the real power of what we’re seeing in AI is we’re going to see an entirely new genre of entertainment, and I don’t know exactly what it’s going to look like. I predict it’ll be something between video game and film and real life. The film industry is passive storytelling: I sit there and observe, it’s like theater or a podcast. I’m the passive recipient of the entertainment. But in our day to day life, we tell stories to each other, we chat about what we did on the weekend and so on. And that’s a very active kind of interactive storytelling.”

Before that happens, though, Chavez Olmos said he expects a more traditional acceptance curve on AI-generated imagery and actors.

“It’s gonna have the same, I think, reaction that we had when we saw the first ‘Final Fantasy’ movie or ‘The Polar Express’ — something’s going to be not quite there yet, but people are going to start accepting these films,” he said. “And instead of a full CG film, it’s going to be a full AI film, which I think we’re going to see even at the end of this year. I think people are going to get past that, like ‘OK, this is AI,’ people are going to accept that.”

“The important thing,” Black said separately, “and Pixar taught us this very clearly: It’s all about story. It’s all about connecting to the characters. It’s about heart. And if the movie has heart, it doesn’t matter if the characters are AI, I think people will enjoy the movie,” he said. “That doesn’t mean that they’re going to not want human actors. There’s an excitement to knowing it’s real humans like us, but like way better than us, to see a human at the peak of their game, it inspires all of us, and I don’t think that’s going to go away.”