Lights, camera, algorithm: How artificial intelligence is being used to make films

Canadian creatives produce short feature about a balloon-headed man using OpenAI’s new text-to-video model Sora

When Walter Woodman and his team of creatives were working on one of their latest film productions, they kept hitting a snag.

They couldn’t seem to get the character at the centre of their picture, a man with a balloon for a head, to look quite right.

“It would draw a face on the balloon and we didn’t really want that,” Mr Woodman tells The National.

“If we even mentioned the word ‘face’ it would put a human face inside the balloon … and so I think we learnt after a while to say 'the balloon man is expressionless'.”

The “it” Mr Woodman is referring to is Sora, a new text-to-video AI model from OpenAI, the artificial intelligence research organisation behind viral chatbot ChatGPT.

Instead of using their broad technical skills in filmmaking, such as animation, to overcome obstacles in the process, Mr Woodman and his team relied only on the model to generate footage for them, shot by shot.

“We just continued generating and it was almost like post-production and production in the same breath,” says Patrick Cederberg, who also worked on the project.

The film, aptly named air head, follows the balloon-like character as he navigates the world, with its windy days, cactus shops and cat claws.

“I’m reminded every day that life is fragile,” the character says in the one minute and 21-second-long video. “We’re all just a pinprick away from deflation.”

Watch in full – air head

The highly realistic scenes Sora can generate from a simple text prompt can be startling.

Examples released by OpenAI include a woman walking down a Tokyo street under the city’s trademark glowing neon lights, as well as several woolly mammoths plodding through a snowy meadow.

The model can create cute, animated characters not unlike Disney Pixar classics, along with near-perfect drone shots over tourist havens such as the Greek island of Santorini.

“It was immediately clear this was powerful and could give us a lot of good stuff,” says Mr Cederberg.

OpenAI’s models are constantly improving and the footage Sora generates is only expected to get better.

A video generated by Sora shows wooly mammoths walking in front of a snowy mountain. Photo: Screengrab / OpenAI

Mr Woodman said that where his team’s usual skills in animation relate more to mathematics, using Sora was all about English.

“It definitely requires a grasp of the English language,” he says, as colleagues Sidney Leeder and Mr Cederberg smile and nod in agreement.

The trio make up part of a Toronto-based creative collective called shy kids, which has produced content for the likes of Netflix, HBO and Lionsgate.

They were some of the first filmmakers to get their hands on Sora after they were invited by OpenAI to work with the new software.

“You’re basically writing out a scene,” says Ms Leeder, who is a producer at shy kids.

Mr Cederberg, whose work usually focuses on animation, adds: “But also trying to fool the computer into doing what you want it to do … by understanding how it’s reading your language.”

Sidney Leeder, Walter Woodman and Patrick Cederberg are part of a Toronto-based creative collective called shy kids. Joshua Longmore / The National

The film took about a week and a half to finish – a quick turnaround for such an undertaking – but Mr Woodman says he and his colleagues were particularly energised during the process.

“The time was definitely due to our years of experience – not that it was somehow easier,” he says. “It was still arduous.”

AI techniques such as text-to-video generation could one day upend the entertainment industry. Its use was a major force behind writers' and actors' strikes in Hollywood last year.

There is creeping anxiety among some workers in the creative industries that machines could replace them in the future or that powerful studio heads could use AI tools to cut them out.

Members of the Writers Guild of America (WGA) demonstrate in front of Walt Disney Studios in Burbank, California, in May last year. EPA

“That horror scenario where executives are just typing out the films they want us all to see, I wish for that future,” Mr Woodman says jokingly, after admitting he shares concerns about his future, too.

“I think when you finally see what is rattling around those executives' brains, that is when we will know that we need artists more than ever.”

But Mr Woodman also thinks AI tools like Sora can create a new world of opportunity for filmmakers and studios.

OpenAI's Sora was able to create a video based on the prompt, 'A young man at his 20s is sitting on a piece of cloud in the sky, reading a book.'

He likens the current situation to the development and use of computer-generated imagery (CGI) at Industrial Light and Magic, the famed visual effects company behind much of the dazzling scenes seen in the Star Wars films.

“There was a mutiny about the old guard who made things with puppets and models and the new guard who made things with CGI,” he says.

“Now we live in a world where we are trying to defend CGI … when 20 years ago people were saying films are being ruined by it.”

In all of these cases, including with AI, Mr Woodman says, you still need a team of dedicated and creative people to make something that moves audiences.

“I don’t think that is ever going to change, no matter how good any technology gets.”