@remotion/three

is a package for integrating React Three Fiber with Remotion.

<ThreeCanvas /> will allow you to use useCurrentFrame() and other Remotion hooks within a R3F Canvas. Animations are now not inside a useFrame() hook, but directly rendered into the markup.
useVideoTexture() allows you to use a Remotion <Video /> as a texture map.
useOffthreadVideoTexture() is an equivalent to useVideoTexture(), but displays the exact frame as an Three.ImageTexture during rendering, simillarly to <OffthreadVideo />.

These are the only two APIs provided - for everything else you can use the standard React Three Fiber APIs.

Starter template

Check out remotion-template-three, a minimal boilerplate for Remotion and React Three Fiber. It is a template repository, you can click "Use this template" on the GitHub repo to get started.

0:00 / 0:10

The template features a 3D phone with a video inside which you can effortlessly swap out. Just as easily, you can change properties like the color, size, thickness, corner radius of the phone.

The template serves as a soft introduction on how to use <ThreeCanvas /> and useVideoTexture(). You can easily delete everything inside the canvas to start working on a different idea.

Installation

npm
yarn
pnpm

bash
npm i three @react-three/fiber @remotion/three @types/three

bash
npm i three @react-three/fiber @remotion/three @types/three

bash
yarn add three @react-three/fiber @remotion/three @types/three

bash
yarn add three @react-three/fiber @remotion/three @types/three

bash
pnpm i three @react-three/fiber @remotion/three @types/three

bash
pnpm i three @react-three/fiber @remotion/three @types/three

You are now set up and can render a <ThreeCanvas /> in your project.

Note on `<Sequence>`

A <Sequence> by default will return a <div> component, which is not allowed inside a <ThreeCanvas>. To avoid an error, pass layout="none" to <Sequence>.

Note on server-side rendering

Three.JS does not render with the default OpenGL renderer - we recommend to set it to angle. The config file of new projects includes by default:

ts
import {Config} from '@remotion/cli/config';
 
Config.setChromiumOpenGlRenderer('angle');

ts
import {Config} from '@remotion/cli/config';
 
Config.setChromiumOpenGlRenderer('angle');

Since the config file does not apply to server-side rendering, you need to explicitly add

json
"chromiumOptions": {
  "gl": "angle"
}

json
"chromiumOptions": {
  "gl": "angle"
}

to server-side rendering APIs like renderMedia(), renderFrames(), getCompositions() and renderMediaOnLambda().

Thanks

A big thanks to Björn Zeutzheim for researching and discovering the techniques needed for React Three Fiber integration and for doing the initial implementation of the @remotion/three APIs.