AI Style Transfer Dance

Final Output

vpd_Final_wAudio.mp4

Visual Break Down

WWu_VPDF24_FinalWBreakdown.mp4

Process

For this project, I used an old project made in UE5 to style transfer the video into a Arcane style animation. This project used UE5 (original footage), ComfyUI (style transfer), FluxGym (Train LoRa), and Ebsynth (generate style transferred sequence).

The video

Training LoRa

To begin with, I wanted to train a model with an Arcane style. I found 30 high res images of the show Arcane on Reddit: Arcane 1, 2

After compiling the images set, I trained the model with FluxGym through Pinokio.

16G VRAM; 30 trains per images, 4 epochs, 30 images => 3600 training steps; took 4 hours

Testing the ArcaneStyleWW LoRa

Tested the model through ComfyUI with text prompt.

Here are some interesting outputs:

Result: Great with environments, not necessarily with characters.

So I looked for a free online Arcane style LoRa on civitai in case that I need it for style transferring the character in my video.

UE to Flux

Moving onto style transferring the video, I exported jpg sequence from UE and selected images for keys so that I can use them on EbSynth.

In the process of style transferring these key frames, I fixed the seed and also use ChatGPT to create text prompt to ensure that the style stay consistent.

Style Transferring the Character

Ebsynth doesn't work great with the character when I tried out to do the whole thing at once, so I have to rotoscope out the dancer through Runway AI. After doing so, I generated an Arcane version of the dancer with the Civitai model.

Then I use this image and the rotoscoped footage of the dancer on Viggle to create a greenscreen footage of this Arcane dancer.

Post Production

After having everything prepared, I used After Effects to edit the video. When the composition is mostly done, I found a copyright free audio online edited it with Reaper to match the composition.

Page updated

Google Sites

Report abuse