Face AR in Unreal

This week I learned how to use ARKit to act behind the character in unreal.

Below shows the animations of blendshapes using FaceIt in Blender.

Process

Should have imported the skeleton not being rigged from makehuman. This image shows the rigged skeleton in blender. Hard to separate the jaw.

Here are the processes of me using FaceIt and Blender to create blendshapes for Unreal and ARKit.

This is Todd using his iPhone to do some serious acting in my Unreal through ZeroTier VPN and Zoom .

Problems encountered that was solved by Todd’s Office Hour

  1. Should check off FaceIt Collections and modifiers to keep the shapekeys (which is the blendshapes for ARKit)
  2. Should check No Rig in MakeHuman before exporting to Blender for FaceIt
  3. The usage of VPN is to have access to the iPhone of others
  4. In order to record the animation, just drag the character and add animation in the sequencer

I can also rig the skeleton in Maya with the AutoRigging feature. Workflow is similar to following:

Rigging – Skeleton – AutoRig – simliar to FaceIt – adding bones to the skeleton

One more problem

When I was trying to rig the Defeated Mixamo skeleton and use the one Todd recorded on the face of Tuna, it does not work because the skeleton does match with one another.

Character LiveLink Capture Animations – to be recorded

  1. Cry out of joy for the rain
  2. Feeling distraught and annoyed – Look down
  3. Crying to sleep
  4. Surprised

Details for the Developed Story

Since I have enriched the story a little bit. There are several things I need to reconsider:

  1. Depressing color-scheme:
    1. Hot-> orangish-brown for the plateau
    2. light yellow atmospheric fog as always
    3. Light grayish-blue as the master color scheme
  2. Features
    1. Occasional seagulls flying – indicating that ocean is not too far away
    2. Few bushes to show existence of life between the cracks of the stones
    3. Dry texture of the rocks and road – make sure it is hot to touch with your bare feet
    4. coffin-shaped cave made up with stones – Tuna’s so-called home
    5. Wild fire everywhere
    6. Steaming hot surroundings
  3. Places & actions that can take place – Mixamo and RADiCAL
    1. Within the Coffin
      1. Tuna sleeps and turns back and forth – insomnia
      1. Laying shrugging
      2. laying sleeping
      3. waking
      4. walking out of the coffin – female start walking, walking, female stop walking, walking in circle, female stop and start
      5. Goes out and does her daily routine – shows the player the surrounding
    2. On the Plateau – The rainy day
      1. crying
      2. dancing
    3. The Escape
      1. running, running tired, turn to running, injured run
      2. climbing down the side of the plateau
        1. climbing down wall
        2. climbing
    4. Animations to recapture
      1. Sense of salvation
        1. Joyful jump and dance for a while, cry on the floor while kneeling down
      2. Over-joyously swimming in the ocean
      3. Look up to the sky while taking in all the water she can in the ocean

Exploring FaceAR for Android for Unreal

I was thinking to find a workflow for Android since I do not have iPhone and I really wanna try the FaceAR sample. But turns out it is possible to use Android to create AR apps through Unreal but not necessarily the face tracking. So the following links are for AR dev for android with Unreal.

Here is the following links I found so far.

https://docs.unrealengine.com/en-US/Platforms/AR/HandheldAR/ARCorePrerequisites/index.html

https://developers.google.com/ar/develop/unreal/augmented-faces/developer-guide

Augmented Faces developer guide for Unreal

Copied the story I wrote from my previous blog

Side notes – Marvelous Designer

It is so fun to play with it! I always wanted to do fashion design since I was a kid now I can finally do so in 3D haha!

Published by Yiting Liu

NYU ITP '21

One thought on “Face AR in Unreal

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create your website with WordPress.com
Get started
<span>%d</span> bloggers like this: