basic insights on DeepfakES

Custom Deepfake content creation is a topic I wanted to share some ‘basic’ insights on. 

Recently at Headfelt XR we were required to deliver a prototype to a client. The brief was ‘to enable a user to replace a photo of their own head into multiple custom still photo scenes (not video…yet), and produce a natural and convincing outcome’. Obviously this is a bit outside the Augmented (AR) and Virtual Reality (VR) box, but hey it sounded like fun.

Surprisingly the result was a lot harder to achieve than I expected. I thought deepfake had well progressed into video content, and as a result a deepfake of a still should be a walk in the park. 

Well after a lot of research I discovered, surprisingly, it isn’t.

Yes, there are plenty of face swap social filters (too many), dedicated mobile apps like Reface  and Place My Face, and software services like DeepSwap and Facehub. The later use AI and do a great, fast, job of adding your likeness into stills, famous movie scenes and even custom videos. Be warned, there are some very dubious references to signing away your biometric data….cant be that important, right?

Speaking of movies, Machine Learning (ML) and AI, there is a great video of the boys at the Corridor Crew using Dreambooth to create a fictional storyboard using staff faces. This provides a great background to understanding what is involved in adding your own face into custom scenes using AI.  Minimum requirements include crazy custom hardware capabilities and days of rendering. 

Back to my brief. There are also few very basic ‘click bait’ sites such as Face-in-a-hole and loonapix that sort of, but not really, achieve the required result.

Hell, you can even build your own system should you have the time to dig a very large rabbit hole using tools like First Order Model, Marek’s Face Swap and Flutter options.

BUT understandably ignoring the custom build option, the problems of these options are: 

• The solution does not meet the basic brief of placing ‘only the users head’ in a custom scene I.e. face swaps is almost always means swapping two faces within the same photo
  
  
• The outcome is ridiculously unnatural and unconvincing
  
  
• Many do not allow for custom scenes
  
  
• Those that use the more sophisticated AI solutions come close, but produce only a “Likeness” and not a straight swap. Thats right, most AI technologies only provide a face-merge outcome and not a 1:1 head swap. This is understandable due to things like head shape, scale, angle, tilt, skin tone, hair style, position, lighting etc. If interested a good overview of the processes types can be found here 

So what I discovered is that if you want to simply enable your users to place their whole ‘head’ into your own custom still scene that results in a natural and convincing outcome, then there are very, very, few solutions. The answer is in the fashion industry and try ons. Although still not as progressed as you think, there are some customisable solutions starting to emerge. This also has a direct relationship to the Extended Reality (XR) industry, and this the connection to this project…phew.