Custom Deepfake content creation is a topic I wanted to share some ‘basic’ insights on. 

Recently at Headfelt XR we were required to deliver a prototype to a client. The brief was ‘to enable a user to replace a photo of their own head into multiple custom still photo scenes (not video…yet), and produce a natural and convincing outcome’. Obviously this is a bit outside the Augmented (AR) and Virtual Reality (VR) box, but hey it sounded like fun.

Surprisingly the result was a lot harder to achieve than I expected. I thought deepfake had well progressed into video content, and as a result a deepfake of a still should be a walk in the park. 

Well after a lot of research I discovered, surprisingly, it isn’t.

Yes, there are plenty of face swap social filters (too many), dedicated mobile apps like Reface  and Place My Face, and software services like DeepSwap and Facehub. The later use AI and do a great, fast, job of adding your likeness into stills, famous movie scenes and even custom videos. Be warned, there are some very dubious references to signing away your biometric data….cant be that important, right?

Speaking of movies, Machine Learning (ML) and AI, there is a great video of the boys at the Corridor Crew using Dreambooth to create a fictional storyboard using staff faces. This provides a great background to understanding what is involved in adding your own face into custom scenes using AI.  Minimum requirements include crazy custom hardware capabilities and days of rendering. 

Back to my brief. There are also few very basic ‘click bait’ sites such as Face-in-a-hole and loonapix that sort of, but not really, achieve the required result.

Hell, you can even build your own system should you have the time to dig a very large rabbit hole using tools like First Order Model, Marek’s Face Swap and Flutter options.

BUT understandably ignoring the custom build option, the problems of these options are: 

• The solution does not meet the basic brief of placing ‘only the users head’ in a custom scene I.e. face swaps is almost always means swapping two faces within the same photo
  
  
• The outcome is ridiculously unnatural and unconvincing
  
  
• Many do not allow for custom scenes
  
  
• Those that use the more sophisticated AI solutions come close, but produce only a “Likeness” and not a straight swap. Thats right, most AI technologies only provide a face-merge outcome and not a 1:1 head swap. This is understandable due to things like head shape, scale, angle, tilt, skin tone, hair style, position, lighting etc. If interested a good overview of the processes types can be found here 

So what I discovered is that if you want to simply enable your users to place their whole ‘head’ into your own custom still scene that results in a natural and convincing outcome, then there are very, very, few solutions. The answer is in the fashion industry and try ons. Although still not as progressed as you think, there are some customisable solutions starting to emerge. This also has a direct relationship to the Extended Reality (XR) industry, and this the connection to this project…phew.

A major step towards world scale augmented reality has been made with the May 2022 announcements by both Google and Niantic, regarding availability of their separate world scale visual positioning solutions. In short, both organisations have separately released their stage 1 tech to AR developers, and officially entered the next era of 3D mapping our world.

The primary purpose of this tech is to provide the ability for developers and users to create and experience real world Augmented Reality (indoor and outdoor), that has detailed geometric understanding of a location.

Google is leveraging its map, satellite and street view data and technology, while Niantic is using the public, surveyors, and developers to collectively and manually 3D map the world as they play.

This is significant in three ways. Firstly it enables the ability for applications to locate and orientate a user and anchored AR content in the real world within centimetres of accuracy. This is opposed to the current GPS ‘meters’ accuracy option.

Secondly this enables the anchored content to be persistent and consistent. This means it will always be in exactly the same spot when you come back. 

Finally, but as important, it lays the foundation for simultaneous, multi-user experiences i.e. multiple users can interact with the same AR content, at the same time, at the same location.

The Niantic approach provides mapping interiors, such as museums and non-road accessible locations like parks such more accessible.  Google obtains and matches their data from street view. Obviously there are many location you might want to map that do not have a road running through it.

For details on Niantic’s Visual Positioning System (VPS) view 14:20 at https://event.lightshipsummit.com/. 

There is also a recent interview with Brian McClendon – SVP Engineering for AR, Research, & Mapping at Niantic Labs, Inc. the https://youtu.be/wb5Vr64v_Bw?t=344. Encode and audio quality is a little dodgy, but worth watching.

The release of the Google’s Spatial Mapping API can be viewed at https://www.youtube.com/watch?v=pFn11hYZM2E. The whole thing is worth watching. The Google API currently provides the ability to set latitude, longitude and altitude of an asset, as well as determine the users location and orientation in relation to the asset. As they mention Geospatial understanding of the surrounding environment is on the way soon.

This sojourn into the realm of Augmented Reality Content Management Systems (CMS) is a high level pass for digital managers, producers and generalists new to the world of AR. 

For context this research is a result of investigations into an augmented reality solution for a client that had approx. 50 domestic products already available for purchase online. Their high level requirements included:

• Enable the management of many 3D models
• Provide an online 3D model view with interaction
• Enable AR world placement
• Provide a no-code interface for publishing to online devices
• Enable seamless integration into an existing 3rd party website store

Research on augmented reality CMSs will include results from AR platforms such as Zappar, Blippar, 8th Wall and Niantic Lightship. Although these meet most of the requirements, they all seem to fail in at least one criteria e.g. the need for a ‘no-code’ publishing interface, or management of many 3D models. 

The primary reason for this is all are ‘development platforms’ providing  AR SDKs, or minimal/partial code systems. Some also provide infrastructure options to host and publish. All are leading in this tech, but you will need a solid 3D/JS frontend developer to get the job done (Not to mention a 3D modeller). 

It is also worth noting that all of the platforms referenced above have been derived from stream lining their own  studio production systems when building custom projects for their own clients. This is good business, but these solutions are development platforms to help other development companies build ‘custom’ projects for clients, like most Website CMSs. For good reason the tech is no where near the realm of WYSISYG editors like Square, Webflow or Wix.  

That said Zap Works is the closest, with a nice no-code option. I don’t think any offer a straight online 3D viewer option, but I will continue to look.

Blippar does have a no-code interface, but it is equivalent to the more complex Zappar Studio, and is not for someone with no 3D model knowledge. 

So what other options do we have? Well, the world of Augmented Reality Content Management Systems (CMS) does seem to be evolving. There are established enterprise level solutions such as Vuforia, that seem to have recently hit a reasonable level of maturity, but they are expensive, complex and overkill for our basic needs. The most suitable options I have found so far are:

Vntana:  Released in 2021, and still seems to be a Beta release.  Created by the Khronos Group, the grandad of 3D web, you can expect quality. The video demos look quite extensive in regards to the features and asset management aspect. 

I have since had a look at the VNTANA 3D CMS and was quite impressed with what it offers. The key features I would use it for are:

1.  Automated file conversion & optimisation. Injecting native formats e.g. .obj’s automatically provide impressively optimised web viewable Gltfs. This alone can save you a reasonable amount of money & time getting your 3D assets converted correctly.

2. User permission access, at folder and file level, user flows & basic communication systems make it a powerful 3D asset management system.

But, It is very expensive costing at least $US 20K p.a.

Plattar: Following initial tests I can confirm it is a simple and easy system with a single focus, to store, edit and publish basic 3D viewer and augmented reality real world placement. A bit pricey, but that is par for the course considering the early stage of this product type’s lifecycle.

All this said, I am still willing to go out on a limb and make a predication that, “in the very near future I think that as a standard offering, a majority of the well known website CMSs, online web builders, and SaaS ecommerce/web publishing systems, will integrate the ability to manage, publish and view 3D models and configure basic  augmented reality experiences.

WebXR is a new standard being created by the World Wide Web Consortium. This standard replaced the now replaced WebVR standard.

The Immersive Web Working is continuing to extend and lobby for this new standard’s implementation across all major browsers.

It’s basic purpose is to allow the development and delivery of VR and AR applications and experiences via the web, to browsers.

In essence this means that any device with a browser can provide an Augmented or Virtual reality experience, without needing to download a native application from that devices App store. It basically will redefine XR content as ‘just’ another type of content and experience delivered via the internet (Marketers and hype masters call this a metaverse…..ummm right).

At Headfelt XR we are strong advocates of this standard. Why? Because it will enable businesses to more easily leverage their existing web assets and investments, significantly increase innovation, and provide access to XR experiences to everyone with an internet connection.

A couple of examples and sources worth following up on if this is ya thing!

• I suggest get straight to it and blow your mind on either desktop, or VR headset at Mozilla Hello World. Yes this is all web Mozilla WebXR Hello World

• A-frame – Their ‘Hello World’ gives you a taste on desktop, but the point being, it will work on VR headsets A-Frame WebXR Hello World

• If you have a basic appreciation for HTML this gives you a brief explanation of how it all works A-Frame WebXR Overview

• 8th Wall – Basically provides prebuilt WebXR templates to get you started. The AR templates they provide are all immediately usable on your own mobile device 8th Wall Templates. Give ‘em a crack and if you want one built, just give us a ring.

If you are interested in learning more about WebXR, or looking to develop a project, please get in touch.

If you are like the team at Headfelt XR, you just can’t help but listen to industry podcasts while you drive, jog, excercise, surf and sleep.

If your topic of choice is Extended Reality (XR), then we can steer you towards a few that we think are the better ones. We found these to be a limitless pool of insight and understanding into the mysteries of the XR She-beast.

• Everything VR & AR Podcast – Also available on Spotify here.

• The AR Show with Jason McDowall – Available on Spotify here.

• For VR specifc content a Youtube channels worthy of note is ThrillSeeker.

• If you are into your VR Gaming, FReality VR podcast.

Well, if you are any way into Augmented, Virtual or Extended Reality you should be well aware that the 2021 AWE conference was recently held in Santa Clara, California on Nov 9th – 11th. As part of this event they also conduct the annual industry Auggies Awards. If you are not sure who the AWE are, then go here.

In short, it is a massive event for the XR industry, and the first ‘in person’ for a couple of years. The Headfelt XR Team did manage some early starts and attend the online event. Even though the content was good, and your schedule full, at the time it was a bit disappointing how only the main stage streamed. This meant you missed 5 other stages, which was a lot of content, choice, and overall dampened my vibe. The saving grace was that all other stage streams were made available to online ticket holders a week later.

They have now released these all to public, so it is definitely worth your time if you want to learn the current state of play. Go to the AWE USA 2021 playlists at the AWE YouTube Channel. The ‘John Hanke (CEO Niantic) Keynote’ on Augmented Reality and their Lightship platform is especially enlightening.

In addition, the Auggie Awards and nominations were a great indicator of ‘where the industry is currently at’. By this, I mean all entries were accessible too, and voted on by the public prior to the event. I spent many October hours trawling through and learning from the 100’s of entries. I will attempt to add some of my stand outs here. If you are reading this sentence, then I haven’t gotten around to it yet.

So as we all now know augmented reality (AR) is about placing virtual objects (graphics or data) on top of real world vision. There are two main approaches to achieving this.

The first involves the more hyped smart glasses, whereby images are projected onto the lenses of the glasses (e.g. Microsoft Hololens 2 and Magic Leap 1), or directly onto the eye.

The second uses external cameras. These cameras “pass through” the real world view, and add the virtual elements on the way. Think of VR headsets that are fully enclosed, but you can still see the real world through its external cameras. These cameras are common on VR headsets and mostly used for safety reasons.

At Headfelt XR we believe that it is the second, VR headset passthrough, that will offer the more immediate and functional delivery of consumer accessible Augmented Reality. There are a number of reasons for this.

Firstly, the failings of Augmented Reality Smart Glasses/Headset:

• AR Smart glasses, well the cool looking ones, are not yet a reality. Word on the street is saying 2024 – 2025 before a worthy consumer product is available to the masses.
• The current AR Hololens and Magic Leap headsets are enterprise (business) units. They are expensive and not overly robust. Not for the consumer!
• Smart Glasses/Headset image quality and Field of View (FOV) still require significant improvement

In favour of the Virtual Reality headset:

• There is a large, existing VR consumer base
• Devices are cost effective; $500 – $1500
• Full colour VR passthrough will soon be available in most consumers VR devices. Well, much sooner than smart glasses
• VR hardware is already starting to provide the required sensors and processing needed to deliver high quality AR experiences  
• AR features and apps will be a natural extension to the existing VR  experience and offerings.

The reason Virtual Reality  passthrough provides a better quality Augmented Reality experience is that it is not trying to map virtual assets onto real world elements that you are seeing directly with your eye. Passthrough ‘pre-renders’ the virtual assets onto the relevant real world elements in the video stream, before it is displayed to you. This is far more efficient, more robust, and produces a much more immersive result.

Now consider Oculus’ recent release of the Quest 2 API to developers. In short, they are starting to open up access to the camera for developers to start creating this augmented content and applications. Although the current camera is B&W, it is expected that the new Quest (Project Cambria), due for release in early 2022, will have a colour feed.

Once this happens we expect a massive growth in the creation and use of functional consumer focused AR applications. Just imagine the VR headset as more than only an entertainment device, such that it is specifically worn at times when functional AR needs arise e.g. While cooking dinner and following a recipe, or when repairing the lawnmower, learning how to knit, or redesigning the backyard.

Below are some recent, ‘early’ videos that show the foundations of what we will be building upon.

• Cubism Passthrough
• Quest2 Passthrough testing

When you are interested in discussing the potential of this emerging tech, let us know.