We use cookies to help us improve, promote, and protect our services. By continuing to use the site, you agree to our cookie policy.

Accept
July 5, 2018

How we brought our product mascot to life

Mateusz Tarnaski
« Go back

Short answer: using a browser-based Augmented Reality application. For the long answer, read below.

The idea of playing with AR started as a random interesting experiment; in our company we strive to stay at the edge of the curve, sharing technical novelties and new technologies with each other on a semi-regular basis. Since we are mostly dealing with web technologies, the concept of AR in the browser really took off.

  • However, since AR is mostly an entertainment technology, a practical application was not obvious to us from the start. Luckily, 2 unrelated things happened at the same time: we had just created a mascot for our product - Hubert,
  • we had to do a marketing booth at devoxxPL 2018

We decided to bring Hubert to life during the event in the form of an AR app for people to play with. In our heads, users should be able to:

  • render Hubert on a wall background in their phones
  • take a picture of the rendered model
  • tweet the photo (not the subject of this article)

The end result is available on glitch.com, scaled down and rotated to be suitable for a desktop experience (you can also take a quick look into the source code).

Rendering Hubert in real time

We used AR.js (version from this commit) as the main building block of our AR app - it packages webRTC camera access, marker recognition, and 3D scene rendering. We liked it mostly because you can have a basic demo running in around 20 lines of code (how cool is that).

Under the hood, AR.js can use either three.js or a-frame implementations to render your 3D scenes.

  • three.js offers fine-grained control of 3D rendering, is javascript-based, and you have probably heard about it in the context of rendering 2D and 3D scenes in the browser
  • a-frame is a web framework designed specifically for building VR and AR experiences, it has an html-like markup that is more declarative than three.js, but sacrifices some of the control in favor of ease of use

We didn’t have a VR or 3D expert (except Mrówa, who prepared the 3D model) and a-frame’s html-like declarative syntax looked more familiar to us, we opted for a-frame to do the rendering.

Here you can see the code for rendering Hubert, 20 lines on the dot (we omitted some options and a-frame tweaking for the sake of simplicity, refer to the repo to see it all):

{{CODE}}

<head>

 <!-- include a-frame -->

 <script id="aframe" src='./vendor/aframe/build/aframe-v0.8.2.min.js'></script>

 <script id="aframe-ar" src='./build/aframe-ar.min.js'></script>

</head>

<body>

 <!-- Define your scene -->

 <a-scene antialias="true" embedded arjs='trackingMethod: best;'>

   <a-assets>

     <a-asset-item

                   id='bercik-obj'

                   src='./models/smoothHubert/bercik2.obj'

     ></a-asset-item>

     <a-asset-item

                   id='bercik-mtl'

                   src='./models/smoothHubert/bercik2.mtl'

     ></a-asset-item>

   </a-assets>

   <!-- Create a anchor to attach your augmented reality -->

   <a-anchor hit-testing-enabled='true'>

     <a-entity

               obj-model="obj: #bercik-obj; mtl: #bercik-mtl"

               scale="0.1 0.1 0.1"

               rotation="-90 0 0"

     ></a-entity>

</a-anchor>

<!-- Define a static camera -->

<a-camera-static/>

</a-scene>

</body>

{{ENDCODE}}

This gives us Hubert nicely rendered in the web browser in real time.

Capturing a photo to tweet

Unfortunately, we don’t have a single video feed rendering the whole scene. There is the video from your camera and a rendered 3D scene. We quickly figured out we will have to capture a frame from both sources and put them together for a nice photo of Hubert.

Taking frames out of a webRTC video stream is pretty straightforward. The best material on the subject can be found here:

https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Taking_still_photos. If your browser has the appropriate API, you need 2 elements:

  • a reference to your source <video/> tag
  • a destination <canvas/> element to put your frame in

Then, it’s just a simple matter of drawing a 2D image from video to canvas. In our case, both of these are a bit tricky.

The video take we are using is generated and embedded by AR.js and we had no idea how to get it gracefully, so we hacked our way around it with a loop and a DOM selector:

{{CODE}}

let videoElement = null

const getVideoElement = () => new Promise(resolve => {

 videoElement = document.querySelector('video')

 if (videoElement) {

   resolve(videoElement);

   return

 }

 setTimeout(() => resolve(getVideoElement()), 300)

})

getVideoElement().then(video => { /* do your thing */ })

{{ENDCODE}}

The other thing that we needed to hack around is some scaling. AR.js doesn’t present the raw video feed to the user, they scale it to fill the screen without losing aspect ratio. That means we need to apply the same scaling to our frame, otherwise, our screenshot will have ‘more’ of the video feed than is shown on the screen and we don’t want to confuse the users here.

what user sees:

if we take a frame without scaling and just try to copy from point (0,0) we lose margins imposed by AR.js, which is a totally different picture from what is presented to the user:

Suffice it to say we just reverse-engineered the scaling and figured out the bounding box of what the user sees:

{{CODE}}

const topOffset = -parseFloat(video.style['margin-top'])

const leftOffset = -parseFloat(video.style['margin-left'])

const relativeTopOffset = topOffset / parseFloat(video.style.height)

const relativeLeftOffset = leftOffset / parseFloat(video.style.width)

const {videoWidth, videoHeight} = video

const partPresentedToTheUser = {

 x: videoWidth*relativeLeftOffset,

 y: videoHeight*relativeTopOffset,

 width: videoWidth*(1-2*relativeLeftOffset),

 height: videoHeight*(1-2*relativeTopOffset)

}

{{ENDCODE}}

To achieve this final result (the same as what is presented live to the user, give or take some camera shake):

Now we just need to get Hubert in the picture. Again, the API for that is very straightforward: to capture a screenshot of a rendered a-frame scene, we need to get the scene’s canvas and copy the relevant part to our destination canvas, on top of the previously taken video frame.

{{CODE}}

const scene = document.querySelector('a-scene')

scene.components.screenshot.data.width = width

scene.components.screenshot.data.height = height

const sceneCanvas = scene.components.screenshot.getCanvas('perspective')

const destinationContext = destinationCanvas.getContext('2d')

destinationContext.drawImage(

 sceneCanvas,

 box.x, box.y, box.width, box.height, // source rect

 -box.x, 0, box.width, box.height // destination rect

)

{{ENDCODE}}

Getting the relevant part is the tricky bit in our case; thanks to the AR.js scaling we cannot simply get the `perspective` shot of the scene and use that - it will look too wide or too short, depending on orientation.

For landscape mode (width > height), the scaling method we used for video works perfectly well.

For portrait mode, it works great on a PC… However, once you enter the realm of mobile devices, the scaling breaks and the screenshot doesn’t look nice, you get this skinny Hubert:

Instead of our lovely, bubbly mascot in all his glory:

We are still not sure why exactly that is the case and we made the mistake of not testing it out thoroughly on actual mobile devices, thinking it would work the same as it does on the development machine (yes, we know how bad that sounds, but that’s the reality of it). During the actual conference, we managed to figure out the formula for portrait scaling and introduced a fix:

{{CODE}}

if (portraitMode()) {

 context.drawImage(

   sceneCanvas,

   box.x, box.y, box.width, box.height, // source rect

   -(box.width/box.height * box.width*0.55), 0, 1.7*box.width, box.height //destination rect

 )

 return

}

{{ENDCODE}}

It’s not pretty. It’s one of those “it’s late, it works, just leave it” fixes. The values presented above produced a satisfactory result and we left it at that.

With that, we have a picture of Hubert in the real world! It can be retrieved from the destination canvas element and displayed on the page or sent to the server to tweet out.

Summary

AR in the browser is possible. Even more, it is possible on mid-grade mobile hardware (as of June 2018). Getting it to work on every phone and browser is still a long shot, so don’t count on it for wide, diverse userbases. However, if you have a somewhat controlled environment, augmented reality on a phone can be used to create unique experiences that don’t require special hardware or workstations and that is a big, big plus. Just make sure to test it on actual devices ahead of time ;).

Tagged:
API
,  
tips
,  
« Go back
Related posts