Poor Marker Tracking detection in Camera Kit for Web

Tavius Koktavy

When on a webpage, marker tracking is very difficult to activate. I've tried using the image tracking template marker and a custom image with good contrast, feature points, and asymmetry.

The tracking runs well when testing the lens in Snapchat. I've tried with the template Mark character and with a simple cube to rule out the fidelity of AR content impacting the web performance. I've tested te website on Pixel 7, iPhone 12 Pro, and from my laptop's webcam without improvement.

Is there something that needs to be changed to improve detection stability on the web? An expected aspect ratio? Targeting specific phone cameras? Any leads are appreciated!

Michael Mase

Hi @Tavius Koktavy, when setting your Camera Kit source, are you using { cameraType: 'back' }?

For example:

const mediaStream = navigator.mediaDevices.getUserMedia({
  video: { width: 1280, height: 720 },
});
const source = createMediaStreamSource(mediaStream, { cameraType: 'back ' });
await session.setSource(source);

Kevando

Tavius!

Tavius Koktavy

That fixed it, thank you!

I didn't realize that also needed to be set in the session source, I thought it was handled by the 'facingMode' property in the mediaStream.

My code is now:

    const mediaStream = await window.navigator.mediaDevices.getUserMedia({
      video: {
        facingMode: 'environment' // 'environment' for rear camera, 'user' for front
      }
    });

    const source = createMediaStreamSource(mediaStream, { cameraType: 'back' });
    await session.setSource(source);
    await session.play();

p.s. hey Kevin!

Tavius Koktavy

For feedback, I'm looking back at the samples and docs to try to find where I might have discovered the cameraType param on my own >>>

---

I see it here (https://docs.snap.com/camera-kit/integrate-sdk/web/web-configuration#camera-type) with a somewhat nondescript note that 'back' should be used "for proper rendering." Expanding the back facing camera use cases to "(e.g. World AR, Marker Tracking, Body Tracking)" may help when trying to search around this issue.

IMO the "Putting it all together" section is also a bit buried in the middle of the doc. Perhaps it could be linked to near the top of the docs page, or shown as an MVP example before the "Bootstrapping the SDK" section?

---

In my search I've also discovered the API reference docs. I had a difficult time finding the CameraKitDeviceInfo type that actually provides the cameraType option, but I see better how it works now.

---

From my current perspective, the ideal developer experience would have the session source automatically update it's cameraType param based on project info from the Lens.

I'll be reading the Web SDK setup docs more closely moving forwards!