algorithm - Augment reality like zookazam

Question

1. What algorithms are used for augmented reality like zookazam ?

   

   I think it analyze image and find planes by contrast, but i don't know how.

2. What topics should I read before starting with app like this?

Answer 1

[Prologue]

This is extremly broad topic and mostly off topic in it's current state. I reedited your question but to make your question answerable within the rules/possibilities of this site

You should specify more closely what your augmented reality:

1. should do

   • adding 2D/3D objects with known mesh ...
   • changing light conditions
   • adding/removing body parts/clothes/hairs ...

   a good idea is to provide some example image (sketch) of input/output of what you want to achieve.

2. what input it has

   • video,static image, 2D,stereo,3D. For pure 2D input specify what conditions/markers/illumination/LASER patterns you have to help the reconstruction.
   • what will be in the input image? empty room, persons, specific objects etc.

3. specify target platform

   many algorithms are limited to memory size/bandwidth, CPU power, special HW capabilities etc so it is a good idea to add tag for your platform. The OS and language is also a good idea to add.

[How augmented reality works]

1. acquire input image

   if you are connecting to some device like camera you need to use its driver/framework or something to obtain the image or use some common API it supports. This task is OS dependent. My favorite way on Windows is to use VFW (video for windows) API.

   I would start with some static file(s) from start instead to ease up the debug and incremental building process. (you do not need to wait for camera and stuff to happen on each build). And when your App is ready for live video then switch back to camera...

2. reconstruct the scene into 3D mesh

   if you use 3D cameras like Kinect then this step is not necessary. Otherwise you need to distinguish the object by some segmentation process usually based on the edge detections or color homogenity.

   The quality of the 3D mesh depends on what you want to achieve and what is your input. For example if you want realistic shadows and lighting then you need very good mesh. If the camera is fixed in some room you can predefine the mesh manually (hard code it) and compute just the objects in view. Also the objects detection/segmentation can be done very simply by substracting the empty room image from current view image so the pixels with big difference are the objects.

   you can also use planes instead of real 3D mesh as you suggested in the OP but then you can forget about more realistic quality of effects like lighting,shadows,intersections... if you assume the objects are standing straight then you can use room metrics to obtain the distance from camera. see:

   • selection criteria for different projections
   • estimate measure of photographed things

   For pure 2D input you can also use the illumination to estimate the 3D mesh see:

   • Turn any 2D image into 3D printable sculpture with code

3. render

   Just render the scene back to some image/video/screen... with added/removed features. If you are not changing the light conditions too much you can also use the original image and render directly to it. Shadows can be achieved by darkening the pixels ... For better results with this the illumination/shadows/spots/etc. are usually filtered out from the original image and then added directly by rendering instead. see

   • White balance (Color Suppression) Formula?
   • Enhancing dynamic range and normalizing illumination

   The rendering process itself is also platform dependent (unless you are doing it by low level graphics in memory). You can use things like GDI,DX,OpenGL,... see:

   • Graphics rendering

   You also need camera parameters for rendering like:

   • Transformation of 3D objects related to vanishing points and horizon line

[Basic topics to google/read]

1. 2D

   • DIP digital image processing
   • Image Segmentation

2. 3D

   • Vector math
   • Homogenous coordinates
   • 3D scene reconstruction
   • 3D graphics
   • normal shading

3. paltform dependent

   • image acquisition
   • rendering