google project tango - Getting color data in Unity

Question

The release notes mention that it is not yet possible to access color data in Unity, but I believe you can attach a callback to a video overlay listener and receive a YUV12 byte array in Unity and manually convert it into RGB565 to display on screen, perhaps with a shader.

Has anyone had success accessing and displaying color data (say in an AR overlay) in Unity? If so, could you share some pointers/code on how to get started?

Answer 1

The video overlay callback works in Unity. Theoretically, you could have the YUV to RGBA shaders and render it in Unity. The only downside of doing that is it will required multiple "memcpy" due to the way Unity texture/color formatting.

The YV12 data packing of Tango is explained in the C-API header: https://developers.google.com/project-tango/apis/c/struct_tango_image_buffer

It says: "Y samples of width x height are first, followed by V samples, with half the stride and half the lines of the Y data, followed by a U samples with the same dimensions as the V sample."

So I think you could have three Unity Texture each include one of the Y,U,V channel, and pass them into a single shader to compute the RGBA color..

More information about YUV image and the conversion: http://en.wikipedia.org/wiki/YUV