c# - Convert OpenCV Mat to Texture2D?

Question

Meta Context:

I'm currently working on a game that utilizes opencv as a substitute for ordinary inputs (keyboard, mouse, etc...). I'm using Unity3D's C# scripts and opencv in C++ via DllImports. My goal is to create an image inside my game coming from opencv.

Code Context:

As done usually in OpenCV, I'm using Mat to represent my image. This is the way that I'm exporting the image bytes:

cv::Mat _currentFrame;

...

extern "C" byte * EXPORT GetRawImage()
{
    return _currentFrame.data;
}

And this is how i'm importing from C#:

[DllImport ("ImageInputInterface")]
private static extern IntPtr GetRawImage ();

...

public static void GetRawImageBytes (ref byte[] result, int arrayLength) {
    IntPtr a = GetRawImage ();
    Marshal.Copy(a, result, 0, arrayLength);
    FreeBuffer(a);
}

Judging by the way I understand OpenCV, I expect the byte array to be structured in this way when serialized in a uchar pointer:

b1, g1, r1, b2, g2, r2, ...

I'm converting this BGR array to a RGB array using:

public static void BGR2RGB(ref byte[] buffer) {
    byte swap;
    for (int i = 0; i < buffer.Length; i = i + 3) {
        swap = buffer[i];
        buffer[i] = buffer[i + 2];
        buffer[i + 2] = swap;
    }
}

Finally, I'm using Unity's LoadRawTextureData to load the bytes to a texture:

this.tex = new Texture2D(
    ImageInputInterface.GetImageWidth(),
    ImageInputInterface.GetImageHeight(),
    TextureFormat.RGB24,
    false
);

...

ImageInputInterface.GetRawImageBytes(ref ret, ret.Length);
ImageInputInterface.BGR2RGB(ref ret);
tex.LoadRawTextureData(ret);
tex.Apply();

Results:

The final image seems to be scattered in someway, it resembles some shapes, but it seems to triple the shapes as well. This is me holding my hand in front of the camera:

[Me, my hand and the camera]

Doing some tests, I concluded that I decoded the channels correctly, since, using my phone to emit RGB light, I can reproduce the colors from the real world:

[Red Test]

[Blue Test]

[Green Test]

There are also some strange lines in the image:

[Spooky Lines]

There is also my face to compare these images to:

[My face in front of the camera]

Questions:

Since I'm able to correctly decode the color channels, what have I assumed wrong in decoding the OpenCV array? It's that I don't know how the Unity's LoadRawTextureData works, or have I decoded something in the wrong way?

How is the OpenCV Mat.data array structured?

UPDATE

Thanks to @Programmer, his solution worked like magic.

[Me Happy]

I changed his script a little, there was no need to do some stuff. And in my case i needed to use BGR2RGBA, not RGB2RGBA:

extern "C" void EXPORT GetRawImage( byte *data, int width, int height )
{
    cv::Mat resizedMat( height, width, _currentFrame.type() );
    cv::resize( _currentFrame, resizedMat, resizedMat.size(), cv::INTER_CUBIC );

    cv::Mat argbImg;
    cv::cvtColor( resizedMat, argbImg, CV_BGR2RGBA );
    std::memcpy( data, argbImg.data, argbImg.total() * argbImg.elemSize() );
}

Answer 1

Use SetPixels32 instead of LoadRawTextureData. Instead of returning the array data from C++, do that from C#. Create Color32 array and pin it in c# with GCHandle.Alloc, send the address of the pinned Color32 array to C++, use cv::resize to resize the cv::Mat to match the size of pixels sent from C#. You must do this step or expect some error or issues.

Finally, convert cv::Mat from RGB to ARGB then use std::memcpy to update the array from C++. The SetPixels32 function can then be used to load that updated Color32 array into Texture2D. This is how I do it and it has been working for me without any issues. There might be other better ways to do it but I have never found one.

C++:

cv::Mat _currentFrame;

void GetRawImageBytes(unsigned char* data, int width, int height)
{
   //Resize Mat to match the array passed to it from C#
    cv::Mat resizedMat(height, width, _currentFrame.type());
    cv::resize(_currentFrame, resizedMat, resizedMat.size(), cv::INTER_CUBIC);

    //You may not need this line. Depends on what you are doing
    cv::imshow("Nicolas", resizedMat);

    //Convert from RGB to ARGB 
    cv::Mat argb_img;
    cv::cvtColor(resizedMat, argb_img, CV_RGB2BGRA);
    std::vector<cv::Mat> bgra;
    cv::split(argb_img, bgra);
    std::swap(bgra[0], bgra[3]);
    std::swap(bgra[1], bgra[2]);
    std::memcpy(data, argb_img.data, argb_img.total() * argb_img.elemSize());
}

---

C#:

Attach to any GameObject with a Renderer and you should see the cv::Mat displayed and updated on that Object every frame. Code is commented if confused:

using System;
using System.Runtime.InteropServices;
using UnityEngine;

public class Test : MonoBehaviour
{
    [DllImport("ImageInputInterface")]
    private static extern void GetRawImageBytes(IntPtr data, int width, int height);

    private Texture2D tex;
    private Color32[] pixel32;

    private GCHandle pixelHandle;
    private IntPtr pixelPtr;

    void Start()
    {
        InitTexture();
        gameObject.GetComponent<Renderer>().material.mainTexture = tex;
    }


    void Update()
    {
        MatToTexture2D();
    }


    void InitTexture()
    {
        tex = new Texture2D(512, 512, TextureFormat.ARGB32, false);
        pixel32 = tex.GetPixels32();
        //Pin pixel32 array
        pixelHandle = GCHandle.Alloc(pixel32, GCHandleType.Pinned);
        //Get the pinned address
        pixelPtr = pixelHandle.AddrOfPinnedObject();
    }

    void MatToTexture2D()
    {
        //Convert Mat to Texture2D
        GetRawImageBytes(pixelPtr, tex.width, tex.height);
        //Update the Texture2D with array updated in C++
        tex.SetPixels32(pixel32);
        tex.Apply();
    }

    void OnApplicationQuit()
    {
        //Free handle
        pixelHandle.Free();
    }
}