Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
293 views
in Technique[技术] by (71.8m points)

c# - Kinect Depth and Image Frames Alignment

I am playing around with new Kinect SDK v1.0.3.190. (other related questions in stackoverflow are on previous sdk of kinect) I get depth and color streams from Kinect. As the depth and RGB streams are captured with different sensors there is a misalignment between two frames as can be seen below.

Only RGB RGB

Only Depth Depth

Depth & RGB RGB & Depth equally blended

I need to align them and there is a function named MapDepthToColorImagePoint exactly for this purpose. However it doesn't seem to work. here is a equally blended (depth and mapped color) result below which is created with the following code

            Parallel.For(0, this.depthFrameData.Length, i =>
        {
            int depthVal = this.depthFrameData[i] >> 3;
            ColorImagePoint point = this.kinectSensor.MapDepthToColorImagePoint(DepthImageFormat.Resolution640x480Fps30, i / 640, i % 640, (short)depthVal, ColorImageFormat.RgbResolution640x480Fps30);
            int baseIndex = Math.Max(0, Math.Min(this.videoBitmapData.Length - 4, (point.Y * 640 + point.X) * 4));

            this.mappedBitmapData[baseIndex] = (byte)((this.videoBitmapData[baseIndex]));
            this.mappedBitmapData[baseIndex + 1] = (byte)((this.videoBitmapData[baseIndex + 1]));
            this.mappedBitmapData[baseIndex + 2] = (byte)((this.videoBitmapData[baseIndex + 2]));
        });

where

depthFrameData -> raw depth data (short array)

videoBitmapData -> raw image data (byte array)

mappedBitmapData -> expected result data (byte array)

order of the parameters, resolution, array sizes are correct (double checked).

The result of the code is: depth and MapDepthToColorImagePoint

The misalignment continues! What is even worse is that, result image after using MapDepthToColorImagePoint is exactly the same with the original image.

Would be appreciated if someone could help me to find out my mistake or at least explain me what is MapDepthToColorImagePoint for (assuming that I misunderstood its functionality)?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

This will always happen slightly because the two sensors are mounted at slightly different places.

Try it:

Look at some object with your two eyes, then try using only your left eye, then only your right eye. Things look slightly different because your two eyes are not in exactly the same place.

However: it is possible to correct a lot of the issues with some of the API codes.

I'm using Kinect for Windows 1.5, so the API's are slightly different from the 1.0.

short[] depth=new short[320*240];
// fill depth with the kinect data
ColorImagePoint[] colorPoints=new ColorImagePoint[320*240];
// convert mappings
kinect.MapDepthFrameToColorFrame(DepthImageFormat.Resolution320x240Fps30,
            depth, ColorImageFormat.RgbResolution640x480Fps30, colorPoints);
// now do something with it
for(int i=0;i<320*240;i++)
{
  if (we_want_to_display(depth[i]))
  {
    draw_on_image_at(colorPoints[i].X,colorPoints[i].Y);
  }  
}

That's the basics. If you look at the greenscreen example in the Kinect Developer Toolkit 1.5 it shows a good use for this.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...