matlab - Detecting edge only in particular portion of an image

Question

I want to detect the edge only in the red marked region as shown in the image below:

Answer 1

A few suggestions. I assume that the red region is input by mouse and that you now have a mask of the region that you want to include in the edge search.

My proposed algorithm

1. Do Edge detection
2. Write your own Hough routine but only count edges if they should be included according to the mask. 
3. Pick the edge with the best score in the Hough space. 

Of course you don't need to run the edge detection on the complete image but if you don't make sure that you handle the border of your search area (so you don't get edges there). Simply mirroring the area might work.

---

Update

Okay, different approach:

Use the hough routines in matlab. houghlines, hough, houghpeaks are the relevant functions. If only one line intersects your region of interest, you are done. The line is the result you want.

If more than one line intersect the region of interest, you need to do a bit more. I'd suggest counting the number of pixels along the line that are within the ROI. So, if the line intersects the ROI for 10 pixels, that line's score is 10. Do this for all lines and then pick the line with the highest score.

Note that none of the approaches are optimized for speed. However, they are easy to understand.