python - Is there a way to slice an image using either numpy or opencv such that the sliced image has at least one instance of the objects of interest?

Question

Essentially, my original image has N instances of a certain object. I have the bounding box coordinates and the class for all of them in a text file. This is basically a dataset for YoloV3 and darknet. I want to generate additional images by slicing the original one in a way such that it contains at least 1 instance of one of those objects and if it does, save the image, and the new bounding box coordinates of the objects in that image.

The following is the code for slicing the image:

x1 = random.randint(0, 1200)
width = random.randint(0, 800)
y1 = random.randint(0, 1200)
height = random.randint(30, 800)
slice_img = img[x1:x1+width, y1:y1+height]
plt.imshow(slice_img)
plt.show()

My next step is to use template matching to find if my sliced image is in the original one:

w, h = slice_img.shape[:-1]

res = cv2.matchTemplate(img, slice_img, cv2.TM_CCOEFF_NORMED)
threshold = 0.6
loc = np.where(res >= threshold)
for pt in zip(*loc[::-1]):  # Switch columns and rows
    cv2.rectangle(img, pt, (pt[0] + w, pt[1] + h), (0, 0, 255), 5)

cv2.imwrite('result.png', img)

At this stage, I am quite lost and not sure how to proceed any further. Ultimately, I need many new images with corresponding text files containing the class and coordinates. Any advice would be appreciated. Thank you.

P.S I cannot share my images with you, unfortunately.

question from:https://stackoverflow.com/questions/65830668/is-there-a-way-to-slice-an-image-using-either-numpy-or-opencv-such-that-the-slic

Answer 1

Template matching is way overkill for this. Template matching essentially slides a kernel image over your main image and compares pixels of each, performing many many computations. There's no need to search the image because you already know where the objects are within the image. Essentially, you are trying to determine whether one rectangle (bounding box for an object) overlaps sufficiently with the slice, and you know the exact coordinates of each rectangle. Thus, it's a geometry problem rather than a computer vision problem.

(As an aside: the correct term for what you are calling a slice would probably be crop; slice generally means you're taking an N-dimensional array (say 3 x 4 x 5) and taking a subset of data that is N-1 dimensional by selecting a single index for one dimension (i.e. take index 0 on dimension 0 to get a 1 x 4 x 5 array).

Here's a brief example of how you might do this. Let x1 x2 y1 y2 be the min and max x and y coordinates for the crop you generate. Let ox1 ox2 oy1 oy2 be the min and max x and y coordinates for an object:

  NO_SUCCESSFUL_CROPS = True
  while NO_SUCCESSFUL_CROPS:

    # Generate crop
    x1 = random.randint(0, 1200)
    width = random.randint(0, 800)
    y1 = random.randint(0, 1200)
    height = random.randint(30, 800)
    x2 = x1 + width
    y2 = y1 + height

    # for each bounding box
    #check if at least (nominally) 70% of object is within crop
    threshold = 0.7
    for bbox in all_objects:
      #assign bbox to ox1 ox2 oy1 oy2
      ox1,ox2,oy1,oy2 = bbox 

      # compute percentage of bbox that is within crop
      minx = max(ox1,x1)
      miny = max(oy1,y1)
      maxx = min(ox2,x2)
      maxy = min(oy2,y2)
      area_in_crop = (maxx-minx)*(maxy-miny)
      area of bbox = (ox2-ox1)*(oy2-oy1)
      ratio = area_in_crop / area_of_bbox

      if ratio > threshold:
          # break loop
          NO_SUCCESSFUL_CROPS = False
          
          # crop image as above
          crop_image =  image[y1:y2,x1:x2] # if image is an array, may have to do y then x because y is row and x is column. Not sure exactly which form opencv uses
          cv2.imwrite("output_file.png",crop_image)

          # shift bbox coords since (x1,y1) is the new (0,0) pixel in crop_image
          ox1 -= x1
          ox2 -= x1
          oy1 -= y1
          oy2 -= y2

          break # no need to continue (although you could alternately continue until you have N crops, or even make sure you get one crop with each object)