python - How to load Pix3D voxel data from .mat files in Tensorflow 2 as labels

Question

I am working on a 3D voxel model reconstruction network and I am struggling with loading .mat voxel models (from Pix3D dataset) as labels in TF2. I had two ideas:

1. Using tf.data.Dataset.from_tensor_slices(img_paths, voxel_mat_paths) and then train_ds.map(self.process_path) with scipy.io.loadmat function. However loadmat can't handle string tensor from TF.

def process_path(self, img_path, voxel):
   label = self.get_label(label_path)
   img = tf.io.read_file(img_path)
   img = self.decode_img(img)
   return img, voxel

def get_label(self, voxel_path):
   mat = scipy.io.loadmat(voxel_path)
   return mat['voxel']

2. The second idea was to load all .mat files into a list and then pass it to from_tensor_slices

train_img_paths = [cfg.PIX3D_PATH + example['img'] for example in self.dataset_desc_train]
train_voxel_paths = [cfg.PIX3D_PATH + example['voxel'] for example in self.dataset_desc_train]
train_voxel = [scipy.io.loadmat(path)['voxel'] for path in train_voxel_paths]
train_ds = tf.data.Dataset.from_tensor_slices((train_img_paths, train_voxel))
train_ds = train_ds.map(self.process_path, num_parallel_calls=1)

Then it should be possible to decode image in process_path and pass through voxel data. However, tf.data.Dataset.from_tensor_slices((train_img_paths, train_voxel)) never finishes.

Basically, I want to load images (which works fine) and voxels as labels for these images, however, I am not sure how should I implement this.

question from:https://stackoverflow.com/questions/65864778/how-to-load-pix3d-voxel-data-from-mat-files-in-tensorflow-2-as-labels

Answer 1

Waitting for answers