pandas - creating H5 Database Error with large amount of data

Question

Im trying to train my own data in Pointnet. The issue is that other than 3 axis of point cloud, there are 15 other variables. I tried to create a H5 database using this way. The filenames file contains all the filenames of ply data and the database contains 1 column with name as ASSEMBLY_NAME containing all the filenames of ply data and the other 14 variables.

import h5py
import numpy as np
from plyfile import PlyData, PlyElement
import pandas as pd

filenames = [line.rstrip() for line in open("filelist", 'r')]

f = h5py.File("./hdf5_data/data_training.h5", 'w')
#f = h5py.File("./hdf5_data/data_testing.h5", 'w')

a_data = np.zeros((len(filenames), 2048, 3))
a_pid = np.zeros((len(filenames), 15), dtype = np.float64)
for i in range(0, len(filenames)):
    plydata = PlyData.read("./" + filenames[i] + ".ply")
    #piddata = [line.rstrip() for line in open("./points_label/" + filenames[i] + ".seg", 'r')]
    data = pd.read_csv('./Database/DATABASE.csv', index_col ="ASSEMBLY_NAME")
    req_data =  data.loc[filenames[i]]
    print(req_data.head)
    for j in range(0, 15):
        a_data[i, j] = [plydata['vertex']['x'][j], plydata['vertex']['y'][j], plydata['vertex']['z'][j]]
        a_data[i,j] = req_data[j]

The Error I'm getting is:

<bound method NDFrame.head of AssetID       NaN
Asset Name    NaN
OffsetX       NaN
OffsetY       NaN
OffsetZ       NaN
Matrix a1     NaN
Matrix b1     NaN
Matrix c1     NaN
Matrix a2     NaN
Matrix b2     NaN
Matrix c2     NaN
Matrix a3     NaN
Matrix b3     NaN
Matrix c3     NaN
KeyPARAM      NaN
Name: scene0002_00_vh_clean_2, dtype: object>
<bound method NDFrame.head of AssetID       NaN
Asset Name    NaN
OffsetX       NaN
OffsetY       NaN
OffsetZ       NaN
Matrix a1     NaN
Matrix b1     NaN
Matrix c1     NaN
Matrix a2     NaN
Matrix b2     NaN
Matrix c2     NaN
Matrix a3     NaN
Matrix b3     NaN
Matrix c3     NaN
KeyPARAM      NaN
Name: scene0003_00_vh_clean_2, dtype: object>
<bound method NDFrame.head of AssetID       NaN
Asset Name    NaN
OffsetX       NaN
OffsetY       NaN
OffsetZ       NaN
Matrix a1     NaN
Matrix b1     NaN
Matrix c1     NaN
Matrix a2     NaN
Matrix b2     NaN
Matrix c2     NaN
Matrix a3     NaN
Matrix b3     NaN
Matrix c3     NaN
KeyPARAM      NaN
Name: scene0004_00_vh_clean_2, dtype: object>
<bound method NDFrame.head of AssetID       NaN
Asset Name    NaN
OffsetX       NaN
OffsetY       NaN
OffsetZ       NaN
Matrix a1     NaN
Matrix b1     NaN
Matrix c1     NaN
Matrix a2     NaN
Matrix b2     NaN
Matrix c2     NaN
Matrix a3     NaN
Matrix b3     NaN
Matrix c3     NaN
KeyPARAM      NaN
Name: scene0005_00_vh_clean_2, dtype: object>
<bound method NDFrame.head of                                                       AssetID Asset Name  
ASSEMBLY_NAME                                                              
scene0006_00_vh_clean_2  213a60dd-2f9f-4750-ac25-68c53ceefec5    Chair:2   
scene0006_00_vh_clean_2  213a60dd-2f9f-4750-ac25-68c53ceefec5    Chair:1   

                            OffsetX     OffsetY    OffsetZ     Matrix a1  
ASSEMBLY_NAME                                                              
scene0006_00_vh_clean_2  331.718156  617.190251   1.558245  7.400000e-15   
scene0006_00_vh_clean_2  277.194912  642.098488  12.850805 -2.100000e-27   

                            Matrix b1  Matrix c1     Matrix a2     Matrix b2  
ASSEMBLY_NAME                                                                  
scene0006_00_vh_clean_2 -2.090000e-27       -1.0  3.020000e-14 -1.000000e+00   
scene0006_00_vh_clean_2 -1.960000e-15       -1.0 -1.000000e+00 -2.850000e-14   

                            Matrix c2   Matrix a3   Matrix b3  Matrix c3  
ASSEMBLY_NAME                                                              
scene0006_00_vh_clean_2  2.310000e-27  277.194912  642.098488  12.850805   
scene0006_00_vh_clean_2  2.150000e-27  331.718156  617.190251   1.558245   

                        KeyPARAM  
ASSEMBLY_NAME                     
scene0006_00_vh_clean_2      NaN  
scene0006_00_vh_clean_2      NaN  >
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
c:userslenovoappdatalocalprogramspythonpython36libsite-packagespandascoreindexesase.py in get_loc(self, key, method, tolerance)
   2894             try:
-> 2895                 return self._engine.get_loc(casted_key)
   2896             except KeyError as err:

pandas\_libsindex.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas\_libsindex.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas\_libshashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas\_libshashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 0

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
<ipython-input-14-fc9112b14f14> in <module>
     19     for j in range(0, 15):
     20         a_data[i, j] = [plydata['vertex']['x'][j], plydata['vertex']['y'][j], plydata['vertex']['z'][j]]
---> 21         a_data[i,j] = req_data[j]
     22 

c:userslenovoappdatalocalprogramspythonpython36libsite-packagespandascoreframe.py in __getitem__(self, key)
   2904             if self.columns.nlevels > 1:
   2905                 return self._getitem_multilevel(key)
-> 2906             indexer = self.columns.get_loc(key)
   2907             if is_integer(indexer):
   2908                 indexer = [indexer]

c:userslenovoappdatalocalprogramspythonpython36libsite-packagespandascoreindexesase.py in get_loc(self, key, method, tolerance)
   2895                 return self._engine.get_loc(casted_key)
   2896             except KeyError as err:
-> 2897                 raise KeyError(key) from err
   2898 
   2899         if tolerance is not None:

KeyError: 0

I would really appreciate any help. Thank you in advance

EDIT*** these are the ply format scans opened in 3d viewer.All these scans are scannet data converted into ply format. I have also added a database image I am currently using. enter image description here

Answer 1

等待大神答复