python - steps_per_epoch and validation_steps for infinite Dataset in Keras Model

Question

I have a huge dataset of csv files having a volume of around 200GB. I don't know the total number of records in the dataset. I'm using make_csv_dataset to create a PreFetchDataset generator.

I'm facing problem when Tensorflow complains to specify steps_per_epoch and validation_steps for infinite dataset....

1. How can I specify the steps_per_epoch and validation_steps?

2. Can I pass these parameters as the percentage of total dataset size?

3. Can I somehow avoid these parameters as I want my whole dataset to be iterated for each epoch?

I think this SO thread answer the case when we know to total number of data records in advance.

Here is a screenshot from documentation. But I'm not getting it properly.

What does the last line mean?

question from:https://stackoverflow.com/questions/65949227/steps-per-epoch-and-validation-steps-for-infinite-dataset-in-keras-model

Answer 1

I see no other option than iterating through your entire dataset.

ds = tf.data.experimental.make_csv_dataset('myfile.csv', batch_size=16, num_epochs=1)

for ix, _ in enumerate(ds, 1):
    pass

print('The total number of steps is', ix)

Don't forget the num_epochs argument.