Usually, the data range I got each day should be over 100,000 records. But when I setup 2 bigquery transfer (one for each owner id), use the same suffix, so the data goes to the same table. the transfer is missing a lot of data.
The transfer status shows successful but somedays I only get a 4000-5000 records instead of 100,000+ records. But somedays I got 100,000 records.
Question 1:
Is that possible bigquery transfer only finishes transferring partial of the data? My understanding is bigquery transfer is "all or none". If it is successful, then all the transfer is done. If it fails, zero records transferred.
Although, I do see in some cases that bigquery failed but status shows complete.
Question 2:
When I set up two transfers, both transfers sending data to a same table, I can see the failure rate increases hugely.
Is it possible the two transfers somehow conflict with each other in the process?
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…