Python file open/close every time vs keeping it open until the process is finished

Question

I have about 50 GB of text file and I am checking the first few characters each line and writing those to other files specified for that beginning text.

For example. my input contains:

cow_ilovecow
dog_whreismydog
cat_thatcatshouldgotoreddit
dog_gotitfromshelter
...............

So, I want to process them in cow, dog and cat (about 200) categories so,

if writeflag==1:
    writefile1=open(writefile,"a") #writefile is somedir/dog.txt....
    writefile1.write(remline+"
")
    #writefile1.close()

so, what is the best way, should I close? Otherwise if I keep it open, is writefile1=open(writefile,"a") doing the right thing?

question from:https://stackoverflow.com/questions/11349020/python-file-open-close-every-time-vs-keeping-it-open-until-the-process-is-finish

Answer 1

You should definitely try to open/close the file as little as possible

Because even comparing with file read/write, file open/close is far more expensive

Consider two code blocks:

f=open('test1.txt', 'w')
for i in range(1000):
    f.write('
')
f.close()

and

for i in range(1000):
    f=open('test2.txt', 'a')
    f.write('
')
    f.close()

The first one takes 0.025s while the second one takes 0.309s