Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
2.7k views
in Technique[技术] by (71.8m points)

python - How to filter s3 objects by last modified date with Boto3

Is there a way to filter s3 objects by last modified date in boto3? I've constructed a large text file list of all the contents in a bucket. Some time has passed and I'd like to list only objects that were added after the last time I looped through the entire bucket.

I know I can use the Marker property to start from a certain object name,so I could give it the last object I processed in the text file but that does not guarantee a new object wasn't added before that object name. e.g. if the last file in the text file was oak.txt and a new file called apple.txt was added, it would not pick that up.

s3_resource = boto3.resource('s3')
client = boto3.client('s3')

def list_rasters(bucket):

    bucket = s3_resource.Bucket(bucket)

    for bucket_obj in bucket.objects.filter(Prefix="testing_folder/"):
        print bucket_obj.key
        print bucket_obj.last_modified
See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

The following code snippet gets all objects under specific folder and check if the file last modified is created after the time you specify :

Replace YEAR,MONTH, DAY with your values.

import boto3
import datetime
#bucket Name
bucket_name = 'BUCKET NAME'
#folder Name
folder_name = 'FOLDER NAME'
#bucket Resource
s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket_name)    
def lambda_handler(event, context):
     for file in bucket.objects.filter(Prefix= folder_name):
         #compare dates 
         if (file.last_modified).replace(tzinfo = None) > datetime.datetime(YEAR,MONTH, DAY,tzinfo = None):
             #print results
             print('File Name: %s ---- Date: %s' % (file.key,file.last_modified))

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...