python cbpro api problems appending data

Question

This script should be printing every order for every second for a two minute duration, but the csv file just has the the same row repeated. Sample data from the csv is below.

import cbpro
import time
import pandas as pd
import os
import json

public_client = cbpro.PublicClient()
res = json.dumps(public_client.get_product_ticker(product_id='BTC-USD'))
csv_file = "cbpro-test-1.csv"
df = pd.DataFrame()  
timeout = time.time() + 60*2 
while True:
    converted = json.loads(res)
    df = df.append(pd.DataFrame.from_dict(pd.json_normalize(converted), orient='columns'))
    if time.time() > timeout:
        break
    
df.to_csv(csv_file, index=False, encoding='utf-8')

here is some sample output of the csv:

trade_id,price,size,time,bid,ask,volume
127344793,32750.24,0.00113286,2021-01-29T06:18:58.637859Z,32750.24,32755.06,41795.68551358
127344793,32750.24,0.00113286,2021-01-29T06:18:58.637859Z,32750.24,32755.06,41795.68551358
127344793,32750.24,0.00113286,2021-01-29T06:18:58.637859Z,32750.24,32755.06,41795.68551358
127344793,32750.24,0.00113286,2021-01-29T06:18:58.637859Z,32750.24,32755.06,41795.68551358

edit: I moved the public client and the res variable to inside the loop and it works somewhat, it skips a second data looks like this now:

127347670,32620.2,0.00307689,2021-01-29T06:33:50.16111Z,32610,32620.12,41966.5764529
127347670,32620.2,0.00307689,2021-01-29T06:33:50.16111Z,32610,32620.12,41966.5764529
127347671,32614.11,0.00146359,2021-01-29T06:33:52.491186Z,32610,32610.01,41966.5764529
127347671,32614.11,0.00146359,2021-01-29T06:33:52.491186Z,32610,32610.01,41966.5764529

it goes from 06:33:50 to 06:33:52, the rest of the file follows the same format

tried with this while loop:

while True:
    public_client = cbpro.PublicClient()
    res = json.dumps(public_client.get_product_ticker(product_id='BTC-USD'))
    converted = json.loads(res)
    df = df.append(pd.DataFrame.from_dict(pd.json_normalize(converted), orient='columns'))
    if time.time() > timeout:
        break

question from:https://stackoverflow.com/questions/65949792/python-cbpro-api-problems-appending-data

Answer 1

You fetch only one quote, before you enter the loop. Then you repeatedly process that same data. You never change res, you simply keep appending the same values to you DF, one iteration after another. You need to fetch repeatedly using get_product_ticker.

After OP update: Yes, that's how to get the quotations rapidly. You can do it better if you move that first line above the loop: you don't need to re-create the client object on every iteration.

Several lines are identical because you're fetching real-time quotations. If nobody changes the current-best bid or ask price, then the quotation remains static. If you want only the changes, then use the unique method of PANDAS to remove the duplicates.