pagination - GitHub API - Different number of results for jq filtered response

Question

I am trying to use the GitHub API to make a list of well documented, open source Java libraries. To do so, I went through the GitHub API documentation and made this simple curl.

curl -G  https://api.github.com/search/repositories?q=language:Java+stars:%3E=500+library+java+in:readme > output1.txt

The output of this is a giant txt file, containing information about all of the repositories found. In this example, there was a total of 736 matches. However, the file from the command above is quite unreadable, so I decided to do some parsing using jq, which resulted in the following code:

curl -G  https://api.github.com/search/repositories?q=language:Java+stars:%3E=500+library+java+in:readme 
 | jq ".items[] | {name, description, language, watchers_count, html_url}" > parsedOutput1.txt

After this, instead of 736 results, I got something around 30 repositories, which is unacceptable for my purposes.

Doing this search: language:java stars:>=500 java library in:readme in the GitHub search box gives me the same 736 results. I don't really know what i am doing wrong so I could use the help.

Answer 1

It was a paging problem, as presented in the documentation, the api only gives you 30 items per request, so you need to add some code to include all the pages. I was using bash so my code ended up like this:

 for i in `seq 1 34`;
        do
            URL="https://api.github.com/search/repositories?q=language:Java+stars:%3E=500+library+java+in:readme&page=$i"
            echo $URL
            curl -G  $URL 
            | jq ".items[] | {name, description, language, watchers_count, html_url}" >> parsedOutput1.txt
        done

On another note, when doing a lot of requests you should authenticate, otherwise you will end up with the API rate limit exceeded message.