Products Search… Join this community

Home
How to append several dictionaries while looping through pagination (API) - Ask Question

PUBLIC python 3
Questions Asked 2 years, 9 months ago Active 2 years, 9 months ago Viewed 989 times

Tags

Users The Overflow Blog

COLLECTIVES Communities are a catalyst for technology

development
Explore Collectives
Report this ad
Podcast 363: Highlights from our 2021
FIND A JOB Developer Survey
I would like to loop through several pages of data which I am accessing through an API, and
Jobs concatenate the pages to the same dictionary as the loop progresses.
Featured on Meta
1
Companies So far I managed to loop through all the pages and could print out as the loop goes on, so when
Join me in Welcoming Valued Associates:
the loop is over, all the pages are printed out. #945 - Slate - and #948 - Vanny
TEAMS

But my dictionary contains only the last page!

Stack Overflow for
Teams – Collaborate
and share knowledge page = 5
with a private group. i = 0
for i in range(0, page):
url =f'http://hotell.difi.no/api/json/npd/wellbore/withcoordinates?page={i+1}'

dataset_all
Join Stack Overflow to learn, share knowledge, = requests.get(url)
and build your career. Sign up with email Sign up with Google Sign up with GitHub Sign up with Facebook
json_dataset_all = dataset_all.json()
json_dataset_all.update()
print(json_dataset_all)
i = i+1

I am not succeeding in going through the loop and updating my dictionary page1 with page2,
Report this ad
page3, page4 and page5.

The 'print' shows that the loop through the pages works, but it's not storing the pages in the
dictionary as it progresses. It only contains the latest page.

Thanks in advance for help.

Loadsmart
Chicago, IL
python dictionary for-loop pagination concatenation
We have great benefits!
Share Improve this question Follow edited Oct 12 '18 at 1:36 asked Oct 11 '18 at 20:32 Competitive Payment
PythonSherpa AnneE
International Environment/Career
2,205 2 14 34 13 3
Unlimited Paid Vacations
Add a comment
Building a Rapidly-Growing Tech Company

+ 5 more benefits
1 Answer Active Oldest Votes

Learn more
The pages have an identical "key" called "entries" and it's value is a list of dictionaries. When you
update your dictionary, it will therefore just overwrite its value (the list of dictionaries).
1 Linked
If you want to have one single list with all the dictionaries, you could do something like this. You
can then also easily put it in a Pandas DataFrame. 1 How do I append the datasets from
pagination responses in Python?

import requests
import pandas as pd Related

page = 5 0 Pagination for PHP search results

i = 0
all_data_list = [] 663 How do I append one string to another in
for i in range(0, page): Python?
url =f'http://hotell.difi.no/api/json/npd/wellbore/with-coordinates?page={i+1}'
1 CodeIgniter Pagination - first page’s link
dataset_all = requests.get(url) doesn’t work, why?
dataset_all_json = dataset_all.json()
0 Next and Previous Logic in e-reader
number_of_entries = len(dataset_all_json['entries'])
application in Python?

for x in range(0, number_of_entries): 0 Looping through multiple pages python +

all_data_list.append(dataset_all_json['entries'][x]) beautiful soup

i = i+1 0 how to navigate to other pages when

pagination exists in the URL
# Put all the results in a Pandas DataFrame
0 How to loop multiple page on selenium
df = pd.DataFrame(all_data_list)
python BeautifulSoup
print(df)

Hot Network Questions

(by the way, your URL is missing the '-' in "withcoordinates")
Share Improve this answer Follow edited Oct 12 '18 at 7:27 answered Oct 11 '18 at 21:06
PythonSherpa
2,205 2 14
Is my "SNSP-001A (FRG)" SNES a one-chip or two-
chip SNES?
Are there physicists who had no formal education
before they became an accepted member of the
34 established community and also famous?

How can a simple physical situation give rise to

Thanks, this helps, but. I do now have a complete list, while I was hoping for a complete dictionary. The next two different possible outcomes?
step is to put everything into a panda dataframe, with a row per well. I could do this from the code I posted Can the fish topple the bowl?
at the top, but again that was working only for the last page... (sorry for the missing '-') – AnneE Oct 11 '18
at 23:58 Should I be paid the original advertised amount as
that was substantially more than what's in my
contract?
Ok, you didn't mention that in your question. I added the code to put the data in a DataFrame. –
PythonSherpa Oct 12 '18 at 7:29 How can I submit jobs to an HPC scheduler?

Ok cool, this is working fine now! By just running 'df' without 'print', I got now the result I wished for! Supert, Choosing reaction stoichiometry in organic
thanks a lot :-) – AnneE Oct 12 '18 at 13:23 synthesis

How is data transferred between devices?

Add a comment
How to determine if a point is just a point or a
valid public key?

How and why did Disney reuse animation?

Your Answer
Why is merging at center in Blender
deleting/deselecting my vertex, or why can’t I
select it?

How can I learn map reading for navigation?

Can you become a U.S. senator if you have dual

citizenship?

How can I draw a mini wave?

How to draw overlapping “rainbow” rings

Do copper blocks oxidize indoors?

Meaning of "-0.0" in Python?

What is this metal rod with attached razor blade?

Found in Nana's Kitchen

I'm using for-each loop and want to highlight a

Sign up or log in Post as a guest single row in LWC using onmouseover but only the
topmost row in the loop is getting highlighted
Name
Sign up using Google Invert the ST_DWithin() function

Nirvana with residue/non-abiding nirvana

Sign up using Facebook Email Is it true that Apple won't allow users to download
Required, but never shown full resolution photos from iCloud Photo to
Windows computers?
Sign up using Email and Password
Is it normal for professors to give out exercises
very similar to those done in class for the exam?

Main character has to go to a school for magicless

Post Your Answer By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy people

Question feed

Not the answer you're looking for? Browse other questions tagged python dictionary for-loop

pagination concatenation or ask your own question.

STACK OVERFLOW PRODUCTS COMPANY STACK EXCHANGE Blog Facebook Twitter LinkedIn Instagram
NETWORK
Questions Teams About
Technology
Jobs Talent Press
Life / Arts
Developer Jobs Directory Advertising Work Here
Culture / Recreation
Salary Calculator Enterprise Legal
Science
Help Privacy Policy
Other
Mobile Terms of Service
Disable Responsiveness Contact Us
Cookie Settings
site design / logo © 2021 Stack Exchange Inc; user contributions
Cookie Policy licensed under cc by-sa. rev 2021.8.3.39902