Downloading historical data from

Coinmarketcap
Daniel Cimring
Aug 2, 2018 · 3 min read

Coinmarketcap is a great resource for crypto prices and data. Despite it’s 7aws it
remains a hugely popular site for checking cryto prices and I <nd myself going back
there daily out of habit despite other compelling and arguably better options like
onchainfx. Habits take e@ort to change and coinmarketcap is a habit for many people
myself included which will be hard to break for as long as the site remains “good
enough”.

I recently found myself wanting to download historical cryto data in order to do some
data analysis. Although they do a have a public API this does not seem to have a facility
for downloading any historical data. There is also a notice that this API will be taken
oGine in Dec 2018 and replaced by a Professional API. The Professional API will allow
for historical data to be downloaded but at the very steep price of $699/month for
access to 12 months of historical data and you need to “inquire for pricing” for access to
up to 5 years of historical data.

That seems a bit steep and I found myself a little shocked since crypto is all about the
elimination or reduction of middle men and rent seeking activities. Something as basic
as price data should surely be free. The legacy <nancial system often charges crazy
amounts for price data, surely the crypto ecosystem can do better than that.

I decided to build a script to just download the data myself but luckily decided to take a
look at the terms of use governing the site <rst. Be completely honest, who actually does
that. And there it was. You are not allowed to scrape or copy any data from the site.
Again this seems very anti-crypto to me. Imagine a public blockchain having something
like this in their terms.

(extract from the Terms of Use)

Prohibited Activities

You agree that you will not:

Use or introduce to the Service any data mining, crawling,

“scraping”, robot or similar automated or data gathering or
extraction method, or manually access, acquire, monitor or copy any
portion of the Service, or download or store Content (unless
expressly authorized by CMC). Certain data and other information
within the Service is available by subscription, or for a fee, at
https://pro.coinmarketcap.com;
Given the above this article is now purely hypothetical. How would I download data
from coinmarket cap *if* I was allowed to. Strictly hypothetically this is how I would do
it in Python.

First I would import BeautifulSoup and a few other handy packages.

from bs4 import BeautifulSoup

import requests
import pandas as pd

Then I would fetch the relevant page and <nd the table that contains all the data

url = "https://coinmarketcap.com/currencies/ripple/historical-data/?
start=20130428&end=20180802"
content = requests.get(url).content
soup = BeautifulSoup(content,'html.parser')
table = soup.find('table', {'class': 'table'})

Then use a nested list comprehension to go through the table rows and cells building up
a list of lists with the data.

data = [[td.text.strip() for td in tr.findChildren('td')]

for tr in table.findChildren('tr')]

Finally convert the list of lists to a DataFrame and do some cleaning up such as
converting text dates to real dates, removing empty rows, naming the columns, setting
an index, and sorting.

df = pd.DataFrame(data)
df.drop(df.index[0], inplace=True) # first row is empty
df[0] = pd.to_datetime(df[0]) # date
for i in range(1,7):
df[i] = pd.to_numeric(df[i].str.replace(",","").str.replace("-
 ","")) # some vol is missing and has -
df.columns = ['Date','Open','High','Low','Close','Volume','Market
Cap']
df.set_index('Date',inplace=True)
df.sort_index(inplace=True)

And that’s it. Python really does make life a pleasure. Now if only we were allowed to do
it. If you know of a good alternative to getting historical data please post something in
the replies. Quandl have some data but nothing for many of the coins and some of their
data has errors (e.g. the coinbase data after BCH forked got all messed up).

If you enjoyed this article or got any bene<t please give a few claps below so that others
can <nd it too.

Python Coinmarketcap Data Analysis Data Science Beautifulsoup

About Help Legal