Data Science Portfolio

Scrape Crypto Prices with Selenium

This project demonstrates the use of Selenium to scrape cryptocurrency price data from crypto.com

Inspect crypto.com source code to obtain relevant XPaths

First we go to https://crypto.com/price, which contains the price data for various cryptocurrencies
The price data is stored in a table
- Right-click the first row in the table and click “inspect”

Examine table element

This is a dynamic website, so the table containing the price data is rendered in JavaScript
Each cryptocurrency’s price data is in a <tr> element
Each data point (column) for a particular row is in a <td> element
We need the XPaths for both the <tr> and <td> elements to scrape the data

Extract XPath for `<tr>` element

Right-click the <tr> element and copy the XPath
The XPath for the <tr> element is /html/body/div[1]/div[3]/div[2]/div/div[4]/div[1]/table/tbody/tr[1]

Extract XPath for `<td>` element

Now we select one of the <td> elements and copy the XPath in the same manner
- We can also easily infer the XPath for <td> based on the XPath for <tr>
The XPath for the <td> element is /html/body/div[1]/div[3]/div[2]/div/div[4]/div[1]/table/tbody/tr[1]/td[1]

Import libraries

Now we can begin writing our scraper
First we import the required libraries

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import NoSuchElementException

Instantiate the WebDriver

For this project we will use headless ChromeDriver
- Headless browsers don’t have a UI, so they run faster
We will also set the headless browser window size to 1920x1080
We will use the sleep() method to give the page time to load

url = 'https://crypto.com/price'

chrome_options = Options()  
chrome_options.add_argument('--headless')
chrome_options.add_argument('--window-size=1920x1080')

driver = webdriver.Chrome(executable_path='./chromedriver', options=chrome_options)
driver.get(url)

time.sleep(3)

print(driver.title)

Top 50 Cryptocurrency Prices, Coin Market Cap, Price Charts And Historical Data | Crypto.com

Calculate the number of rows and columns in the table

Using both of the XPaths we copied, we can calculate the total rows and columns in the table
- We will use this for our scraping operation
We do this using the find_elements_by_xpath() method
To find the total rows and columns we remove the index (in brackets) from the XPaths we copied
- This will enable us to extract all the necessary <tr> and <td> elements

tr_xpath = '/html/body/div[1]/div[3]/div[2]/div/div[4]/div[1]/table/tbody/tr'
td_xpath = '/html/body/div[1]/div[3]/div[2]/div/div[4]/div[1]/table/tbody/tr[1]/td'

rows = 1+len(driver.find_elements_by_xpath(tr_xpath))
cols = 1+len(driver.find_elements_by_xpath(td_xpath))

print(rows)
print(cols)

51
10

Initialize empty lists for the desired data

There are six pieces of relevant data that we can extract from the table
- Name
- Symbol
- Price
- 24-hour change
- 24-hour volume
- Market cap
The rest of the data points can be discarded (ignored) during the scraping operation

name = []
symbol = []
price = []
one_day_chg = []
one_day_vol = []
market_cap = []

Loop through the table and collect the data

We will use a double for loop to iterate through each column of each row
- The range will be adjusted accordingly to ignore the columns we don’t need
A try except block is used to keep the scraper going in case it doesn’t find an element
For each column, we save the value by using the find_element_by_xpath method
- We replace the bracketed <tr> and <td> indexes with the row index i and column index j, respectively
- We use the text attribute to extract the text rather than a Selenium page object
Then we check which column we are on and append the value to the appropriate list
The cryptocurrency name and symbol are in the same column, separated by \n
- We use the split() method to extract the name and symbol separately for this special case

xpath_prefix = '/html/body/div[1]/div[3]/div[2]/div/div[4]/div[1]/table/tbody/tr['

for i in range(1, rows):
    for j in range(3, cols-2):
        try:
            value = driver.find_element_by_xpath(xpath_prefix+str(i)+']/td['+str(j)+']').text
            if j == 3:
                name.append(value.split('\n')[0])
                symbol.append(value.split('\n')[1])
            elif j == 4:
                price.append(value)
            elif j == 5:
                one_day_chg.append(value)
            elif j == 6:
                one_day_vol.append(value)
            elif j == 7:
                market_cap.append(value)
        except NoSuchElementException:
            pass

Store data in pandas DataFrame

Now that our lists have been populated with the data, we can store the data in a pandas DataFrame
- This will enable easy post-processing of the data for a multitude of use cases

crypto_data = {'Name': name, 'Symbol': symbol, 'Price': price, '24 Hour Change': one_day_chg,\
               '24 Hour Volume': one_day_vol, 'Market Cap': market_cap}

df = pd.DataFrame(data=crypto_data)

df.head()

	Name	Symbol	Price	24 Hour Change	24 Hour Volume	Market Cap
0	Bitcoin	BTC	$37,059.32	-5.09%	$19.06 B	$700.84 B
1	Ethereum	ETH	$2,706.02	-3.98%	$13.84 B	$321.49 B
2	Tether	USDT	$1.01	+0.00%	$44.06 B	$77.98 B
3	BNB	BNB	$367.60	-4.82%	$1.28 B	$60.81 B
4	USD Coin	USDC	$1.00	-0.01%	$2.72 B	$50.42 B

Quit the driver

We can see from printing df.head() that our entries have been recorded successfully
Now we close the WebDriver and finish the program

driver.quit()

Complete scraper

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import NoSuchElementException

url = 'https://crypto.com/price'

chrome_options = Options()  
chrome_options.add_argument('--headless')
chrome_options.add_argument('--window-size=1920x1080')

driver = webdriver.Chrome(executable_path='./chromedriver', options=chrome_options)
driver.get(url)

time.sleep(3)

print(driver.title)

tr_xpath = '/html/body/div[1]/div[3]/div[2]/div/div[4]/div[1]/table/tbody/tr'
td_xpath = '/html/body/div[1]/div[3]/div[2]/div/div[4]/div[1]/table/tbody/tr[1]/td'

rows = 1+len(driver.find_elements_by_xpath(tr_xpath))
cols = 1+len(driver.find_elements_by_xpath(td_xpath))

print(rows)
print(cols)

name = []
symbol = []
price = []
one_day_chg = []
one_day_vol = []
market_cap = []

xpath_prefix = '/html/body/div[1]/div[3]/div[2]/div/div[4]/div[1]/table/tbody/tr['

for i in range(1, rows):
    for j in range(3, cols-2):
        try:
            value = driver.find_element_by_xpath(xpath_prefix+str(i)+']/td['+str(j)+']').text
            if j == 3:
                name.append(value.split('\n')[0])
                symbol.append(value.split('\n')[1])
            elif j == 4:
                price.append(value)
            elif j == 5:
                one_day_chg.append(value)
            elif j == 6:
                one_day_vol.append(value)
            elif j == 7:
                market_cap.append(value)
        except NoSuchElementException:
            pass

crypto_data = {'Name': name, 'Symbol': symbol, 'Price': price, '24 Hour Change': one_day_chg,\
               '24 Hour Volume': one_day_vol, 'Market Cap': market_cap}

df = pd.DataFrame(data=crypto_data)

df.head()

driver.quit()

Data Science Portfolio

Scrape Crypto Prices with Selenium

Inspect crypto.com source code to obtain relevant XPaths

Examine table element

Extract XPath for <tr> element

Extract XPath for <td> element

Import libraries

Instantiate the WebDriver

Calculate the number of rows and columns in the table

Initialize empty lists for the desired data

Loop through the table and collect the data

Store data in pandas DataFrame

Quit the driver

Complete scraper

Extract XPath for `<tr>` element

Extract XPath for `<td>` element