Professional Documents
Culture Documents
You have 2 free member-only stories left this month. Upgrade for unlimited access.
M. Makkawi Follow
May 18 · 19 min read
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 1/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
In this tutorial, we’ll use that data to plot the weather conditions on a world
map, deploy this map online so it can be accessed by anyone, and automate
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 2/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
the whole process such that it updates the weather regularly without you
having to edit run the scripts manually.
3. Creating a Shiny app and deploying the world weather map online
However, if we‘re looking to get the most recent weather data on a more
regular basis, writing a Python script is the way to go. The script will
essentially contain the same code as in the notebook except it won’t be
chunked up into blocks and instead will just have one continuous flow.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 3/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
For our exercise, however, it doesn’t seem necessary to get an updated list of
cities every time we look to update the weather info. This is because the
populations of the world’s 200 most populous are unlikely to change
drastically in a short period of time. On top of that, web-scraping code can
be a tricky affair (unusable, obsolete) if the code is not maintained.
Instead, what we’ll do is run it once manually, and store the list of cities and
their populations as a CSV file in our data/working directory.
If the file is present, then it will just assign local list variables with the
values from the file and we’ll use those to send requests to the Weather API.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 4/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
If for whatever reason the file is not present (ex: due to corruption of the
file), then it will go through the most populous cities extraction web-
scraping code, tally the city names and their populations, and then store
them in the appropriate lists.
Going about it this way will both save us time when executing the script and
give us extra safety from running obsolete scraping code that could damage
the pipeline.
B) Weather Update
After this, the script will loop through the top cities, calling the
OpenWeatherMap API and getting a weather report as a JSON for each city
in the list.
From each JSON response, we‘ll extract the information we need and then
store them in a dictionary.
Temperature in Celsius
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 5/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
Wind Speed
Latitude
Longitude
Then, each dictionary will be stored in an overall list that will be converted
into a dataframe which we’ll output and save in our directory as a CSV.
Screenshot of the Pandas dataframe containing weather information for each city
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 6/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
There are many tools/packages that allow us to plot geo-data on a map. For
this project, we’ll use R and specifically the Leaflet R package that is a
wrapper for the main Leaflet library in JavaScript. The Leaflet library
makes generating interactive maps easy and allows us to display anything of
interest on a map with about 10 lines of code.
The first step is to get the map working and displaying the information in
the format that we want before deploying it as an application to Shiny.
For this, the environment we’ll work in the R-Studio IDE. Inside R-Studio, I
prefer working in an R-Markdown because it is easy to separate different
steps in the code into chunks (similar to a Jupyter Notebook). Code chunks
will make it easier to quickly iterate and debug the map during
development.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 7/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
the use-case, even combining more than one on top of each other if need
be. For a full list of leaflet map providers, see here.
For our map, we’ll use the Esri World Terrain basemap because of its
aesthetic appeal and simplicity. After all we just need a simple map without
street-level information or other extra details.
library(dplyr)
library(leaflet)
leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain)
Beyond this, we’ll need to set the view to be zoomed out and centred to
capture all of the cities in one view. To do this, we’ll use setView using a
zoom level of 2 and coordinates of 30, 30 (which appears to be the centre of
the civilized world).
leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2)
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 8/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70 )
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 9/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
Leaflet Basemap
Next, we’ll use the coordinate data to pinpoint the cities on the map.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 10/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
We’ve got the dataframe with all the necessary columns to create our map
for 196 cities.
To plot the cities on the map, we’ll add them as Markers using addMarkers
and specify the dataset that we’ll use in the data attribute and the
respective latitude and longitude columns for the lat and lng attributes.
leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70) %>%
addMarkers(data = city_info,
lng = ~Longitude,
lat = ~Latitude)
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 11/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
Weather Icons
We’ll simplify the weather conditions into seven main conditions (clear,
cloudy, overcast, rainy, snowy, thunderstorms, haze) and create an icon for
each. For the icons, I found the following stock image on the internet that
we’ll crop and use for each individual condition.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 12/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
Stock image used to crop the icons for the weather map
To create the iconList, it’s as simple as calling makeIcon for each condition in
our conditions column with the attributes: iconUrl (location of the png file,
here in the img folder), the iconWidth, and the iconHeight. And doing this
for all seven of our weather conditions.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 13/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
On the data side, we’ll need make sure our mapping column contains only
one of these seven conditions. The following is a code snippet that I used
that is admittedly messy and far from perfect but does the job. Essentially
what’s happening here is we’re creating a new column called ‘conditions’
that will be used to map the icons, and using many if-else statements to
ensure that there are only the main seven conditions in the column.
leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70) %>%
addMarkers(data = city_info,
lng = ~Longitude,
lat = ~Latitude,
icon = ~weatherIcons[city_info$conditions])
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 14/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
Now, our map has taken shape and looks pretty appealing. A couple of
tweaks to the design left and we’ll start working on the Shiny app with this
template.
Information Pop-up
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 15/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
The last cool feature in Leaflet that we’ll cover in this walkthrough is the
popup. After specifying the latitude, longitude, and icon corresponding to
the weather for each city, one thing that sticks out is that we’re not sure
which city is which, especially in areas that have multiple large cities
adjacent to one another (looking at you, China).
To solve this, we can add a popup on each icon such that the user can click
on the icon and get presented with a display showing the temperature,
population, and even the time the weather was last updated; all the data
that we’ve already extracted and is available in the dataframe for each city.
The popup also comes in the form of an attribute to addMarkers and will
contain the following information:
City, Country
Population
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 16/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
A collection of most of the details that a user might look for when checking
the weather for a particular city.
leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70) %>%
addMarkers(data = city_info,
lng = ~Longitude,
lat = ~Latitude,
icon = ~weatherIcons[city_info$conditions],
popup = paste(
"<b>",city_info$City,", ", city_info$Country,"</b>","<br>",
"<b>Updated: </b>",city_info$DateTime,"<br>",
"<b>Population: </b>",city_info$Population,"<br>",
"<b>Weather: </b>",city_info$weather_main,"<br>",
"<b>Temperature: </b>",city_info$temp, " C","<br>",
"<b>Wind Speed: </b>",city_info$Wind_Speed, " km/h",
sep=""))
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 17/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
3. Shiny Application
Our map looks good in our local environment. The next step is turning it
into a live Shiny application and deploying it.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 18/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
We’ll sign up for the freemium model which allows up to 5 applications and
a monthly limit on visitor viewing time, but nonetheless serves us well here
where the project is a simple application.
Signing Up
First things first, go to shinyapps.io and Sign Up. In the shinyapps.io
console, you’ll find a 3 point instruction manual on how to set up.
In summary, install the rsconnect package, copy the secret token issued to
you by shinyapps.io, and in your R Console connect your Shiny account to
your R-Studio environment using rsconnect::setAccountInfo() where
you’ll specify your token.
Now, you’re ready to go and are just a few clicks from deploying the app.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 19/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
# Loading packages
library(shiny)
library(leaflet)
library(dplyr)
# Data Processing
Server Logic
The server logic will contain the brains of our weather map, which is almost
all of the logic that we used to render the map earlier in the RMarkdown.
We’ll wrap our code in the renderLeaflet function and assign that to the
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 20/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
$map variable in output. The output will be used later in the UI section to
dictate what gets displayed.
UI Logic
The UI logic will take care of how the entire page is laid out and how the
server logic gets displayed. As a design, we’re going to want the map to take
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 21/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
over the entire page. For starters, however, let’s keep it simple and just
render the map to see how it holds up when we try to run the application.
ui <- fluidPage(
leafletOutput(outputId = "map", width = 1000, height = 1000)
)
The function fluidPage creates a fluid layout that scales the components to
fit the available browser width so we don’t have to worry about sizes or
measurements. Its output is passed to the UI variable.
specifically deals with leaflet maps that are created in the server chunk.
Inside, we assign the outputId argument with map variable that we
specified in the server logic.
This is how the two sections communicate with each other. In addition,
we’ve arbitrarily specified the width and the height of our map to be
1000px.
shinyApp function
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 22/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
Finally, at the bottom of our application, we combine the server and UI logic
and call the shinyApp function:
And press Run App in the top right corner of the R-Studio IDE.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 23/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
Output from running the Shiny app in the R-Studio environment — Map is smaller than screen
Looks great. But one thing that remains unappealing is that the map doesn’t
take up the entirety of the window.
To solve this, we can create a CSS style sheet file (called styles.css) and add
it in our working directory. Inside the CSS file, we’ll have:
# styles.css
div.outer {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
overflow: hidden;
padding: 0;
}
Not a CSS expert so don’t take my word here for this. But essentially for the
outer class that we’ll use in the UI, we’ve specified that the padding and all
of the sides consume 0 pixels to fill up the page. There’s an abundance of
CSS resources online that can help you further understand what’s going on
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 24/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
here and potentially guide you to add more stylistic functionality to the app,
but for this tutorial, we’ll stick to this.
Now we can modify the UI by specifying the class as outer and including
the CSS file in our fluidPage call and using “100%” for the width and height
in the leafletOutput. This should give us the full-page view.
ui <- fluidPage(
class = "outer",
tags$head(includeCSS("styles.css")),
leafletOutput(outputId = "map",width = "100%",height = "100%")
)
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 25/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
We’ve got a full-page map with all the information that we need with the
code written in the Shiny format.
Just as a reminder, since we’ve added a few files to our working directory
that may have been difficult to keep track of, the working directory should
contain the following files and folders:
WorldWeatherMap
|
|- app.R
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 26/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
|- styles.css
|- img/ -- Clear.png, Clouds.png ...
|- data/ -- weather.csv
This is important to note as when deploying the app, Shiny packages all
these files and folders in a container of sorts and runs them every time the
web-page is opened. Which means you’ll want everything ordered and
organized in the folder and make sure its all-encompassing (i.e. containing
all the resources your app needs including data and images).
To deploy the app, ensure your credentials are loaded (see: Signing Up
section), and make sure you’re in the correct working directory in your R
Console. Then type the following:
library(rsconnect)
rsconnect::deployApp()
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 27/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
It is as simple as that. This might take a few minutes while it uploads the
container and gets the instances running. But after that, you should be able
to find your live web application at the following domain:
{username}.shinyapps.io/{name_of_app}/
So for the final order of business, we’ll configure the pipeline to update our
weather map on a regular basis.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 28/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
As a reminder, our data is currently static. As in, it comes packaged with the
entire application container that gets deployed. So to update it would
require re-deploying the app with every update - a tedious affair.
Once this is hooked up, we’ll automate the update of the file on Dropbox
that will feed the map data using a scheduled job, and this will save us from
having to update the file manually.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 29/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
1. Convert the input source to read a data file from a dropbox folder
instead of a file stored locally
2. Run a cronjob to run the script that we developed in Part 1 that updates
this data file on a regular basis.
To connect our R app with Dropbox, we need to install the rdrop2 package
and set up a token or a key that allows our R application to access the files
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 30/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
in dropbox.
The following is taken directly from the rdrop2 docs, which is a great
resource on how to set up. Highly recommend you read through it to
familiarize yourself with the process and check out some examples on how
it is used.
# In the console
drop_auth()
# This will launch your browser and request access to your Dropbox
account. You will be prompted to log in if you aren't already logged
in.
# Once completed, close your browser window and return to R to
complete authentication.
# The credentials are automatically cached (you can prevent this) for
future use.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 31/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
Once authenticated, in app.R, we’ll add the following chunk before we load
the data
library(rdrop2)
drop_download(path = '/Apps/weather_app/weather.csv',
local_path = 'data/weather.csv',
overwrite=TRUE)
# Data Processing
Set the overwrite parameter to true so it updates the existing file every
time it is called.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 32/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
Now, instead of having little control over the data file that the application is
reading from after it is deployed, we have the ability to download the data
file from a dropbox folder every time the app starts.
If we are able to update the file in the dropbox folder, we are able to update
the data in the application without re-deploying.
We’re only halfway done, because we’d still have to update the file in the
dropbox folder manually.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 33/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
Once that’s done, go to ‘Generate Access Token’ and copy the token into the
config.py file that you have in your directory (the one that also has the API
key to OpenWeatherMap).
Upload.py Script
Then, create a new script in your directory called upload.py . This script will
simply take the data found in a particular directory (in our case
data/weather.csv) and upload it to the Dropbox folder that you specified
above.
# upload.py
import dropbox
import config
access_token = config.drop_key
file_from = 'data/weather.csv'
file_to = '/weather_csv/weather.csv'
dbx = dropbox.Dropbox(access_token)
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 34/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
print("Upload complete")
folder
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 35/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
First, let’s simplify the two Python scripts by combining them into a shell
script upload.sh that will just execute one after the other with one
command.
#!/usr/bin/env bash
python3.6 ~/WorldWeatherMap/updateWeather.py
&&
python3.6 ~/WorldWeatherMap/csv_upload.py
Note: You might have to specify full paths for both Python and the two
scripts, depending on the location of your Python path and working
directory on your local PC.
Now that the upload process is simplified into one command, we want to
automate the execution of this script on our computer. Enter cronjobs.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 36/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
There are different ‘flavors’ of cron jobs that have different specs. The one
we’ll use is cron is native to Unix environments. The one downside to cron
is that it won’t run if the computer is not on at the time it is scheduled to run
and won’t run the missed job again in the future.
There are other versions that run the backlog of missed jobs as soon as the
computer turns back on. Try to keep that in mind when choosing the
version of cron to meet your needs. There are also more advanced solutions
using public servers to run the job but these are out of scope here.
crontab -e
An empty file should appear. Inside this file, we specify the time period that
the script should run as well as the path to the script that we want to
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 37/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
This specifies to run the job at the 0th minute of every hour. In addition,
we’ve piped in a log file that can store any potential errors that occurred
while attempting to run the script. This can be useful while debugging.
Wait until the hour break, and check the Dropbox directory, if the data has
been updated, then the cron job works. If not, check your back.log file in
the logs directory where you can find the error that occurred.
Next check your Shiny app, you should be able to see the Updated At: in
your popup reflecting the most recent weather report.
Conclusion
And voila, you’ve got a regularly updating weather map deployed online.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 38/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
Every once in a while, check the deployed application to make sure things
are running smoothly. Often times there might be a small detail that was
overlooked or a weather condition that we forgot to specify in our data
processing section that are causing the map to malfunction and we would
need to add or fix something to correct it.
This was a good project to work on the various parts that go into creating a
useful visualization online, regularly updating the flow of data, and sharing
it with others. Shiny is an extremely handy tool and can be used for a
variety of use-cases, your imagination being the only limit.
Check out the link to the live weather map here. To see the full code that
went into this, check it out on the Github page for the project. There may be
a few things that are different there but all of the core parts are the same.
Finally, if there was anything that was unclear or could use further
clarification from my end, please don’t hesitate to comment below.
Happy coding!
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 39/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 40/40