You are on page 1of 40

11/19/2020 Creating a Live World Weather Map using Shiny | by M.

Makkawi | The Startup | Medium

You have 2 free member-only stories left this month. Upgrade for unlimited access.

Creating a Live World Weather


Map using Shiny
Combining OpenWeatherMap API, Python/Shell scripts, Dropbox,
Scheduled Jobs to create a live world weather map in Shiny that
updates every hour

M. Makkawi Follow
May 18 · 19 min read

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 1/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

World Weather Map using Shiny

In the previous tutorial, we introduced sending requests to and receiving


information from the OpenWeatherMap API and parsing that information
in a way to create a dataset containing various data points related to the
weather for the world’s 200 most populous cities. At the end of that tutorial,
we compiled the weather conditions and other metrics of interest in a
Pandas dataframe and downloaded it to our local computer as a CSV.

In this tutorial, we’ll use that data to plot the weather conditions on a world
map, deploy this map online so it can be accessed by anyone, and automate

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 2/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

the whole process such that it updates the weather regularly without you
having to edit run the scripts manually.

To accomplish this, we will perform the following steps:

1. Transforming OpenWeatherMap API calls into a Python script

2. Visualizing the static data on a world map using Leaflet package in R

3. Creating a Shiny app and deploying the world weather map online

4. Updating the weather data regularly using cronjobs and Dropbox

1. OpenWeatherMap Python script (Quick recap)


In the previous post, the environment we used to call the OpenWeatherMap
API and generate the CSV was a Jupyter notebook. This was due to its
overall simplicity and our ability to iterate quickly with code chunks.

However, if we‘re looking to get the most recent weather data on a more
regular basis, writing a Python script is the way to go. The script will
essentially contain the same code as in the notebook except it won’t be
chunked up into blocks and instead will just have one continuous flow.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 3/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

The flow of the script will go as follows:

A) Top Cities Loading/Extraction

To send requests to the OpenWeatherMap API, we would need the name or


names of the cities as a string or a list of strings as the input to the request.
In the last post, we used web-scraping to get the names of the cities and
stored them in a list that we loop through later.

For our exercise, however, it doesn’t seem necessary to get an updated list of
cities every time we look to update the weather info. This is because the
populations of the world’s 200 most populous are unlikely to change
drastically in a short period of time. On top of that, web-scraping code can
be a tricky affair (unusable, obsolete) if the code is not maintained.

Instead, what we’ll do is run it once manually, and store the list of cities and
their populations as a CSV file in our data/working directory.

Then, as we go through our weather update script, we will check if there is


a most populous cities file present in the directory.

If the file is present, then it will just assign local list variables with the
values from the file and we’ll use those to send requests to the Weather API.
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 4/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

If for whatever reason the file is not present (ex: due to corruption of the
file), then it will go through the most populous cities extraction web-
scraping code, tally the city names and their populations, and then store
them in the appropriate lists.

Going about it this way will both save us time when executing the script and
give us extra safety from running obsolete scraping code that could damage
the pipeline.

B) Weather Update

After this, the script will loop through the top cities, calling the
OpenWeatherMap API and getting a weather report as a JSON for each city
in the list.

From each JSON response, we‘ll extract the information we need and then
store them in a dictionary.

Amongst other values, each city’s dictionary will contain:

Main Weather condition (ex: Clear, Clouds, Rain…)

Temperature in Celsius

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 5/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Wind Speed

Latitude

Longitude

Time of Last Update

Then, each dictionary will be stored in an overall list that will be converted
into a dataframe which we’ll output and save in our directory as a CSV.

Screenshot of the Pandas dataframe containing weather information for each city

2. (Static) World Weather Map in R using Leaflet


Now onto the good stuff, plotting the cities on a world map with each city
displaying its weather conditions.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 6/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

There are many tools/packages that allow us to plot geo-data on a map. For
this project, we’ll use R and specifically the Leaflet R package that is a
wrapper for the main Leaflet library in JavaScript. The Leaflet library
makes generating interactive maps easy and allows us to display anything of
interest on a map with about 10 lines of code.

The first step is to get the map working and displaying the information in
the format that we want before deploying it as an application to Shiny.

For this, the environment we’ll work in the R-Studio IDE. Inside R-Studio, I
prefer working in an R-Markdown because it is easy to separate different
steps in the code into chunks (similar to a Jupyter Notebook). Code chunks
will make it easier to quickly iterate and debug the map during
development.

Quick intro to Leaflet


Adding the Base Maps

To create a Leaflet map, we start by calling leaflet() which creates the


map widget. Beyond that, we add layers on top that include the type of map
( addTiles ) and markers on a map ( addMarkers ) to modify the widget.
There are a variety of maps that one can choose as the base depending on

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 7/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

the use-case, even combining more than one on top of each other if need
be. For a full list of leaflet map providers, see here.

For our map, we’ll use the Esri World Terrain basemap because of its
aesthetic appeal and simplicity. After all we just need a simple map without
street-level information or other extra details.

library(dplyr)
library(leaflet)

leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain)

Beyond this, we’ll need to set the view to be zoomed out and centred to
capture all of the cities in one view. To do this, we’ll use setView using a
zoom level of 2 and coordinates of 30, 30 (which appears to be the centre of
the civilized world).

leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2)

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 8/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Lastly, because leaflet allows unlimited dragging/scrolling, we want to


make sure that the user viewing the map cannot drag the map outside of
the main view. For this, we’ll use setMaxBounds and set them a little beyond
the maximum coordinates of our cities in all four directions.

leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70 )

Put together, our map now looks like this:

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 9/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Leaflet Basemap

Plotting the cities using addMarkers

Next, we’ll use the coordinate data to pinpoint the cities on the map.

First, we’ll load our data in the R session

city_info <- read.csv("data/weather.csv")

Screenshot of loaded dataset in R-Studio

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 10/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

We’ve got the dataframe with all the necessary columns to create our map
for 196 cities.

To plot the cities on the map, we’ll add them as Markers using addMarkers

and specify the dataset that we’ll use in the data attribute and the
respective latitude and longitude columns for the lat and lng attributes.

leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70) %>%
addMarkers(data = city_info,
lng = ~Longitude,
lat = ~Latitude)

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 11/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Leaflet Basemap with Cities Markers

Weather Icons

Instead of displaying the weather conditions as a number (temperature) or


text (conditions) on the Markers, we can make things a little more
appealing visually by giving an icon representing each main weather
condition, and have that display as the main button for each city.

To do this, there’s a useful integration within Leaflet called IconList that


allows us to specify a path to a custom image and have that map to a
particular value in the conditions column.

We’ll simplify the weather conditions into seven main conditions (clear,
cloudy, overcast, rainy, snowy, thunderstorms, haze) and create an icon for
each. For the icons, I found the following stock image on the internet that
we’ll crop and use for each individual condition.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 12/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Stock image used to crop the icons for the weather map

To create the iconList, it’s as simple as calling makeIcon for each condition in
our conditions column with the attributes: iconUrl (location of the png file,
here in the img folder), the iconWidth, and the iconHeight. And doing this
for all seven of our weather conditions.

weatherIcons <- iconList(


Clear= makeIcon(iconUrl="img/Clear.png",iconWidth=20,iconHeight=20),
Rainy= makeIcon(iconUrl="img/Rain.png", iconWidth=20,iconHeight=20),
etc...
)

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 13/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

On the data side, we’ll need make sure our mapping column contains only
one of these seven conditions. The following is a code snippet that I used
that is admittedly messy and far from perfect but does the job. Essentially
what’s happening here is we’re creating a new column called ‘conditions’
that will be used to map the icons, and using many if-else statements to
ensure that there are only the main seven conditions in the column.

city_info <- city_info %>%


mutate(conditions = factor(
ifelse(Main_Weather=="overcast clouds", "Overcast",
ifelse(Weather == "Drizzle", "Rain",
ifelse(Weather == "Mist", "Rain,
etc...
as.character(Weather))..))

And we add them in addMarkers in the icon argument:

leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70) %>%
addMarkers(data = city_info,
lng = ~Longitude,
lat = ~Latitude,
icon = ~weatherIcons[city_info$conditions])

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 14/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Leaflet Map with Custom Weather Icons

Now, our map has taken shape and looks pretty appealing. A couple of
tweaks to the design left and we’ll start working on the Shiny app with this
template.

Information Pop-up

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 15/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

The last cool feature in Leaflet that we’ll cover in this walkthrough is the
popup. After specifying the latitude, longitude, and icon corresponding to
the weather for each city, one thing that sticks out is that we’re not sure
which city is which, especially in areas that have multiple large cities
adjacent to one another (looking at you, China).

To solve this, we can add a popup on each icon such that the user can click
on the icon and get presented with a display showing the temperature,
population, and even the time the weather was last updated; all the data
that we’ve already extracted and is available in the dataframe for each city.

The popup also comes in the form of an attribute to addMarkers and will
contain the following information:

City, Country

Time of Last Update

Population

Detailed Weather Condition

Temperature (in Celsius)

Wind Speed (in km/h)

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 16/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

A collection of most of the details that a user might look for when checking
the weather for a particular city.

leaflet() %>%
addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70) %>%
addMarkers(data = city_info,
lng = ~Longitude,
lat = ~Latitude,
icon = ~weatherIcons[city_info$conditions],
popup = paste(
"<b>",city_info$City,", ", city_info$Country,"</b>","<br>",
"<b>Updated: </b>",city_info$DateTime,"<br>",
"<b>Population: </b>",city_info$Population,"<br>",
"<b>Weather: </b>",city_info$weather_main,"<br>",
"<b>Temperature: </b>",city_info$temp, " C","<br>",
"<b>Wind Speed: </b>",city_info$Wind_Speed, " km/h",
sep=""))

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 17/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Popup showing extended weather information

And now we’ve got a good-looking template to work with.

3. Shiny Application
Our map looks good in our local environment. The next step is turning it
into a live Shiny application and deploying it.

Shiny is a platform built for R that allows developers to build interactive


web applications that can be used to visualize data in various forms and
makes it easily accessible. Shiny takes care of the hosting part of the
application on their servers so that developers don’t have to worry too
much about the ‘web development’ part of the process and can focus more
on what goes into their visualizations.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 18/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

We’ll sign up for the freemium model which allows up to 5 applications and
a monthly limit on visitor viewing time, but nonetheless serves us well here
where the project is a simple application.

Signing Up
First things first, go to shinyapps.io and Sign Up. In the shinyapps.io
console, you’ll find a 3 point instruction manual on how to set up.

In summary, install the rsconnect package, copy the secret token issued to
you by shinyapps.io, and in your R Console connect your Shiny account to
your R-Studio environment using rsconnect::setAccountInfo() where
you’ll specify your token.

Now, you’re ready to go and are just a few clicks from deploying the app.

Building the Shiny App


Turning our map into an R-Shiny app requires a few tweaks to our Leaflet
code to fit the form that the Shiny engine expects. Namely, we have to
separate the server logic from the UI (User Interface) logic and call the
shinyApp function which combines both and renders the interactive map.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 19/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

We begin by creating a folder in our directory called app.R. Inside app.R,


we’ll load the necessary libraries that we’ll use followed by the data that
we’ll need to process for the visualization.

# Loading packages

library(shiny)
library(leaflet)
library(dplyr)

# Data Processing

city_info <- read.csv("data/weather.csv")

city_info <- city_info %>%


mutate(conditions = factor(ifelse(
Main_Weather=="overcast clouds", "Overcast",
ifelse(Weather == "Drizzle", "Rain",
ifelse(Weather == "Mist", "Rain",
etc...
, as.character(Weather)))))

Server Logic

The server logic will contain the brains of our weather map, which is almost
all of the logic that we used to render the map earlier in the RMarkdown.
We’ll wrap our code in the renderLeaflet function and assign that to the
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 20/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

$map variable in output. The output will be used later in the UI section to
dictate what gets displayed.

server <- function(input, output) {


output$map <- renderLeaflet({

weatherIcons <- iconList(


Clear=makeIcon(iconUrl="img/Clear.png",iconWidth=20,iconHeight=20),
Clouds=makeIcon(iconUrl="img/Clouds.png",iconWidth=20,iconHeight=20)
etc...
)

leaflet(options = leafletOptions(minZoom = 2)) %>%


addProviderTiles(providers$Esri.WorldTerrain) %>%
setView(lat = 30, lng = 30, zoom = 2) %>%
setMaxBounds(lng1 = -140, lat1 = -70, lng2 = 155, lat2 = 70) %>%
addMarkers(data = city_info,
lng = ~Longitude,
lat = ~Latitude,
icon = ~weatherIcons[city_info$conditions],
popup = paste("<b>",city_info$City,", ", etc...))
})
}

UI Logic

The UI logic will take care of how the entire page is laid out and how the
server logic gets displayed. As a design, we’re going to want the map to take

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 21/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

over the entire page. For starters, however, let’s keep it simple and just
render the map to see how it holds up when we try to run the application.

ui <- fluidPage(
leafletOutput(outputId = "map", width = 1000, height = 1000)
)

The function fluidPage creates a fluid layout that scales the components to
fit the available browser width so we don’t have to worry about sizes or
measurements. Its output is passed to the UI variable.

Inside the fluidPage function, we call leafletOutput , a function that

specifically deals with leaflet maps that are created in the server chunk.
Inside, we assign the outputId argument with map variable that we
specified in the server logic.

This is how the two sections communicate with each other. In addition,
we’ve arbitrarily specified the width and the height of our map to be
1000px.

shinyApp function

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 22/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Finally, at the bottom of our application, we combine the server and UI logic
and call the shinyApp function:

shinyApp(ui = ui, server = server)

And press Run App in the top right corner of the R-Studio IDE.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 23/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Output from running the Shiny app in the R-Studio environment — Map is smaller than screen

Looks great. But one thing that remains unappealing is that the map doesn’t
take up the entirety of the window.

To solve this, we can create a CSS style sheet file (called styles.css) and add
it in our working directory. Inside the CSS file, we’ll have:

# styles.css

div.outer {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
overflow: hidden;
padding: 0;
}

Not a CSS expert so don’t take my word here for this. But essentially for the
outer class that we’ll use in the UI, we’ve specified that the padding and all
of the sides consume 0 pixels to fill up the page. There’s an abundance of
CSS resources online that can help you further understand what’s going on

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 24/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

here and potentially guide you to add more stylistic functionality to the app,
but for this tutorial, we’ll stick to this.

Now we can modify the UI by specifying the class as outer and including
the CSS file in our fluidPage call and using “100%” for the width and height
in the leafletOutput. This should give us the full-page view.

ui <- fluidPage(
class = "outer",
tags$head(includeCSS("styles.css")),
leafletOutput(outputId = "map",width = "100%",height = "100%")
)

Running the application again:

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 25/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

World Weather Map with full-page view

We’ve got a full-page map with all the information that we need with the
code written in the Shiny format.

Deploying the Shiny App


We’re now ready to deploy the application to the Shiny servers.

Just as a reminder, since we’ve added a few files to our working directory
that may have been difficult to keep track of, the working directory should
contain the following files and folders:

WorldWeatherMap
|
|- app.R

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 26/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

|- styles.css
|- img/ -- Clear.png, Clouds.png ...
|- data/ -- weather.csv

This is important to note as when deploying the app, Shiny packages all
these files and folders in a container of sorts and runs them every time the
web-page is opened. Which means you’ll want everything ordered and
organized in the folder and make sure its all-encompassing (i.e. containing
all the resources your app needs including data and images).

Furthermore, the name of your directory (here: WorldWeatherMap) will


become the name of the web app and be included in the domain name for
the application, so make sure you name it accordingly.

To deploy the app, ensure your credentials are loaded (see: Signing Up
section), and make sure you’re in the correct working directory in your R
Console. Then type the following:

library(rsconnect)
rsconnect::deployApp()

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 27/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

It is as simple as that. This might take a few minutes while it uploads the
container and gets the instances running. But after that, you should be able
to find your live web application at the following domain:

{username}.shinyapps.io/{name_of_app}/

If your application is live, congratulations on making it this far. If not,


double check the steps you took and don’t hesitate to ask in the comments if
anything was unclear.

4. Updating the Weather Regularly


We’re almost home. But our application is missing one crucial element that
makes any weather map a weather map… recency. The application
wouldn’t be of much use if it displayed weather conditions from years ago
or even a day ago.

So for the final order of business, we’ll configure the pipeline to update our
weather map on a regular basis.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 28/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

As a reminder, our data is currently static. As in, it comes packaged with the
entire application container that gets deployed. So to update it would
require re-deploying the app with every update - a tedious affair.

What we’ll do instead is take advantage of a Dropbox-R interface called


rdrop2 that allows us to feed the data that we use as input from a file stored
on Dropbox instead of from a file saved locally. This will save us from
having to re-deploy the application every time we want to update the
weather data.

Once this is hooked up, we’ll automate the update of the file on Dropbox
that will feed the map data using a scheduled job, and this will save us from
having to update the file manually.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 29/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

A visual summary of the pipeline for the Weather Map

Two main steps to accomplish this:

1. Convert the input source to read a data file from a dropbox folder
instead of a file stored locally

2. Run a cronjob to run the script that we developed in Part 1 that updates
this data file on a regular basis.

Changing the Data Source to Dropbox


R-Studio allows a few ways to load data from outside the application
environment that includes databases and cloud based storages. You can
read about them here. One of these ways is Dropbox, a familiar easy to use
cloud-based storage system.

Authenticating with rdrop2

To connect our R app with Dropbox, we need to install the rdrop2 package
and set up a token or a key that allows our R application to access the files
https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 30/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

in dropbox.

The following is taken directly from the rdrop2 docs, which is a great
resource on how to set up. Highly recommend you read through it to
familiarize yourself with the process and check out some examples on how
it is used.

# In the console

drop_auth()

# This will launch your browser and request access to your Dropbox
account. You will be prompted to log in if you aren't already logged
in.
# Once completed, close your browser window and return to R to
complete authentication.
# The credentials are automatically cached (you can prevent this) for
future use.

# If you wish to save the tokens, for local/remote use

token <- drop_auth()


saveRDS(token, file = "token.rds")

Connecting Dropbox with Shiny

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 31/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Once authenticated, in app.R, we’ll add the following chunk before we load
the data

library(rdrop2)

# Data Download (from Dropbox)

token <- drop_auth(rdstoken = 'token.rds')

drop_download(path = '/Apps/weather_app/weather.csv',
local_path = 'data/weather.csv',
overwrite=TRUE)

# Data Processing

city_info <- read.csv("data/weather.csv")

In summary, we use the drop_download function:

Specify the path to get the data from (path)

Specify the location to download the data to (local_path)

Set the overwrite parameter to true so it updates the existing file every
time it is called.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 32/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Now, instead of having little control over the data file that the application is
reading from after it is deployed, we have the ability to download the data
file from a dropbox folder every time the app starts.

If we are able to update the file in the dropbox folder, we are able to update
the data in the application without re-deploying.

We’re only halfway done, because we’d still have to update the file in the
dropbox folder manually.

Uploading an updated data file to Dropbox


To upload a file to Dropbox from a Python script on a local computer, we
would need a separate Dropbox interface for Python and separate
authentication.

Connecting Python with Dropbox

To get an authentication token, go to dropbox.com/developers. Log in and


go to the App Console. Inside the console, click on ‘Create a New App’, fill in
all the necessary information and specify a path in the App folder where
you’ll upload the data file.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 33/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Once that’s done, go to ‘Generate Access Token’ and copy the token into the
config.py file that you have in your directory (the one that also has the API
key to OpenWeatherMap).

Upload.py Script

Download the dropbox Python package: pip install dropbox

Then, create a new script in your directory called upload.py . This script will

simply take the data found in a particular directory (in our case
data/weather.csv) and upload it to the Dropbox folder that you specified
above.

# upload.py

import dropbox
import config

access_token = config.drop_key

file_from = 'data/weather.csv'

file_to = '/weather_csv/weather.csv'

dbx = dropbox.Dropbox(access_token)

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 34/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

with open(file_from, 'rb') as f:


dbx.files_upload(f.read(),file_to,
mode=dropbox.files.WriteMode.overwrite)

print("Upload complete")

The script is fairly self-explanatory. Do note however: the mode =

dropbox.files.WriteMode.overwrite is very important otherwise you’ll get an


error any time it tries to write a file to Dropbox and it finds another file
there.

As a quick recap, there’s now:

updateWeather.py — Script that generates an up-to-date weather CSV


and stores it in the data/ folder

upload.py — Script that uploads weather.csv from data/ to a Dropbox

folder

rdrop2 integration in app.R that downloads fresh data from Dropbox on


application startup

Scheduling Upload Job using Cron


Since all the pieces are in place now, we need to automate the pipeline.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 35/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

First, let’s simplify the two Python scripts by combining them into a shell
script upload.sh that will just execute one after the other with one
command.

#!/usr/bin/env bash

python3.6 ~/WorldWeatherMap/updateWeather.py
&&
python3.6 ~/WorldWeatherMap/csv_upload.py

Note: You might have to specify full paths for both Python and the two
scripts, depending on the location of your Python path and working
directory on your local PC.

Now that the upload process is simplified into one command, we want to
automate the execution of this script on our computer. Enter cronjobs.

A cronjob is a job that executes a script or command every given period of


time on a scheduled basis. We’ll use this tool to run the script our upload.sh

script every hour.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 36/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

An hour is a reasonable period of time to update the weather. A shorter


period and we might violate the limit of API calls that we can make to the
OpenWeatherMap API and a longer period and we risk showing obsolete
weather data.

There are different ‘flavors’ of cron jobs that have different specs. The one
we’ll use is cron is native to Unix environments. The one downside to cron
is that it won’t run if the computer is not on at the time it is scheduled to run
and won’t run the missed job again in the future.

There are other versions that run the backlog of missed jobs as soon as the
computer turns back on. Try to keep that in mind when choosing the
version of cron to meet your needs. There are also more advanced solutions
using public servers to run the job but these are out of scope here.

To set up a job with cron, in the terminal:

crontab -e

An empty file should appear. Inside this file, we specify the time period that
the script should run as well as the path to the script that we want to

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 37/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

execute. Using your preferred text editor (vim, nano, etc…):

0 * * * * cd ~/WorldWeatherMap/ && ~/WorldWeatherMap/upload.sh >


~/WorldWeatherMap/logs/back.log 2>&1

This specifies to run the job at the 0th minute of every hour. In addition,
we’ve piped in a log file that can store any potential errors that occurred
while attempting to run the script. This can be useful while debugging.

Checking if the job is working

Wait until the hour break, and check the Dropbox directory, if the data has
been updated, then the cron job works. If not, check your back.log file in
the logs directory where you can find the error that occurred.

Next check your Shiny app, you should be able to see the Updated At: in
your popup reflecting the most recent weather report.

Conclusion
And voila, you’ve got a regularly updating weather map deployed online.

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 38/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Every once in a while, check the deployed application to make sure things
are running smoothly. Often times there might be a small detail that was
overlooked or a weather condition that we forgot to specify in our data
processing section that are causing the map to malfunction and we would
need to add or fix something to correct it.

This was a good project to work on the various parts that go into creating a
useful visualization online, regularly updating the flow of data, and sharing
it with others. Shiny is an extremely handy tool and can be used for a
variety of use-cases, your imagination being the only limit.

Check out the link to the live weather map here. To see the full code that
went into this, check it out on the Github page for the project. There may be
a few things that are different there but all of the core parts are the same.

Finally, if there was anything that was unclear or could use further
clarification from my end, please don’t hesitate to comment below.

Happy coding!

Shiny Python API Visualization Web Applications

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 39/40
11/19/2020 Creating a Live World Weather Map using Shiny | by M. Makkawi | The Startup | Medium

Learn more. Make Medium yours. Share your thinking.


Medium is an open platform where 170 Follow the writers, publications, and topics If you have a story to tell, knowledge to
million readers come to find insightful and that matter to you, and you’ll see them on share, or a perspective to offer — welcome
dynamic thinking. Here, expert and your homepage and in your inbox. Explore home. It’s easy and free to post your thinking
undiscovered voices alike dive into the heart on any topic. Write on Medium
of any topic and bring new ideas to the
surface. Learn more

About Help Legal

https://medium.com/swlh/creating-a-live-world-weather-map-using-shiny-f2ad05a08a13 40/40

You might also like