Schedule automated near real-time data updates

Gonzalo Espinoza is an expert on the integration of global and large datasets into applied research, especially in the fields of water resources and environmental engineering. He is part of the Living Atlas of the World team at Esri and has a keen interest in Web GIS, data science, and education.

Water levels in reservoirs change constantly: decreasing during the dry months when water is being consumed and increasing during the rainy season.

In this tutorial, you'll configure a layer in ArcGIS Online to consume a near real-time web service provided by the Texas Water Development Board. You'll create a feature service in ArcGIS Online using water data for Texas. You'll create a feature view to preserve symbology when the data is updated and configure a script to overwrite the feature service when new data is available. Finally, you'll schedule a task on a local computer to automate the updates.

This tutorial was last tested on August 6, 2024.

Requirements

• ArcGIS organizational account with a Publisher, Facilitator, or Administrator role (see options for software access)

Outline

Publish a feature service

Access near real-time reservoir data for Texas and create a feature service from a GeoJSON file in ArcGIS Online.

Create a hosted feature layer view

Create a hosted feature layer view of the Texas reservoirs feature service, and update and save its symbology.

Update a hosted feature layer

Use the OverwriteFS Python script to update the reservoirs feature service.

Create and schedule a task

Create and schedule an overwrite task to automate the updates of the reservoirs data.

Publish a feature service

In this tutorial, you’ll create a hosted feature layer in ArcGIS Online to display near real-time reservoir data for Texas. First, you’ll download reservoir data for Texas. Next, you’ll use this information to publish a hosted feature layer and configure a view of it. You'll configure a Python script to update the feature layer and set up a Windows task to run it. The Texas reservoirs data feature layer will be overwritten as new data becomes available, so any web maps based on the feature view will always display the latest information.

Access near real-time reservoir data for Texas

First, you'll access reservoir information from the Water Data for Texas website. Then, you'll download the data in GeoJSON format and store it locally.

1. In a browser, go to the Water Data for Texas website. Scroll down and under the Statewide Conservation Storage chart, click Recent Averaged Storage Conditions and click GeoJSON .

The website redirects you to https://www.waterdatafortexas.org/reservoirs/statewide/recent-conditions.geojson, which contains the near real-time reservoir data for Texas. This is a GeoJSON file. For each reservoir in Texas, it contains information such as location (coordinates), name, conservation capacity, percent full, and last updated time (time stamp).

Note:

The Water Data for Texas website is dynamic. It is frequently updated, as soon as new data is available. The time stamp value for each reservoir displays the time of the last update.

Note:

Check the file in your local folder after you save it to verify that the file extension is .geojson and not .geojson.txt .

Add an item to ArcGIS Online

You'll use the local GeoJSON file of reservoir data for Texas to create a hosted feature layer in ArcGIS Online .

1. Sign in to your ArcGIS organizational account.

Note:

• For Title , type TX_reservoirs_recent_ and add your name or initials.
• For Tags , type Texas, Reservoir, Recent Conditions and press Enter after each.
• For Summary , type Near real-time reservoir information for Texas.

The process will take a few moments. After it completes, the Overview page for the TX_reservoirs_recent_(your name) hosted feature layer appears. In addition to this layer, a copy of the TX_reservoirs_recent GeoJSON file is added to your My Content section.

You accessed near real-time reservoir information for Texas and created a hosted feature service in ArcGIS Online . Next, you'll use the feature service to create a feature view. This feature view can be customized, shared publicly, and added to your apps. Creating the feature view allows the feature service to be updated without disrupting the symbology and pop-ups configured for the view. This simplifies the process of updating the data later.

Create a hosted feature layer view

Now you'll create a hosted feature layer view from the hosted feature layer, share the hosted feature layer view publicly, update its symbology, and configure its pop-up.

Create the hosted feature layer view

The Texas reservoirs hosted feature layer that you created will be overwritten when new information is available. The overwrite process will remove any custom symbology and pop-up settings and restore it to the default settings. To avoid changing how the data is displayed in your web maps and apps, you will create a hosted feature layer view in ArcGIS Online and configure its symbology and pop-up settings. These will remain the same and display the latest data after the hosted feature layer is overwritten.

1. On the TX_reservoirs_recent_(yourname) feature layer Overview page, click Create View Layer and click View layer .

In the Included layers pane you can set layer definitions To learn more about these options, view the Create hosted feature layer views documentation.

• For Title , type Texas Reservoirs Recent Conditions (yourname) .
• For Folder , you can optionally specify a folder where you want to save the view layer.
• For Tags , type Water, Texas, Reservoirs, Environment, Natural Resources, Live Feeds, livefeeds , pressing Enter after each.
• For Summary , you can optionally provide additional text about the view layer.

A new hosted feature layer view item is created from the hosted feature layer and the item details page for the new feature layer view appears.

There are now three items in your My Content page: the GeoJSON file, the hosted feature layer, and the view.

Note:

It is a good practice to use a hosted feature layer view to share a hosted feature layer, keeping the original hosted feature layer private. You can change the pop-up and the symbology of the hosted feature layer view without modifying the original hosted feature layer. During an update process, the hosted feature layer can be overwritten, and the new values will be reflected in the hosted feature layer view. The symbology of the hosted feature layer view is saved with it and is not affected when the original hosted feature layer is overwritten.

Change symbology

The hosted feature layer view has the reservoir data for Texas. You'll customize its symbology to better represent the information in the layer using the reservoirs' capacity for different feature sizes and using the current percent-full level for different colors.

1. Click the Texas Reservoirs Recent Conditions_(Your Name) feature layer to open the hosted feature layer view item page.

The attributes are added to the Styles pane.

To see the name of a color ramp, point to the color ramp.

The symbology for the layer and basemap are now set. The final map shows the recent conditions of reservoirs in Texas. The color of each point represents the percentage full of each reservoir, and the size of the point represents the total reservoir capacity.

Configure the pop-up

You will improve the pop-up for the layer to display only the relevant information. You'll show the name, capacity, and percent full for each reservoir.

1. On the Settings toolbar and click Pop-ups .

The Add field window appears.

A text box appears.

The attribute field names in curly braces will be replaced with the values from those fields when the pop-up is opened by clicking a feature. The pop-up will now show the title and text you configured when a feature is clicked. You can still improve the format of the fields by configuring how each of the fields is displayed.

• For Title , type Texas Reservoirs (Your Name)
• Add Tags , Water , Texas , and Reservoirs .
• For Summary , type A near real-time map of Texas reservoirs showing percent full and capacity .

You've created a hosted feature layer view in ArcGIS Online from the reservoirs hosted feature layer. The hosted feature layer view allows you to customize the display of the Texas reservoir layer and retain these customizations even when the hosted feature layer is overwritten. Next, you'll overwrite the Texas reservoirs hosted feature layer using a simple Python script. The updated information will be reflected automatically in the hosted feature layer view.

Update a hosted feature layer

Now you'll configure a short script to overwrite the Texas reservoirs hosted feature layer. First, you'll download the overwrite script from ArcGIS Online . Then, you'll retrieve the hosted feature layer parameters such as item title and item ID. Finally, you'll run the script locally using the Python Command Prompt. The overwrite process allows you to update the hosted feature layer with the latest information.

Download the overwrite script

First, download the script for overwriting a hosted feature layer in ArcGIS Online .

1. Download the OverwriteFS.zip file to your computer.

The .zip file contains two files: the Python script ( OverwriteFS.py ) and a document ( ReadMe.pdf ).

You will need to know the title and the item ID for this hosted feature layer in order to update it. You will save this information in a text document for reference.

In this example, the item title is TX_reservoirs_recent_(yourname) and the item ID is 4309a6d6bb77477f986acba13b5f9ef6 . The item ID is unique for each item in ArcGIS Online . Your item ID will be different.

Store ArcGIS Online credentials locally

ArcGIS Online requires login credentials to overwrite items. You'll store ArcGIS credentials locally in a secure profile. This profile will be used in the Python script to overwrite the reservoirs hosted feature layer.

1. Open the Windows Start menu, search for Python Command Prompt , and open it.
2. Type cd and type the path to the folder where you extracted the OverwriteFS.py script and press Enter.

For example, if you extracted the script to C:\Temp , you would type cd C:\Temp .

This starts the Python console in the Python Command Prompt window. The Python version is noted in the console and the input prompt changes to >>> . Next you will import the arcgis Python module.

Note:

The Python command prompt and the arcgis Python module are included with the ArcGIS Pro installation.

Note:

Your ArcGIS Online credentials will be securely saved in the learn_user profile. You can learn more about storing ArcGIS Online credentials locally in a profile by visiting the Working with different authentication schemes page.

Overwrite the hosted feature layer

Now use the overwrite Python script, the ArcGIS Online profile, and the hosted feature layer parameters (item ID and title) to update the reservoirs hosted feature layer.

1. In the Python command prompt, in the C:\Temp\ folder that contains the OverwriteFS.py script, run the following command: python OverwriteFS.py learn_user , replacing and with the parameters of the hosted feature layer you saved in your text file.

In this code, learn_user is the ArcGIS Online profile saved locally. Replace and with the parameters of the hosted feature layer that you saved in your text file. In the example, you would replace with 3b2b385c923147349dec02025e10aa55 and with TX_reservoirs_recent_(yourname). You would replace is the URL of the data source with the updated data, https://www.waterdatafortexas.org/reservoirs/statewide/recent-conditions.geojson.

Note:

Because you've already created the hosted feature layer using the most recent data, the script won’t make any changes. Instead, you can temporarily use a file from https://downloads.esri.com/learnarcgis/schedule-automated-near-real-time-data-updates/recent-conditions.geojson, which contains a snapshot of the recent-conditions.geojson file for December 26, 2019.

Message showing that the overwrite process completed successfully." />

If you run it with the URL to the live Texas water site, the command will resemble the following (the item ID will be different): python OverwriteFS.py learn_user 20af483d68e54ff5835753de0e90ce27 TX_reservoirs_recent_(yourname) https://www.waterdatafortexas.org/reservoirs/statewide/recent-conditions.geojson If you download the snapshot of the recent-conditions.geojson file for December 26, 2019, save it to the C:\Temp\ folder that you are working in (replace the existing file). The command will resemble the following (the item ID will be different, and the path will be a local file path instead of a URL): python OverwriteFS.py learn_user 20af483d68e54ff5835753de0e90ce27 TX_reservoirs_recent_(yourname) c:\Temp\recent-conditions.geojson Press Enter to run the command. The command prompt displays the results. The Performing Overwrite… Success! Elapsed Time message at the end means that the script correctly updated the hosted feature layer.

The pop-up shows the updated time and percent-full values. The symbology of the hosted feature layer view remains as you set it.

You ran a Python script to overwrite a hosted feature layer in ArcGIS Online . Next, you'll create and schedule a task in Windows to automatically run the overwrite script at regular intervals.

Create and schedule a task

You ran a Python script to update a hosted feature layer. Now, you'll automate running the script by scheduling a task in Windows. First, you'll create a .bat file. Then, you'll create a task in Windows Task Scheduler. The task will regularly run the .bat file, which will run the overwrite Python script, which will check for updates and overwrite the hosted feature layer if new data is available.

Create a .bat file

Windows .bat files are simple text files with a set of instructions to be run by the command line interpreter.

1. Open a text editor such as Notepad or Notepad++.
2. Copy and paste the following text into the text editor:

"C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" "C:\Temp\OverwriteFS.py" learn_user 3b2b385c923147349dec02025e10aa55 TX_reservoirs_recent_(yourname) https://www.waterdatafortexas.org/reservoirs/statewide/recent-conditions.geojson

Next, you'll create a task to run the .bat file.

Create a basic task

Now that you have a .bat file, you’ll schedule a task to run it at regular intervals.

1. Open the Windows Start menu. Search for and open Task Scheduler .
2. In the Actions panel, click Create Basic Task .

The Create Basic Task Wizard window appears and starts on the Create a Basic Task tab.

The wizard updates to the Daily tab.

Note:

If you saved the .bat file in another folder, ensure you enter the correct path to the .bat file. You can also click the browse button and browse to the location of the file.

The Finish tab appears in the wizard.

The task will run at the scheduled time, updating the hosted feature layer with the latest information from the Water Data for Texas website. The hosted feature layer view will show the updated information from the hosted feature layer while retaining the same symbology and pop-up configuration.

In this tutorial, you downloaded data from the Water Data for Texas website and created a hosted feature layer. You created, customized, and shared a hosted feature layer view of the hosted feature layer to isolate the public item from the update process. You adapted, downloaded, and used a Python script to obtain the latest data from the website and update the hosted feature layer. You also created a .bat file and scheduled a Windows task to run the Python script. When the task runs, the overwrite Python script will update the hosted feature layer. You configured the task to run daily, so any web maps or apps that display the hosted feature layer view will show the latest data each day.

Acknowledgements

• Water Data For Texas from Texas Water Development Board (TWDB).
• Cartography, symbology, and images by Emily Meriam.

Send Us Feedback

Please send us your feedback regarding this tutorial. Tell us what you liked as well as what you didn't. If something in the tutorial didn't work, let us know what it was and where in the tutorial you encountered it (the section name and step number). Use this form to send us feedback.

Share and repurpose this tutorial

Sharing and reusing these tutorials are encouraged. This tutorial is governed by a Creative Commons license (CC BY-SA-NC). See the Terms of Use page for details about adapting this tutorial for your use.