data-portal-tools

This repo is intended to house classes to be used for import and export to PSRC's data portal hosted on ArcGIS Online.

This has been confirmed to work as of ArcGIS Pro 3.7.1

How to run it

Setup

First, your agency Enterprise GIS Portal must have an "Application" object in its contents. PSRC has one called "PSRC Python Scripting" which should be available for the agency to use. The important thing about this object is its Client ID, which is a unique ID that will allow the script to authenticate as a valid member of the organization, and thus to access the layers in the Enterprise Portal. Find that object in the Portal, and copy the contents of the Client ID to an environment variable ENTERPRISE_PORTAL_CLIENT_ID on your machine.

You will need to create two other environment variables AGOL_ADMIN_USERNAME and AGOL_ADMIN_PW, which will store your ArcGIS username and password.

This conda setup routine worked as of 1/29/2026. Since the ArcGIS Online is continuously evolving, you might need to make sure you have a fresh version of the arcgis library.

These libraries might need to be reinstalled and updated to keep aligned with both the ArcPro versions you have on your desktops and the current version of arcgis api for python that is needed for AGOL.

conda create -n dpt python=3.13.7 (since ArcGIS Pro currently ships with Python 3.13.7 as of this writing, 7/30/2026)
conda activate dpt
conda install -c esri pathlib
conda install -c esri pandas
conda install -c esri geopandas
conda install -c esri arcgis
conda install -c esri arcpy

To export a layer from ElmerGeo or a table from Elmer:

1. Create a config file (in yaml format) for the layer/table in the /Config/run_files directory. Give it a filename that is very close or identical to the layer name itself. This will make it easier to keep track of your files later.
2. Open the file export_runner.py and adjust the final block to search for your new yaml file.
3. Run export_runner.py

The main code is run by executing the file export_runner.py, which in turn looks to the configuration files in Config/run_files/ for a lists of data sets to export. Each YAML file in this directory defines the parameters for a single dataset (i.e. a spatial layer or a tabular data set). A single run of export_runner.py iterates through all the config files in Config/run_files/, exporting each in turn. The config parameters for each data set are as follows:

• share_level (string): allow the exported layer to be share globally (the default) or just with members of our organization. The two possible values are 'everyone' (the default) or 'org'.
• description (string): this is the layer description that will be listed in ArcGIS Online.
• allow_edits (True or False): (feature pending) This will control whether the layer will be editable in Arc Online.
• tags (string): A comma-delimited list of tags for the layer in ArcGIS Online

Config file structure

The yaml files in folder Config\run_files define the details of each layer to be published. These details include things like the source data to be published, metadata, and some other related information. The elements in the yaml are defined as follows

• layer_params
  • accessInformation:
  • allow_edits: This should be set to False, unless you want users to be able to edit the data set
  • licenseInfo: If you want to publish the data with any particular license, you can specify the license here.
  • metadata
    • contact_city: Seattle
    • contact_email: email address of the PSRC staff member
    • contact_name: The PSRC staff member available to respond to questions about the data set
    • contact_phone: The phone number of the PSRC staff member
    • contact_state: Washington
    • contact_street_address: 1201 3rd Ave. Ste. 500
    • contact_zip: 98101
    • data_lineage:
    • fields: a list of pairs of title and description elements, with one pair for each field in the data set
      • - title: The title of a column
      • description: The description of the column
    • data_source: Organizations or locations that originally provided the data to PSRC, even if it has been reshaped since.
    • date_last_updated: The date of the last modification to the data.
    • organization_name: Puget Sound Regional Council
    • psrc_website: URL of web page discussing the program or project around the data (or just www.psrc.org)
    • summary: A description of the data set. This could be a short sentence or a long paragraph.
    • summary_addendum: Anything entered here will appear as a secondary paragraph in the description/summary
    • summary_footer: Anything entered here will appear as a third paragraph in the description/summary
    • supplemental_info: This is appended as an element in the layer's metadata xml, but is not currently included in ESRI's metadata display.
    • time_period: The span of time represented by the data.
    • update_cadence: The frequency at which the data set gets updated
  • share_level: a member of the set ["everyone", "org", "owner"]. This determines how widely the data will be shared. Usually this will be "everyone".
  • snippet: If provided, this is appended to the top of the layer's summary. It is often a very brief description of the data, with or without a version number.
  • groups: Groups are groupings of data on the Data Portal.
  • spatial_data: [True, False]. True means it is a spatial layer, False means it will be published in tabular format.
  • tags: Any tags to be associated with the layer, for searchability. At least one tag should be supplied.
  • title: The title
  • 'use_constraints`: Any special information regarding appropriate use of the data layer.
• source
  • is_simple: Set this to True if the spatial layer does not contain polygons with donut holes. For tabular data, set to True if the table will be exported in its entirety as it exists in Elmer (False if it is defined by a sql_query)
  • feature_dataset: If the spatial layer exists as a layer in ElmerGeo, list its feature dataset container here
  • schema_name: The schema in which a table resides in Elmer (ignored unless is_simple == True AND spatial_data == False)
  • sql_query: A Select query runnable against Elmer (if spatial_data == False) or against ElmerGeo (if spatial_data == True).
  • table_name: The name of a table in Elmer (ignored unless is_simple == True AND spatial_data == False)

Internals

See the module PortalExporter for classes to use for import and export from our internally-facing SQL Server databases. The module contains several classes:

• PortalConnector provides a connection object that binds an internally-facing database to the data portal.
• PortalResource defines a resource (a table, view, or user-defined query) on an internally-facing database, and provides an export method that allows that data to be easily published to the data portal as a single CSV.
• PortalSpatialResource is similar to PortalResource, but is for versioned spatial data layers such as those in ElmerGeo. The export method in this class publishes a spatial layer on the data portal in GeoJSON format.