Miami-Dade County Open Data Hub (2018). Miami-Dade Major Roads [gis dataset]. University of Miami GDSC. https://gis-mdc.opendata.arcgis.com/datasets/MDC::street/about. last updated: 0001-01-01 (update frequency: 1 week). A line feature class of Miami-Dade County streets with seamless address ranges. Displays main roads including expressway ramps, expressways, highways, and major roads. A line feature class of Miami-Dade County streets with seamless address ranges. This feature class (FC) is one of a set of FC's derived from our main Streets FC, which were created to display street information. Key words: Florida, Miami-Dade County, MDC, Infrastructure Open Geospatial Consortium Standards: dynamic vector tiles | | WMS (coming soon) | WFS (coming soon) Analytical Functions: gdsc_nearest_neighbor(table=mdc_major_roads,lat=25.718,lng=-80.279,id=streetid) ETL documentation

ETL for mdc_major_roads

######### # GDSC ETL is performed in two steps: shell script to ETL data into postgres # and then additional transformations with SQL in postGIS (if needed). #### ######### # Step 1 - bash script (pseudo code) #### # create directory structure and move into it mkdir -p /data/mdc_major_roads/{download,etl} && cd /data/mdc_major_roads # set update flag based on last update and update frequency do_update=0 if date() < last_update + update_frequency else do_update = 1 # Create copy of geom dependency pg_dump -d $POSTGRES_DB -U $POSTGRES_USER -p $POSTGRES_PORT -h gdsc-postgis-node-r96xb6p4fpgnqrmc -t mdc_streets | sed 's/mdc_streets/mdc_major_roads/g' | psql -d $POSTGRES_DB -U $POSTGRES_USER -p $POSTGRES_PORT -h gdsc-postgis-node-bgorngopq01xs8x7 # drop geom_local and subset as per custom parameters psql -d $POSTGRES_DB -U $POSTGRES_USER -p $POSTGRES_PORT -h gdsc-postgis-node-bgorngopq01xs8x7 -c " ALTER TABLE mdc_major_roads DROP COLUMN IF EXISTS geom_local; DELETE FROM mdc_major_roads WHERE NOT (class in ('0','1','2','3')); COMMIT;" # add local geometry column and reproject existing geometries into local EPSG: SELECT AddGeometryColumn ( 'mdc_major_roads', 'geom_local', 2236, 'multilinestring', 2 ); UPDATE mdc_major_roads SET geom_local=ST_MakeValid(ST_RemoveRepeatedPoints(ST_Transform(ST_Multi(geom),2236))); CREATE INDEX mdc_major_roads_geom_local_idx ON mdc_major_roads USING GIST (geom_local); NOTIFY pgrst, 'reload schema'; # Move into corrrect directory and create derivative directory in data package on osgeo cd /data/mdc_major_roads/ mkdir -p derived # Create shapefile of table in derivate directory on osgeo ogr2ogr -f "ESRI Shapefile" -overwrite derived/mdc_major_roads.shp PG:"" mdc_major_roads # Create downloadable tarfile of shp in derivative directory on osgeo rm -f derived/mdc_major_roads.shp.tar.gz tar -czf derived/mdc_major_roads.shp.tar.gz meta_dcat_mdc_major_roads.json -C derived mdc_major_roads.shp mdc_major_roads.prj mdc_major_roads.shx mdc_major_roads.dbf # Move into correct directory cd /data/mdc_major_roads/