UM Geospatial Digital Special Collections

Citation: Miami-Dade County (2014). Miami-Dade Stormwater Lines [gis dataset]. University of Miami GDSC. https://gis-mdc.opendata.arcgis.com/datasets/MDC::stormwater-line/about. last updated: 2024-05-01 (update frequency: Annually). Abstract: A line feature class that represents all drainage pipes and conveyance systems such as outfalls, culverts, infiltration trenches, ditches in the drainage network within Miami-Dade County. A line feature class that represents all drainage features for the Unincorporated Miami-Dade County Service Area (UMSA), which can be represented by lines in a map, are represented in this layer, such as outfalls, culverts, infiltration trenches, ditches in the drainage network. The data was collected through permit files, surveys and as-built(?), representing the best available information for each location at the time the layer was published. Key words: Miami-Dade County, Stormwater Line Open Geospatial Consortium Standards: vector tiles | WMS (coming soon) | WFS (coming soon) Analytical Functions: gdsc_get_attributes(tablename=mdc_stormwater_lines,lat=25.718,lng=-80.279,identifier=objectid,attrs=[id,map_label,type,layer,standard_detail,from_x1y1,to_x2y2,depth,width,length,bottom_elevation,diameter,invert_z1,invert_z2,material,treatment,headwall,headwall_height,pipe_shape,notes,entry_type,canalname,exit_type,wing_angle_headwall,skew_angle,column_no,beam,source,sourcedate,maintained_by,munic_nsme,state_road,vertical_datum,basin,control_structure,sw_pump_station,weir_elevation,originally_drawn_by,last_modified_date,last_modified_by,fdot_type,fdot_detail1,fdot_detail2,drainage_system_id,label_survey],normal='') ETL documentation

ETL for mdc_stormwater_lines

######### # GDSC ETL is performed in two steps: shell script to ETL data into postgres # and then additional transformations with SQL in postGIS (if needed). #### ######### # Step 1 - bash script (pseudo code) #### # create directory structure and move into it mkdir -p /data/mdc_stormwater_lines/{download,etl} && cd /data/mdc_stormwater_lines # set update flag based on last update and update frequency do_update=0 if date() < last_update + update_frequency else do_update = 1 # download if update flag is set: if do_update == 1 then ogr2ogr -nln mdc_stormwater_lines download/mdc_stormwater_lines.geojson "https://services.arcgis.com/8Pc9XBTAsYuxx9Ny/arcgis/rest/services/StormWaterLine_gdb/FeatureServer/0/query?outFields=*&where=1%3D1&outSR=4326&f=geojson" # replace column names Shape__Length and Shape__Area to avoid conflict when # providing as service on ArcServer. sed -i 's/Shape__Length/shape_length/gI; s/Shape__Area/shape_area/gI' download/mdc_stormwater_lines.geojson # load into postGIS with: ogr2ogr -lco GEOMETRY_NAME=geom -f PostgreSQL PG:"" download/mdc_stormwater_lines.geojson -nlt multilinestring -nln mdc_stormwater_lines ######### # Step 2 - SQL script (pseudo code): #### # remove duplicate points and make geometries valid: UPDATE mdc_stormwater_lines SET geom=ST_MakeValid(ST_RemoveRepeatedPoints(geom));" # add local geometry column and reproject existing geometries into local EPSG: SELECT AddGeometryColumn ( 'mdc_stormwater_lines', 'geom_local',2236,'multilinestring',2 ); UPDATE mdc_stormwater_lines SET geom_local=ST_Transform(ST_Multi(geom),2236); CREATE INDEX mdc_stormwater_lines_geom_local_idx ON mdc_stormwater_lines USING GIST (geom_local); NOTIFY pgrst, 'reload schema';"