Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,17 @@ The following diagram shows the flow of data from the county into UTRANS.
flowchart TD
subgraph CLI
direction TB
updateFeatureClass@{ shape: lean-r , label: "Update Feature Class (County Data)"} --> get-recent-edits
baseFeatureClass@{ shape: lean-r , label: "Base Feature Class (Previously Obtained Data)"} --> get-recent-edits
get-recent-edits --> RoadCenterline_Recents@{ shape: lean-r}
RoadCenterline_Recents --> etl
countyBoundaries@{ shape: lean-r, label: "County Boundaries" } --> etl
updateFeatureClass@{ shape: lean-r , label: "Update Feature Class (--update-features)"} --> getRecentEdits["get-recent-edits"]
countyBaseFeatureClass@{ shape: lean-r , label: "County Base Feature Class (--base-features)"} --> getRecentEdits
getRecentEdits --> RoadCenterline_Recents@{ shape: lean-r , label: "RoadCenterline_Recents (--source-features)"}

RoadCenterline_Recents --> etl["etl"]
utransRoads@{ shape: lean-r , label: "UTRANS Roads Schema (--utrans-roads)"} --> etl
countyBoundaries@{ shape: lean-r , label: "County Boundaries (--county-boundaries)"} --> etl
etl --> etlOutput@{ shape: lean-r , label: "county_etl_YYYYMMDD"}

etlOutput --> detectChanges["detect-changes"]
utransBaseFeatureClass@{ shape: lean-r , label: "UTRANS Feature Class (--utrans-features)"} --> detectChanges
detectChanges --> productionDfc@{ shape: lean-r , label: "UTRANS/DFC_RESULT"}
end
```
2 changes: 1 addition & 1 deletion add-in/UGRC.UtransTools/Config.daml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://schemas.esri.com/DADF/Registry file:///C:/Program%20Files/ArcGIS/Pro/bin/ArcGIS.Desktop.Framework.xsd">

<AddInInfo id="{f893be2f-ad95-46be-9d60-fd6dba457152}" version="1.14" desktopVersion="3.6.59527" product="ArcGISPro">
<AddInInfo id="{f893be2f-ad95-46be-9d60-fd6dba457152}" version="1.16" desktopVersion="3.6.59527" product="ArcGISPro">
<Name>UGRC_UtransTools</Name>
<Description>UGRC_UtransTools description</Description>
<Image>Images\AddinDesktop32.png</Image>
Expand Down
197 changes: 67 additions & 130 deletions cli/PROFILES.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,27 @@
# County Profiles Configuration

`profiles.json` defines flat, per-county settings shared by UTRANS commands. `get-recent-edits` uses its field-mapping settings to run ArcGIS Pro's **Detect Feature Changes** tool between two versions of a county road centerline dataset. `etl` uses the same county key and its `fips` setting to select the matching county transformation.
`profiles.json` defines flat, per-county settings shared by UTRANS commands. `etl` uses the transformation settings to convert county roads into the UTRANS schema. `get-recent-edits` uses its change-detection settings to run ArcGIS Pro's **Detect Feature Changes** tool between two versions of a county road centerline dataset. `detect-changes` uses only `fips` from the profile; its DFC match and comparison fields are fixed by the command.

Each key in the JSON object is a county identifier (e.g. `"grand"`, `"davis"`). The script resolves the correct profile at runtime using the `--county` argument.

`saltlake` is the Salt Lake County identifier for every command. The former `vecc` identifier is not supported.

## Shared ETL Field
## Profile Fields

### `fips`

**Type:** `string` | **Required by `etl`**
**Type:** `string` | **Required by:** `etl`, `detect-changes`

**Used by:** `etl`, `detect-changes`

The county's five-digit FIPS code plus the name written to `COUNTY_L` and `COUNTY_R` during ETL. For example, `"49035 - Salt Lake"`.

### `field_mappings`

**Type:** `string` | **Required by `etl`**

**Used by:** `etl`

A semicolon-delimited list of `UTRANS_FIELD=COUNTY_FIELD` assignments. These map
county source fields into the target schema after colliding source fields have been
renamed with an underscore. When the UTRANS destination has a coded-value domain,
Expand All @@ -33,36 +37,52 @@ are reported with the field and value.

### `value_mappings`

**Type:** `object` | **Optional** — defaults to `{}`
**Type:** `object` | **Optional** - defaults to `{}`

**Used by:** `etl`

Per-destination-field mappings for county source values that do not match a
UTRANS coded-value domain code or description. The outer field name is the
destination field name in the UTRANS output data. Each inner mapping uses this form:

Per-target mappings for source values that do not match a UTRANS coded-value
domain code or description. Each target field contains an object mapping the
source value to the desired target value. Field names and source values are
matched case-insensitively after surrounding whitespace is removed. The mapped
target value is then resolved against the destination coded-value domain, so it
can be either a coded value or a domain description.
```text
"county source value": "value written to the UTRANS output field"
```

The inner key on the left is the value read from the county source field. The
inner value on the right is the value written to the destination UTRANS field
after it is resolved against that field's coded-value domain. Field names,
source keys, and source values are matched case-insensitively after surrounding
whitespace is removed. The mapping value should be a valid destination domain
code or description, and mapping entries should represent translations rather
than identity mappings.

```json
"value_mappings": {
"ONEWAY": {
"ONE DIRECTION": "Y",
"TWO WAY": "N"
"DOT_SRFTYP": {
"PAVED ASPHALT": "P-ASP"
},
"DOT_CLASS": {
"LOCAL ROAD": "L"
}
}
```

Mappings run after `field_mappings` selects a source field and before normal
destination-domain matching. If no configured mapping or valid domain match is
found, the original source value is handled as an invalid value: it is appended
to `UTRANS_NOTES` and copied only when it fits the target field length.
In this example, `DOT_SRFTYP` and `DOT_CLASS` are UTRANS destination fields. A
county value of `PAVED ASPHALT` is written as `P-ASP` to the output
`DOT_SRFTYP` field, and a county value of `LOCAL ROAD` is written as `L` to the
output `DOT_CLASS` field.
The mapped value is resolved against the destination coded-value domain. Values
that remain invalid are recorded in `UTRANS_NOTES`.

---

### `rules`

**Type:** `string[]` | **Optional** — defaults to `[]`

**Used by:** `etl`

An ordered list of supported post-mapping transformations. Rules run after field
and full-address mappings, but before the feature is normalized and appended. Unknown
or duplicate names are rejected.
Expand All @@ -80,6 +100,8 @@ Supported rules:

**Type:** `string` | **Optional**

**Used by:** `etl`

Selects a built-in, county-specific transformation for behavior that requires
multiple fields or conditional assignments. Handler names are resolved from the
CLI's fixed registry; profile files cannot name Python modules or functions.
Expand Down Expand Up @@ -114,57 +136,32 @@ Currently supported handlers:

**Type:** `string` | **Optional**

**Used by:** `etl`

A semicolon-delimited list of `COUNTY_FIELD=TARGET` values used to parse full street
names. `TARGET` is `PRIMARY`, `A1`, or `A2`.

### `translate_vertical_levels`

**Type:** `boolean` | **Optional** — defaults to `false`

When `true`, legacy vertical values `1`, `2`, and `3` are translated to `0`, `1`,
and `2` before domain-value matching. Invalid values are still preserved when they
fit within the destination field length and are always appended to `UTRANS_NOTES`.

### `exclude_if_any`

**Type:** `string` | **Optional**

A semicolon-delimited list of `COUNTY_FIELD=VALUE,VALUE` exclusions. A feature is
removed when any configured field contains a listed value.
**Type:** `string` or `object` | **Optional** - defaults to no exclusions

County selection is strict:
**Used by:** `etl`

- `--county` must exactly match a top-level key in the active profiles file
- By default, the active profiles file is `cli/profiles.json`
- Use `--profiles <path-to-json>` to supply a custom profiles file
- Matching is case-sensitive
- Aliases and spaced variants are not supported

---

## Fields Reference

### `match_fields`

**Type:** `string` | **Required**

A space-separated field pair (`update_field base_field`) used to confirm that two spatially proximate road segments are the **same road** across the two datasets. This is passed directly to the `match_field` parameter of `DetectFeatureChanges`.

The tool first finds candidate matches within `--search-distance`, then uses this field to disambiguate — e.g. confirming two nearby segments share the same street name before treating them as the same feature.
Excludes a staged feature when any configured field contains one of the listed
case-insensitive values. The string form uses semicolon-delimited `FIELD=VALUE1,VALUE2`
pairs. The object form maps field names to arrays of values.

```json
"match_fields": "STREETNAME STREETNAME"
"exclude_if_any": "S_SURF=400,410,420,430,440"
```

When both datasets use the same schema, the field name is repeated. If the schemas differ, the two names can be different (e.g. `"S_NAME STREETNAME"`).

> **Note:** The specific field chosen per county was determined during original script development and reflects the primary name field in that county's road schema.

---

### `compare_fields`

**Type:** `string` | **Optional in profile**
**Type:** `string` | **Required**

**Used by:** `get-recent-edits`

A semicolon-delimited list of `update_field base_field` pairs passed to the `compare_fields` parameter of `DetectFeatureChanges`. These are the attribute fields checked for changes after two segments are matched.

Expand All @@ -176,111 +173,51 @@ A segment is included in the output recents layer if any of these fields differ

Fields that don't exist in one or both datasets are silently dropped before the tool runs, with a warning logged.

If omitted from the profile, it must be provided via `--compare-fields` at runtime.
Every active county profile must provide this setting.

---

### `text_fields`

**Type:** `string[]` | **Optional** — defaults to `[]`
### `match_fields`

Fields that will be text-normalized before change detection runs. Normalization:
**Type:** `string` | **Required**

- Collapses internal whitespace to single spaces
- Trims leading/trailing whitespace
- Converts `None`, `NULL`, and blank values to empty string `""`
**Used by:** `get-recent-edits`

This prevents false positives caused by whitespace or null inconsistencies between datasets.
A space-separated `update_field base_field` pair used to confirm that two
spatially proximate road segments are the same road across the two datasets.
When both datasets use the same schema, the field name is repeated. If the
schemas differ, the two names can be different.

```json
"text_fields": ["PREDIR", "STREETNAME", "STREETTYPE", "SUFDIR"]
"match_fields": "STREETNAME STREETNAME"
```

---

### `numeric_fields`

**Type:** `string[]` | **Optional** — defaults to `[]`

Fields that will be numeric-normalized before change detection runs. Normalization:

- Converts `None`, `NULL`, and blank values to `0`

This prevents false positives caused by null vs. zero inconsistencies in address range fields.

```json
"numeric_fields": ["L_F_ADD", "L_T_ADD", "R_F_ADD", "R_T_ADD"]
```
The specific field is selected per county because source schemas differ.

---

### `uppercase_normalize_fields`

**Type:** `string[]` | **Optional** — defaults to `[]`

A subset of `text_fields` whose values will also be uppercased during normalization. Use this when a county's data may have mixed case values that should compare as equal.
**Used by:** `get-recent-edits`

```json
"uppercase_normalize_fields": ["STREETNAME", "STREETTYPE", "ACSALIAS"]
```

---

### `required_fields`

**Type:** `string[]` | **Optional** — defaults to `[]`

Fields that must exist in **both** the update and base feature classes. If any are missing, the script raises an error before doing any work. Useful for counties whose schemas are known to be strict or whose pipelines depend on specific fields being present.
A list of string fields whose values will also be uppercased during normalization.
Use this when a county's data may have mixed case values that should compare as equal.

```json
"required_fields": ["RoadName", "LeftFrom", "LeftTo", "RightFrom", "RightTo"]
"uppercase_normalize_fields": ["STREETNAME", "STREETTYPE", "ACSALIAS"]
```

---

### `dfc_output_name`

**Type:** `string` | **Optional** — defaults to `"DFC_CountyToCounty"`

Name of the output feature class written by `DetectFeatureChanges`. Output is placed in the same workspace as the update feature class.

---

### `stats_table_name`

**Type:** `string` | **Optional** — defaults to `"stats_county_to_county"`

Name of the statistics table written by `DetectFeatureChanges`.

---

### `recents_name`

**Type:** `string` | **Optional** — defaults to `"RoadCenterline_Recents"`

Name of the final output feature class containing only roads that changed (i.e. those where `CHANGE_TYPE <> 'NC'`). This is the primary deliverable of the script.

---

## Minimal Profile Example

```json
"example": {
"fips": "49000 - Example",
"field_mappings": "NAME=STREETNAME",
"match_fields": "STREETNAME STREETNAME",
"compare_fields": "PREDIR PREDIR; STREETNAME STREETNAME; STREETTYPE STREETTYPE; L_F_ADD L_F_ADD; L_T_ADD L_T_ADD; R_F_ADD R_F_ADD; R_T_ADD R_T_ADD",
"text_fields": ["PREDIR", "STREETNAME", "STREETTYPE"],
"numeric_fields": ["L_F_ADD", "L_T_ADD", "R_F_ADD", "R_T_ADD"]
"compare_fields": "PREDIR PREDIR; STREETNAME STREETNAME; STREETTYPE STREETTYPE; L_F_ADD L_F_ADD; L_T_ADD L_T_ADD; R_F_ADD R_F_ADD; R_T_ADD R_T_ADD"
}
```

## CLI Overrides

Most profile values can be overridden at runtime without editing `profiles.json`:

| Profile field | CLI argument |
| ------------------ | -------------------- |
| `match_fields` | `--match-fields` |
| `compare_fields` | `--compare-fields` |
| `dfc_output_name` | `--dfc-output-name` |
| `stats_table_name` | `--stats-table-name` |
| `recents_name` | `--recents-name` |
18 changes: 16 additions & 2 deletions cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,14 +45,28 @@ pip install --upgrade pathspec hatchling
utrans get-recent-edits --county Carbon --update-features "Z:\Documents\gdb\Carbon20231019.gdb\Roads" --base-features "Z:\Documents\gdb\Carbon20230208.gdb\CC_Roads"
```

The output is written to the geodatabase containing `--update-features`.

Run `utrans get-recent-edits --help` for the complete option reference and supported county profiles.

### ETL

Convert county roads to the UTRANS schema with explicit inputs:

```powershell
utrans etl --county saltlake --source-features "Z:\Documents\gdb\SaltLake.gdb\RoadCenterline_Recents" --utrans-roads "Z:\schemas\UtahRoadsNGSchema.gdb\Roads_Edit" --output-workspace "Z:\Documents\gdb\output.gdb" --county-boundaries "Database Connections\internal.sde\SGID.BOUNDARIES.Counties"
utrans etl --county saltlake --source-features "Z:\Documents\gdb\SaltLake.gdb\RoadCenterline_Recents" --utrans-roads "Z:\schemas\UtahRoadsNGSchema.gdb\Roads_Edit" --county-boundaries "Database Connections\internal.sde\SGID.BOUNDARIES.Counties"
```

The output is written to the geodatabase containing `--source-features`. It contains complete road features that intersect the selected county boundary; it does not clip features at the boundary. Run `utrans etl --help` for all options.

### Detect Changes

Process ETL output against a previous UTRANS road feature class:

```powershell
utrans detect-changes --county Carbon --update-features "Z:\Documents\gdb\Carbon.gdb\county_etl" --utrans-features "Z:\Documents\gdb\UTRANS.gdb\Roads" --append-target "Database Connections\internal.sde\UTRANS.DFC_RESULT"
```

The output contains complete road features that intersect the selected county boundary; it does not clip features at the boundary. Run `utrans etl --help` for all options.
The command writes outputs to the geodatabase containing `--update-features`. It uses the `NAME` matching and comparison fields, adds DFC editor metadata, creates `TEST_DFC_RESULT` with `NC` and `D` changes excluded, and appends the filtered result to `--append-target`.

Run `utrans detect-changes --help` for the remaining options.
28 changes: 28 additions & 0 deletions cli/scripts/county_datasets.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
county,profile,source_features
Beaver,beaver,C:\Mac\Home\Documents\working\utrans\county_obtained\Beaver\BeaverCo_20260604.gdb\Beaver_Roads2025
BoxElder,boxelder,C:\Mac\Home\Documents\working\utrans\county_obtained\BoxElder\BoxElder_20260624.gdb\BECO_RDS
Cache,cache,C:\Mac\Home\Documents\working\utrans\county_obtained\Cache\CacheCo_20260824.gdb\CacheCountyRoads
Carbon,carbon,C:\Mac\Home\Documents\working\utrans\county_obtained\Carbon\Carbon20241031.gdb\Roads
Daggett,daggett,C:\Mac\Home\Documents\working\utrans\county_obtained\Daggett\DaggettCo_20210515.gdb\Centerlines
Davis,davis,C:\Mac\Home\Documents\working\utrans\county_obtained\Davis\DavisCounty_20260722.gdb\DavisRoads
Duchesne,duchesne,C:\Mac\Home\Documents\working\utrans\county_obtained\Duchesne\DuchesneCo_20191009.gdb\Centerlines
Emery,emery,C:\Mac\Home\Documents\working\utrans\county_obtained\Emery\EmeryCo_20260611.gdb\Roads
Garfield,garfield,C:\Mac\Home\Documents\working\utrans\county_obtained\Garfield\Garfield20260427.gdb\Garfield_Streets_Roads
Grand,grand,C:\Mac\Home\Documents\working\utrans\county_obtained\Grand\GrandCo_20260722.gdb\GC_ROADS_07_09_26
Iron,iron,C:\Mac\Home\Documents\working\utrans\county_obtained\Iron\Iron_20260625.gdb\Iron_Roads06_25_2026
Kane,kane,C:\Mac\Home\Documents\working\utrans\county_obtained\Kane\KaneCo_20250915.gdb\Kane_Street_Centerlines
Millard,millard,C:\Mac\Home\Documents\working\utrans\county_obtained\Millard\Millard_20251217.gdb\Millard_Roads_Dec_2025
Morgan,morgan,C:\Mac\Home\Documents\working\utrans\county_obtained\Morgan\MorganCo_20260519.gdb\Centerlines_May26
Piute,piute,C:\Mac\Home\Documents\working\utrans\county_obtained\Piute\PiuteCo_20260414.gdb\Roads_2026_April
Rich,rich,C:\Mac\Home\Documents\working\utrans\county_obtained\Rich\RichCo_20211115.gdb\E911_RichRoads11_3_21
SanJuan,sanjuan,C:\Mac\Home\Documents\working\utrans\county_obtained\SanJuan\SanJuanRoads_20250808.gdb\T2025SanJuanRoads
SaltLake,saltlake,C:\Mac\Home\Documents\working\utrans\county_obtained\VECC\SaltLakeVECC_20260728.gdb\Centerlines
Sevier,sevier,C:\Mac\Home\Documents\working\utrans\county_obtained\Sevier\SevierCo_20260514.gdb\SC911DataUpdate_260511
Summit,summit,C:\Mac\Home\Documents\working\utrans\county_obtained\Summit\Summit_20260722.gdb\SummitRoads
Tooele,tooele,C:\Mac\Home\Documents\working\utrans\county_obtained\Tooele\TooeleCo_20260514.gdb\TCRoads
Uintah,uintah,C:\Mac\Home\Documents\working\utrans\county_obtained\Uintah\UintahCo20260423.gdb\Roads
Utah,utah,C:\Mac\Home\Documents\working\utrans\county_obtained\Utah\UtahCo_20260824.gdb\RoadCenterline
Wasatch,wasatch,C:\Mac\Home\Documents\working\utrans\county_obtained\Wasatch\WasatchCounty20260604.gdb\WC_ROAD
Washington,washington,C:\Mac\Home\Documents\working\utrans\county_obtained\Washington\Washington20260722.gdb\WashCo_RoadCenterlines
Wayne,wayne,C:\Mac\Home\Documents\working\utrans\county_obtained\Wayne\WayneCo_20260414.gdb\WayneCoRds_20260409
Weber,weber,C:\Mac\Home\Documents\working\utrans\county_obtained\Weber\WeberCounty_20260729.gdb\weber_centerline
Loading