Combine AD + ServiceNow pre-population extractors

ad-pre-population-extractor/README.md

@@ -0,0 +1,114 @@
+# AD Pre-Population Extractor for Dr Migrate (DMC)
+
+PowerShell extractor that pulls server inventory, workload signals, and topology
+from Active Directory into CSVs ready to be merged into a DMC cloud-migration
+assessment.
+
+AD is the lowest-friction pre-population source available — every customer with
+Windows servers has it. This tool is read-only, runs from any domain-joined
+workstation, and needs only an authenticated domain user. It is not a
+replacement for DMC; it's a feeder that gives you a complete server list, OS
+metadata, network topology, and Kerberos SPN-derived workload hints before any
+scanning starts.
+
+## Requirements
+
+- PowerShell 5.1 or 7.x
+- RSAT `ActiveDirectory` module (Windows client) **or** AD DS role tools
+  (Windows Server)
+- LDAP/LDAPS connectivity to a domain controller (389/636) or Global Catalog
+  (3268/3269)
+- Read-only credentials — any authenticated domain user is sufficient
+
+## Usage
+
+```powershell
+# Default: writes to ./ad-extract using current user credentials
+.\Extract-ADInventory.ps1
+
+# Pin to a specific DC and write a timestamped snapshot
+.\Extract-ADInventory.ps1 -Server dc01.contoso.com `
+                          -OutputDir C:\dmc\customer-x `
+                          -TimestampOutput
+
+# Alternate credentials (e.g. cross-domain run)
+$cred = Get-Credential
+.\Extract-ADInventory.ps1 -Server dc-othercorp.example.com -Credential $cred
+
+# Tighten the stale-server window
+.\Extract-ADInventory.ps1 -StaleThresholdDays 60
+```
+
+Logging is written to **stderr** so stdout stays clean for piping.
+
+## Outputs
+
+| File                              | Contents                                                                     |
+| --------------------------------- | ---------------------------------------------------------------------------- |
+| `ad_servers.csv`                  | Authoritative enabled-server list with OS, last logon, OU, owner             |
+| `ad_spns_computer.csv`            | SPNs registered against computer objects (services running as Computer$)     |
+| `ad_spns_service_accounts.csv`    | SPNs against domain user accounts — captures most SQL/Exchange/IIS workloads |
+| `ad_spns_gmsa.csv`                | SPNs against Group Managed Service Accounts                                  |
+| `ad_workload_classification.csv`  | Derived: per-host workload hints from joined SPN data + reference table      |
+| `ad_sites.csv`                    | AD Sites (typically 1:1 with datacentres / offices)                          |
+| `ad_subnets.csv`                  | Subnet-to-site mapping (correlate any IP back to a site)                     |
+| `ad_ou_distribution.csv`          | Top-3-level OU counts — often encodes BU / env / app grouping                |
+| `ad_stale_servers.csv`            | Servers inactive past `StaleThresholdDays` (default 90)                      |
+| `ad_domain_controllers.csv`       | DCs (usually out of migration scope, identify and treat separately)          |
+| `ad_coverage_summary.csv`         | Single-row summary for the customer conversation                             |
+| `ad_run_status.csv`               | Per-query status with row counts and any errors (partial output is fine)     |
+
+All CSVs are UTF-8 with BOM, comma-delimited, hostname-keyed (lowercased).
+
+## Hostname normalisation
+
+Both `Name` (NetBIOS) and `DNSHostName` (FQDN) are emitted in lowercase so
+downstream merging with SCCM/SCOM extracts works without further normalisation.
+Pick NetBIOS or FQDN for your join key and stick with it across all sources.
+
+## Recommended merge order
+
+1. Start with `ad_servers.csv` as the **authoritative server list** — it's the
+   only source guaranteed to be complete and current.
+2. Outer-join `sccm_servers.csv` for hardware/OS detail; flag servers in AD
+   but not in SCCM (no inventory data).
+3. Outer-join `scom_*` for topology/utilisation; flag servers in AD but not
+   in SCOM (no monitoring coverage).
+4. Use `ad_workload_classification.csv` (or the raw SPN CSVs) as
+   workload-classification hints joined by hostname.
+
+## What AD cannot tell you
+
+DMC remains required for:
+
+- Hardware specifications (CPU/memory/disk)
+- Live utilisation
+- Network dependencies / process attribution
+- Application boundaries
+- Workgroup / non-domain-joined servers
+- Azure AD-only resources
+- Linux/Unix servers (unless centralised identity is used — rare)
+- Software inventory
+
+Always emit the coverage summary alongside the extraction. The customer
+conversation hinges on it.
+
+## SPN reference
+
+`spn_workload_reference.csv` lists the ServiceClass → Workload mapping the
+script applies. The same map is embedded in the script for self-contained runs;
+the CSV is provided for downstream tooling that doesn't run PowerShell.
+
+## Caveats
+
+- `LastLogonTimestamp` replicates every 9–14 days — fine for migration
+  assessment, not for live activity reporting.
+- `OperatingSystem` is set at domain join and updated on OS upgrades — usually
+  reliable but can lag in-place upgrades.
+- Multi-domain forests: re-run with `-Server` per domain.
+- LAPS-managed devices: `msLAPS-Password` requires elevated rights and is
+  sensitive — this script does **not** read it.
+- Workgroup servers are invisible to AD. Flag this gap explicitly in the
+  customer conversation.
+- Disabled computer objects are excluded from the main inventory (they're
+  surfaced via the stale-server query if relevant).

ad-pre-population-extractor/spn_workload_reference.csv

@@ -0,0 +1,24 @@
+ServiceClass,Workload,Notes
+MSSQLSvc,SQL Server,SQL Server instance; SPN target encodes host:port
+HTTP,IIS / SharePoint / Web,IIS, SharePoint, web apps
+exchangeMDB,Exchange,Exchange mailbox database
+exchangeRFR,Exchange,Exchange referral
+SMTPSVC,Exchange,Exchange SMTP transport
+TERMSRV,Terminal Services / RDS,Remote Desktop Session Host
+WSMAN,WinRM endpoint,Server is WinRM-reachable
+host,IGNORE,Default computer SPN registered against every machine
+RestrictedKrbHost,IGNORE,Kerberos host SPN; not workload-bearing
+ldap,Domain Controller,DC role
+GC,Domain Controller,Global Catalog role
+kadmin,Domain Controller,Kerberos admin
+DNS,Domain Controller,Integrated DNS
+MSOMHSvc,SCOM,SCOM agent / management server
+MSOMSdkSvc,SCOM,SCOM SDK service
+Hyper-V Replica Service,Hyper-V,Hyper-V host
+MSServerClusterMgmtAPI,Failover cluster,Windows Server Failover Cluster
+IMAP,Mail service,IMAP server
+POP,Mail service,POP3 server
+SMTP,Mail service,Generic SMTP service
+ftp,FTP,FTP server
+cifs,File server,SMB/CIFS share host (extra-detected; host SPNs already cover SMB)
+vmrc,VMware / vCenter,VMware management

servicenow-cmdb-export/.env.example

@@ -0,0 +1,3 @@
+SNOW_INSTANCE=acme.service-now.com
+SNOW_USER=cmdb_reader
+SNOW_PASSWORD=changeme

servicenow-cmdb-export/.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+venv/
+.env
+*.xlsx
+!sample.xlsx
+.pytest_cache/

servicenow-cmdb-export/README.md

@@ -0,0 +1,55 @@
+# servicenow-cmdb-export
+
+CLI that pulls CMDB reference records from ServiceNow and writes a denormalized
+device-to-business-app-and-environment mapping to an Excel file. Servers and
+client devices in one sheet, joined to apps via `cmdb_rel_ci` (with an
+assigned-user fallback for clients).
+
+## Install
+
+```bash
+pip install -e .
+```
+
+## Configure
+
+Set environment variables (see `.env.example`):
+
+- `SNOW_INSTANCE` — e.g. `acme.service-now.com`
+- `SNOW_USER` / `SNOW_PASSWORD` — basic-auth credentials with read on
+  `cmdb_ci_server`, `cmdb_ci_pc_hardware`, `cmdb_ci_business_app`,
+  `cmdb_ci_environment`, `cmdb_rel_ci`, `sys_user`.
+
+## Run
+
+```bash
+snow-cmdb-export server-app-map --out cmdb.xlsx
+snow-cmdb-export server-app-map --out servers-only.xlsx --device-type server
+snow-cmdb-export server-app-map --out filtered.xlsx \
+    --query "operational_status=1^company.name=Acme"
+snow-cmdb-export server-app-map --out audit.xlsx --raw
+```
+
+## Output schema (`device_app_map` sheet)
+
+| column | source |
+| --- | --- |
+| device_type | `server` or `client` |
+| name | `cmdb_ci_*.name` |
+| fqdn | `cmdb_ci_*.fqdn` (falls back to `dns_domain`) |
+| environment | `cmdb_ci_environment.name` joined via `cmdb_rel_ci` |
+| assigned_user | `sys_user.name` resolved from `assigned_to` |
+| assigned_user_email | `sys_user.email` |
+| business_app | `cmdb_ci_business_app.name` |
+| app_owner | `business_owner` (fallback `managed_by`) |
+| criticality | `business_criticality` |
+| support_group | `support_group` on the CI |
+
+## Joining logic
+
+- **Servers → apps**: relationship rows on `cmdb_rel_ci` whose `type` display
+  value is one of `Runs on::Runs`, `Depends on::Used by`, `Hosted on::Hosts`.
+- **Clients → apps**: same relationship lookup first; if none, fall back to
+  business apps where the client's assigned user is `business_owner`,
+  `managed_by`, or `owned_by`. Capped by `--max-apps-per-user` (default 10).
+- **Environment**: any `cmdb_rel_ci` edge from the CI to a `cmdb_ci_environment`.

servicenow-cmdb-export/pyproject.toml

@@ -0,0 +1,20 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "servicenow-cmdb-export"
+version = "0.1.0"
+description = "Export CMDB server/client to business-app and environment mappings from ServiceNow into Excel."
+requires-python = ">=3.10"
+dependencies = [
+    "requests>=2.31",
+    "openpyxl>=3.1",
+    "click>=8.1",
+]
+
+[project.scripts]
+snow-cmdb-export = "servicenow_cmdb_export.cli:main"
+
+[tool.setuptools.packages.find]
+include = ["servicenow_cmdb_export*"]

servicenow-cmdb-export/requirements.txt

@@ -0,0 +1,3 @@
+requests>=2.31
+openpyxl>=3.1
+click>=8.1

servicenow-cmdb-export/servicenow_cmdb_export/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

servicenow-cmdb-export/servicenow_cmdb_export/__main__.py

@@ -0,0 +1,4 @@
+from servicenow_cmdb_export.cli import main
+
+if __name__ == "__main__":
+    main()

servicenow-cmdb-export/servicenow_cmdb_export/cli.py

@@ -0,0 +1,74 @@
+"""CLI: snow-cmdb-export."""
+from __future__ import annotations
+
+import sys
+
+import click
+
+from servicenow_cmdb_export.client import ServiceNowClient, ServiceNowError
+from servicenow_cmdb_export.exporter import write_workbook
+from servicenow_cmdb_export.queries import build_rows, fetch_snapshot
+
+
+@click.group()
+@click.version_option()
+def main() -> None:
+    """Export CMDB device-to-app-and-environment mappings from ServiceNow."""
+
+
+@main.command("server-app-map")
+@click.option(
+    "--out",
+    "-o",
+    required=True,
+    type=click.Path(dir_okay=False, writable=True),
+    help="Output .xlsx path.",
+)
+@click.option(
+    "--device-type",
+    type=click.Choice(["all", "server", "client"]),
+    default="all",
+    show_default=True,
+)
+@click.option(
+    "--query",
+    default=None,
+    help="Optional ServiceNow encoded query applied to device tables (sysparm_query).",
+)
+@click.option(
+    "--max-apps-per-user",
+    type=int,
+    default=10,
+    show_default=True,
+    help="Cap apps per client (via assigned-user fallback). 0 = unlimited.",
+)
+@click.option(
+    "--raw/--no-raw",
+    default=False,
+    help="Also dump raw CMDB tables as additional sheets.",
+)
+def server_app_map(
+    out: str, device_type: str, query: str | None, max_apps_per_user: int, raw: bool
+) -> None:
+    """Pull CMDB and write a denormalized device/app/env mapping to Excel."""
+    try:
+        client = ServiceNowClient()
+    except ServiceNowError as e:
+        click.echo(f"error: {e}", err=True)
+        sys.exit(2)
+
+    click.echo("Fetching CMDB snapshot from ServiceNow...", err=True)
+    snap = fetch_snapshot(client, device_filter=device_type, extra_query=query)
+    click.echo(
+        f"  servers={len(snap.servers)} clients={len(snap.clients)} "
+        f"apps={len(snap.apps_by_id)} envs={len(snap.envs_by_id)}",
+        err=True,
+    )
+
+    rows = build_rows(snap, max_apps_per_user=max_apps_per_user)
+    n = write_workbook(out, rows, snapshot=snap, include_raw=raw)
+    click.echo(f"Wrote {n} rows to {out}", err=True)
+
+
+if __name__ == "__main__":
+    main()