Fix MSSentinelSearch environment name and connection check in AzureSearchDriver by Copilot · Pull Request #871 · microsoft/msticpy

Copilot · 2026-01-21T17:30:43Z

AzureSearchDriver inherited EFFECTIVE_ENV="MSSentinel" from its parent AzureMonitorDriver, causing the QueryProvider to report the wrong environment name. Additionally, the inherited query() method checked for _query_client (used by parent) instead of _auth_header (used by this driver), causing "Workspace not connected" errors even after successful connection.

Changes

Override environment in __init__: Set EFFECTIVE_ENV to "MSSentinelSearch" after parent initialization
Override query filter: Include "MSSentinelSearch" in supported environments tuple
Extract connection check: Add _ensure_connected() helper to check _auth_header is not None
Override query method: Use _ensure_connected() instead of parent's _query_client check

Example

# Before: failed with "Workspace not connected" 
qry_prov = QueryProvider('MSSentinelSearch')
qry_prov.connect(workspace='BasicLogs')
print(qry_prov.environment)  # Showed "MSSentinel" (wrong)
df = qry_prov.exec_query('SyslogBasic_CL | take 1', start=start, end=end)  # Failed

# After: works correctly
qry_prov = QueryProvider('MSSentinelSearch')
qry_prov.connect(workspace='BasicLogs')  
print(qry_prov.environment)  # Shows "MSSentinelSearch" (correct)
df = qry_prov.exec_query('SyslogBasic_CL | take 1', start=start, end=end)  # Succeeds

Original prompt

This section details on the original issue you should resolve

<issue_title>[Bug]: New experimenal MSSentinelSearch data provider doesn't correctly use the AzureSearchDriver</issue_title>
<issue_description>Describe the bug

The MSSentinelSearch query provider / data environment seems to get confused between using the MSSentinel vs MSSentinelSearch data environments and fails to correctly connect the AzureSearchDriver.

To Reproduce

Steps to reproduce the behavior:

Clone from this git repo and checkout main in order to test PR
Ianhelle/az monitor search driver 2025 02 05 #825 included in main.
Create an editable venv from the source and activate.
Configure msticpyconfig.yaml with a 'Sentinel' workspace that inlcudes a basic table.
Run a test script with the AzureSearchDriver on a table with the 'basic' plan.

import datetime

# Set debug logging
import logging
logging.basicConfig(level=logging.DEBUG)

# Inherit log level
import msticpy
print(f'msticpy version: {msticpy.__version__}')

# Config
msticpy.init_notebook()

# ws_config = msticpy.common.wsconfig.WorkspaceConfig(workspace="MyWorkspace")
# print(f'Workspace config: {ws_config}')
#qry_prov_basic_search = msticpy.QueryProvider(data_environment='MSSentinelSearch', ws_config=ws_config, workspace='BasicLogs')
qry_prov_basic_search = msticpy.QueryProvider('MSSentinelSearch')
qry_prov_basic_search.connect(workspace='BasicLogs')
print(f'Query provider driver: {qry_prov_basic_search.driver_class}')
print(f'Query provider environment: {qry_prov_basic_search.environment}')
print(f'Query provider connections: {qry_prov_basic_search.list_connections()}')

# Prep a small time range to limit basic logs query costs
lookback_period = datetime.timedelta(hours=1)
ingest_grace_period = datetime.timedelta(minutes=15)
end = datetime.datetime.now(datetime.timezone.utc) - ingest_grace_period
start = end - lookback_period
print(f'Start: {start}, End: {end}')

# Test query
df = qry_prov_basic_search.exec_query('SyslogBasic_CL | take 1', start=start, end=end)
print(df)

Expected behavior

AzureSearchDriver is connected and used with the corresponding MSSentinelSearch data environment.

Screenshots and/or Traceback

INFO:msticpy.data.drivers.azure_monitor_driver:AzureMonitorDriver loaded. connect_str  None, kwargs: {'data_environment': <DataEnvironment.MSSentinelSearch: 25>}
INFO:msticpy.data.core.data_providers:Using data environment MSSentinel
INFO:msticpy.data.core.data_providers:Driver class: AzureSearchDriver
...
INFO:msticpy.data.core.data_providers:Calling connect on driver
INFO:msticpy.data.drivers.azure_monitor_driver:WorkspaceConfig created from workspace name BasicLogs
...
INFO:msticpy.data.drivers.azure_monitor_driver:WorkspaceConfig created from workspace name BasicLogs
INFO:msticpy.data.drivers.azure_search_driver:Created HTTP-based query client using /search endpoint.
connected
INFO:msticpy.data.core.data_providers:Adding query pivot functions
Query provider driver: <class 'msticpy.data.drivers.azure_search_driver.AzureSearchDriver'>
Query provider environment: MSSentinel
Query provider connections: ['Default: BasicLogs']
Start: 2025-02-16 19:20:23.193644+00:00, End: 2025-02-16 20:20:23.193644+00:00
INFO:msticpy.data.core.query_provider_connections_mixin:Executing query 'SyslogBasic_CL | take 1...'
DEBUG:msticpy.data.core.query_provider_connections_mixin:Full query: SyslogBasic_CL | take 1
DEBUG:msticpy.data.core.query_provider_connections_mixin:Query options: {'start': datetime.datetime(2025, 2, 16, 19, 20, 23, 193644, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2025, 2, 16, 20, 20, 23, 193644, tzinfo=datetime.timezone.utc)}
Traceback (most recent call last):
  File "/home/***/SecurityNotebooks/demo/_bugs/msticpy/msticpy_cannot_query_azure_monitor_basic_logs/test_data_prov_ms_sentinel_search.py", line 39, in <module>
    df = qry_prov_basic_search.exec_query('SyslogBasic_CL | take 1', start=start, end=end)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/***/SecurityNotebooks/demo/_bugs/msticpy/msticpy_cannot_query_azure_monitor_basic_logs/src/msticpy/msticpy/data/core/query_provider_connections_mixin.py", line 99, in exec_query
    return self._query_provider.query(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/***/SecurityNotebooks/demo/_bugs/msticpy/msticpy_cannot_query_azure_monitor_basic_logs/src/msticpy/msticpy/data/drivers/azure_monitor_driver.py", line 294, in query
    raise MsticpyNotConnectedError(
msticpy.common.exceptions.MsticpyNotConnectedError: ('Workspace not connected.', 'Please run connect() to connect to the workspace', 'before running a query.', 'https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProv-MSSentinel.html')

Environment (please complete the following information):

Python Version: 3.111
OS: Ubuntu
Python environment: editable venv from cloned msticpy source with main branch...

Fixes [Bug]: New experimenal MSSentinelSearch data provider doesn't correctly use the AzureSearchDriver #831

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

…d override query method Co-authored-by: ianhelle <13070017+ianhelle@users.noreply.github.com>

…ation Co-authored-by: ianhelle <13070017+ianhelle@users.noreply.github.com>

Co-authored-by: ianhelle <13070017+ianhelle@users.noreply.github.com>

msticpy/data/drivers/azure_search_driver.py

- Add drop_duplicates(subset=['query']) before merge in get_whois_df to prevent row multiplication from duplicate whois results - Change net_df fixture scope from module to function for test isolation with random sampling - Add autouse fixture to clear LRU caches (get_whois_info, _whois_lookup) between tests to prevent state leakage

ianhelle · 2026-02-02T16:41:35Z

Thx for the review @FlorianBracq -
These 3 PRs were all Co-pilot authored from issues - a bit of an experiment but not bad responses. However, I should have looked at them with a more critical eye than I did. I took your suggestions on this.

FlorianBracq

Sorry, a few more changes would probably help IMO.
Feel free to correct me if you feel otherwise!

FlorianBracq · 2026-02-03T07:16:32Z

msticpy/data/drivers/azure_monitor_driver.py

            the underlying provider result if an error.

        """
-        if not self._connected or self._query_client is None:


I thought we would overload _ensure_connected (or the property connected) in this class (and AzureSearch) to ensure both conditions (self._connected being True and self._query_client/auth_header not being None) are validated?

FlorianBracq · 2026-02-03T07:18:50Z

msticpy/data/drivers/azure_monitor_driver.py

-                title="Workspace not connected.",
-                help_uri=_HELP_URL,
-            )
+        self._ensure_connected("Azure Monitor")


I would remove the hardcoded parameter as this would impact inheritance. Instead, we maybe want to add a new class variable that would contain the "display name" of the driver and use this?

FlorianBracq · 2026-02-03T07:19:36Z

msticpy/data/drivers/azure_search_driver.py

        self._connected = True
        logger.info("Created HTTP-based query client using /search endpoint.")

+    def query(


Isn't this the same implementation as in the parent class?

FlorianBracq · 2026-02-03T07:24:44Z

msticpy/data/drivers/driver_base.py

+            If the driver is not connected.
+
+        """
+        if not self._connected:


I read again the code for DriverBase, and I'm wondering why we are using _connected and not the property connected here?
It would be easier to overload (and probably to understand) the definition of the property connected in child class

FlorianBracq · 2026-02-03T07:26:45Z

msticpy/data/drivers/prismacloud_driver.py

        )
        # Check if authentication token is present
-
-        if "X-Redlock-Auth" not in self.client.headers:


this should probably move in the definition of property connected for this driver?

FlorianBracq · 2026-02-03T07:47:21Z

msticpy/data/drivers/azure_search_driver.py

+        self.set_driver_property(
+            DriverProps.EFFECTIVE_ENV, DataEnvironment.MSSentinelSearch.name
+        )
+        # Override query filter to include MSSentinelSearch


s/override/extend/

Initial plan

20c5b8a

Copilot AI assigned Copilot and ianhelle Jan 21, 2026

Copilot started work on behalf of ianhelle January 21, 2026 17:31 View session

Copilot AI and others added 4 commits January 21, 2026 17:37

Fix AzureSearchDriver to properly set MSSentinelSearch environment an…

8bcb2e5

…d override query method Co-authored-by: ianhelle <13070017+ianhelle@users.noreply.github.com>

Extract connection check to _ensure_connected method to reduce duplic…

fd6dd24

…ation Co-authored-by: ianhelle <13070017+ianhelle@users.noreply.github.com>

Move import statement to module-level in test file

93e838e

Co-authored-by: ianhelle <13070017+ianhelle@users.noreply.github.com>

Apply pre-commit formatting fixes (ruff)

ca67e11

Co-authored-by: ianhelle <13070017+ianhelle@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix MSSentinelSearch data provider connection issue with AzureSearchDriver~~ Fix MSSentinelSearch environment and connection handling in AzureSearchDriver Jan 21, 2026

Copilot AI requested a review from ianhelle January 21, 2026 17:46

Copilot finished work on behalf of ianhelle January 21, 2026 17:46

Merge branch 'main' into copilot/fix-azure-search-driver-connection

7bfef92

Copilot started work on behalf of ianhelle January 21, 2026 20:43 View session

Copilot AI changed the title ~~Fix MSSentinelSearch environment and connection handling in AzureSearchDriver~~ Fix MSSentinelSearch environment name and connection check in AzureSearchDriver Jan 21, 2026

Copilot finished work on behalf of ianhelle January 21, 2026 20:51

ianhelle approved these changes Jan 22, 2026

View reviewed changes

Merge branch 'main' into copilot/fix-azure-search-driver-connection

6a2af17

ianhelle marked this pull request as ready for review January 22, 2026 20:56

ianhelle requested review from FlorianBracq, petebryan and ryan-detect-dot-dev January 30, 2026 18:30

Merge branch 'main' into copilot/fix-azure-search-driver-connection

5a83f49

ianhelle enabled auto-merge January 30, 2026 18:30

FlorianBracq requested changes Jan 31, 2026

View reviewed changes

msticpy/data/drivers/azure_search_driver.py Outdated Show resolved Hide resolved

msticpy/data/drivers/azure_search_driver.py Outdated Show resolved Hide resolved

ianhelle added 5 commits February 1, 2026 17:00

Added _ensure_connected to DriverBase

5f4f2d1

Removing incorrectly committed file

c0b5205

Merge branch 'main' into copilot/fix-azure-search-driver-connection

68eddd7

Merge branch 'main' into copilot/fix-azure-search-driver-connection

bf4525e

ianhelle requested a review from FlorianBracq February 2, 2026 16:38

Merge branch 'main' into copilot/fix-azure-search-driver-connection

2c8f053

FlorianBracq requested changes Feb 3, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix MSSentinelSearch environment name and connection check in AzureSearchDriver#871

Fix MSSentinelSearch environment name and connection check in AzureSearchDriver#871
Copilot wants to merge 14 commits intomainfrom
copilot/fix-azure-search-driver-connection

Copilot AI commented Jan 21, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

ianhelle commented Feb 2, 2026

Uh oh!

FlorianBracq left a comment

Uh oh!

FlorianBracq Feb 3, 2026

Uh oh!

FlorianBracq Feb 3, 2026

Uh oh!

FlorianBracq Feb 3, 2026

Uh oh!

FlorianBracq Feb 3, 2026

Uh oh!

FlorianBracq Feb 3, 2026

Uh oh!

FlorianBracq Feb 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Example

Uh oh!

Uh oh!

Uh oh!

ianhelle commented Feb 2, 2026

Uh oh!

FlorianBracq left a comment

Choose a reason for hiding this comment

Uh oh!

FlorianBracq Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

FlorianBracq Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

FlorianBracq Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

FlorianBracq Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

FlorianBracq Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

FlorianBracq Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jan 21, 2026 •

edited

Loading