feat: committees implementation [CM-1066] by mbani01 · Pull Request #3995 · linuxfoundation/crowd.dev

mbani01 · 2026-04-02T13:58:00Z

This pull request introduces support for "committees" as a new platform throughout the data pipeline, including database, integrations, and attribute configuration. The main changes add new activity types for committee membership, implement the data extraction and transformation logic for committee events, and update the type system and configuration to recognize "committees" as a first-class source.

Support for "committees" platform and data pipeline:

Added two new activity types, added-to-committee and removed-from-committee, to the activityTypes table for tracking committee membership events.
Introduced a new source query and transformer for the "committees" platform in the Snowflake connectors, enabling extraction and transformation of committee membership data. [1] [2]
Registered the committees source and transformer in the integrations index, and defined its data source name. [1] [2] [3]

Type system and configuration updates:

Added "committees" to the PlatformType, OrganizationSource, and OrganizationAttributeSource enums, and included it in the organization attribute source priority list, ensuring proper handling and prioritization. [1] [2] [3] [4]
Defined new types and scoring grid for committee activities in the integrations library, and re-exported them for usage across the codebase. [1] [2]

Note

Medium Risk
Introduces a new Snowflake ingestion path and DB activity types for committee membership events, which can affect exported data volume and downstream identity/org attribution. Risk is moderate due to new joins/timestamps/dedup logic and new enum values being used across services.

Overview
Adds first-class committees platform support end-to-end: new PlatformType.COMMITTEES, OrganizationSource/OrganizationAttributeSource.COMMITTEES, and priority ordering so committee-derived org attributes can be applied.

Extends the data pipeline with a new Snowflake connector source (DataSourceName.COMMITTEES_COMMITTEES) including a multi-join source query (with incremental export support and non-prod limiting) and a transformer that emits added-to-committee / removed-from-committee activities, builds member identities (email/LF username), and derives organization domain identities.

Adds a DB migration inserting the two new committee membership activityTypes, plus an integrations-library CommitteesActivityType + scoring grid export used by the transformer.

^{Written by Cursor Bugbot for commit cb61fdd. This will update automatically on new commits. Configure here.}

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

github-actions · 2026-04-02T13:58:17Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2026-04-02T13:58:32Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2026-04-02T13:59:54Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

cursor · 2026-04-02T14:03:08Z

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

+      platform: PlatformType.COMMITTEES,
+      timestamp: (row.LASTMODIFIEDDATE as string | null) || null,
+      score: COMMITTEES_GRID[type].score,
+      sourceId: committeeId,


Activity sourceId uses committeeId causing uniqueness collisions

High Severity

The sourceId on the activity is set to committeeId (from row.COMMITTEE_ID), which identifies the committee itself, not the individual membership record. The database has a unique index on (tenantId, platform, type, sourceId, segmentId, channel), so multiple members added to the same committee in the same segment will collide, causing all but one membership activity to be lost. Every other transformer uses a per-record unique ID (e.g., registrationId, enrollmentId, certificateId). The row's SFID (Salesforce record ID), which is already selected and used for dedup in the SQL query, is the correct unique identifier here.

Additional Locations (1)

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts#L29-L30

@mbani01 I think you were right, same committee will result in deduplication to fail.... because everything else will be different.

Also channel is missing which I think is required. Let's add committeeName to channel and for sourceId.... perhaps we can go with your suggestion commiteId + unique identifier.

Copilot

Pull request overview

Adds “committees” as a first-class platform across the pipeline (types/config, Snowflake connector extraction+transform, and DB activity type registration) so committee membership events can be exported and represented as activities.

Changes:

Added committees to platform/org source enums and organization attribute source priority.
Implemented Snowflake connector source query + transformer for committee membership events.
Added DB migration to register new committee membership activity types.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
services/libs/types/src/enums/platforms.ts	Adds `PlatformType.COMMITTEES`.
services/libs/types/src/enums/organizations.ts	Adds `COMMITTEES` to org source enums.
services/libs/integrations/src/integrations/index.ts	Re-exports committees integration types.
services/libs/integrations/src/integrations/committees/types.ts	Defines committee activity types + scoring grid.
services/libs/data-access-layer/src/organizations/attributesConfig.ts	Adds committees to org attribute source priority list.
services/apps/snowflake_connectors/src/integrations/types.ts	Adds committees datasource name.
services/apps/snowflake_connectors/src/integrations/index.ts	Registers committees platform and datasource.
services/apps/snowflake_connectors/src/integrations/committees/committees/buildSourceQuery.ts	Adds Snowflake query for committee membership export.
services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts	Transforms exported rows into committee membership activities.
backend/src/database/migrations/V1775064222__addCommitteesActivityTypes.sql	Inserts committees activity types into `activityTypes`.
backend/src/database/migrations/U1775064222__addCommitteesActivityTypes.sql	Undo migration placeholder (empty).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-02T14:04:55Z

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

+    const activity: IActivityData = {
+      type,
+      platform: PlatformType.COMMITTEES,
+      timestamp: (row.LASTMODIFIEDDATE as string | null) || null,
+      score: COMMITTEES_GRID[type].score,
+      sourceId: committeeId,
+      sourceParentId: null,
+      member: {


activity.sourceId is set to committeeId (COMMITTEE_ID). That value is shared by many rows/members, so activities will collide on the unique key (tenant/platform/type/sourceId/segmentId) and later rows can overwrite or fail to insert. Use a per-membership unique identifier like row.SFID for sourceId (and keep COMMITTEE_ID as an attribute and/or sourceParentId).

Same as above

Copilot · 2026-04-02T14:04:56Z

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

+      return null
+    }
+
+    const committeeId = (row.COMMITTEE_ID as string).trim()


committeeId is computed via (row.COMMITTEE_ID as string).trim() without null/undefined protection. If COMMITTEE_ID is missing or not a string for any row, this will throw and safeTransformRow will skip the row. Consider using optional chaining + explicit skip/log when COMMITTEE_ID is not present.

Suggested change

const committeeId = (row.COMMITTEE_ID as string).trim()

const rawCommitteeId = row.COMMITTEE_ID

const committeeId =

typeof rawCommitteeId === 'string' && rawCommitteeId.trim().length > 0

? rawCommitteeId.trim()

: null

if (!committeeId) {

log.warn(

{ sfid: row.SFID, rawCommitteeId: row.COMMITTEE_ID, email },

'Skipping row: missing or invalid committeeId',

)

return null

}

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

github-actions · 2026-04-02T14:47:29Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

CLAassistant · 2026-04-02T14:51:09Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

joanagmaia · 2026-04-02T15:56:52Z

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

+      identities.push({
+        platform: PlatformType.COMMITTEES,
+        value: email,
+        type: MemberIdentityType.USERNAME,
+        verified: true,
+        verifiedBy: PlatformType.COMMITTEES,


We should also add type email on this case no?

joanagmaia · 2026-04-02T16:01:37Z

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

+      platform: PlatformType.COMMITTEES,
+      timestamp: (row.LASTMODIFIEDDATE as string | null) || null,
+      score: COMMITTEES_GRID[type].score,
+      sourceId: committeeId,


@mbani01 I think you were right, same committee will result in deduplication to fail.... because everything else will be different.

Also channel is missing which I think is required. Let's add committeeName to channel and for sourceId.... perhaps we can go with your suggestion commiteId + unique identifier.

joanagmaia · 2026-04-02T16:01:45Z

services/apps/snowflake_connectors/src/integrations/committees/committees/transformer.ts

+    const activity: IActivityData = {
+      type,
+      platform: PlatformType.COMMITTEES,
+      timestamp: (row.LASTMODIFIEDDATE as string | null) || null,
+      score: COMMITTEES_GRID[type].score,
+      sourceId: committeeId,
+      sourceParentId: null,
+      member: {


Same as above

feat: committees implementation (WIP)

5f5e903

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

mbani01 self-assigned this Apr 2, 2026

Copilot AI review requested due to automatic review settings April 2, 2026 13:58

Copilot started reviewing on behalf of mbani01 April 2, 2026 13:58 View session

cursor bot reviewed Apr 2, 2026

View reviewed changes

Copilot AI reviewed Apr 2, 2026

View reviewed changes

fix: activity timestamp and use cncf for staging tests

bdc9da9

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

mbani01 changed the title ~~feat: committees implementation~~ feat: committees implementation [CM-1066] Apr 2, 2026

Merge branch 'main' into feat/snowflake_committees_implementation

cb61fdd

mbani01 requested a review from joanagmaia April 2, 2026 14:48

joanagmaia reviewed Apr 2, 2026

View reviewed changes

-    const committeeId = (row.COMMITTEE_ID as string).trim()
+    const rawCommitteeId = row.COMMITTEE_ID
+    const committeeId =
+      typeof rawCommitteeId === 'string' && rawCommitteeId.trim().length > 0
+        ? rawCommitteeId.trim()
+        : null
+    if (!committeeId) {
+      log.warn(
+        { sfid: row.SFID, rawCommitteeId: row.COMMITTEE_ID, email },
+        'Skipping row: missing or invalid committeeId',
+      )
+      return null
+    }

Conversation

mbani01 commented Apr 2, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor bot Apr 2, 2026

Choose a reason for hiding this comment

Activity sourceId uses committeeId causing uniqueness collisions

Uh oh!

joanagmaia Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

joanagmaia Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

CLAassistant commented Apr 2, 2026

Uh oh!

joanagmaia Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

joanagmaia Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

joanagmaia Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mbani01 commented Apr 2, 2026 •

edited by cursor bot

Loading