Markings by jalphonso · Pull Request #3469 · NationalSecurityAgency/datawave

jalphonso · 2026-03-23T17:25:34Z

No description provided.

jsduncan2 · 2026-04-08T00:47:57Z

+    @SuppressWarnings({"unchecked", "rawtypes"})
+    public static Markings<?> combine(MarkingFunctions<?> markingFunctions, Collection<? extends Markings<?>> markings) throws MarkingFunctions.Exception {
+        return ((MarkingFunctions) markingFunctions).combine(markings);
+    }


I think we can be a little more specific with generics here:

Suggested change

@SuppressWarnings({"unchecked", "rawtypes"})

public static Markings<?> combine(MarkingFunctions<?> markingFunctions, Collection<? extends Markings<?>> markings) throws MarkingFunctions.Exception {

return ((MarkingFunctions) markingFunctions).combine(markings);

}

public static <T extends Markings<?>, U extends MarkingFunctions<T>> T combine(U markingFunctions, Collection<T> markings) throws MarkingFunctions.Exception {

return markingFunctions.combine(markings);

}

lets see if we can address this down the road on another MR

jsduncan2 · 2026-04-08T00:48:53Z

+                    if (markings1 instanceof AccessExpressionMarkings && markings2 instanceof AccessExpressionMarkings) {
+                        AccessExpressionMarkings aem1 = (AccessExpressionMarkings) markings1;
+                        AccessExpressionMarkings aem2 = (AccessExpressionMarkings) markings2;
+                        return combine(markingFunctions, List.of(aem1, aem2));


If the above generic suggestion works, i think you'll need to modify this slightly:

Suggested change

if (markings1 instanceof AccessExpressionMarkings && markings2 instanceof AccessExpressionMarkings) {

AccessExpressionMarkings aem1 = (AccessExpressionMarkings) markings1;

AccessExpressionMarkings aem2 = (AccessExpressionMarkings) markings2;

return combine(markingFunctions, List.of(aem1, aem2));

if (markingFunctions instanceof MarkingFunctions.Default && markings1 instanceof AccessExpressionMarkings && markings2 instanceof AccessExpressionMarkings) {

MarkingFunctions<AccessExpressionMarkings> defaultMarkingFunctions = (MarkingFunctions.Default) markingFunctions;

AccessExpressionMarkings aem1 = (AccessExpressionMarkings) markings1;

AccessExpressionMarkings aem2 = (AccessExpressionMarkings) markings2;

return combine(defaultMarkingFunctions, List.of(aem1, aem2));

lets see if we can address this down the road on another MR

jsduncan2 · 2026-04-08T01:04:43Z

+import com.fasterxml.jackson.annotation.JsonTypeInfo;
+
+@JsonTypeInfo(use = JsonTypeInfo.Id.CLASS)
+public interface Markings<T> {


Can we restrict T here at all?

lets see if we can address this down the road on another MR

jsduncan2 · 2026-04-09T01:14:16Z

 public interface HasMarkings {

-    void setMarkings(Map<String,String> markings);
+    void setMarkings(Markings<?> markings);

-    Map<String,String> getMarkings();
+    Markings<?> getMarkings();
 }


I'm guessing there's some issue you'll eventually run into, but here's a naive suggestion (which will require a lot of associated changes, some of which may be impossible):

public interface HasMarkings <T> { void setMarkings(Markings<T> markings); Markings<T> getMarkings(); }

lets see if we can address this down the road on another MR

…expression markings are present

jalphonso · 2026-05-08T17:51:19Z

-    public void inheritFromRootPayload(RawRecordContainer event, Map<String,Collection<Object>> metadata) {
-        // TODO: No-op here, but need to refactor the following in upstream
-        // if (metadata == null)
-        // return;
-        //
-        // IBaseDataObject payload = (IBaseDataObject) event.getAuxData();
-        // if (payload != null) {
-        // for (String field : helper.getInheritedPayloadFields()) {
-        // if (metadata.containsKey(field) && (!payload.hasParameter(field))) {
-        // payload.putParameter(field, metadata.get(field));
-        // }
-        // }
-        // }
-    }
-
-    public void addMetadataFromParms(RawRecordContainer event, Map<String,Collection<Object>> metadata, String id) {
-
-        for (Map.Entry<String,Collection<Object>> entry : metadata.entrySet()) {
-            CharSequence key = entry.getKey();
-            for (Object value : entry.getValue()) {
-                if (value == null)
-                    continue;
-
-                if (helper.getUuids().contains(key)) {
-                    event.getAltIds().add(String.valueOf(value));
-                }
-                if (this.helper.getSecurityMarkingFieldDomainMap().containsKey(key)) {
-                    addSecurityMetadataFromParms(key, value, event);
-                }
-            }
-        }
-
-        Multimap<String,String> newMetadata = HashMultimap.create();
-        Multimap<String,MetadataIdParser> fieldParsers = helper.getMetadataFieldParsers();
-        for (String field : fieldParsers.keySet()) {
-            if (metadata.containsKey(field) && metadata.get(field) != null) {
-                for (MetadataIdParser parser : fieldParsers.get(field)) {
-                    for (Object v : metadata.get(field)) {
-                        if (v == null)
-                            continue;
-                        try {
-                            parser.addMetadata(event, newMetadata, v.toString());
-                        } catch (Exception e) {
-                            log.error("Unable to apply " + parser + " to " + field, e);
-                            event.addError(RawDataErrorNames.FIELD_EXTRACTION_ERROR);
-                        }
-                    }
-                }
-            }
-        }
-
-        // If the ID is a UUID (i.e., contains no date), try to get
-        // event metadata from the parameters
-        if ((id != null) && (id.length() >= UUID_LENGTH) && id.matches(UUID_PATTERN)) {
-            fieldParsers = helper.getMetadataFieldUuidParsers();
-            for (String field : fieldParsers.keySet()) {
-                if (metadata.containsKey(field) && metadata.get(field) != null) {
-                    for (MetadataIdParser parser : fieldParsers.get(field)) {
-                        for (Object v : metadata.get(field)) {
-                            if (v == null)
-                                continue;
-                            try {
-                                parser.addMetadata(event, newMetadata, v.toString());
-                            } catch (Exception e) {
-                                log.error("Unable to apply " + parser + " to " + field, e);
-                                event.addError(RawDataErrorNames.FIELD_EXTRACTION_ERROR);
-                            }
-                        }
-                    }
-                }
-            }
-        }
-    }
-
-    protected void addSecurityMetadataFromParms(CharSequence key, Object value, RawRecordContainer event) {
-        // If fieldName is a security marking field (as configured by EVENT_SECURITY_MARKING_FIELD_NAMES),
-        // then put the marking value into this.securityMarkings, where 'key' maps to the domain for the marking
-        // (as configured by EVENT_SECURITY_MARKING_FIELD_DOMAINS)
-        if (!StringUtils.isEmpty(key.toString()) && !StringUtils.isEmpty(value.toString())) {
-            event.addSecurityMarking(this.helper.getSecurityMarkingFieldDomainMap().get(key.toString()), value.toString());
-        }
-    }
-


I don't think this is used anywhere but please doublecheck. There are other references to EVENT_SECURITY_MARKING_FIELD_NAMES and EVENT_SECURITY_MARKING_FIELD_DOMAINS that can be cleaned up once confirmed

jalphonso · 2026-05-08T18:31:35Z

+    @Override
+    public void setMarkings(Markings<?> markings) {
        this.markings = markings;
-        this.setSecurityMarkings(markings.get("security"));


what is the key "security" used for?

jalphonso self-assigned this Mar 23, 2026

jalphonso marked this pull request as draft March 23, 2026 17:25

jalphonso force-pushed the accumulo-access branch 8 times, most recently from e86e7ba to 0e0b4d2 Compare March 26, 2026 01:35

ddanielr reviewed Mar 26, 2026

View reviewed changes

Comment thread core/base-rest-responses/src/main/java/datawave/marking/ColumnVisibilitySecurityMarking.java Outdated

jalphonso force-pushed the accumulo-access branch 3 times, most recently from 07765cd to 65db1ad Compare March 27, 2026 17:27

keith-turner reviewed Mar 27, 2026

View reviewed changes

jalphonso force-pushed the accumulo-access branch 5 times, most recently from 8bc250c to 98f8567 Compare April 2, 2026 17:24

jalphonso force-pushed the accumulo-access branch from 98f8567 to b07ead2 Compare April 6, 2026 21:01

jsduncan2 reviewed Apr 7, 2026

View reviewed changes

Comment thread core/utils/accumulo-utils/src/main/java/datawave/marking/AccessExpressionUtil.java Outdated

jalphonso force-pushed the accumulo-access branch 2 times, most recently from cceab28 to e72fd54 Compare April 7, 2026 17:48

jsduncan2 reviewed Apr 8, 2026

View reviewed changes

Comment thread core/utils/accumulo-utils/src/main/java/datawave/marking/AccessExpressionUtil.java

jsduncan2 reviewed Apr 8, 2026

View reviewed changes

jsduncan2 reviewed Apr 9, 2026

View reviewed changes

jschmidt10 reviewed Apr 9, 2026

View reviewed changes

Comment thread core/utils/accumulo-utils/src/main/java/datawave/marking/MarkingFunctions.java

Joe Alphonso added 25 commits May 7, 2026 17:44

tag cloud fixes

795ddb7

fix FieldBase and FieldCardinalityBase column visibility

f4721a0

cleanup dead marking code

f6e97bc

fix SecurityMarking interface

88596d2

marking fixes

30f1c4f

marking class updates

a09b942

rebase fix

923a864

restore flatten functions in mapreduce handler

d042a91

bugfixes

378bcce

remove EMPTY_MARKINGS

780d9fd

marking combine bugfixes

2774f80

rebase fixes

bd6528d

fix MarkingFunctions to return empty ColumnVisibility when no access …

402a29f

…expression markings are present

DoucmentTransformer bugfix

6a0ed7e

more bugfixes

c0cdc02

fix CacheableQueryRowImpl addColumn

1f0c378

cleanup AccessExpressionUtils and MarkingFunctions.Default usage

64efb4f

updating comment in MarkingFunctions

0e27364

cleanup

6cb143b

add unit tests for AccessExpressionMarkings and Utils

402bb29

remove instanceof AccessExpressionMarkings usage

625d3a1

fix RemoteEventQueryLogicIT flaky test

d366cd9

restore translateFromColumnVisibilityForAuths

5f54327

remove instanceof AccessExpression usage

b0bd8bb

restore previous exception msgs and handling

1f76bb2

jalphonso force-pushed the accumulo-access branch from 628a365 to 1f76bb2 Compare May 7, 2026 21:44

Joe Alphonso added 2 commits May 8, 2026 09:54

fix ThreadedRangeBundlerIteratorTest flaky test

0469aa7

dictionary fixes

78fd86e

jalphonso commented May 8, 2026

View reviewed changes

Conversation

jalphonso commented Mar 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jalphonso May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jalphonso May 7, 2026 •

edited

Loading