Upserted document construction in `Update` operations ignores valid `$eq` filters if inside `$and` clauses

[toptobes]

While both of the following filters are semantically equivalent, when upserted in an update command, age and key aren't included in the upserted documents due to being inside of $and statements.

"filter": { "$and":[{ "age":12 },{ "key":"k_6a7ab6e493e8f14b314f12a7_0" }] }
"filter": {"age":12, "key":"k_6a7ab6e493e8f14b314f12a7_0 }

This is problematic for clients which provide filter builders, such as Java, C#, and Go, as they use $ands to glue together separate filters:

filter.And(filter.Eq("age", 12), filter.Eq("key", t.Key(0)))

The offending commands are:

• Collection.UpdateOne
• Collection.UpdateMany
• Collection.FindOneAndUpdate

For reference, with Mongo, $anded filters are properly reconstructed inside of documents

$ db.test.updateOne(
  { $and: [{ a: 1 }, { $and: [{ b: 2 }, { c: 3 }] }] },
  { $set: { d: 4 } },
  { upsert: true }
);

$ db.test.findOne();it does indeed upsert the $anded fields:
{ a: 1, b: 2, c: 3, d: 4 }

It probably goes without saying, but non-$eqs and fields in $or are excluded. Explicit $eqs are ofc included, and dot notation works as well.

For validation precedence, both of these rightfully fail:

// MongoServerError: Plan executor error during update :: caused by :: cannot infer query fields to set, path 'a' is matched twice
db.test.updateOne({ $and: [{ a: 1 }, { a: 2 }] }, { $set: { z: 9 } }, { upsert: true })

// MongoServerError: Plan executor error during update :: caused by :: cannot infer query fields to set, both paths 'a.b' and 'a' are matched
db.test.updateOne({ "a.b": 1, a: { c: 2 } }, { $set: { z: 9 } }, { upsert: true })

[vkarpov15]

Just following up on my comments on #2571 - it would be helpful to support $or since $and is typically used with $or (at least in MongoDB). Specifically, $and is often used to compose multiple $or conditions, because there's no other way to represent an AND of ORs in MongoDB filter syntax. Example:

Enterprise test> db.tests.updateOne({ $and: [{ $or: [{ foo: 'bar', baz: 42 }] }, { $or: [{ qux: 42 }] }] }, { $set: { answer: 'test' } }, { upsert: true })
{
  acknowledged: true,
  insertedId: ObjectId('6a95e91b765e6cf73835d80b'),
  matchedCount: 0,
  modifiedCount: 0,
  upsertedCount: 1
}
Enterprise test> db.tests.findOne()
{
  _id: ObjectId('6a95e91b765e6cf73835d80b'),
  qux: 42,
  baz: 42,
  foo: 'bar',
  answer: 'test'
}
Enterprise test> 

The filter builder approach is an interesting wrinkle, I haven't seen people make heavy use of $and to build general queries. #2571 is a meaningful improvement in that case. I don't think that supporting $or is a blocker for #2571, but I just wanted to call out that supporting $or would also be useful.

[toptobes]

Ah good call! I didn't realize this was the case, but after testing myself, it looks like this is only the case when an $or has a single element in its array; otherwise it's ignored.

For example, in your case, or with something like { $or: [{ $and: [{ a: 1 }, { b: 2 }] }] }, it'll still insert { a: 1, b: 2}

But if I had { $or: [{ $or: [{ a: 1 }, { b: 2 }] }] }, it'd ignore both fields.

Now where we handle this, that's a question which I'd want to discuss with Aaron or Hazel, as I can see two paths here:

• Either the reconstruction code checks if the $or has just a single element, and recurses on that, or
• When building the filter DTO itself, it can simplify an $or: [x] to just x