Skip to content

Golden set: 29 additional issues on the cal.com PRs, each with public evidence - #66

Open
pranav100000 wants to merge 1 commit into
withmartian:mainfrom
pranav100000:golden-set/cal-dot-com-additions
Open

pranav100000 wants to merge 1 commit into
withmartian:mainfrom
pranav100000:golden-set/cal-dot-com-additions

Conversation

@pranav100000

@pranav100000 pranav100000 commented Oct 7, 2026 •

Copy link
Copy Markdown

Golden set: 29 additional issues on the calcom/cal.com PRs

Each entry below is an issue that a review of the benchmark PR surfaced, that the current gold set does not contain, and that is corroborated by at least one of: a recorded reproduction in the review, or the same finding appearing among other published tools' unmatched candidates in results/anthropic_claude-opus-4-5-20251101/evaluations.json (the methodology's "multiple strong tools agree against the gold set" signal).

Disclosure: these were found by Aether Review, a vendor tool, on public forks in the aether-bench GitHub org. Every linked review comment is public and states the trigger; the entries stand or fall on that evidence.

Adding them changes the gold set for every tool, so this is a new golden-set version in the sense of #52; re-running step 3 against stored candidates re-scores existing tools without re-running them.

PR proposed golden comment severity / category corroborated by reproduced evidence
#7232 Non-immediate cancellation only marks the reminder row and defers the SendGrid cancellation to the 15-minute cron, but SendGrid rejects cancellations within ten minutes of send_at, so a reminder cancelled shortly before its send time is still delivered. High / bug greptile-v5, propel no review comment
#7232 A single rejected SendGrid cancellation request throws out of the loop before the accumulated lazy Prisma deletes reach Promise.all, so earlier successful cancellations are never persisted and every later reminder in the batch is skipped. Medium / bug copilot-v2, cubic-dev, devin, greptile-v5, qodo-extended-v2, qodo-v22, sourcery yes review comment
#8330 When a date override matches, the availability check returns true before evaluating busy.every, so existing bookings, calendar conflicts and buffers no longer exclude slots on override dates. High / bug claude-code, propel-v2, qodo-extended, qodo-extended-v2 yes review comment
#8330 The override check rejects a slot if any same-day override excludes it, treating multiple overrides on one date as an intersection instead of a union, so a schedule with two override windows (e.g. 09:00-12:00 and 14:00-17:00) loses all of its slots. High / bug augment, copilot-v2, gemini-v2, kodus-v2, qodo-extended, qodo-extended-summary, qodo-extended-v2, qodo-v2, qodo-v2-2, qodo-v22 yes review comment
#8330 The weekly working-hours check rejects a slot when any interval for that weekday excludes it rather than requiring containment in one interval, so ordinary split-shift schedules produce no available slots. High / bug greptile-v5, propel-v2, qodo-v2 yes review comment
#8330 Applying the timezone offset by extracting only hour and minute drops the date carry, so an override that crosses midnight after conversion (e.g. 22:00-23:59 UTC viewed at UTC+1) is reversed or attached to the wrong day and discarded by buildSlots. Medium / bug — yes review comment
#10600 In the NextAuth authorize callback the password check is skipped whenever totpCode is non-empty, and the new backup-code branch never validates that TOTP, so a valid backup code plus any dummy totpCode value authenticates the user without the correct password. Critical / security cubic-dev, cubic-v2, greptile-v4, qodo-extended-v2, qodo-v2-2 yes review comment
#10600 After successful enablement the modal no longer calls onEnable and only the explicit Close button on the backup-code step invalidates viewer.me, so dismissing that step with Escape or an outside click leaves the settings page showing two-factor as disabled although the server enabled it. Medium / bug claude-code, gitar, greptile-v5, qodo-extended, qodo-v22 no review comment
#10600 The modal passes an explicit type="password" to PasswordField, and Input spreads caller props after its own visibility-controlled type, so the new show/hide toggle changes its icon but never reveals the password. Low / bug claude-code, cubic-dev no review comment
#10600 Setting tabIndex={-1} on the shared password visibility button removes the only show/hide control from keyboard navigation for every PasswordField consumer (login, signup, settings), not just the modal the change targets. Low / bug codeant, codeant-v2, coderabbit, copilot-v2, cubic-dev, qodo-v2-2, qodo-v22 no review comment
#10967 evt.destinationCalendar is initialized to null when neither the event type nor the organizer has a destination, so the optional-chained push of co-host destinations is a no-op and every connected co-host calendar is silently discarded for collective bookings. High / bug codeant-v2, coderabbit, copilot-v2, entelligence, gitlab, greptile-v4-1, kodus-v2, qodo-extended-summary, qodo-extended-v2, qodo-v2 yes review comment
#10967 The per-credential creation loop passes the full organizer-first destination array to every adapter, but Office365, Lark and the base CalDAV service still select element zero, so a second host's credential writes the event into the organizer's calendar (or fails) instead of its own. High / bug codeant-v2, coderabbit, copilot-v2, greptile-v5, mesa, propel-v2, qodo-v22 no review comment
#10967 The new conditional drops result.credentialId from every video reference even though createMeeting supplies it, and handleCancelBooking only calls deleteMeeting for references that carry a credentialId, so cancelling a booking leaves the external video meeting alive. High / bug claude, copilot-v2, entelligence, qodo-v2 no review comment
#10967 In the all-remaining-occurrences branch the loop still iterates only the organizer's credentials and selects each occurrence's first calendar reference, ignoring the co-host credential resolved by the new outer loop, so co-host calendar events of a recurring collective booking are never deleted. Medium / bug copilot, copilot-v2, greptile-v5, macroscope, qodo-v2-2 no review comment
#10967 Request-reschedule cleanup still builds its deletion map only from the requesting user's credentials keyed by provider type, so the co-host calendar events that EventManager now creates under other hosts' credentials cannot be deleted when a reschedule is requested. Medium / bug coderabbit, greptile-v5, qodo-v2-2 no review comment
#11059 The webhook indexes appStoreMetadata with the database app slug (e.g. google-calendar) while the generated metadata is keyed by directory name (googlecalendar, office365calendar, zoomvideo), so valid installed apps are rejected with 404. Medium / bug — yes review comment
#11059 Replacing credential.key on sync leaves the invalid flag untouched, so a credential that was marked invalid stays excluded by consumers such as getCalendarsEvents even after valid replacement keys arrive. Medium / bug — yes review comment
#11059 The Webex sync path still validates the delegated response with webexRefreshedTokenSchema, which requires scope, token_type, refresh_token and refresh_token_expires_in, so the documented minimal access-token-plus-expiry response from the sync endpoint is always rejected. Medium / api — no review comment
#11059 The Lark sync path passes the delegated response through handleLarkError, which requires code === 0 and a nested data payload, so the flat access-token response from the sync endpoint is thrown as an error before any credential is updated. Medium / api cubic-dev, entelligence no review comment
#14740 The handler rebuilds the calendar event with Booking.description, which stores the booker's additional notes, and publishes it as the event description to the calendar provider and the guest emails regardless of the event type's hideCalendarNotes setting, exposing notes that were hidden at booking time. High / security codeant-v2, coderabbit, kodus no review comment
#14740 The reconstructed event omits team.members, and Google Calendar's updateEvent replaces the entire attendee list derived from that field, so adding a guest to a collective booking removes the other hosts from the calendar event. High / bug qodo-v2-2 no review comment
#14740 The rebuilt event omits the persisted iCalUID and iCalSequence, so generateIcsString falls back to the raw booking UID with sequence zero and existing recipients receive an invitation that does not update the calendar entry created by the original invite. Medium / bug — yes review comment
#14740 updateCalendarAttendees discards the results of CalendarManager.updateEvent, which converts provider failures into success:false, so the mutation commits the new attendees and reports success even when no calendar invitation was updated. Medium / bug baz, greptile-v5, mergemonkey, mesa no review comment
#14740 Being an attendee grants permission to add guests even when the event type has disableGuests set, so a booked attendee can invite arbitrary additional people to an event whose host disabled guests. Medium / security — no review comment
#14943 The retry-exhaustion cleanup deletes every reminder whose retryCount exceeds the limit regardless of whether it was eventually scheduled, so a reminder that succeeded on a later attempt is deleted along with the Twilio SID needed to cancel it, leaving an uncancellable SMS scheduled. High / data — yes review comment
#14943 The retryCount update inside the catch block can itself reject (for example when the reminder row was deleted by a booking cancellation while the Twilio call was pending), and that rejection escapes the per-reminder error boundary and aborts processing of every remaining reminder in the batch. Medium / bug — yes review comment
#22532 ENV_FILE="../.env" is resolved relative to the caller's working directory rather than the script's location, so running the script from the repository root edits or creates a .env file in the parent directory instead of the one the app loads. Low / bug augment, copilot, cubic-dev, propel yes review comment
#22532 Deleting the calendar cache only invokes the optional onChanged callback and never invalidates viewer.calendars.connectedCalendars, so in the event-type Advanced tab (which passes no callback) the UI keeps showing the deleted cache's timestamp and delete action. Medium / bug baz, cubic-dev, greptile-v5, qodo-extended-v2, qodo-v2-2 no review comment
#22532 The cache-status lookup queries by the credential IDs shown in the UI, but delegated connections expose the synthetic ID -1 while their cache rows are keyed by the real credential ID, so delegated users never see cache status or the delete action. Medium / bug claude-code, codeant, greptile-v5 no review comment

Each added comment is an issue the current gold set lacks, with a public
review comment stating the trigger, and in most cases the same finding among
other published tools' unmatched candidates or a recorded reproduction.
See the pull request description for the per-entry evidence table.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant