WithTimestampResolution option for RedisQueue

[josephbuchma]

Currently timestamps are truncated to second resolution. This change allows to override that, so it's possible to enqueue 200ms into the future, for example.

[taylorchu]

do you have thousands of jobs that are enqueued within 1 second?
I am not sure what this solves.

[josephbuchma]

do you have thousands of jobs that are enqueued within 1 second?

Yes, but that's not the reason. Although I agree that use case I have is rather uncommon (more on that below), having this option (or just using milliseconds by default) does not cost much.

I don't want to take too much of your time, but if you are interested in my use case, here are details.

• We have records stored in DB, each record has a version that is atomically incremented on every update.
• Every time record is created or updated, we enqueue a job with ID equal to the record id, and with record version in payload.
• In the job handler, we fetch the record, and check if its version is still the same as in the job payload, and if it's not, then we just return, since there is another job already enqueued for this record (this is why issue Lost job #68 matters), otherwise we proceed to our time-consuming job.
• Sub-second job delay is useful when we insert new record into DB. Ideally we'd want to enqueue the job without delay at all, but if something goes wrong after inserting the record, the job won't be enqueued, and we can't allow that. So we enqueue job ~100ms into the future before running DB insert.

[taylorchu]

For your use case, you can use uuid as job id, and put both record id and version number in the payload. On job execution, compare version number, and decide whether to proceed.

It sounds like you should only enqueue without delay "after" the row is committed. Because if any error happens during commit, the version number won't be updated, and this job does not need to run. Also if your database does not commit within 100ms, then it will cause race condition.

[josephbuchma]

For your use case, you can use uuid as job id, and put both record id and version number in the payload. On job execution, compare version number, and decide whether to proceed.

That is how it's currently done. I'll do a bit more investigation regarding your concerns about expensiveness of deduplication logic.

It sounds like you should only enqueue without delay "after" the row is committed.

We do that on updates. On inserts we enqueue first, because it's basically a no-op if we fail to insert the record right after. After few job retries we consider insert failed and give up.

Similarly with 100ms delay. In most cases insert will happen much faster. In rare cases when it doesn't, job will pick it up on retry.