Change Traces Grammar #148

Activity

1. raymondwjang commented on Nov 11, 2025

   

   raymondwjang

   on Nov 11, 2025

   MemberAuthor

   More actions

   Only the peek-size is ever necessary, I think, except for few cases. (like merge or delete)
   
   So, traces should work like this:
       1. Some epochs are cached in memory. This is self.array_.
       2. Appending to numpy is expensive. It recreates the whole array. That means as
           the array grows, appending gets slower.
       3. So, after the memory array gets to a certain point, we should flush the
           array to storage, with only the peek-size remaining. 
       4. Deciding whether to do this in-memory, or by zarr.
           (a) If we do this in-memory, we can use a list of numpy arrays. Appending to
               list is very cheap. It takes nanoseconds.
           (b) Tested with zarr's append_dim, and it's relatively cheap as well. Not sure
               why the time goes up as the saved array gets larger, but even with a
               1000 x 1000000 array does not take more than 20ms. This does have a danger
               of slowing down when we start getting to days to weeks realm.
   

   

   where x is number of appends (appending 1000 x 1000 array each time) and y is microseconds.
   so assuming 20 fps, 1000 frames takes 50 seconds, and if we have 1000 components to track, this translates to about 50,000 seconds of data. This is roughly 13.8 hours. The overhead seems to scale linearly with the accumulated array size, so it might be reasonable to assume that the save operation will eventually start taking 100ms at 5-6 day mark. still very decent considering we're only doing this every 1000 frames, but something to consider nonetheless.
   On the other hand, keeping traces in memory costs 64bits * 1000 * 20 *60 * 60 * 24 = 13.8 GB per day. As much as appending to list takes a lot less longer, if we want to do multi-day, using zarr seems inevitable.
2. raymondwjang

   linked a pull request that will close this issueLong Recording #150on Dec 4, 2025
3. raymondwjang

   closed this as completedon Dec 4, 2025