Skip to content

[测试] 分片夹具布局不真实,缺少属性测试和超预算规模测试 #223

Closed

Description

@codeasier

父 issue:#217

问题

  • tests/core/test_dataset_attribution.py:33-53 的 case() 给 8 个分片塞了同一张 block 表,后续分片才分配的 block 也出现在第 0 片。这不是生产端的真实布局,"归属分片找漏 block"这类错误测不出来
  • 没有任何超出工作预算规模的测试;docs/dataset-performance-baseline.json 只用了一个 581 KB 夹具
  • 文档未说明按默认参数导出的数据集在当前实现下会查询失败

建议

  • 加入真实布局的随机差分属性测试:每个分片只放跨入窗口的 block 和窗口内新分配的 block,覆盖地址复用、preexisting/static、空栈和 NULL 栈;切片后与单 DB 结果对比(审阅时用这种方式跑了 200 个种子 × 20 种查询形态,0 处不一致)
  • 增加默认导出规模的规模/性能测试(可标记为 slow),并更新性能基线
  • 在 H1–H4 修复前,先在文档中注明当前的规模限制

验收

  • 属性测试进入常规测试套件
  • 至少一个默认导出规模的端到端测试

Activity

  1. codeasier commented on Oct 10, 2026

    @codeasier
    OwnerAuthor

    复核结论(2026-10-10,未修改产品代码)

    基于 commit 7c33464906aced190698452ebb3a8de8e8e9ab05、非 editable 安装的 pt-snap-cli 0.4.0.post1.dev31、Python 3.12.14 实测。结论支持本 issue 的“需要真实布局与默认容量测试”,但两处现状描述应收窄;随机差分还发现了一个低优先级舍入一致性问题。

    1. 已证实的测试缺口与需要修正的表述

    • case() 确实把同一批 10 个 block 插入全部 8 个分片,包括尚未在该片窗口分配的 block。这不能有效检验 owner-event 片的成员完整性。
    • 性能基线的源文件确为 581,140 bytes,容量设置 2000;分片模式仅 8092 个真实事件、5 片,不是默认容量压力证据。见固定版本基线。
    • 不能说“没有任何超预算测试”:现有 test_work_limits… 已把工作预算调低到 4/16 行、1 byte 测拒绝路径。缺的是使用真实默认预算、跨过 100000 行与默认每片 500000 事件的规模/E2E 测试。
    • 不能说文档完全没写限制:英文 / 中文 已写 100000 行 / 64 MiB、重复读取计费、超限整体失败。需要补充的是把 DEFAULT_CAPACITY=500000 与“按默认参数生成的单片完整扫描也可能失败”直接说明。

    2. 真实成员布局随机差分:200 种子 × 20 形态

    独立脚本仅生成 JSON/SQLite,不加载任何 pickle,不导入仓库 tests,不修改/猴子补丁产品;实际调用安装包的 SnapshotAnalyzer.execute_query。每种子 80 个真实事件,8 片 × 10 事件;每片仅有窗口内新分配或从前一窗口仍存活进入的 block。block 的 alloc/free 全局生命周期 ID 在各片保持一致,已完成释放的 block 不进入后续片。

    覆盖地址复用、preexisting/static、NULL/空栈、字面量 [missing callstack]、空白栈、跨窗口分配/请求释放/完成释放。200 个种子均包含各类边界,累计 2753 条已完成的跨窗口生命周期。20 形态覆盖 memory_peak、allocator_gap、allocation、event、block、leak_detection、freed_block_lifetime、callstack_analysis、preexisting_live、active_blocks_at_event、active_memory_callstack_at_event,含区间、排序、分页和 top_n。

    • 4000 次比较中 3998 次严格相等,2 次仅百分比舍入不同;不能写“0 处不一致”。
    • seed 136 / E39:80/2048 的百分比为 dataset 3.9062、single 3.9063;16/2048 为 0.7812 / 0.7813。
    • seed 196 / E40:720/2048 为 35.1562 / 35.1563。
    • 4000 次的被比较非百分比字段都一致,包括行集合/排序、字节、分组和总数;实际查询前后 SQLite 文件 SHA-256 未变。
    • 单次运行 71.155s;这仅为本机一次执行时间,不是性能承诺。

    比较边界必须明确:dataset 独有的来源/身份元数据不与单库强行逐字段比较;定点 block 的存储 state 不代表请求事件状态,因此此字段不参与该项差分;global block 的 single 参考 state 采用同一个 latest-containing-observation 口径。全局 event/allocation 显式 min_id=0,避免把 standalone 允许的负边界行与 dataset 明确排除的边界行混为 bug。这个测试验证兼容 v1 的查询/成员布局,不是 exporter/replay 生产链 E2E,也不覆盖 native v2/多设备。

    舍入差异已最小化到 2 个 block(大小 1 与 127)、2 个事件、2 个单事件分片:E1 的小组百分比 dataset 0.7812,single 0.7813。原因是 Python round 与 SQL ROUND 的中点舍入口径不同。建议把该最小用例收入一致性测试,而不是用过宽容差掩盖它。

    3. 默认规模证据和验收建议

    同次复核的 #218 测试实际生成了 1,000,000 事件 / 2 片 × 500,000 的兼容 v1 SQLite;block 只出现在已分配及后续存活窗口,6 个全局模板都在累计 100096 fetched rows 时触发预算错误;未插桩的安装版 CLI memory_peak 也 exit 1。这支持缺少默认容量测试,但仍是合成 SQLite + 正式查询路径,不能冒称执行过真实 exporter/import E2E。

    建议验收拆成:

    1. 常规 suite 纳入确定种子、真实成员布局的差分测试,并固定地址复用、跨片来源、NULL/空栈与舍入中点最小回归。
    2. slow 标记使用真实默认预算,至少覆盖 100000 门槛和 500000 默认每片规模;短栈/长栈分别验证行/字节预算,不用固定耗时断言。
    3. 另加受信且经 fixture gate 的 producer→export→query E2E,真正覆盖默认 exporter,而不只手工建库。
    4. 完成 H1–H4 前,文档显式关联默认容量与失败后果;修复后同步基线与测试预期。

    可复现脚本

    按 #217 本次汇总评论的 git/venv/pip 步骤建立上述 commit 的非 editable安装环境后运行,下方只写新建输出目录,不读 pickle。精确运行方式:

    PYTHONPATH= venv/bin/python realistic_diff.py --out diff200 --seeds 200

    脚本输出 diff200/results.json、每个种子的 JSON 时间线和 SQLite;failures 中应保留上述 2 个百分比差异,而非宣称全通过。

    展开完整独立差分脚本 realistic_diff.py
    """Independent compatibility-v1 SQLite differential: no repository imports or pickle."""
    import argparse, hashlib, json, os, random, sqlite3, sys, time
    from contextlib import closing
    from pathlib import Path
    import pt_snap_cli
    from pt_snap_cli.api import SnapshotAnalyzer
    from pt_snap_cli.core.dataset_contract import ACTION_NAMES, BLOCK_STATES, validate_dataset
    
    N=80; CAP=10
    SCHEMA='''CREATE TABLE dictionary ("table" TEXT,"column" TEXT,"key" TEXT,"value" TEXT);
    CREATE TABLE trace_entry_0 (id INTEGER PRIMARY KEY, action INTEGER,address INTEGER,size INTEGER,stream INTEGER,allocated INTEGER,active INTEGER,reserved INTEGER,callstack TEXT);
    CREATE TABLE block_0 (id INTEGER PRIMARY KEY,address INTEGER,size INTEGER,requestedSize INTEGER,state INTEGER,allocEventId INTEGER,freeEventId INTEGER);'''
    
    def timeline(seed):
        rng=random.Random(seed)
        blocks={-2:dict(id=-2,address=900,size=48,requestedSize=40,alloc=-1,free=-1,request=-1,stack=None),-1:dict(id=-1,address=800,size=96,requestedSize=80,alloc=-1,free=-1,request=-1,stack=None)}
        trace=[]; nextaddr=2000
        fixed={0:('a',1000,'shared'),1:('a',1100,None),2:('a',1200,''),3:('a',1300,'[missing callstack]'),4:('a',1400,' \n'),17:('r',-1),18:('f',-1),22:('r',0),23:('f',0),24:('a',1000,'reuse'),60:('r',24),61:('f',24)}
        protected={-2,-1,0,1,2,3,4,24}
        for event in range(N):
            if event in fixed: op=fixed[event]
            else:
                live=[b['id'] for b in blocks.values() if b['free']==-1 and b['id'] not in protected]
                pending=[i for i in live if blocks[i]['request']>=0]
                ready=[i for i in live if blocks[i]['request']==-1]
                coin=rng.random()
                if coin<0.23 and pending: op=('f',rng.choice(pending))
                elif coin<0.48 and ready: op=('r',rng.choice(ready))
                elif coin<0.87:
                    available=[b['address'] for b in blocks.values() if b['free']>=0 and b['address']>=2000 and all(c['address']!=b['address'] or c['free']>=0 for c in blocks.values())]
                    address=rng.choice(available) if available and rng.random()<0.6 else nextaddr
                    if address==nextaddr: nextaddr+=100
                    op=('a',address,rng.choice(['shared','stack-A','stack-B',None,'','[missing callstack]',' \n','长栈']))
                else: op=('w',)
            kind=op[0]
            if kind=='a':
                size=rng.choice([16,32,64,128,256]); req=size-rng.randrange(0,size//4+1)
                blocks[event]=dict(id=event,address=op[1],size=size,requestedSize=req,alloc=event,free=-1,request=-1,stack=op[2])
                action,address,stack=4,op[1],op[2]
            elif kind in ('r','f'):
                b=blocks[op[1]]; size,address=b['size'],b['address']; stack=rng.choice(['free-A','free-B',None,''])
                if kind=='r': b['request']=event; action=5
                else: b['free']=event; action=6
            else: action,address,size,stack=7,0,0,rng.choice(['workspace',None,''])
            alive=[b for b in blocks.values() if b['free']==-1]
            active=sum(b['size'] for b in alive); allocated=sum(b['size'] for b in alive if b['request']==-1)
            reserved=((active+255)//256)*256+1024
            trace.append((event,action,address,size,rng.choice([0,1]),allocated,active,reserved,stack))
        return trace,list(blocks.values())
    
    def member(b,lo,hi): return (b['alloc']==-1 or b['alloc']<=hi) and (b['free']==-1 or b['free']>=lo)
    def state_at_left(b,lo): return 0 if b['request']>=0 and b['request']<lo else 1
    def row(b,state): return (b['id'],b['address'],b['size'],b['requestedSize'],state,b['alloc'],b['free'])
    def db(path,trace,rows,boundary=True):
        with closing(sqlite3.connect(path)) as c,c:
            c.executescript(SCHEMA)
            c.executemany('INSERT INTO dictionary VALUES (?,?,?,?)',[('trace_entry_0','action',str(i),n) for i,n in enumerate(ACTION_NAMES)]+[('block_0','state',str(i),n) for i,n in BLOCK_STATES])
            c.executemany('INSERT INTO trace_entry_0 VALUES (?,?,?,?,?,?,?,?,?)',trace)
            if boundary: c.execute("INSERT INTO trace_entry_0 VALUES (-1,2,0,1048576,0,999999,999999,999999,'boundary-excluded')")
            c.executemany('INSERT INTO block_0 VALUES (?,?,?,?,?,?,?)',rows)
    
    def build(out,seed):
        root=out/f'seed-{seed:04d}'; root.mkdir(); dataset=root/'dataset'; (dataset/'device_0').mkdir(parents=True)
        trace,blocks=timeline(seed); slices=[]; layout=[]
        for i,lo in enumerate(range(0,N,CAP)):
            hi=lo+CAP-1; rows=[row(b,state_at_left(b,lo)) for b in blocks if member(b,lo,hi)]
            assert all(b[5]==-1 or b[5]<=hi for b in rows)
            assert all(b[6]==-1 or b[6]>=lo for b in rows)
            file=f'device_0/slice_{i:05d}.db'; db(dataset/file,trace[lo:hi+1],rows)
            slices.append(dict(index=i,startEventId=lo,endEventId=hi,file=file,ready=True))
            layout.append(dict(slice=i,lo=lo,hi=hi,block_ids=[r[0] for r in rows]))
        manifest=dict(schemaVersion=1,status='complete',sourceFile='/synthetic/no-pickle',cacheHash='',eventsPerSlice=CAP,devices={'0':dict(eventCount=N,sliceCount=len(slices),readySlices=list(range(len(slices))),slices=slices)})
        (dataset/'manifest.json').write_text(json.dumps(manifest))
        single=root/'single.db'
        # Global block uses latest containing observation, matching its documented state_scope;
        # point-event comparisons exclude stored state, which is not state at requested E.
        finalrows=[row(b,state_at_left(b,max(i*CAP for i in range(N//CAP) if member(b,i*CAP,(i+1)*CAP-1)))) for b in blocks]
        db(single,trace,finalrows)
        (root/'timeline.json').write_text(json.dumps(dict(seed=seed,trace=trace,blocks=blocks,layout=layout),ensure_ascii=False,indent=2))
        validate_dataset(dataset)
        return root,dataset,single,trace,blocks
    
    def shapes(seed):
        event=random.Random(seed+10000).choice([0,9,10,17,18,19,20,22,23,24,29,30,39,40,59,60,61,69,70,79])
        return [
          ('memory_peak',{}),('memory_peak',{'start_id':10,'end_id':61}),
          ('allocator_gap',{}),('allocator_gap',{'start_id':10,'end_id':61}),
          ('allocation',{'min_id':0}),('allocation',{'min_id':10,'max_id':61,'order_by':'active','order_dir':'DESC','limit':7,'offset':2}),
          ('event',{'min_id':0}),('event',{'min_id':0,'order_by':'callstack','order_dir':'ASC','limit':9,'offset':2}),
          ('event',{'min_id':0,'action':4}),('event',{'id':event}),
          ('block',{}),('block',{'address':1000}),('block',{'min_size':32,'order_by':'id','order_dir':'ASC','limit':6,'offset':2}),
          ('leak_detection',{}),('freed_block_lifetime',{}),('callstack_analysis',{'min_count':1}),
          ('preexisting_live',{'event_id':event}),('active_blocks_at_event',{'event_id':event}),
          ('active_memory_callstack_at_event',{'event_id':event,'top_n':-1}),
          ('active_memory_callstack_at_event',{'event_id':event,'top_n':1,'include_static':False,'min_size':16}),
        ]
    
    def hashes(root): return {str(p.relative_to(root)):hashlib.sha256(p.read_bytes()).hexdigest() for p in root.rglob('*.db')}
    
    def run(out,seeds):
        failures=[]; checked=0; coverage=dict(address_reuse_seeds=0,preexisting_seeds=0,static_seeds=0,null_stack_seeds=0,empty_stack_seeds=0,cross_window_lifetimes=0)
        start=time.perf_counter()
        for seed in range(seeds):
            root,dataset,single,traces,blocks=build(out,seed); before=hashes(root)
            for key in ['address_reuse_seeds','preexisting_seeds','static_seeds','null_stack_seeds','empty_stack_seeds']: coverage[key]+=1
            coverage['cross_window_lifetimes']+=sum(b['alloc']>=0 and b['free']>=0 and b['alloc']//CAP!=b['free']//CAP for b in blocks)
            with SnapshotAnalyzer(dataset) as d, SnapshotAnalyzer(single) as s:
                for idx,(name,params) in enumerate(shapes(seed)):
                    try:
                        actual=d.execute_query(name,params,exact_total=True); expected=s.execute_query(name,params,exact_total=True)
                        fields=list(expected['rows'][0]) if expected['rows'] else []
                        if name=='active_blocks_at_event' and 'state' in fields: fields.remove('state')
                        ar=[{k:r[k] for k in fields} for r in actual['rows']]; er=[{k:r[k] for k in fields} for r in expected['rows']]
                        assert ar==er, json.dumps(dict(actual=ar,expected=er),ensure_ascii=False)
                        assert actual['total']==expected['total'], f"totals {actual['total']} != {expected['total']}"
                        assert actual['returned']==expected['returned']
                        if name=='active_blocks_at_event':
                            event=params['event_id']; want={b['id'] for b in blocks if (b['alloc']==-1 or b['alloc']<=event) and (b['free']==-1 or b['free']>event)}
                            assert {r['id'] for r in actual['rows']}==want
                            assert sum(r['size'] for r in actual['rows'])==traces[event][6]
                            assert actual['scope']['source_coverage']['allocation_source_complete']
                        checked+=1
                    except Exception as e:
                        failures.append(dict(seed=seed,shape=idx,template=name,params=params,error=f'{type(e).__name__}: {e}'))
            assert before==hashes(root), 'read-only mutation'
            if seed%20==0: print(json.dumps(dict(progress_seed=seed,checked=checked,failures=len(failures))),flush=True)
        result=dict(package_path=pt_snap_cli.__file__,python=sys.version,seeds=seeds,shapes_per_seed=20,comparisons_passed=checked,failures=failures,coverage=coverage,seconds=time.perf_counter()-start,scope='synthetic compatibility-v1; real installed QueryService/SQLite, no producer or pickle; state excluded only for point-event block comparison; global state aligned to documented latest-containing observation; dataset-only provenance excluded from common-schema row comparison')
        (out/'results.json').write_text(json.dumps(result,ensure_ascii=False,indent=2)); print(json.dumps(result,ensure_ascii=False),flush=True)
    
    if __name__=='__main__':
        ap=argparse.ArgumentParser(); ap.add_argument('--out',type=Path,required=True); ap.add_argument('--seeds',type=int,default=200); a=ap.parse_args(); a.out.mkdir(parents=True,exist_ok=False); run(a.out,a.seeds)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions