Skip to content

TICKET-055: Deeper DDPG coverage — actor Q-improvement trend (update_actor) #124

Description

@belarusian

Gap

The DDPG integration suite (tests/test_ddpg_integration.py, 5 tests) covers the
critic-loss trend (TICKET-054) and the in-loop Polyak soft-update, but the
actor side is only ever checked for finiteness.

  • test_closed_loop_training_step (line 51) calls update_actor(states) once
    and asserts only math.isfinite(actor_loss) (line 71).
  • test_update_actor_gradient_ascent (test_actor_critic.py:390) computes
    q_before / q_after around a single update_actor call, but the
    assertions are only np.isfinite(...) — the docstring says "Q-values should
    generally increase (gradient ascent)" yet the increase is never asserted.

There is no test that runs repeated update_actor steps and asserts the actor's
Q-value (the quantity the actor is trained to maximise) actually improves.

Impact

  • False confidence on actor learning. A regression that silently breaks the
    actor's gradient (sign flip, detached tape, zeroed actor LR) would still pass
    every existing test.
  • Asymmetric coverage. The critic-learning trend is asserted (TICKET-054);
    the actor-learning trend is not.

Verified empirically (prototype, seed=42 fixture, 50 update_actor steps)

  • Mean Q rises +0.12691 -> +0.13858 (delta +0.01166), identical across 5 runs.
  • Robust across seeds 0/1/7/42 (delta +0.0017 to +0.008), all positive.
  • actor_loss (= -mean Q) falls correspondingly.

Fix

Add one integration test to tests/test_ddpg_integration.py: run repeated
update_actor steps on a fixed batch and assert the actor's mean Q-value
strictly improves (later > earlier) and the actor loss decreases.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions