Replication of results for standard PPO

[MatthewCWeston]

I've been looking at using evidential PPO for a project I've been working on, and, to that ends, I've tried to replicate the comparison results published in the paper. While the results obtained using evidential PPO are in line with those in the paper, the results I get when using standard, non-evidential PPO look quite a bit higher than the ones that were published. In particular, looking at the HalfCheetah environment with the front-one paralysis strategy:

The results I obtained were, for EPPO:

AULC: 3833.0048828125
Final Return: 4082.907470703125

And, for standard PPO:

AULC: 3481.70068359375
Final Return: 3648.4755859375

Loss curves:

EPPO does consistently do better than PPO, but the margin is substantially smaller than what I see in the paper's results. Hyperparameters used are identical to the defaults provided in this repository. The code for my vanilla PPO implementation is here, and I'm reasonably confident that no trace of the evidential critic remains:

evidential_ppo_paper.ipynb

Is there a difference between my implementation and yours?

[MatthewCWeston]

Also, I noticed that removing the priors by setting regularization_coeff to zero yields consistently better performance from EPPO (though not from vanilla PPO, that remains much the same).

I've tried using one of the exploration mechanisms ("ind"), and found that it needs the priors set to avoid producing NaN values with its exploration incentives, by default. Zeroing out NaN values lets me continue using it. Side by side, the two look like this (the custom loss is another experiment I was running, and can be ignored here):

In short, the exploration incentive does seem to be a modest benefit relative to the base setup, a lack of priors seems to yield a much larger improvement (though combining a lack of priors and an exploration incentive via the means above doesn't seem to give us anything extra).

Am I making a mistake somewhere in this chain of reasoning, or are the priors and the associated regularization term a substantial detriment to model performance?

[aportekila]

Thank you for your interest in our work!

Regarding the PPO Baseline Results

1. Looking at your PPO implementation, we noticed that you've included regularization. In our experiments, we did not apply regularization to the vanilla PPO baseline.

2. The standard error for PPO is high even with 15 repetitions. We suggest repeating the experiment at least 5 times with different seeds and comparing the average. The variance is high for PPO-based experiments in general, and we believe non-stationarity increases this variance further.

Regarding Priors and Regularization

The priors are primarily added to address numerical instability caused by the deep evidential regression framework, which has been reported in several prior works that we reference in the paper. We also encountered these issues in our experiments.

We suggest trying other settings and environments, the instability issue will likely become more visible. Adding priors does not degrade performance but provides numerical stability and flexibility to learn.

[MatthewCWeston]

@aportekila Thanks for the response, it was very informative. My interest in EPPO comes from the fact that I've been working on a challenging curriculum learning task where critic plasticity is the performance bottleneck, and the best results I've been able to get thus far come from crude methods like reinitializing the critic's weights after handpicked milestones in the curriculum.

I've been doing more replication work over the past week, verifying EPPO's performance improvements over competing probabilistic value solutions, re-implementing it in the RLlib framework, and writing code to adapt the experiment to a discrete target environment to get an idea of how it will perform for me in practice.

Unfortunately, the performance I've been able to get on tasks resembling my target seems to be lower than baseline PPO, consistent over a number of trials, and I'm not sure why. An exploratory run on my tough curriculum learning setting showed a particularly wide gap, leading me to believe that something might be amiss with how I'm approaching this (hyperparameters are here, if relevant).

I notice that the paper mentions continuous control problems throughout. Is there a known limitation when operating on tasks with discrete action spaces, like my target environment? I've looked through the algorithm, and the improvements to the critic seem sufficiently isolated from the actor for a discrete action head to slot in neatly.

EPPO is of particular importance to me because the next step in my work is transferring the trained agent to a self-play setting, in which the environment will inherently be both highly nonstationary and bereft of any clear transition threshold that I would be able to use when applying fixes like weight reinitialization. Do you have any advice on how I should approach this?