Skip to content

Update verinet#107

Open
MarcelBulpr wants to merge 10 commits into
dlshriver:developfrom
MarcelBulpr:update-verinet
Open

Update verinet#107
MarcelBulpr wants to merge 10 commits into
dlshriver:developfrom
MarcelBulpr:update-verinet

Conversation

@MarcelBulpr

Copy link
Copy Markdown

This PR changes the install method of the verinet verifier, from pulling a fixed version from the dnnv github to installing it from pipenv. This will allow dnnv to always get the newest version of verinet (as long as there are no major API changes). It takes the verinet implementation provided by https://github.com/vas-group-imperial/VeriNet.

Some caveats:

  1. The installer reports that the installation fails, even if it succeeded and verinet can be used. But I also had the same issue before the change and with other verifiers
  2. Other verifiers install a specific commit as supposed to the most up-to-date version, this can be done here also if required.
  3. This verinet version uses a different optimizer, Xpress instead of Gurobi.

@codecov

codecov Bot commented Mar 27, 2023

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.72%. Comparing base (f71c983) to head (5c84bce).

❗ There is a different number of reports uploaded between BASE (f71c983) and HEAD (5c84bce). Click for more details.

HEAD has 3 uploads less than BASE
Flag BASE (f71c983) HEAD (5c84bce)
6 3
Additional details and impacted files
@@             Coverage Diff              @@
##           develop     #107       +/-   ##
============================================
- Coverage    89.90%   77.72%   -12.18%     
============================================
  Files          140      140               
  Lines         7982     7982               
  Branches      1440     1440               
============================================
- Hits          7176     6204      -972     
- Misses         579     1634     +1055     
+ Partials       227      144       -83     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dlshriver dlshriver self-assigned this Apr 8, 2023
@dlshriver dlshriver added the enhancement New feature or request label Apr 8, 2023
@MarcelBulpr

MarcelBulpr commented May 3, 2023

Copy link
Copy Markdown
Author

The code I provided to execute the new Verinet version does not quite work. I am currently trying to get it to work, but I have a question, if you have time @dlshriver .

How does DNNV change the networks before passing them to the verifiers?
I am testing with image classification networks on MNIST. The original model correctly classifies the image as 1 and gives a corresponding (10,1) output. The model given to Verinet produces [16.436, 0], which I understand comes from the property using np.argmax(). But I do not understand a) why it has 2 outputs; and b) why does it not return only 0 or 1 due to the comparison in the property.
I provided the property I am working with below.

from dnnv.properties import *
import numpy as np

N = Network("N")
x = Image("properties/0015/mnist/image2.npy")
x = x.reshape(N.input_shape[0])

true_class = 1

epsilon = 0.015

Forall(
    x_, Implies(x - epsilon <= x_ <= x + epsilon, np.argmax(N(x_)) == true_class)
)

I hope I provided enough information.

@dlshriver

Copy link
Copy Markdown
Owner

Hi Marcel, thanks for the PR. This is something I've needed to update for a while, and I appreciate your effort to do so. The reduction used should be the same as the one used by DNNF as described in this paper. The negation of the property is converted to DNF, then each disjunct is converted to a halfspace polytope representation which is used to construct two new layers for the output. The new output of the network has 2 values, the second of which is always 0 and the first of which is strictly greater than 0 if and only if the encoded disjunct is false.

Hopefully that helps a bit. Let me know if you have more questions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants