xmpp_tests.SendAndReceiveTest fails intermittently with AssertionError 'message_recieved' != 'non-existent'

[scragraham]

Near the end of the test we have code that looks like this:

    # Ensure the XMPP message has been received by the application.
    message_received = False
    for _ in range(5):
      response = self.http_get('/xmpp')
      xmpp_info = json.loads(response.payload)
      self.assertEquals(response.status, 200)
      self.assertTrue(xmpp_info['status'])
      if xmpp_info['state'] == 'message received!':
        message_received = True
        break
      time.sleep(1)

    self.assertTrue(message_received)

    # finally, clean up the mess we made for this test
    response = self.http_delete('/xmpp')
    self.assertEquals(response.status, 200)

    # and we should see no metadata
    response = self.http_get('/xmpp')
    xmpp_info = json.loads(response.payload)
    self.assertEquals(response.status, 200)
    self.assertTrue(xmpp_info['status'])
    self.assertEquals(xmpp_info['state'], 'non-existent')

Looking at the java application logs:

INFO hawkeye Response: 200 http://40.121.41.66:8080/java/xmpp
Server: nginx/1.10.3 (Ubuntu)
Date: Thu, 01 Aug 2019 21:42:28 GMT
Content-Type: application/json
Content-Length: 45
Connection: keep-alive
Cache-Control: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT

{"state":"message received!","status":true}

INFO hawkeye Request: DELETE http://40.121.41.66:8080/java/xmpp
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.22.0
Content-Length: 0


INFO hawkeye Response: 200 http://40.121.41.66:8080/java/xmpp
Server: nginx/1.10.3 (Ubuntu)
Date: Thu, 01 Aug 2019 21:42:29 GMT
Content-Length: 0
Connection: keep-alive
Cache-Control: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT


INFO hawkeye Request: GET http://40.121.41.66:8080/java/xmpp
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.22.0


INFO hawkeye Response: 200 http://40.121.41.66:8080/java/xmpp
Server: nginx/1.10.3 (Ubuntu)
Date: Thu, 01 Aug 2019 21:42:29 GMT
Content-Type: application/json
Content-Length: 45
Connection: keep-alive
Cache-Control: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT

{"state":"message received!","status":true}

ERROR hawkeye tests.xmpp_tests.SendAndReceiveTest.runTest - failed with error:
Traceback (most recent call last):
  File "/home/jenkins/jobs/Hawkeye-Test-Azure-Image/workspace/hawkeye/test-suite/hawkeye_test_runner.py", line 410, in runTest
    self.run_hawkeye_test()
  File "/home/jenkins/jobs/Hawkeye-Test-Azure-Image/workspace/hawkeye/test-suite/tests/xmpp_tests.py", line 63, in run_hawkeye_test
    self.assertEquals(xmpp_info['state'], 'non-existent')
AssertionError: u'message received!' != 'non-existent'

For some reason the key doesn't look like it is getting removed from the datastore:

INFO hawkeye Request: DELETE http://40.121.41.66:8080/java/xmpp
INFO hawkeye Response: 200 http://40.121.41.66:8080/java/xmpp
INFO hawkeye Request: GET http://40.121.41.66:8080/java/xmpp
INFO hawkeye Response: 200 http://40.121.41.66:8080/java/xmpp
{"state":"message received!","status":true}

This might be a timing issue.