Problems for capacity test: 1. deadlock 2. low TpmC

[X1anWang]

Hi,

I want to test the capacity of EdgelessDB (e.g., throughput). Given open-sourced TPC-C is for innoDB and sysbench does not support certificate authentication, I have to build a testing program myself (BTW, what tool do you use to gain the experimental result, EdgelessDB delivers close to the native performance?), which is finally decided to be built on the TPC-C GitHub repo.

While the testing program works well on my MySQL, the test on EdgelessDB results in:

1. Continuous deadlock report.
   
   There are deadlock reports for new order and payment operations through the whole process. (But there is 0 failure, as the final testing report showed. So I guess there is no bug for my program and EdgelessDB, since there is no failure, and possibly just because of the workload is too heavy?)
2. Low Transaction Per Minute for tpc-C (TpmC)
   The values of TpmC for EdgelessDB and my MySQL under the same experimental setting are 3196.839 TpmC and 6860.572 TpmC, respectively.

While I know that there are many possible reasons for the result and have a few plans to evaluate them, it will be my honor if I can know your ideas and opinions about the issues. Hearty thanks for your time and patience : )

[thomasten]

Hi,
We used https://github.com/Percona-Lab/sysbench-tpcc for benchmarks. Also see Benchmarks section of this blog post.

What CPU do you run this on?

But there is 0 failure, as the final testing report showed. So I guess there is no bug for my program and EdgelessDB, since there is no failure, and possibly just because of the workload is too heavy?

I'd guess so, too. It might also be related to RocksDB. Do you use InnoDB or Myrocks as storage engine for the MySQL baseline?

[X1anWang]

Hi,

Thanks for your quick response.

1. You used https://github.com/Percona-Lab/sysbench-tpcc but I think it connects database by socket while EdgelessDB needs key/certificate authentication. May I know how you gap the bridge and implement the adaptation for the tpcc testing?
2. We have 8 Intel(R) Xeon(R) CPUs (E3-1280 v6 @ 3.90GHz, with 4 cores each) for the testing. I ran the experiments several times and realized that:
   1) the deadlocks are possibly related to RocksDB, since the deadlocks still existed even with much light workload and my experiment based on innoDB worked well. So I think my testing program is not suitable for EdgelessDB/RocksDB and try to base on your suggested repo.
   2) It seems the result "EdgelessDB is slightly slower than native MariaDB and better than MySQL innoDB" has certain limitation, i.e. workload. Because when I increase the thread number from 4 to 32 in the experiment, the average throughput of MySQL-innoDB increased and even far outperformed EdgelessDB at thread 32, while the counterpart of EdgelessDB decreased.

[thomasten]

1. Use this as manifest:

{
    "sql": [
        "GRANT ALL ON *.* TO root"
    ]
}

Then you can connect with user root without password or certificate.

2. We did the benchmark on an Icelake CPU. Prior generations like your CPU suffer from small EPC size, so performance difference to native MariaDB is expected to be larger.

2.2 That's probably true.

[zeuson0]

1. Use this as manifest:

{
    "sql": [
        "GRANT ALL ON *.* TO root"
    ]
}

Then you can connect with user root without password or certificate.

2. We did the benchmark on an Icelake CPU. Prior generations like your CPU suffer from small EPC size, so performance difference to native MariaDB is expected to be larger.

2.2 That's probably true.

Hi, @thomasten . I use the manifest like you provided, but I could not connect edb anyway. And the problem mentioned here no longer appear.
My question is:

1. How should I make the benchmark ?
2. Has the issue mentioned above been fixed ?

[zeuson0]

@thomasten Could you help me at your convenience？

[thomasten]

Hi @zeuson0,
Sorry for the late response. So if I run EdgelessDB

docker run --pull always -t --name my-edb -p3306:3306 -p8080:8080 -e OE_SIMULATION=1 ghcr.io/edgelesssys/edgelessdb-sgx-1gb

and then initialize it with that manifest

echo '{"sql":["GRANT ALL ON *.* TO root"]}' > manifest.json
curl -k --data-binary @manifest.json https://localhost:8080/manifest

I can connect to it with

mysql -h127.0.0.1 -uroot

Does this work for you? If not, please post any error messages you see.

[zeuson0]

mysql -h127.0.0.1 -uroot

Thanks for your response @thomasten.
I did it like you provided, but It's not work for me. The error message post by mysql client is:

ERROR 1045 (28000): Access denied for user 'root'@'172.17.0.1' (using password: NO)

And edb didn't not post any error.

[thomasten]

Does it make a difference if you run it on another port with

docker run --pull always -t --name my-edb -p3307:3306 -p8080:8080 -e OE_SIMULATION=1 ghcr.io/edgelesssys/edgelessdb-sgx-1gb

and connect with

mysql -h127.0.0.1 -P3307 -uroot

?

[zeuson0]

Does it make a difference if you run it on another port with

docker run --pull always -t --name my-edb -p3307:3306 -p8080:8080 -e OE_SIMULATION=1 ghcr.io/edgelesssys/edgelessdb-sgx-1gb

and connect with

mysql -h127.0.0.1 -P3307 -uroot

?

I originally used port 3307 to try it because the port was occupied. And I tried port 3306 just now, it caused the same error.

[thomasten]

Try debug mode with

docker run --pull always -t --name my-edb -p3306:3306 -p8080:8080 -e OE_SIMULATION=1 -e EDG_EDB_DEBUG=1 ghcr.io/edgelesssys/edgelessdb-sgx-1gb

and manifest

echo '{"sql":["GRANT ALL ON *.* TO root"],"debug":true}' > manifest.json

Then edb may print something if you try to connect.