latest TWS using protobufs - does ib_async need to adapt?

[nealeyoung]

Just a heads up... Someone on the TWS API group pointed out that the latest TWS API is starting to use google protobufs. Jurgen states that all clients (including ib_async) will have to be adapted to this.

-Neal

[mattsta]

lol well that'll be fun. Thanks for pointing it out!

At least the current architecture has all message parsing isolated to decoder.py so it should be the only place we need to update protocol logic (in addition to any writing adapters we need to change for type formatting sending to newer gateway versions).

for now, I'm fine with the condition of:

or they are limited to requests, responses, and classes available in 10.34.02.

I agree with rossh here:

• https://groups.io/g/twsapi/message/54068
• https://groups.io/g/twsapi/message/54078

Looking at my own usage, I'm using Linux gateway 10.34.1c and it's... fine. No rush to spend weeks updating things currently.

Could also be a clean "ib_async 3.0" break for a protobuf-only version with all the legacy protocol reading/writing removed.

[gnzsnz]

i'm not sure how this will work with the license, or rather not work.

my discussion with grok ai

https://grok.com/share/bGVnYWN5_b0aae868-5a9c-4525-b85d-e0c7c889d666

my basic assumption here is that we need the protobuf definition for the data objects. this is generated by the protobuf compiler, not really writen by IBKR, but it's part of the software bundle.

Conclusion

Use: You can use the protobuf definition only if your implementation is for your own personal, non-commercial use to manage your IB account. If your API is for a commercial purpose or for third-party use, you cannot use it under this license without further permission from IB.

Distribute: You cannot distribute the protobuf definition under any circumstances without violating the license. If distribution is necessary for your project, you would need to negotiate a different agreement with IB (e.g., by contacting Shail Mangla at opensource@interactivebrokers.com as suggested in Section 0).

Recommendation

If your goal is to create a third-party API that others will use (commercially or otherwise), or if you need to distribute the protobuf definition, you should reach out to Interactive Brokers directly to discuss your use case and seek a commercial license or explicit permission. Without that, your intended use and distribution would likely violate the terms of this Non-Commercial License.

Disclaimer: Grok is not a lawyer; please consult one. Don't share information that can identify you.

[mattsta]

protip: never put "it may be illegal if we do this" in writing.

Duplicating APIs isn't really enforceable. Something similar to: https://en.wikipedia.org/wiki/Google_LLC_v._Oracle_America,_Inc.

One would hope it's just an oversight they didn't include the .proto files, but it's not like they can hide them since the generated files contain all the original information anyway?

Here's a reasonable re-construction of their X_pb2.py files. It's just the same information in the current API just with direct numbers and types attached. Overall it's a much nicer structure than the current protocol method of essentially zero-schema delimited data just sent sequentially and indexed by read position (further adjusted per-server and per-client-version).

ComboLeg.proto

protobufCopysyntax = "proto3";

package protobuf;

option java_package = "com.ib.client.protobuf";
option java_outer_classname = "ComboLegProto";
option csharp_namespace = "IBApi.protobuf";

message ComboLeg {
  optional int32 conId = 1;
  optional int32 ratio = 2;
  optional string action = 3;
  optional string exchange = 4;
  optional int32 openClose = 5;
  optional int32 shortSalesSlot = 6;
  optional string designatedLocation = 7;
  optional int32 exemptCode = 8;
  optional double perLegPrice = 9;
}

Contract.proto

protobufCopysyntax = "proto3";

package protobuf;

import "ComboLeg.proto";
import "DeltaNeutralContract.proto";

option java_package = "com.ib.client.protobuf";
option java_outer_classname = "ContractProto";
option csharp_namespace = "IBApi.protobuf";

message Contract {
  optional int32 conId = 1;
  optional string symbol = 2;
  optional string secType = 3;
  optional string lastTradeDateOrContractMonth = 4;
  optional double strike = 5;
  optional string right = 6;
  optional double multiplier = 7;
  optional string exchange = 8;
  optional string primaryExch = 9;
  optional string currency = 10;
  optional string localSymbol = 11;
  optional string tradingClass = 12;
  optional string secIdType = 13;
  optional string secId = 14;
  optional string description = 15;
  optional string issuerId = 16;
  optional DeltaNeutralContract deltaNeutralContract = 17;
  optional bool includeExpired = 18;
  optional string comboLegsDescrip = 19;
  repeated ComboLeg comboLegs = 20;
}

DeltaNeutralContract.proto

protobufCopysyntax = "proto3";

package protobuf;

option java_package = "com.ib.client.protobuf";
option java_outer_classname = "DeltaNeutralContractProto";
option csharp_namespace = "IBApi.protobuf";

message DeltaNeutralContract {
  optional int32 conId = 1;
  optional double delta = 2;
  optional double price = 3;
}

ExecutionDetailsEnd.proto

protobufCopysyntax = "proto3";

package protobuf;

option java_package = "com.ib.client.protobuf";
option java_outer_classname = "ExecutionDetailsEndProto";
option csharp_namespace = "IBApi.protobuf";

message ExecutionDetailsEnd {
  optional int32 reqId = 1;
}

ExecutionDetails.proto

protobufCopysyntax = "proto3";

package protobuf;

import "Contract.proto";
import "Execution.proto";

option java_package = "com.ib.client.protobuf";
option java_outer_classname = "ExecutionDetailsProto";
option csharp_namespace = "IBApi.protobuf";

message ExecutionDetails {
  optional int32 reqId = 1;
  optional Contract contract = 2;
  optional Execution execution = 3;
}

ExecutionFilter.proto

protobufCopysyntax = "proto3";

package protobuf;

option java_package = "com.ib.client.protobuf";
option java_outer_classname = "ExecutionFilterProto";
option csharp_namespace = "IBApi.protobuf";

message ExecutionFilter {
  optional int32 clientId = 1;
  optional string acctCode = 2;
  optional string time = 3;
  optional string symbol = 4;
  optional string secType = 5;
  optional string exchange = 6;
  optional string side = 7;
  optional int32 lastNDays = 8;
  repeated int32 specificDates = 9;
}

ExecutionRequest.proto

protobufCopysyntax = "proto3";

package protobuf;

import "ExecutionFilter.proto";

option java_package = "com.ib.client.protobuf";
option java_outer_classname = "ExecutionRequestProto";
option csharp_namespace = "IBApi.protobuf";

message ExecutionRequest {
  optional int32 reqId = 1;
  optional ExecutionFilter executionFilter = 2;
}

Execution.proto

protobufCopysyntax = "proto3";

package protobuf;

option java_package = "com.ib.client.protobuf";
option java_outer_classname = "ExecutionProto";
option csharp_namespace = "IBApi.protobuf";

message Execution {
  optional int32 orderId = 1;
  optional string execId = 2;
  optional string time = 3;
  optional string acctNumber = 4;
  optional string exchange = 5;
  optional string side = 6;
  optional string shares = 7;
  optional double price = 8;
  optional int64 permId = 9;
  optional int32 clientId = 10;
  optional bool isLiquidation = 11;
  optional string cumQty = 12;
  optional double avgPrice = 13;
  optional string orderRef = 14;
  optional string evRule = 15;
  optional double evMultiplier = 16;
  optional string modelCode = 17;
  optional int32 lastLiquidity = 18;
  optional bool isPriceRevisionPending = 19;
  optional string submitter = 20;
}

Also gotta love their change logs for introducing major architecture changes: https://ibkrguides.com/releasenotes/prod-2025.htm

[nealeyoung]

Further relevant discussion by Jurgen on the twsapi group on this topic:

https://groups.io/g/twsapi/topic/112061798#msg54225

Notably, Jurgen (perhaps the most competent / authoritative poster on twsapi) discusses the difficulties with using protobuf, and how / why he is avoiding doing that by not advancing to newer versions of the TWS api that require it.

[mattsta]

Thanks for keeping up with the notes over there!

Yeah, just staying on the pre-protobuf version is fine for now.

Attempting to re-use the auto-generated protobuf code is also not great as they mention, but the auto-generated protobuf code basically contains a blob of the original protobuf spec which can be re-run through the protobuf compiler to re-generate the implementations again in a clean/automated way instead of copy/pasting the already generated code elsewhere.

I understand their reason for wanting a better serialization format (instead of the current IBKR binary API fragile serialization method of "every command and reply is just dozens of input strings concatenated together indexed by position in the string"), but whether protobufs is a good choice or another just more structured reusable format is out of our hands at this point.

Hopefully the IBKR dev team will at least provide their official protobuf specs for re-use eventually, especially if they replace all data input/output with these specs. Shared re-usable formats like that would also make upgrading to new API endpoints/features/versions much nicer too since the shared spec formats could allow for "drop-in" feature additions or replacements instead of needing to partially reverse engineer every new binary serialization stream format update each time.

[IanMcDevitt]

I plan to use this library for connection to IBKR; as long as I use a version older than 10.34 I shouldn't have any problems right?

Huge fan of this project btw!

[mattsta]

Correct! IBKR usually has their "stable" version being months or even a year older than their "latest" release too. They retain the old one for a long time while still updating "newer" or weekly ones.

Also the gateway reads the "max supported version" advertised from the API client when an API client connects to know which replies it can send, so you should be able to use new gateways too, but it will just not include their newer API additions (which aren't in the API client anyway yet).

[IanMcDevitt]

Thanks so much for such a thorough answer!

[gnzsnz]

update https://ibkrguides.com/releasenotes/prod-2025.htm

10.40 claims "TWSAPI supports protocol buffers in all requests/responses." so this is getting serious

I can spend some time to start migrating methods to protobuf, but this will take time. @mattsta could we have a protobuf branch with stable methods, at the moment zero, and start merging methods as we implement them.

I would suggest to start creating a test suit, this is not a minor effort and errors will pop-up everywhere. decoder.py, wrapper.py and client.py

as said, i'm whiling to help on the migration.

[mattsta]

Good find. I stopped checking the release notes (https://ibkrguides.com/releasenotes/api.htm) years ago because it was almost never updated (the "gateway beta" version changes every week and there's never any notes about what changes!), but looks like they added new notes for those changes.

We should look over it soon, but for now I think they are just adding parallel features and not replacing or adding new features outside of "the old way" so far.

We can branch off a new protobuf-dev version for ongoing work as to not pollute the current known-working primary branch.

For development simplicity, the protobuf-dev version should probably be 100% protobuf and remove any manual binary encoding/decoding so we don't need to maintain parallel feature states going forward.

The other fun part will be creating more of a live integration test suite against live gateway components in paper trading mode (which we've never had) instead of only developing against "live account testing" as everything has been in the past.

If it helps, this is what I use to auto-deploy gateways automatically (without the license accept input):

bash ./ibgateway-latest-standalone-linux-x64.sh -q -Vsys.licenseAccepted=true

and for fully configuring the gateway:

• copy your ibgateway.vmoptions for extended java params (6 GB+ heap sizes, etc) into the ibgateway extract directory
• also copy a known-good version of your ~/Jts directory for gateway+ibkr configs (gateway restart time, instrument and data settings etc), but that directory also contains things like some account details and your "encrypted weekly session cookie" so it's not clean to just have in a repo itself unless we can make a version without account data inside of it.

🚀

[gnzsnz]

quick update

• I managed to connect and perform the initial sync
• I have implemented protobuf only, no compatibility with current serialization. Requires ibgateway 10.40+
• still a lot of glitches and unexpected errors, but i can connect and get account and trades information.
• data model is evolving and diverging from ib_async. i had to catch-up with a few data objects and there are more to come.
• I have to say that there is no evident benefit to use protobuf from ib_async point of view. there is more code needed to serialize and unserialize. But for ibkr team should help to maintain their api in multiple language. I would expect an improvement in speed as we are using binary format, should be measurable but not significant.
• I have created dedicated files to do serialize and unserialize, which will simplify decoder.py a lot, and keep it shorter.
• Currently, "decoder" is not a clear cut, in the sense that delegates "easy stuff" to wrapper. I'm keeping a one to one mirroring, decoder is now always decoding and wrapper is always working with ib_async dataclasses and "deciding", this will keep architecture and code cleaner.
• I have migrated a only few methods to protobuf just the minimum to connect and synchronize. still a lot of work and tests ahead.
• I will probably implement pre-commit to run ruff and code quality checks, there are a lot of ruff linting problems, so this will take time. Mypy is complaining a lot too 🫠
• I need to do some clean up and quite a few fixes and i will do a first push.
• don't expect anything functional, just minimal connection and a few methods working.
• some initial tests (pytest) in place, key focus at the moment is in data structures.
• for testing i'm using docker to run ibgateway, testinfra is a pytest pluging that allows to spin a docker image. we can delay that for the moment.

Some notes on compatibility:

• 10.40+ only supported, this is because is very challenging to support both versions in parallel. ibapi pythonclient is doing that
• proto only
• ib_async API compatibility, all ib instance methods should work as they work right now. 🤞
• data objects will be mostly compatible with current objects. I think here we need to aim to be compatible with ibkr api as it evolves.

[gnzsnz]

Another quick update

• connection is working
• connection "handshake" -> account synchronization is working. We have trades, executions, commissionReport, account values, positions, portfolioItems, etc, basically everything called on connectionAsync working and syncing on connection.
• ib.qualifyContracts/ib.reqContractDetails both working. This is request/response cycle completed.
• next stop historical data

[mattsta]

Sounds exciting! Will be interesting to try compare minor API performance differences too.

Looking forward to see how everything is being organized and extracted/compared/validated against the standard ibapi stuff too. Hopefully having better and more logical data structure assembly, sending, and re-parsing for inbound data again is more stable over all (especially when adding new API features instead of just "magic positional fields compared against version number capability" for every new update).

[gnzsnz]

More updates pushed into #185

@mattsta I have implemented a few changes that would require your blessing. Please review it and come back with your comments.

I have simplified(or aimed to simplify) wrapper.py, there's quite a lot of "state" there in particular the dictionaries used to track and match requests to responses, which i have removed and replaced by a "reactive" solution. this uses eventkit for reactive pipelines, it's now used on all "request-response" methods, like reqHistoricalData, and i have implemented it too on streaming data like tickers, and "event" data like trades, executions, commission report (still WIP) but fully working for the initial sync.

this approach simplifies a lot wrapper, as there is no need to keep "status" of all requests and tickers. we do keep some state but it's simplified.

Another change i have done to simplify wrapper.py is to move "ticker related logic" to ticker.py now Ticker object manages itself.

One of my main concerns was eventkit performance. as the slot system does not scales well. but for the order of magnitude that we are dealing with here, there are not a problem. We can not handle efficiently 100K listeners, but we are efficient handling 100K events. TWS is limited to 100 tickers, so worst case scenario we will be dealing 1K events, and this is an extreme scenario. I have built a small util that writes to disk trades and fills, and i can handle 10K fills per second, with most of the overload coming from disk operations.

I have a draft PR for eventkit to improve slot performance. coming soon.

there is still work and clean-up to be done, but right now we can connect, get account information, positions, qualify contracts, get historical data, option chain and tickers. All working with protobuf using a reactive "bus".

Other changes worth mentioning:

• an enum for tick types, i.e. TickType.BID, TickType.ASK, rather than 0,1 for the ticktype. there are 105 tick types so this improves code readability.
• a MessageId class, so we manage in&out messages by name rather than id.
• protobuf_converters module, with methods to convert from/to protobuf and ib_async objects. this is quite verbose, but it has greatly simplified client.py

[mattsta]

Sounds impressive! Happy to look over the eventkit updates and try to review or brainstorm any performance improvements too. Would also be interesting running the updates under pypy to compare for performance improvements (I've seen 1000x performance boosts on pypy when it can jit nested data structures into native pointers and structs for much faster access instead of the default python "everything is a dispatch lookup table dict" pattern).

Worst case, we could even add simple and tiny compiled extensions (cython or rust extensions) for fastest data access if there's too much "python overhead" we can't escape for accounting maintenance.

It will also be great to have a more self-managing and well-encapsulated request/response system instead of maintaining manual dicts of ids and results and callbacks everywhere.

The next steps are I have for the current project: some bug fix and minor improvement updates in #189 to keep things going for the current branch, but we'll eventually make a new ib_async 3.0 (4.0? 10.x? follow IBKR API version with minor patch numbers internally?) for the 100% protobuf internal format as well (while keeping our usable API extensions/adapters/helpers on top of it all, etc).

One question with things like protobufs is: do we keep the compiled protobuf sources inside the repo or only keep the protobuf specs/definitions, then the build process creates the protobuf .py files itself? I'm usually not a fan of having auto-generated transpiled things inside repos if they can be auto-generated at build time, but if we have to manually edit/hack around protobuf output selectively, it's useful to have the outputs too (or, in that case, we could just patch the classes externally or have override adapters or something).