Implement protobuf

[gnzsnz]

works with protobuf ONLY, requires 10.40 or greater
replaces wrapper._results with a reactive pipeline implemented in eventkit. This simplifies the solution by removing state management

simple test script

import logging
import ib_async


ib_async.util.startLoop()
logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s %(levelname)s %(name)s: %(message)s',
    filename='debug_connection.log',
    filemode='w')
ib = ib_async.IB()

ib.connect('localhost',7497,clientId=0)
print("connection completed")

aapl = ib_async.Stock("AAPL",'SMART','USD')
aapl_details = ib.reqContractDetails(aapl)
print(aapl)
print(aapl_details)

ib.disconnect()
print("disconnected.")

[gnzsnz]

Still a long way to go, but I wanted to share a version that can connect, sync account related objects and perform some basic requests and disconnect.

Implementation status, with their cancel counterpart:

• reqHistoricalData, working for one-off requests. Subscription pending
• reqHistoricalData keepUpToDate=True
• reqHistoricalTicks,
• reqHeadTimeStamp
• reqHistogramData
• reqHistoricalSchedule
• reqTickByTickData
• reqMarketDataType
• calculateImpliedVolatilityAsync
• calculateOptionPriceAsync
• placeOrder
• cancelOrder - breaking change, new data type OrderCancel in parameter
• whatIfOrder
• reqGlobalCancel
• tests: i have a few tests in place. i need to do some clean-up and i will push.
• tests for data converters from/to protobuf
• reqMktData,
• cancelMktData
• reqRealTimeBars,
• reqScannerSubscription,
• reqScannerDataAsync
• cancelScannerSubscription
• reqScannerParameters
• reqPnL,
• reqPnLSingle.
• reqUserInfoAsync
• reqTickers
• reqSmartComponentsAsync
• reqFundamentalDataAsync
• request throttling
• reqMktDepthExchanges
• reqMktDepth
• cancelMktDepth
• getWshMetaDataAsync
• reqWshMetaData
• getWshEventDataAsync
• reqWshEventData
• reqNewsProvidersAsync
• reqNewsArticleAsync
• reqHistoricalNewsAsync
• reqNewsBulletins
• newsTicks
• newsBulletins
• requestFAAsync
• replaceFA
• reqWshMetaData

[gnzsnz]

added ticker protobuf support

[q-phi]

This has been open for a while, are people mainly using ib_async with v1034 of TWS only?

From my research it appears that ib_async would be broken on anything above v1034, but I am not fully sure as I have not tested it myself.

[mattsta]

Yeah, I'm currently running Gateway 10.40.1d with the regular primary branch here and it works as expected (i.e. no protobuf needed there yet).

The next big update for our project here will be more live testing/reviewing of all these protobuf re-architecture changes then branching off into a new major "protobuf-only" version going forward.

eta: ??? I'm still lagging behind a bit on the other updates for even 3.14 validation which we have workarounds/improvements for already, but haven't been merged back coherently yet into a packaged version update.

[q-phi]

Yeah, I'm currently running Gateway 10.40.1d with the regular primary branch here and it works as expected (i.e. no protobuf needed there yet).

The next big update for our project here will be more live testing/reviewing of all these protobuf re-architecture changes then branching off into a new major "protobuf-only" version going forward.

eta: ??? I'm still lagging behind a bit on the other updates for even 3.14 validation which we have workarounds/improvements for already, but haven't been merged back coherently yet into a packaged version update.

So the latest ib gateway works fine with the current version of ib_async (with no protobuf support), am i understanding you correctly? What about the latest version of TWS?

[mattsta]

Looking over this again to get it out soon. Thanks for figuring all this out.

I think the approach is going to be using your design and integration approaches here but with a more merged/unified architecture in some places, so this exact PR won't be merged, but I'm using the integration points and ideas to build off of it for hopefully a consistent and long-term usable system.

The plan is to have two versions going forward:

• the current next branch will be the last primary version of "legacy binary-only mode"
• an upcomong next-protobuf branch will be off of next and include these changes too for protobuf compliance (and maybe performance improvements? hopefully? it would be said if IBKR added protobuf everywhere and the extra encoding/decoding is actually slower) but will also retain backwards compat with the traditional/legacy raw binary system too.

Hopefully will have these out for review and more testing this week. and as usual, most of the live validation tests can only be run against live gateways during live market hours, so we'll see how it goes. I'll try to run run both APIs in parallel this week for my own use to validate and check and just make sure things work as expected.

ALSO the next-protobuf version will be a new major version with some interface and datatype changes for better usage end to end:

• all prices will be Decimal types
• no more nan/int-max/float-max/float-min/int-min for checking unset values, all unset values will now be None by default so no mistaken math or numerical contamination can happen
• all NamedTuples are now slotted/frozen dataclasses
• better stability and performance and interface compliance in a couple places for faster account data and market data usage (some of these will be added to the pre-cutover next branch too and will be updated a bit more)
• may increase minimum python version up a few versions (or even just make it 3.14 for modernization)

More details and branch updates soon!