Repository navigation
Conversation
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
|
Copy of this comment: Nice 😎 The idea of reimplementing an OpenTelemetry component in a different language has many applications: better performance, lower memory usage, removing a dependency, sharing one implementation across several SDKs, etc. There is clear interest in this idea, and this issue is one example. In order to avoid misunderstandings and to make communication easier and more accurate, I would like to propose a name for this idea: refit. The term comes from the ship industry. A refit replaces a ship's machinery while the ship stays the same ship: same hull, same name, same crew. I am deliberately proposing refit instead of port. The term port already has a different meaning: adapting a program so that it runs in an environment where it could not run before, usually on a different operating system or processor. The result is a second version of the program that runs in the new environment. A port benefits the users of the new environment. After Linux was ported from x86 to ARM, the users of Linux on x86 did not get any benefits. A refit changes the implementation of a component in place, so the users of that component keep the same API and get the benefits of the new implementation. It gives us a small set of consistent terms:
A few sentences to show how it reads:
|
ocelotl
left a comment
There was a problem hiding this comment.
I think we could restructure this document for clarity. Also, there is a typo.
| @@ -0,0 +1,90 @@ | |||
| # OpenTelemetry SDK Wrapper | |||
There was a problem hiding this comment.
I understand the idea you have of this project is to "wrap" the C++ SDK with Python bindings, but I think "wrapper" is not specific enough.
Wrapping an SDK is the technique this project will use and wrapping is a technique used all over software for all kinds of purposes.
What is specific to this project is that wrapping an SDK implemented in language B with bindings for language A creates a relationship between SDK A and SDK B: the implementation of SDK A is replaced by SDK B, while the API of A stays exactly as it is for its users.
That relationship will come up again and again, for other components and other languages, so we will need to name the individual cases. If we use the word "wrapper" then the best we can say is "the Python TracerProvider wrapper is feature complete" or "the Node OTLPTraceExporter wrapper is waiting on async exporter support".
Neither of those sentences tells the reader that the Python or Node API for those components is unchanged or that the underlying implementation is not written in Python or Node respectively. The only thing the reader gets is that some components are being wrapped.
On the other hand, with a term of our own, "the Python TracerProvider new_term is feature complete" and "the Node OTLPTraceExporter new_term is waiting on async exporter support" tell the reader exactly what the previous two sentences could not: the Python and Node APIs for those components are unchanged, and the implementation underneath each one is not written in Python or Node.
I think we would benefit from using a new term to describe this project, and I have proposed one here. If we use that proposal, then the title becomes:
| # OpenTelemetry SDK Wrapper | |
| # OpenTelemetry SDK Refit |
Of course, the definition of refit should follow immediately so the reader can understand what we mean by "refit" in the title.
Coining a term is not unusual here. OpenTelemetry has done this before, both to replace a word that was ambiguous and to name something new:
- Baggage replaced "Distributed Context" (oteps#36)
- Instrumentation Scope replaced "Instrumentation Library" (opentelemetry-proto#362)
- Connector named a fifth Collector component type, alongside receiver, processor, exporter and extension
- Injector named the idea behind The OpenTelemetry Injector Project
- Entity named the central concept of Resources and Entities
- Distribution, or "distro", named a packaging pattern that already existed under other names (docs)
There was a problem hiding this comment.
If the specific goal of this project is "C++ Bindings for Python" then that should be the name of the project. That will help people looking at the calendar, scanning through the project lists, etc. Neither "refit" or "wrapper" really describe what the specific goal is.
| @@ -0,0 +1,90 @@ | |||
| # OpenTelemetry SDK Wrapper | |||
There was a problem hiding this comment.
I would suggest we rename this file (if we decide to use my naming proposal) to sdk-refit.md.
There was a problem hiding this comment.
I don't feel strongly on the name and am happy to let others weigh in here, will rename the file appropriately once the name is agreed.
|
|
||
| This project started as an investigation to mitigate some issues identified around memory allocations and global interpreter lock (GIL) contention in Python. As a result of testing an implementation that binds [Python to a C++ SDK](https://github.com/open-telemetry/community/issues/3483), a potential reduction in heap allocations by 10-15% and as much as 40% was observed. | ||
|
|
||
| The project proposed here provides performance the benefits of an underlying SDK across Python initially and possibly other languages in the future. A secondary benefit, though not a goal, would be to allow end users and API implementers to experiment with features already supported in the underlying SDK, that may not yet be implemented in a native SDK. |
There was a problem hiding this comment.
The project proposed here provides performance the benefits
This seems to state the project has already reached the goal of providing benefits and it seems to be contradicted by this line:
An objective is to prove this approach as part of the project.
I think proving the approach proposed here has benefits is the goal of this project.
|
|
||
| This project started as an investigation to mitigate some issues identified around memory allocations and global interpreter lock (GIL) contention in Python. As a result of testing an implementation that binds [Python to a C++ SDK](https://github.com/open-telemetry/community/issues/3483), a potential reduction in heap allocations by 10-15% and as much as 40% was observed. | ||
|
|
||
| The project proposed here provides performance the benefits of an underlying SDK across Python initially and possibly other languages in the future. A secondary benefit, though not a goal, would be to allow end users and API implementers to experiment with features already supported in the underlying SDK, that may not yet be implemented in a native SDK. |
There was a problem hiding this comment.
performance the benefits of an underlying SDK
This needs more context, an SDK does not have performance benefits on its own.
|
|
||
| This project started as an investigation to mitigate some issues identified around memory allocations and global interpreter lock (GIL) contention in Python. As a result of testing an implementation that binds [Python to a C++ SDK](https://github.com/open-telemetry/community/issues/3483), a potential reduction in heap allocations by 10-15% and as much as 40% was observed. | ||
|
|
||
| The project proposed here provides performance the benefits of an underlying SDK across Python initially and possibly other languages in the future. A secondary benefit, though not a goal, would be to allow end users and API implementers to experiment with features already supported in the underlying SDK, that may not yet be implemented in a native SDK. |
There was a problem hiding this comment.
API implementers
I am not sure API implementers will benefit from this project, end users will, that is correct.
|
|
||
| ### Current challenges | ||
|
|
||
| Reference implementation SDKs in every language offer the flexibility that includes every feature supported by OpenTelemetry. In some implementations, this comes at the cost of: |
There was a problem hiding this comment.
Reference implementation SDKs
I think this is another name for native SDKs which is already being used elsewhere in this PR.
|
|
||
| ### Current challenges | ||
|
|
||
| Reference implementation SDKs in every language offer the flexibility that includes every feature supported by OpenTelemetry. In some implementations, this comes at the cost of: |
There was a problem hiding this comment.
I am having a bit of a hard time understanding what you mean here. What do you mean with an SDK offering "flexibility"?
|
|
||
| Reference implementation SDKs in every language offer the flexibility that includes every feature supported by OpenTelemetry. In some implementations, this comes at the cost of: | ||
|
|
||
| * increased resource consumption (memory usage, artifact size) |
There was a problem hiding this comment.
I think using OpenTelemetry regardless of the language increases resource consumption. I think what you mean here is that there are certain implementations of OpenTelemetry that significantly increase resource consumption, and those implementations are the ones who can benefit the most from the approach proposed in this project.
| Reference implementation SDKs in every language offer the flexibility that includes every feature supported by OpenTelemetry. In some implementations, this comes at the cost of: | ||
|
|
||
| * increased resource consumption (memory usage, artifact size) | ||
| * added complexity |
|
|
||
| ### Goals, objectives, and requirements | ||
|
|
||
| The OTel community is currently working across many languages to support reference implementations of all the features in every language. We have always proposed that the API/SDK separation would make it possible for end-users to swap SDKs without having to re-instrument their application/instrumentations code. The proposal for this project is to leverage that model as well as existing OpenTelemetry component to provide alternative SDKs for specific use-cases. |
There was a problem hiding this comment.
The OTel community is currently working across many languages to support reference implementations of all the features in every language.
I think what you are trying to say here is that the OTel community is currently working to add all the OpenTelemetry features to every native SDK.
|
|
||
| ### Goals, objectives, and requirements | ||
|
|
||
| The OTel community is currently working across many languages to support reference implementations of all the features in every language. We have always proposed that the API/SDK separation would make it possible for end-users to swap SDKs without having to re-instrument their application/instrumentations code. The proposal for this project is to leverage that model as well as existing OpenTelemetry component to provide alternative SDKs for specific use-cases. |
There was a problem hiding this comment.
We have always proposed...
We have always promised that to our users.
|
|
||
| ### Goals, objectives, and requirements | ||
|
|
||
| The OTel community is currently working across many languages to support reference implementations of all the features in every language. We have always proposed that the API/SDK separation would make it possible for end-users to swap SDKs without having to re-instrument their application/instrumentations code. The proposal for this project is to leverage that model as well as existing OpenTelemetry component to provide alternative SDKs for specific use-cases. |
There was a problem hiding this comment.
The proposal for this project is to leverage that model as well as existing OpenTelemetry component to provide alternative SDKs for specific use-cases.
I think the proposal for this project is something else. What I think you want to say here is that this project will use the existing separation between API and SDK to allow end users to use alternative SDKs for specific purposes.
|
|
||
| The OTel community is currently working across many languages to support reference implementations of all the features in every language. We have always proposed that the API/SDK separation would make it possible for end-users to swap SDKs without having to re-instrument their application/instrumentations code. The proposal for this project is to leverage that model as well as existing OpenTelemetry component to provide alternative SDKs for specific use-cases. | ||
|
|
||
| An objective is to prove this approach as part of the project. A requirement of the project is that the performance improvements be worth the investment of time to produce and maintain artifacts moving forward. If the investigations from the project finds that the performance improvements are only applicable to one language or the improvements are negligible, then the deliverables may be changed. |
There was a problem hiding this comment.
An objective is to prove this approach.
A requirement is that the performance improventes are worth the investment.
This seems contradictory or at least hard to understand, an objective is to test this approach and a requirement is that the approach is worth the investment?
|
|
||
| An objective is to prove this approach as part of the project. A requirement of the project is that the performance improvements be worth the investment of time to produce and maintain artifacts moving forward. If the investigations from the project finds that the performance improvements are only applicable to one language or the improvements are negligible, then the deliverables may be changed. | ||
|
|
||
| A non-goal of this project is to replace existing SDK implementations. There are always different use-cases for end users and a native implementation that offers flexibility is likely to always be needed. |
There was a problem hiding this comment.
What do you mean with flexibility here? I agree that this project is not trying to replace existing SDK implementations, but I think the reason is that there is no certainty that a refitted SDK can satisfy every use case existing SDK implementations can because we have not tested all those use cases yet with a refitted SDK.
There was a problem hiding this comment.
Performance is an optimization. If we are optimizing for higher performance and lower resource consumption, that will be better achieved if this project does not attempt to provide all of the plugins, hooks, and configuration options currently available in the existing SDKs. For example, it may be the case that this implementation only provides OTLP via Proto/HTTP as the sole exporter option, in order to reduce file size and optimize container start up time.
I would suggest adding details to this section to clarify that performance is an optimization goal, and this implementation will makes trade-offs to achieve that.
The other important form of flexibility is supporting build/deployment environments where our C++ binding mechanism doesn't work or otherwise creates a lot of trouble.
There was a problem hiding this comment.
Another form of flexibility is "mix-n-match." For example, installing the OTel tracing SDK, but binding to Prometheus for metrics and not installing a logging SDK at all.
This implementation could be all-or-nothing, in order to avoid the complexity that comes with offering those options.
|
|
||
| #### Other Staffing | ||
|
|
||
| Names and expected contribution of any other contributors. |
ocelotl
left a comment
There was a problem hiding this comment.
I think this document captures the essence of this project but can be edited for clarity. Glad to help if help is needed with this part ✌️
Co-authored-by: Diego Hurtado <ocelotl@users.noreply.github.com>
ocelotl
left a comment
There was a problem hiding this comment.
The comments I added before can be addressed later as enhancements, approving ✌️
Co-authored-by: Diego Hurtado <ocelotl@users.noreply.github.com>
|
|
||
| There are a few different approaches available to solve the issue. Phase 1 of the project will provide: | ||
|
|
||
| * a library or artifact that can be used behind an existing language API. The initial proposal is to support at least Python, but support for other languages is not out of scope. |
There was a problem hiding this comment.
I think we should also take look at OBI https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation as one of the possible solutions.
It can already be used with Python. I think it is worth mentioning that it can be already used as an SDK for users of OTel Go API (see: https://opentelemetry.io/docs/zero-code/obi/distributed-traces/#instrument-applications-that-use-the-go-trace-api), similar feature is planned for other languages. We already have issues like:
- 🏆 Epic: OTel API/SDK Integration opentelemetry-ebpf-instrumentation#1140
- Support manual instrumentation inside OBI traces opentelemetry-ebpf-instrumentation#2659
This was initially brought up by @lmolkova 🙏
There was a problem hiding this comment.
Definitely worth investigating! Especially if the goal is to support the same instrumentation we already have, and not to try and create a duplicate set of instrumentation using ebpf targets.
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com>
|
I think that this project is great, my main suggestion is to pick a small scope, as it's possible for this project to sprawl. For example, C++ bindings might be a good idea for improved performance. Rust, eBPF, native-sdk-but-with-a-different-architecture might also prove to work well. It might be different in different languages. But if you pick one approach for the initial investigation, it will be more clear who should join the SIG to work on it. I would suggest continuing with the C++ binding investigation, and determine which would languages benefit from that approach. If someone wants to investigate another approach, they can follow up with a separate project, and we can compare the results. (If there are enough people available to work on two investigations at the same time, great.) |
This PR contains a project proposal as discussed during the 29-Sept-2026 specification/maintainer SIG call.
Closes #3483