Skip to content

perf: per-schema granularity in incremental generator pipeline (#170) #172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pedrosakuma merged 1 commit into from
Aug 29, 2026
Merged

perf: per-schema granularity in incremental generator pipeline (#170) #172

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/copilot-instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,17 @@ This is a **Roslyn incremental source generator** (`IIncrementalGenerator`) that

```
XML schema (*.xml via AdditionalFiles)
→ SBESourceGenerator (entry point, namespace derivation, SchemaContext creation)
→ SBESourceGenerator (entry point, per-schema incremental pipeline, namespace derivation, SchemaContext creation)
→ TypesCodeGenerator (enums, types, composites, sets, derived constants on decimal composites)
→ MessagesCodeGenerator (messages, fields, groups, varData, per-message {Msg}VersionMap when multi-version)
→ DispatcherGenerator (per-schema ISbeMessageHandler + zero-cost SbeDispatcher.Dispatch<T>)
→ UtilitiesCodeGenerator (SpanReader, SpanWriter, endian helpers)
→ UtilitiesCodeGenerator (SpanReader, SpanWriter, endian helpers; emitted once per runtime namespace in a separate incremental step)
→ ValidationGenerator (optional validation)
→ sourceContext.AddSource() per generated file
```

Incremental behavior matters: keep schema `AdditionalText` inputs as per-item `IncrementalValuesProvider`s through to `RegisterSourceOutput`. Only collect genuinely shared/project-wide data (for example semantic-type registrations, or runtime-namespace dedup when emitting shared helpers).

Each generator implements `ICodeGenerator` and returns `IEnumerable<(string name, string content)>`.

### Key Abstractions
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed

- **Per-schema incremental invalidation for XML AdditionalFiles (#170)**: `SBESourceGenerator` no longer `.Collect()`s the entire schema set before generation. Each schema now flows through its own `RegisterSourceOutput` path, so editing one XML file only re-runs the generators for that schema. Runtime helpers (`SpanReader` / `SpanWriter`) are emitted in a separate namespace-deduplicated step to preserve shared helper generation without reintroducing whole-project schema invalidation.

## [1.7.0] - 2026-04-30

### Added
Expand Down
30 changes: 21 additions & 9 deletions src/SbeCodeGenerator/Generators/UtilitiesCodeGenerator.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,28 @@ internal class UtilitiesCodeGenerator : ICodeGenerator

if (context.GeneratedRuntimeNamespaces.Add(runtimeNamespace))
{
// Generate SpanReader once per runtime namespace
StringBuilder sb = new StringBuilder();
new SpanReaderGenerator(runtimeNamespace).AppendFileContent(sb);
yield return (context.CreateHintName(runtimeNamespace, "Runtime", "SpanReader"), sb.ToString());

// Generate SpanWriter once per runtime namespace
sb = new StringBuilder();
new SpanWriterGenerator(runtimeNamespace).AppendFileContent(sb);
yield return (context.CreateHintName(runtimeNamespace, "Runtime", "SpanWriter"), sb.ToString());
foreach (var item in GenerateRuntimeSources(
runtimeNamespace,
typeName => context.CreateHintName(runtimeNamespace, "Runtime", typeName)))
{
yield return item;
}
}
}

internal static IEnumerable<(string name, string content)> GenerateRuntimeSources(
string runtimeNamespace,
Func<string, string> createHintName)
{
// Generate SpanReader once per runtime namespace
StringBuilder sb = new StringBuilder();
new SpanReaderGenerator(runtimeNamespace).AppendFileContent(sb);
yield return (createHintName("SpanReader"), sb.ToString());

// Generate SpanWriter once per runtime namespace
sb = new StringBuilder();
new SpanWriterGenerator(runtimeNamespace).AppendFileContent(sb);
yield return (createHintName("SpanWriter"), sb.ToString());
}
}
}
242 changes: 151 additions & 91 deletions src/SbeCodeGenerator/SBESourceGenerator.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
using System.Linq;
using System.Security.Cryptography;
using System.Text;
using System.Threading;

namespace SbeSourceGenerator
{
Expand Down Expand Up @@ -48,13 +49,28 @@ public void Initialize(IncrementalGeneratorInitializationContext initContext)
}
});

var semanticRegistry = userAttributeResults
.Select(static (results, _) => BuildSemanticRegistry(results));

// Stage 2: Combine with analyzer config options (for SbeAssumeHostEndianness hint) and the registry.
var combined = xmlSchemaFiles.Collect()
var combined = xmlSchemaFiles
.Combine(initContext.AnalyzerConfigOptionsProvider)
.Combine(userAttributeResults);
.Combine(semanticRegistry)
.Select(static (input, _) => (
Path: input.Left.Left.Path,
Schema: input.Left.Left,
Options: input.Left.Right,
SemanticRegistry: input.Right))
.WithTrackingName("PerSchemaGeneration");

var runtimeNamespaces = xmlSchemaFiles
.Select(static (schemaFile, cancellationToken) => TryGetRuntimeNamespace(schemaFile, cancellationToken))
.Collect()
.WithTrackingName("RuntimeNamespaceCollection");

// Stage 3: Register source generation with diagnostic support
RegisterSourceGeneration(initContext, combined);
RegisterRuntimeGeneration(initContext, runtimeNamespaces);
}

/// <summary>
Expand All @@ -69,20 +85,11 @@ private static IncrementalValuesProvider<AdditionalText> CollectXmlSchemaFiles(I
/// Registers source output for each XML schema with diagnostic reporting.
/// </summary>
private static void RegisterSourceGeneration(IncrementalGeneratorInitializationContext initContext,
IncrementalValueProvider<((ImmutableArray<AdditionalText> Schemas, AnalyzerConfigOptionsProvider Options) Left, ImmutableArray<UserAttributeResult> UserRegs)> combined)
IncrementalValuesProvider<(string Path, AdditionalText Schema, AnalyzerConfigOptionsProvider Options, SemanticConverterRegistry SemanticRegistry)> combined)
{
initContext.RegisterSourceOutput(combined, (sourceContext, input) =>
{
var ((text, configOptions), userRegs) = input;

if (text.IsDefaultOrEmpty)
return;

// Build the semantic registry once per generation pass.
var userRegistrations = userRegs.IsDefaultOrEmpty
? ImmutableArray<SemanticConverterRegistration>.Empty
: userRegs.Where(r => r.Registration != null).Select(r => r.Registration!).ToImmutableArray();
var semanticRegistry = SemanticConverterRegistry.Build(userRegistrations);
var (path, additionalText, configOptions, semanticRegistry) = input;

// Read the optional SbeAssumeHostEndianness MSBuild property
string? hostHint = null;
Expand All @@ -92,115 +99,168 @@ private static void RegisterSourceGeneration(IncrementalGeneratorInitializationC
hostHint = hintValue;
}

var emittedRuntimeNamespaces = new HashSet<string>(StringComparer.Ordinal);
var emittedHintNames = new HashSet<string>(StringComparer.Ordinal);

foreach (var additionalText in text)
try
{
try
{
string path = additionalText.Path;
var xmlContent = additionalText.GetText(sourceContext.CancellationToken)?.ToString();
if (string.IsNullOrEmpty(xmlContent))
continue;
var xmlContent = additionalText.GetText(sourceContext.CancellationToken)?.ToString();
if (string.IsNullOrEmpty(xmlContent))
return;

var schema = SchemaReader.Parse(xmlContent!, sourceContext);
var schema = SchemaReader.Parse(xmlContent!, sourceContext);

string ns = GetNamespaceFromSchema(schema, path);
string schemaKey = CreateSchemaKey(path);
string ns = GetNamespaceFromSchema(schema, path);
string schemaKey = CreateSchemaKey(path);

// Create a per-schema context to hold mutable state (sharing runtime tracking)
var context = new SchemaContext(schemaKey, emittedRuntimeNamespaces);
context.SemanticConverters = semanticRegistry;
// Create a per-schema context to hold mutable state.
var context = new SchemaContext(schemaKey);
context.SemanticConverters = semanticRegistry;

if (!string.IsNullOrEmpty(schema.ByteOrder))
{
context.ByteOrder = schema.ByteOrder;
}
if (!string.IsNullOrEmpty(schema.ByteOrder))
{
context.ByteOrder = schema.ByteOrder;
}

context.EndianConversion = ComputeEndianConversion(
context.ByteOrder, hostHint, sourceContext, path);
context.EndianConversion = ComputeEndianConversion(
context.ByteOrder, hostHint, sourceContext, path);

if (!string.IsNullOrEmpty(schema.HeaderType))
context.HeaderType = schema.HeaderType;
if (!string.IsNullOrEmpty(schema.HeaderType))
context.HeaderType = schema.HeaderType;

// Use specialized generators to handle different categories
var typesGenerator = new TypesCodeGenerator();
var messagesGenerator = new MessagesCodeGenerator();
var utilitiesGenerator = new UtilitiesCodeGenerator();
var validationGenerator = new ValidationGenerator();
// Use specialized generators to handle different categories
var typesGenerator = new TypesCodeGenerator();
var messagesGenerator = new MessagesCodeGenerator();
var validationGenerator = new ValidationGenerator();

var generators = new (string phase, ICodeGenerator gen)[]
{
("types", typesGenerator),
("messages", messagesGenerator),
("dispatcher", new DispatcherGenerator()),
("utilities", utilitiesGenerator),
("validation", validationGenerator)
};

foreach (var (phase, gen) in generators)
var generators = new (string phase, ICodeGenerator gen)[]
{
("types", typesGenerator),
("messages", messagesGenerator),
("dispatcher", new DispatcherGenerator()),
("validation", validationGenerator)
};

foreach (var (phase, gen) in generators)
{
try
{
try
foreach (var item in gen.Generate(ns, schema, context, sourceContext))
{
foreach (var item in gen.Generate(ns, schema, context, sourceContext))
try
{
try
if (!emittedHintNames.Add(item.name))
{
if (!emittedHintNames.Add(item.name))
// Roslyn would throw ArgumentException on duplicate hintName,
// aborting the rest of the phase. Suppress and continue so a
// single duplicate doesn't cascade into thousands of CS0246s
// against partially-emitted files.
if (!sourceContext.CancellationToken.IsCancellationRequested)
{
// Roslyn would throw ArgumentException on duplicate hintName,
// aborting the rest of the phase. Suppress and continue so a
// single duplicate doesn't cascade into thousands of CS0246s
// against partially-emitted files.
if (!sourceContext.CancellationToken.IsCancellationRequested)
{
sourceContext.ReportDiagnostic(Diagnostic.Create(
SbeDiagnostics.DuplicateGeneratedSource,
Location.None,
item.name,
phase));
}
continue;
sourceContext.ReportDiagnostic(Diagnostic.Create(
SbeDiagnostics.DuplicateGeneratedSource,
Location.None,
item.name,
phase));
}
sourceContext.AddSource(item.name, item.content);
}
catch (Exception itemEx) when (!sourceContext.CancellationToken.IsCancellationRequested)
{
// Per-item failure must not derail subsequent items in the same phase.
sourceContext.ReportDiagnostic(Diagnostic.Create(
SbeDiagnostics.MalformedSchema,
Location.None,
path,
$"[{phase}] {item.name}: {itemEx.Message}"));
continue;
}
sourceContext.AddSource(item.name, item.content);
}
catch (Exception itemEx) when (!sourceContext.CancellationToken.IsCancellationRequested)
{
// Per-item failure must not derail subsequent items in the same phase.
sourceContext.ReportDiagnostic(Diagnostic.Create(
SbeDiagnostics.MalformedSchema,
Location.None,
path,
$"[{phase}] {item.name}: {itemEx.Message}"));
}
}
catch (Exception genEx) when (!sourceContext.CancellationToken.IsCancellationRequested)
{
sourceContext.ReportDiagnostic(Diagnostic.Create(
SbeDiagnostics.MalformedSchema,
Location.None,
path,
$"[{phase}] {genEx.Message}"));
}
}
}
catch (Exception ex)
{
if (!sourceContext.CancellationToken.IsCancellationRequested)
catch (Exception genEx) when (!sourceContext.CancellationToken.IsCancellationRequested)
{
sourceContext.ReportDiagnostic(Diagnostic.Create(
SbeDiagnostics.MalformedSchema,
Location.None,
additionalText.Path,
ex.Message));
path,
$"[{phase}] {genEx.Message}"));
}
}
}
catch (Exception ex)
{
if (!sourceContext.CancellationToken.IsCancellationRequested)
{
sourceContext.ReportDiagnostic(Diagnostic.Create(
SbeDiagnostics.MalformedSchema,
Location.None,
additionalText.Path,
ex.Message));
}
}
});
}

private static void RegisterRuntimeGeneration(
IncrementalGeneratorInitializationContext initContext,
IncrementalValueProvider<ImmutableArray<string?>> runtimeNamespaces)
{
initContext.RegisterSourceOutput(runtimeNamespaces, (sourceContext, namespaces) =>
{
if (namespaces.IsDefaultOrEmpty)
return;

var emittedRuntimeNamespaces = new HashSet<string>(StringComparer.Ordinal);
foreach (var runtimeNamespace in namespaces)
{
if (string.IsNullOrWhiteSpace(runtimeNamespace))
continue;

var resolvedRuntimeNamespace = runtimeNamespace!;
if (!emittedRuntimeNamespaces.Add(resolvedRuntimeNamespace))
continue;

foreach (var item in UtilitiesCodeGenerator.GenerateRuntimeSources(
resolvedRuntimeNamespace,
typeName => CreateRuntimeHintName(resolvedRuntimeNamespace, typeName)))
{
sourceContext.AddSource(item.name, item.content);
}
}
});
}

private static SemanticConverterRegistry BuildSemanticRegistry(ImmutableArray<UserAttributeResult> userRegs)
{
var userRegistrations = userRegs.IsDefaultOrEmpty
? ImmutableArray<SemanticConverterRegistration>.Empty
: userRegs.Where(r => r.Registration != null).Select(r => r.Registration!).ToImmutableArray();

return SemanticConverterRegistry.Build(userRegistrations);
}

private static string? TryGetRuntimeNamespace(AdditionalText additionalText, CancellationToken cancellationToken)
{
var xmlContent = additionalText.GetText(cancellationToken)?.ToString();
if (string.IsNullOrEmpty(xmlContent))
return null;

try
{
var schema = SchemaReader.Parse(xmlContent!);
return GetNamespaceFromSchema(schema, additionalText.Path);
}
catch
{
return null;
}
}

private static string CreateRuntimeHintName(string runtimeNamespace, string typeName)
{
return string.Concat(runtimeNamespace, "\\Runtime\\", typeName);
}

private static string CreateSchemaKey(string path)
{
string fileName = Path.GetFileNameWithoutExtension(path);
Expand Down
Loading
Loading