Repository navigation
🐛 Pin git source, GitHub token check and mail relay dials to checked addresses #19288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| - **Egress: git knowledge sources and the GitHub token check connect only to the address they checked** ([#19221](https://github.com/hivecommons/hive/issues/19221), part of [#18986](https://github.com/hivecommons/hive/issues/18986)). A git source's host was validated with one DNS lookup and then resolved again by `git` itself on every clone, `ls-remote` and pull, so a name that rebinds between the two was checked as public and fetched as private. Each git command now resolves the host once, refuses the fetch when any answer is private, loopback, link-local or otherwise barred, or when the lookup fails or comes back empty, and pins git to the checked addresses with `-c http.curloptResolve` (git 2.37 or later). Periodic syncs are checked the same way, so a source whose name later moves inward stops syncing. `HIVE_ALLOW_PRIVATE_GIT_SOURCE=true` still allows an internal Git server, but link-local (cloud metadata), multicast and unspecified answers stay refused. The dashboard's GitHub token check now dials through a pinned transport that refuses barred addresses for `api.github.com`, allows a configured API URL to be private while still refusing metadata answers, and refuses cross-host redirects. The hub's mail relay to a hive's self-reported dashboard URL now dials through the same pinned transport as the webhook relay. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| package dashboard | ||
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
| "net" | ||
| "net/http" | ||
| "strings" | ||
| "time" | ||
|
|
||
| "github.com/hivecommons/hive/pkg/netguard" | ||
| ) | ||
|
|
||
| const ( | ||
| // githubTokenDialTimeout and githubTokenDialKeepAlive match the | ||
| // net.Dialer http.DefaultTransport uses. | ||
| githubTokenDialTimeout = 30 * time.Second | ||
| githubTokenDialKeepAlive = 30 * time.Second | ||
|
|
||
| // githubTokenMaxRedirects bounds a token check's same-host redirects. | ||
| githubTokenMaxRedirects = 3 | ||
| ) | ||
|
|
||
| // githubTokenConfiguredTargetKey marks a token-check request whose API URL is | ||
| // a configured override rather than public api.github.com. | ||
| type githubTokenConfiguredTargetKey struct{} | ||
|
|
||
| // githubTokenConfiguredRefusedReasons are the address classes refused even | ||
| // for a configured API URL. A configured target is intentionally chosen and | ||
| // may be private or loopback (an httptest server, an internal mirror), but no | ||
| // GitHub API lives on link-local (cloud instance metadata), multicast or an | ||
| // unspecified address. | ||
| var githubTokenConfiguredRefusedReasons = map[string]bool{ | ||
| netguard.ReasonLinkLocal: true, | ||
| netguard.ReasonMulticast: true, | ||
| netguard.ReasonUnspecified: true, | ||
| netguard.ReasonThisNetwork: true, | ||
| netguard.ReasonBroadcast: true, | ||
| } | ||
|
|
||
| // githubTokenDial connects to an address githubTokenTransport already | ||
| // checked. A package var so TestMain routes it through the suite's | ||
| // loopback-only guard and tests can observe where a dial would go. | ||
| var githubTokenDial netguard.DialFunc = (&net.Dialer{ | ||
| Timeout: githubTokenDialTimeout, | ||
| KeepAlive: githubTokenDialKeepAlive, | ||
| }).DialContext | ||
|
|
||
| // githubTokenTransport is the transport for validateGitHubToken (#19221, | ||
| // part of #18986). The token check sends a user's bearer token to the API | ||
| // host, and a default http.Client resolved the name at dial time with nothing | ||
| // checking the answer. This transport resolves each dial once through | ||
| // privateURLResolver, checks every answer, fails closed when the lookup | ||
| // errors or returns nothing, and connects only to an address it checked. | ||
| // | ||
| // For public api.github.com every class isPrivateURL bars is refused. For a | ||
| // configured API URL (githubTokenConfiguredTargetKey on the request context) | ||
| // the target is intentionally chosen, so only | ||
| // githubTokenConfiguredRefusedReasons are refused; githubTokenCheckRedirect | ||
| // refuses a cross-host redirect, so that allowance never reaches another host. | ||
| var githubTokenTransport http.RoundTripper = netguard.PinnedDialer{ | ||
| Resolve: func(ctx context.Context, host string) ([]string, error) { return privateURLResolver(ctx, host) }, | ||
| Refuse: githubTokenRefuse, | ||
| Dial: func(ctx context.Context, network, addr string) (net.Conn, error) { | ||
| return githubTokenDial(ctx, network, addr) | ||
| }, | ||
| }.Transport() | ||
|
|
||
| func githubTokenRefuse(ctx context.Context, target string) bool { | ||
| configured, _ := ctx.Value(githubTokenConfiguredTargetKey{}).(bool) | ||
| if !configured { | ||
| return isPrivateURL(ctx, "http://"+target) | ||
| } | ||
| host, _, err := net.SplitHostPort(target) | ||
| if err != nil { | ||
| return true | ||
| } | ||
| addr, ok := netguard.ParseHostLiteral(host) | ||
| if !ok { | ||
| return true | ||
| } | ||
| return githubTokenConfiguredRefusedReasons[netguard.DenyReason(addr)] | ||
| } | ||
|
|
||
| // githubTokenRoundTripper is githubTokenTransport, unless http.DefaultTransport | ||
| // was replaced with a non-*http.Transport RoundTripper (tests and | ||
| // instrumented builds inject one there, the same convention as the hub's | ||
| // pinnedOrReplacedDefault); then that replacement is used. | ||
| func githubTokenRoundTripper() http.RoundTripper { | ||
| if _, ok := http.DefaultTransport.(*http.Transport); ok { | ||
| return githubTokenTransport | ||
| } | ||
| return http.DefaultTransport | ||
| } | ||
|
|
||
| // githubTokenCheckRedirect allows only a bounded number of same-host | ||
| // redirects. A cross-host hop would carry the configured-target allowance | ||
| // and the bearer token to a host nobody configured. | ||
| func githubTokenCheckRedirect(req *http.Request, via []*http.Request) error { | ||
| if len(via) >= githubTokenMaxRedirects { | ||
| return fmt.Errorf("stopped after %d redirects", githubTokenMaxRedirects) | ||
| } | ||
| if len(via) > 0 && !strings.EqualFold(req.URL.Host, via[0].URL.Host) { | ||
| return fmt.Errorf("%w: cross-host redirect to %s", netguard.ErrRefused, req.URL.Host) | ||
| } | ||
| return nil | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,194 @@ | ||
| package dashboard | ||
|
|
||
| import ( | ||
| "context" | ||
| "errors" | ||
| "net" | ||
| "net/http" | ||
| "net/http/httptest" | ||
| "net/url" | ||
| "sync" | ||
| "sync/atomic" | ||
| "testing" | ||
|
|
||
| "github.com/hivecommons/hive/pkg/netguard" | ||
| ) | ||
|
|
||
| // Resolve-once and dial pinning for validateGitHubToken (#19221, part of | ||
| // #18986). These tests drive a resolver that answers differently on later | ||
| // lookups and a recording dialer, so they never leave the machine. | ||
|
|
||
| const ( | ||
| tokenPinPublicIP = "140.82.112.6" | ||
| tokenPinHost = "rebind.example" | ||
| ) | ||
|
|
||
| // stubTokenResolver answers first on the first lookup and later afterwards. | ||
| func stubTokenResolver(t *testing.T, first, later []string, err error) *int32 { | ||
| t.Helper() | ||
| var calls int32 | ||
| orig := privateURLResolver | ||
| privateURLResolver = func(_ context.Context, _ string) ([]string, error) { | ||
| if err != nil { | ||
| atomic.AddInt32(&calls, 1) | ||
| return nil, err | ||
| } | ||
| if atomic.AddInt32(&calls, 1) == 1 { | ||
| return first, nil | ||
| } | ||
| return later, nil | ||
| } | ||
| t.Cleanup(func() { privateURLResolver = orig }) | ||
| return &calls | ||
| } | ||
|
|
||
| // recordTokenDials replaces githubTokenDial with one that records each | ||
| // target. When to is set, the dial connects there instead (an httptest | ||
| // server standing in for the checked public address). | ||
| func recordTokenDials(t *testing.T, to string) *[]string { | ||
| t.Helper() | ||
| var mu sync.Mutex | ||
| var got []string | ||
| orig := githubTokenDial | ||
| githubTokenDial = func(ctx context.Context, network, addr string) (net.Conn, error) { | ||
| mu.Lock() | ||
| got = append(got, addr) | ||
| mu.Unlock() | ||
| if to == "" { | ||
| return nil, errors.New("test dialer: no connection") | ||
| } | ||
| return (&net.Dialer{}).DialContext(ctx, network, to) | ||
| } | ||
| t.Cleanup(func() { githubTokenDial = orig }) | ||
| return &got | ||
| } | ||
|
|
||
| // The invariant: a name that rebinds between the check and the dial cannot | ||
| // reach a private address. The pre-fetch check sees the public answer; the | ||
| // dial's own single lookup sees loopback and refuses, and the loopback server | ||
| // receives nothing. | ||
| func TestGitHubTokenTransport_RebindBetweenCheckAndDialRefused(t *testing.T) { | ||
| var hits int32 | ||
| srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { | ||
| atomic.AddInt32(&hits, 1) | ||
| })) | ||
| t.Cleanup(srv.Close) | ||
| u, err := url.Parse(srv.URL) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
| stubTokenResolver(t, []string{tokenPinPublicIP}, []string{"127.0.0.1"}, nil) | ||
| // Real dialer, so a dial to the rebound address would reach srv. | ||
| orig := githubTokenDial | ||
| githubTokenDial = (&net.Dialer{}).DialContext | ||
| t.Cleanup(func() { githubTokenDial = orig }) | ||
|
|
||
| target := "http://" + tokenPinHost + ":" + u.Port() + "/user" | ||
| if isPrivateURL(context.Background(), target) { | ||
| t.Fatal("pre-fetch check should see the public first answer") | ||
| } | ||
| req, err := http.NewRequest(http.MethodGet, target, nil) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
| resp, err := (&http.Client{Transport: githubTokenTransport}).Do(req) | ||
| if err == nil { | ||
| resp.Body.Close() | ||
| t.Fatal("dial after rebind succeeded") | ||
| } | ||
| if !errors.Is(err, netguard.ErrRefused) { | ||
| t.Fatalf("dial after rebind = %v; want netguard.ErrRefused", err) | ||
| } | ||
| if n := atomic.LoadInt32(&hits); n != 0 { | ||
| t.Fatalf("rebound loopback server received %d request(s)", n) | ||
| } | ||
| } | ||
|
|
||
| // Default api.github.com: the dial connects to the address it checked, from | ||
| // one lookup, never to the name. | ||
| func TestValidateGitHubToken_DialsCheckedAddressOnce(t *testing.T) { | ||
| calls := stubTokenResolver(t, []string{tokenPinPublicIP}, []string{"127.0.0.1"}, nil) | ||
| dials := recordTokenDials(t, "") | ||
|
|
||
| _ = validateGitHubToken("ghp_pin_checked_once", "") | ||
| if got := atomic.LoadInt32(calls); got != 1 { | ||
| t.Fatalf("resolver called %d times, want 1", got) | ||
| } | ||
| if len(*dials) != 1 || (*dials)[0] != net.JoinHostPort(tokenPinPublicIP, "443") { | ||
| t.Fatalf("dials = %v, want only the checked %s:443", *dials, tokenPinPublicIP) | ||
| } | ||
| } | ||
|
|
||
| // Default api.github.com: a private answer (any of several), an empty answer | ||
| // or a lookup error refuses the dial before any connection is attempted. | ||
| func TestValidateGitHubToken_DefaultHostRefusesPrivateAndFailsClosed(t *testing.T) { | ||
| cases := []struct { | ||
| name string | ||
| answers []string | ||
| err error | ||
| }{ | ||
| {"loopback", []string{"127.0.0.1"}, nil}, | ||
| {"any answer private", []string{tokenPinPublicIP, "10.1.2.3"}, nil}, | ||
| {"metadata", []string{"169.254.169.254"}, nil}, | ||
| {"empty", nil, nil}, | ||
| {"lookup error", nil, errors.New("dns unavailable")}, | ||
| } | ||
| for i, tc := range cases { | ||
| t.Run(tc.name, func(t *testing.T) { | ||
| stubTokenResolver(t, tc.answers, tc.answers, tc.err) | ||
| dials := recordTokenDials(t, "") | ||
| if got := validateGitHubToken("ghp_pin_refused_"+string(rune('a'+i)), ""); got != "" { | ||
| t.Fatalf("username = %q, want refusal", got) | ||
| } | ||
| if len(*dials) != 0 { | ||
| t.Fatalf("dialled %v despite a barred or failed lookup", *dials) | ||
| } | ||
| }) | ||
| } | ||
| } | ||
|
|
||
| // A configured API URL is intentionally chosen and may be private: an | ||
| // httptest server on loopback answers. Link-local metadata stays refused. | ||
| func TestValidateGitHubToken_ConfiguredTargetMayBePrivateButNotMetadata(t *testing.T) { | ||
| srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { | ||
| _, _ = w.Write([]byte(`{"login":"octo-pin"}`)) | ||
| })) | ||
| t.Cleanup(srv.Close) | ||
| if got := validateGitHubToken("ghp_pin_configured_ok", srv.URL); got != "octo-pin" { | ||
| t.Fatalf("configured loopback API: username = %q, want octo-pin", got) | ||
| } | ||
|
|
||
| stubTokenResolver(t, []string{"169.254.169.254"}, []string{"169.254.169.254"}, nil) | ||
| dials := recordTokenDials(t, "") | ||
| if got := validateGitHubToken("ghp_pin_configured_md", "http://ghe.internal.example"); got != "" { | ||
| t.Fatalf("configured host resolving to metadata: username = %q, want refusal", got) | ||
| } | ||
| if len(*dials) != 0 { | ||
| t.Fatalf("dialled %v for a metadata answer", *dials) | ||
| } | ||
| } | ||
|
|
||
| // The configured-target allowance must not follow a cross-host redirect. | ||
| func TestValidateGitHubToken_CrossHostRedirectRefused(t *testing.T) { | ||
| var otherHits int32 | ||
| other := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { | ||
| atomic.AddInt32(&otherHits, 1) | ||
| _, _ = w.Write([]byte(`{"login":"elsewhere"}`)) | ||
| })) | ||
| t.Cleanup(other.Close) | ||
| ou, err := url.Parse(other.URL) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
| front := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { | ||
| http.Redirect(w, r, "http://localhost:"+ou.Port()+"/user", http.StatusFound) | ||
| })) | ||
| t.Cleanup(front.Close) | ||
|
|
||
| if got := validateGitHubToken("ghp_pin_redirect", front.URL); got != "" { | ||
| t.Fatalf("username = %q after a cross-host redirect, want refusal", got) | ||
| } | ||
| if n := atomic.LoadInt32(&otherHits); n != 0 { | ||
| t.Fatalf("redirect target received %d request(s)", n) | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: this map has the same five reasons as
gitSourceOptInRefusedReasons(src/pkg/knowledge/gitsource.go:568). Two copies of a security allowlist can drift. Consider exporting one set fromnetguard(e.g. aDenyReasonIsMetadataClasshelper) and using it in both places.