[WIP] Add snapshotting#8

Draft

RyandenOtter wants to merge 98 commits intomasterfrom

add-snapshotting

Commits on Nov 23, 2020

feat: tf gcs-bq-ingest module sets bq permissions
Jacob Ferriero
committed
fixup roles
Jacob Ferriero
committed

Commits on Nov 24, 2020

fixup dockerfil ci check
Jacob Ferriero
committed

Commits on Nov 25, 2020

docs: add note on unicode delimiters
Jacob Ferriero
committed

Commits on Nov 30, 2020

fix: update nested values in configs
Jacob Ferriero
committed
chore: improve error message for wrong external table name (#200)
Jacob Ferriero
authored
fix: external configs not found in parent dirs
Jacob Ferriero
committed

Commits on Dec 1, 2020

Commits on Dec 2, 2020

feat: bq project env-var
Jacob Ferriero
committed

Commits on Dec 3, 2020

Merge branch 'feat/bq-project-env-var'
Jacob Ferriero
committed
Merge branch 'fix/configDiscovery'
Jacob Ferriero
committed
Merge branch 'docs/delimiters'
Jacob Ferriero
committed
Merge branch 'tf/multi-project'
Jacob Ferriero
committed

Commits on Dec 4, 2020

Move utility methods into a utils module
Ryan den Otter
committed
Fix sorting issues
Ryan den Otter
committed
Move out constants into their own file
Ryan den Otter
committed

Commits on Dec 5, 2020

fixup! pylint
Jacob Ferriero
committed
fixup! fixup! gcb pylint issue
Jacob Ferriero
committed
Merge pull request #1 from jaketf/feat/move-utils
Jacob Ferriero
authored

Commits on Dec 9, 2020

feat: sequencing with backlog publisher / subscriber
Jacob Ferriero
committed
fixup! mypy pylint
Jacob Ferriero
committed
fixup! flake8
Jacob Ferriero
committed
fixup! mypy tests
Jacob Ferriero
committed
support _config/*.sql for bq tranform sql
Jacob Ferriero
committed
improve performance of wait_on_bq_job
Jacob Ferriero
committed

Commits on Dec 10, 2020

wip
Jacob Ferriero
committed
fixup! handle race condition
Jacob Ferriero
committed
ordering docs and isort single line rule
Jacob Ferriero
committed
docs
Jacob Ferriero
committed
fixup linters
Jacob Ferriero
committed
fixup import style
Jacob Ferriero
committed
typing isort single line exclusion
Jacob Ferriero
committed
fixup gcb no-name-in-module bug
Jacob Ferriero
committed
add test of subscriber after subscriber exit
Jacob Ferriero
committed

Commits on Dec 11, 2020

chores: tf updates, larger machine type, etc.
Jacob Ferriero
committed
terraform fmt
Jacob Ferriero
committed
handle abandoned _BACKFILL and other review feedback
Jacob Ferriero
committed

Commits on Dec 12, 2020

improve tests
Jacob Ferriero
committed

Commits on Dec 14, 2020

fix: handle long running bq jobs
Jacob Ferriero
committed

Commits on Dec 15, 2020

chore: add e2e test, fixup terraform
Jacob Ferriero
committed
ignore pylint redherring import errors
Jacob Ferriero
committed
fixup! e2e tf to support builds where short_sha is set to empty string.
Jacob Ferriero
committed
fix TF_VAR env var
Jacob Ferriero
committed
enable resource manager api
Jacob Ferriero
committed
enable cloud functions api...
Jacob Ferriero
committed
add unit test timeout
Jacob Ferriero
committed
explicit local backend
Jacob Ferriero
committed
debug missing state file
Jacob Ferriero
committed
debug
Jacob Ferriero
committed
relative state path
Jacob Ferriero
committed
typo .[tf]state
Jacob Ferriero
committed
fixup docs
Jacob Ferriero
committed
chore: clean up subscriber
Jacob Ferriero
committed
fix: don't try to regex match _backlog/* items
Jacob Ferriero
committed
don't regex match in triage if ordering enabled (this happens later)
Jacob Ferriero
committed
fix: subscriber monitor get table prefix
Jacob Ferriero
committed
fix: get_table_prefix issues w/ backlog, backfill and historydone
Jacob Ferriero
committed
fix: look_for_config_in_parents should return empty string for empty file
Jacob Ferriero
committed
fix table prefix w/ trailing slash
Jacob Ferriero
committed
use get_table_prefix instead of removesuffix
Jacob Ferriero
committed

Commits on Dec 16, 2020

chore: refactor terraform into pytest fixture to always clean up
Jacob Ferriero
committed
fix don't removesuffix for start backfill file
Jacob Ferriero
committed
fixup isort
Jacob Ferriero
committed
more logging statements fail on untriageable event
Jacob Ferriero
committed
fix pylint
Jacob Ferriero
committed
Merge branch 'master' into sequencing-develop
Jacob Ferriero
authored
feat: env-var t numDmlRowsAffected = 0 as a failure
Jacob Ferriero
committed
[skip ci] add comment to cloudbuild.yaml
Jacob Ferriero
committed
[skip ci] update comment in cloudbuild.yaml
Jacob Ferriero
committed
chore: clean up unused fixture, init files
Jacob Ferriero
committed
chore: improve terraform printint in pytest fixture
Jacob Ferriero
committed
better bq job ids
Jacob Ferriero
committed
fixup regex escaping
Jacob Ferriero
committed
make pylint happy
Jacob Ferriero
committed
[skip ci] more docs
Jacob Ferriero
committed

Commits on Dec 17, 2020

fix default load config return type
Jacob Ferriero
committed

Commits on Jan 7, 2021

fix: fail on failure of children jobs
Jacob Ferriero
committed

Commits on Jan 8, 2021

chore: add test for child job failing behavior
Jacob Ferriero
committed
fixup flake8
Jacob Ferriero
committed
fixup flake8
Jacob Ferriero
committed
fixup flake8
Jacob Ferriero
committed

Commits on Jan 11, 2021

feat: separate bq storage and compute project env vars
Jacob Ferriero
committed

Commits on Jan 22, 2021

fix: don't require escaping braces in sql, still support {dest_dataset} {dest_table} rendering
Jacob Ferriero
committed
happy newyear! copyright 2020 -> 2021
Jacob Ferriero
committed

Commits on Feb 18, 2021

clean up newlines in logs / error messages
Jacob Ferriero
committed

Commits on Mar 3, 2021

improve logging in bq failures
Jacob Ferriero
committed
fixup flake8
Jacob Ferriero
committed
fixup mypy
Jacob Ferriero
committed
fixup pylint
Jacob Ferriero
committed
Merge pull request #6 from jaketf/cf-improve-logging
Jacob Ferriero
authored
fixup BigQueryJobFailure docstring
Jacob Ferriero
committed
Merge pull request #7 from jaketf/cf-improve-logging
Jacob Ferriero
authored

Commits on Mar 25, 2021

setting the snapshotting as enabled and included in tests by default
RyandenOtter
committed