Merge forked changes - #23
Draft
tmck-code wants to merge 66 commits into
Draft
Conversation
* TDL-17513: Upgrade API and SDK version (#135) * added missing test cases * updated sdk version and added new fields * removed abandoned_checkouts from all test cases except discovery * updated all fields test to remove value_type from metafields stream * updated api and sdk version * added comment for abandoned checkouts in test cases * added value_type field in metafield containing type value * resolved pylint error * TDL-18045: Fields Addition in the Schema (#140) * added fields in the schema * updated schema of order_refunds * updated arguments field schema * removed unnecessary space in the schema Co-authored-by: Harsh <80324346+harshpatel4crest@users.noreply.github.com>
…148) Co-authored-by: KrishnanG <kgurusamy@talend.com>
…rm (#149) * updated schema to fix transform errors * added patternProperties for order_refunds stream * updated schema of order_refunds * resolve integration test failure * updated schema for abandoned_checkouts
Co-authored-by: KrishnanG <kgurusamy@talend.com>
* save test logs * logger usage * fix indentation error in start date test Co-authored-by: kspeer <kspeer@stitchdata.com>
* Canonicalize transactions in OrderRefunds * Move canonicalize to base * Version Bump and Changelog
* Changes : - Updated Shopify API and SDK version - Removed below fields from the customer object of orders streams : last_order_id last_order_name orders_count total_spent - Added email_marketing_consent object in customer resource as well as in customer.order resource * changes in CHANGELOG.md and setup.py (#158)
This reverts commit ef2a257. Co-authored-by: btowles <btowles@stitchdata.com>
Co-authored-by: btowles <btowles@stitchdata.com>
* updated the write_bookmark() logic * fixed pylint * test commit * test commit * fixed unittests * added intermediate state and updated integration tests * resolved final state written after every page * removed comments * added code comments * resolved comments * version bump and changelog update * fix typo [skip ci] --------- Co-authored-by: Leslie VanDeMark <38043390+leslievandemark@users.noreply.github.com> Co-authored-by: Leslie VanDeMark <lvandemark@talend.com>
* Add URLError to backoff, write unittest, changelog + version * Whitespace cleanup Co-authored-by: Andy Lu <alu@talend.com>
* update bookmarking for interrupted sync * add unit tests * minor fix * fix reviews * fix style suggestion Co-authored-by: Andy Lu <andy@stitchdata.com> * bump version 1.7.3 --------- Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com> Co-authored-by: Andy Lu <andy@stitchdata.com>
* added backoff for incompleteread error * fixed pylint * bump version 1.7.4 --------- Co-authored-by: Rushikesh Todkar <98420315+RushiT0122@users.noreply.github.com>
This reverts commit 1debdba.
* implemented backoff for ConnectionResetError and added a corresponding test case * updated commit based on review feedback * Update tap_shopify/streams/base.py * add parameterized dependancy for unittests * Update tests/unittests/test_error_handling.py Co-authored-by: Rushikesh Todkar <98420315+RushiT0122@users.noreply.github.com> * remove trailing whitespace * fixing circleci job * removing makefile since it is redundant * made changes based on review * removed extra code and switched locations to stream * remove extra import --------- Co-authored-by: Rushikesh Todkar <98420315+RushiT0122@users.noreply.github.com>
* Add interrupted sync test, WIP * Update tap to fix sync order in target file, update test assertions * Clean up for PR review, remove globals, simplify state * PR review comments round 1 * Missed one review comment * Review comments 2 * Review comments, move common function to base, clean up and simplify * Move actual_oldest_resuming_replication_date inside assertion if statement per PR review * Formatting changes from review comments
* fix all_fields test * pr review comments * pr review comments
* upgrade shopify api to 23-04 * try parallel circle tests * Remove fulfillment_service from line_item resource for 2022_10 ----------------------------- Co-authored-by: alu <alu@talend.com> * Updated config.yml * Updated config.yml * Updated config.yml * Updated config.yml * update schema name to avoid test collisions ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * decrease parallelism for rate limiting errors ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * add fulfillment_service back in since it's nested ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * Remove deprecated orders.total_price_usd field for 2022_10 ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * Add new orders.merchant_of_record_app_id field ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * add new transactions.payment_details fields 2023_01 ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * Add new transactions.payment_id field for 2023_01 ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * make jsonvalidator happy ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * Remove processing_method, payment_details, gateway for 2023_04 ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * Add total_unsettled_set object for 2023_04 ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * Add current / original fees sets objects for 2023_04 ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * Fix json and collapse newly added objects for readability ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * version_bump and changelog ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * adjust parallelism to 2 so that we don't api calls per second limit ----------------------------- Co-authored-by: Dylan Sprayberry <dsprayberry@talend.com> * Fix whitespace --------- Co-authored-by: Leslie VanDeMark <lvandemark@talend.com> Co-authored-by: alu <alu@talend.com> Co-authored-by: Leslie VanDeMark <38043390+leslievandemark@users.noreply.github.com>
* Update platform for tests * fix date
* test tap-shopify on python 3.11 * remove outdated backoff wait generator that was causing errors * pylint updates and version bump * keep on standard base image [skip ci] -----------------------------
* upgrade shopify api to 2024-01 * 2023-07 schema changes * Version bump & changelog * removed deprecated fields from customer schema * removed few keys from actual keys to check as data is not available for them * refactoring * updated the comment
* added backoff and fix interuptible sync * add the metafields query * update query and update the pagination logic * Fix the bookmarking logic * replicate the records exact matching to the bookmark value * fix the pylint issues * fix the failing unit test cases * remove shop owner from the other resources query * remove redundant query * update according to the review comments * remove redundant code * Add new field - inventoryItem * update setup and changelog * remove redundant function call * optimize the bookmarking logic * add space above class initiation --------- Co-authored-by: Vi6hal <vpachpinde@talend.com>
* add new queries * update schemas * schemas for abandoned checkouts, collections and customers * uncomment transactions query * revert the changes for the collects stream * schema changes for customers stream and update the queries * update inventory levels schema * update transactions and order_refunds schema * update schemas for the collections stream * add new schema for the metafield_products, metafields_collections, metafields_customers and metafields_orders * remove redundant queries * update the order_refunds and transactions schema * rename filename * update checkout schema * fix the schemas * update orders schema * fix collections schema * Migrate Shopify Streams from REST API to GraphQL API (#203) * graphql implementataion for custoemrs, events and locations stream * Use graphql endpoint for orders * add the queries in the respective stream class * Update the logic for inventory levels stream * Optimize inventory levels logic * update transactions and order_refunds logic * Initial commit for abandoned checkouts and collections streams * Remove reference for the collects and custom collections * optimize the order_refunds and transactions stream logic * partial logic for collections * update locations query * add collections streams changes and separate out metafields as individual streams * remove gql query file * remove gql query file reference and include the query in the code * fix abandoned checkouts logic * format the code * remove metafields reference from init file * update checkout schema * fix typo * set the default page size to 250 * make pylint happy and format the code * upgrade python packages * fix circleci pylint issues * modify variable name in collections stream * optimize get query params method * move the graphql base code to the base file * removed reference to the gql_base file * fix the pylint * optimize the source code * update unit test cases * update exception in unit test * Refactor the code - 1. Use the single query for inventiry levels to fetch the child object as well. 2. Add Sortkey in the products query 3. Update query names for the respective metafields * move date_window_size to the parent class * make pylint happy * make pylint happy * Make metafields related stream, order_refunds and transactions stream as pseudo incremental * make pylint happy * the similar scencario is tested in the integration test, so we can skip this test. * Update Integration Tests for GraphQL Migration (#205) * make initial base file changes * add individual meta streams * update all fields integration test * update locations replication key in test suite * remove old references of the stream * avoid testing on the events stream * reduce the default page size for metafields * update state file in the bookmarks * include events in the expected streams * fix interrupt sync * update foreign key for transactions table * update the custom field name in the collections stream * print the catalog in the sorted order of streams * update README and changelog
* updated orders schema * update orders schema * reduce query cost by limiting less line items * Add refunds into order schema * updated page limits * Fix pylint issue * removed order_refund from query and schema * updated pagination logic * fix refund schema * update orders pagination * Add missing fields in schema * DIsable pylint for order_refunds query * Fix json decode error * Update version in changelog * fixed review comments * update schema * Update datatype for addresses --------- Co-authored-by: Dylan Sprayberry <dylan@stitchdata.com> Co-authored-by: vishalpachpinde <vpachpinde@talend.com>
* Add missing fields * --------- Co-authored-by: Vishal Pachpinde <vpachpinde@talend.com>
… objects (#212) * remove the pseudo incremental sync logic from the order_refunds stream * setup and changelog update * exiting bookmark value should not decrease * perform pseudo sync on the order_refunds stream * replicate record that exactly matches the bookmark value * update transactions stream bookmarking logic * add pylint fix * add custom get query function * update changelog * make pylint happy
* Implement dynamic query builder * fix pylint and unit test * Include graphql query and the request-id in the logs * add unit test for schema validation * fix pylint * add retry mechanism for the ShopifyError * make the condition in positive tone
* bugfix & updated changelog
* Fix date window bug * Enhance date_window extraction logic * Update changelog * Add unit tests * Update class name in unit tests * Fix pylint issue
* Initial commit for the shipping lines stream * update query * Add order_shipping_lines stream * Update changelog and bump version * Implement dynamic query builder * fix pylint and unit test * Update changelog --------- Co-authored-by: Sourabh Gandhi <sgandhi@talend.com>
…fulfillments) (#219) * Initial commit for the shipping lines stream * update query * Add order_shipping_lines stream * Update changelog and bump version * Implement dynamic query builder * fix pylint and unit test * add retailLocation and DiscountCodeApplication fields to the orders schema * Update changelog * add location id in the orders.fulfillments * update pylint command * Skip the author field on missing `read_users` scope (#223) * Skip author field * mark the author field as unsupported for the public app * fix the unit test and pylint * fix the integration test * include scenario for unsupported field in the discovery metadata check * update review comments * changelog and setup update --------- Co-authored-by: prijendev <prijen.khokhani@qlik.com>
* Add new stream fulfillment * Remove merchant requests * Update pagination logic * Remove heldBy as it require app_reads scope * Fix type error for events field * Fetch closed fulfillment_orders as well * Raise exception on missing scopes * Fix results_per_page type in tests * Add new stream fulfillment * Remove merchant requests * Update pagination logic * Remove heldBy as it require app_reads scope * Fix type error for events field * Fetch closed fulfillment_orders as well * Raise exception on missing scopes * Fix results_per_page type in tests * Update __init__ * Fix base tests * Update bookmark tests
* Bump versions * Bump setup.py version, update changelog
* Set max size for pagination to 30 * Bump tap version * Update change log
* Initial commit for the shipping lines stream * update query * Add order_shipping_lines stream * Update changelog and bump version * Implement dynamic query builder * fix pylint and unit test * add retailLocation and DiscountCodeApplication fields to the orders schema * Update changelog * add location id in the orders.fulfillments * update pylint command * implement bulk query operation * working code * working code * Add requests module in the required * increase the poll time to 1 hour * Update logger statement and add date_window_size in the __init__ * set the max time limit for polling to 2 hours * Revert "Update logger statement and add date_window_size in the __init__" This reverts commit 4214f6f. * Add bulk operation ID in the logger statement * add wait time to 3 hours from 2 hours * fetch the last bulk operation id if it wasn't processed * fix pylint * add location and retail location back to the query * update wait time * Adjust the exceptions and few corner case scenarios * include unsupported fields as well * clear previous bulk id * customer specific, remove older bulk operation ID * update the order ID * Optimise query * include x-request-id in the log * Add new fields in the orders stream * update bulp op id * I have updated the poll wait time to 23 hours * add createdAt field in the fulfillments stream * display JSONL file size * Shopify claims that parent and child records will always be clubbed together * remove requests dependency as it's added in dev * update known missing field in the all fields test * add requests package as required * add customAttributes and number fields (#232) * setup and changelog changes * reduce the date window size for the bulk query operation to complete * clear previous bulk operation details if the date window size is modified * add custom message for the already bulk op running * remove orders from the few integration tests to save the time * make pylint happy * trying to reduce the circleci execution time * increase date window size for tests * remove orders from automatic test to save time --------- Co-authored-by: prijendev <prijen.khokhani@qlik.com> Co-authored-by: Robert Hill <30398720+rjh336@users.noreply.github.com>
* update circle config to use uv * increase parallelism * revert parallelism increase
* upgrade circleci build to use python 3.12 * assertEquals -> assertEqual
* Store the bulk operation details in the state once it is complete for the extraction if the existing fails due to the time limit of 23 hours * add automatic keys properties * update the automatic key name * Increase the chunk size to 1MB * remove chunk size modifications * update setup and changelog * update discovery test * update automatic test for the order_refunds stream * include extra automatic keys logic in the automatic test
* Added Stream Schema files (#238) * SAC-28916: Added schema.json files * SAC-28916: Updated schema files * SAC-28916: Update in schams file * SAC-28916: Updated replication key for resource_feedback * Implement data sync with pagination, bookmark handling (#239) * SAC-28916: Added streams.py files with sync logic * SAC-28916: Updated sync implementation to adderess Copilot and new implementation * SAC-28916: updated unittests * SAC-28916: Addressed copilot issues * SAC-28916: Update for integration tests * SAC-28916: Removed parent field from metadata * parent-tap-stream-id added * SAC-28916: Updated readme.md file * SAC-28916: Removed replication keys for full table streams * SAC-28916: Removed replication keys from schema catalog entry * SAC-28916: Remove article author from integration due access issue * SAC-28916: commit for test how many streams have access issue * SAC-28916: commit for test how many streams have access issue * SAC-28916: Added access scope permission check at discovery * SAC-28916: Updated integration tests * SAC-28916: Updated integration tests test_all_fields * SAC-28916: Updated integration tests test_full_replication * SAC-28916: updated circle.ci for all_field test * SAC-28916: updated circle.ci to run all tests * SAC-28916: updated test_full_replication test * SAC-28916: updated test_bookmark test * SAC-28916: updated base.py for api limit * SAC-28916: updated full_replication.py file * SAC-28916: updated full_pagination and bookmark.py file * SAC-28916: updated test_discovery.py * SAC-28916: updated marketing event replication key * SAC-28916: updated interupted sync.py * SAC-28916: updated test_start_date.py * SAC-28916: updated for running selected test cases * SAC-28916: update test_start_date * SAC-28916: enable all intergration tests and updated readme file with access scope info --------- Co-authored-by: Tushar Mittal <tushar.mittal@qlik.com> * SAC-28916:Addressed review comments * SAC-28916:Addressed review comments --------- Co-authored-by: Tushar Mittal <tushar.mittal@qlik.com> * SAC-28933: Updated setup.py and changelog.md * SAC-28933: Updated changelog.md * SAC-28885:removed parent stream metafield --------- Co-authored-by: Tushar Mittal <tushar.mittal@qlik.com>
* clear state file for the failed bulk operation * update setup and changelog
…ogress (#246) * retry if the bulk operation is already in progress * add factor as 5 * update backoff logic * fix pylint * fixes as per co-pilot * update setup and changelog
* update the bookmark value even if the record is not retrieved * setup and changelog update * add discount applications in the output * fix the test * fix copilot review comments
* add pagination for fulfillment line items * update page size * copilot comments * update setup and changelog
* adding events to expected streams * removing current_bookmark when no data available, uses previously updated * putting back the stream with the fix * removing events as expansion of coverage is not necessary
* Add support of client_cred grant type auth * Fix pylint and unit tests * Fix pylint * update integration test configuration * execute without access_token and expires config variables * Added better error handling * Fix order stream sync * Remove the expiry token variable * run test in memory * make pylint happy * fix the integration tests * Add few more unit test cases * add better logging * update setup and changelog --------- Co-authored-by: Sourabh Gandhi <sgandhi@talend.com>
* Only remove unselected fields at the top-level stream node * Changed the depth check from == 3 to == 2 * make pylint happy * update filtering logic for all the streams * fix: suppress W0613 false positive in FieldRemover.leave_field Rename unused `node` parameter to `_node` to signal intentional non-use per Python convention. The graphql-core Visitor interface requires the positional argument, but leave_field only needs to pop the field stack. No behavior change. * fix: wire field pruning into OrderShippingLines and bump singer-python to 6.8.0 * upgrade the requests library from 2.32.4 to 2.34.2
…ration polling (#252) * fix(tap-shopify): refresh token and retry on 401 during bulk operation polling * update according to the copilot comments * fix pylint * update setup and changelog * SAC-31774: fix Serialize metafield value to JSON string to prevent target type errors * Revert "SAC-31774: fix Serialize metafield value to JSON string to prevent target type errors" This reverts commit b80f37e. --------- Co-authored-by: atttiwari <atul.tiwari@qlik.com>
…errors (#253) * fix: normalize metafields serialization, schema typing, and user_agent metafields support --------- Co-authored-by: Sourabh Gandhi <sgandhi@talend.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of change
(write a short description here or paste a link to JIRA)
QA steps
Risks
Rollback steps