Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEV-11036: Initial Run - No Initial Copy Bug #4245

Open
wants to merge 4 commits into
base: qat
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1135,10 +1135,8 @@ def prepare_orphaned_award_temp_table():
# Due to the size of the dataset, need to keep information about the orphaned transactions in a table.
# If we tried to insert the data directly into a SQL statement, it could break the Spark driver.
with prepare_orphaned_transaction_temp_table(), prepare_orphaned_award_temp_table():
# To avoid re-testing for raw.transaction_normalized, use a variable to keep track. Initially
# assume that the table does exist.
raw_transaction_normalized_exists = True

# To avoid re-testing for raw.transaction_normalized, use a variable to keep track.
# Test to see if raw.transaction_normalized exists
try:
self.spark.sql("SELECT 1 FROM raw.transaction_normalized")
Expand All @@ -1149,12 +1147,16 @@ def prepare_orphaned_award_temp_table():
"Skipping population of transaction_id_lookup table; no raw.transaction_normalized table."
)
raw_transaction_normalized_exists = False
# Without a raw.transaction_normalized table, can't get a maximum id from it, either.
max_id = None
else:
# Don't try to handle anything else
raise e
else:
raw_transaction_normalized_exists = True
# if we're not doing the initial copy, we're also skipping all things raw.transaction_normalized
raw_transaction_normalized_exists = raw_transaction_normalized_exists and not self.no_initial_copy

max_id = None
if raw_transaction_normalized_exists:
self._insert_orphaned_transactions()

# Extend the orphaned transactions to any transactions found in raw.transaction_normalized that
Expand Down