Retrace steps. Make walkthrough from initial files through to query to join message # with entries. Record as procedure to follow. Want model of what we are doing and why and the commands to do it.

[zipperer]

[zipperer]

See email thread with subject 'another little project' from Jan 22.

Pieces:

• BDA ... .txt files
• Title Text file

For the pieces, see Dropbox links from Michael.

Phases:

• data cleanup

  • For 'Title Text file', see the history of edits to the file history 1 history 2
  • For 'BDA ... .txt' files, title_program_numbers_and_bible_verse_timestamp_files/bible_verse_timestamp_files/edits_to_bda_files.txt

• convert to csvs

  • For 'Title Text file': see title_program_numbers_and_bible_verse_timestamp_files/title_program_numbers_into_db

  • space-separated to pipe-separated

  • pipe-separated to pipe-separated with additional columns -- additional columns for the components of the decoded date

  • For 'BDA ... .txt' files: see title_program_numbers_and_bible_verse_timestamp_files/bible_verse_timestamp_files/bible_display_app_file_entries/bible_display_app_files_to_csvs.py

  • for each .txt file, make one or two .csv files. Make two files when the input .txt file contains entries for both morning session and night session.

  • for use of script see use of bible_display_app_files_to_csvs.py in title_program_numbers_and_bible_verse_timestamp_files/bible_verse_timestamp_files/bible_display_app_file_entries/shell_interactions_use_script_to_make_csvs_and_copy_csv_into_database.txt

• import csvs into db

  • For 'Title Text file': see the 'docker*' files in title_program_numbers_and_bible_verse_timestamp_files/title_program_numbers_into_db

  • docker cp file into container

  • make table

  • \copy into table

  • For 'BDA ... .txt' files: see

    • create table docker_postgres_interactions_copy_into_table_and_query.txt
    • script that uses pyscopg2 to COPY each csv into table title_program_numbers_and_bible_verse_timestamp_files/bible_verse_timestamp_files/bible_display_app_file_entries/copy_output_csvs_into_table.py
    • and use of this ^ script in title_program_numbers_and_bible_verse_timestamp_files/bible_verse_timestamp_files/bible_display_app_file_entries/shell_interactions_use_script_to_make_csvs_and_copy_csv_into_database.txt

• query tables and join

  • title_program_numbers_and_bible_verse_timestamp_files/bible_verse_timestamp_files/bible_display_app_file_entries/docker_postgres_interactions_query_to_join_tables_morning_or_night.txt