Skip to content

Move parsed folder and rename zipped#297

Open
PascalEgn wants to merge 5 commits into
SCOAP3:mainfrom
PascalEgn:move-parsed-folder-and-rename-zipped
Open

Move parsed folder and rename zipped#297
PascalEgn wants to merge 5 commits into
SCOAP3:mainfrom
PascalEgn:move-parsed-folder-and-rename-zipped

Conversation

@PascalEgn
Copy link
Copy Markdown
Contributor

@PascalEgn PascalEgn commented Apr 20, 2026

  • renamed ZIPED to RAW and move parsed folder
  • unify jagiellonian process_file with other publishers

@PascalEgn PascalEgn force-pushed the move-parsed-folder-and-rename-zipped branch 2 times, most recently from e985973 to 55c7b02 Compare April 20, 2026 12:26
@PascalEgn
Copy link
Copy Markdown
Contributor Author

Once this is merged on qa and prod we can run something like this to move the already parsed files in all buckets:

#!/bin/bash

set -e

# Configuration
REMOTE="scoap3"
BUCKET="scoap3-<PROD/QA>-workflows-<PUBLISHER>"
SOURCE_PREFIX="extracted/parsed/"
DEST_PREFIX="parsed/"

echo "Starting migration of S3 objects..."
echo "Remote: $REMOTE"
echo "Bucket: $BUCKET"
echo "Source: /$SOURCE_PREFIX"
echo "Destination: /$DEST_PREFIX"
echo ""

echo "Copying objects..."
rclone copy \
    "$REMOTE:$BUCKET/$SOURCE_PREFIX" \
    "$REMOTE:$BUCKET/$DEST_PREFIX" \
    --progress \
    --verbose

echo ""
echo "Deleting source objects..."
rclone delete \
    "$REMOTE:$BUCKET/$SOURCE_PREFIX" \
    --progress \
    --verbose

echo ""
echo "✓ Migration complete!"

@PascalEgn PascalEgn force-pushed the move-parsed-folder-and-rename-zipped branch from 55c7b02 to 7497ff6 Compare April 30, 2026 08:07
@PascalEgn PascalEgn force-pushed the move-parsed-folder-and-rename-zipped branch from 7497ff6 to 05f05a2 Compare April 30, 2026 12:37
@PascalEgn PascalEgn force-pushed the move-parsed-folder-and-rename-zipped branch from 05f05a2 to 700497b Compare April 30, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant