archive_files_to_s3 Normal 0 /home/acave P_ARCHIVE_FOLDER_NAME this will be the sub-bucket name P_ARCHIVE_FOLDER_PATH this is the folder to upload P_AUDIT_ID ID_BATCH Y ID_BATCH CHANNEL_ID Y CHANNEL_ID TRANSNAME Y TRANSNAME STATUS Y STATUS LINES_READ Y LINES_READ LINES_WRITTEN Y LINES_WRITTEN LINES_UPDATED Y LINES_UPDATED LINES_INPUT Y LINES_INPUT LINES_OUTPUT Y LINES_OUTPUT LINES_REJECTED Y LINES_REJECTED ERRORS Y ERRORS STARTDATE Y STARTDATE ENDDATE Y ENDDATE LOGDATE Y LOGDATE DEPDATE Y DEPDATE REPLAYDATE Y REPLAYDATE LOG_FIELD Y LOG_FIELD EXECUTING_SERVER N EXECUTING_SERVER EXECUTING_USER N EXECUTING_USER CLIENT N CLIENT ID_BATCH Y ID_BATCH SEQ_NR Y SEQ_NR LOGDATE Y LOGDATE TRANSNAME Y TRANSNAME STEPNAME Y STEPNAME STEP_COPY Y STEP_COPY LINES_READ Y LINES_READ LINES_WRITTEN Y LINES_WRITTEN LINES_UPDATED Y LINES_UPDATED LINES_INPUT Y LINES_INPUT LINES_OUTPUT Y LINES_OUTPUT LINES_REJECTED Y LINES_REJECTED ERRORS Y ERRORS INPUT_BUFFER_ROWS Y INPUT_BUFFER_ROWS OUTPUT_BUFFER_ROWS Y OUTPUT_BUFFER_ROWS ID_BATCH Y ID_BATCH CHANNEL_ID Y CHANNEL_ID LOG_DATE Y LOG_DATE LOGGING_OBJECT_TYPE Y LOGGING_OBJECT_TYPE OBJECT_NAME Y OBJECT_NAME OBJECT_COPY Y OBJECT_COPY REPOSITORY_DIRECTORY Y REPOSITORY_DIRECTORY FILENAME Y FILENAME OBJECT_ID Y OBJECT_ID OBJECT_REVISION Y OBJECT_REVISION PARENT_CHANNEL_ID Y PARENT_CHANNEL_ID ROOT_CHANNEL_ID Y ROOT_CHANNEL_ID ID_BATCH Y ID_BATCH CHANNEL_ID Y CHANNEL_ID LOG_DATE Y LOG_DATE TRANSNAME Y TRANSNAME STEPNAME Y STEPNAME STEP_COPY Y STEP_COPY LINES_READ Y LINES_READ LINES_WRITTEN Y LINES_WRITTEN LINES_UPDATED Y LINES_UPDATED LINES_INPUT Y LINES_INPUT LINES_OUTPUT Y LINES_OUTPUT LINES_REJECTED Y LINES_REJECTED ERRORS Y ERRORS LOG_FIELD N LOG_FIELD ID_BATCH Y ID_BATCH CHANNEL_ID Y CHANNEL_ID LOG_DATE Y LOG_DATE METRICS_DATE Y METRICS_DATE METRICS_CODE Y METRICS_CODE METRICS_DESCRIPTION Y METRICS_DESCRIPTION METRICS_SUBJECT Y METRICS_SUBJECT METRICS_TYPE Y METRICS_TYPE METRICS_VALUE Y METRICS_VALUE 0.0 0.0 10000 50 50 N Y 50000 Y N 1000 100 cluster N dynamic Y 1 master ${framework.pentaho.server.host} ${framework.pentaho.server.port} ${framework.pentaho.server.webapp} ${framework.pentaho.server.username} ${framework.pentaho.server.password} N N acave 2019/09/18 09:15:30.253 admin 2022/06/03 10:38:41.608 N #Story Moves all the zipped files from the passed in folder to the AWS S3 bucket s3:// s3 long-term storage is about 1% of the cost of the EFS (Elastic File System) used for /mnt/staging After 8 days it is moved by AWS to S3 Glacier storage After 2556 days (7 years) it is deleted by AWS. Logs into localhost using a key file so it can run AWSCLI commands 0 0 358 197 Ubuntu 11 N N 0 0 0 255 205 112 100 100 100 Y #Date: 2019/09/18 #Author: Andrew Cave 384 0 131 72 Segoe UI 9 N N 0 0 0 255 205 112 100 100 100 Y #HowToTest P_AUDIT_ID 100 Process ID of parent process P_ARCHIVE_FOLDER_NAME this will be the sub-bucket name . eg "test_folder" P_ARCHIVE_FOLDER_PATH this is the folder to upload . eg "/Users/<yourlocalusername>/Downloads/test_folder aws.s3bucket.name 448 96 605 104 Segoe UI 9 N N 0 0 0 128 255 128 100 100 100 Y #ParametersIn P_ARCHIVE_FOLDER_NAME this will be the sub-bucket name P_ARCHIVE_FOLDER_PATH this is the folder to upload P_AUDIT_ID aws.s3bucket.name -- this is from kettle.properties 512 0 394 132 Segoe UI 9 N N 0 0 0 255 205 112 100 100 100 Y #ParametersOut none 32 192 97 42 Segoe UI 9 N N 0 0 0 255 205 112 100 100 100 Y #VariablesIn 192 192 74 26 Segoe UI 9 N N 0 0 0 255 205 112 100 100 100 Y #VariablesOut 320 192 85 26 Segoe UI 9 N N 0 0 0 255 205 112 100 100 100 Y Use keyfile for passwordless login 35 343 189 26 Segoe UI 9 N N 0 0 0 255 205 112 100 100 100 Y filter std_err Write std_err from archive files to S3 Y Write std_err from archive files to S3 Abort Y SSH Archive to S3 filter std_err Y filter std_err archive files to S3 log 2 Y Abort Abort Y 1 none 0 Y ABORT

240 480 Y SSH Archive to S3 SSH Y 1 none N aws s3 mv ${P_ARCHIVE_FOLDER_PATH} s3://${aws.s3bucket.name}/${P_ARCHIVE_FOLDER_NAME}/ --recursive; 22 localhost bizcubed Encrypted Y /home/bizcubed/.ssh/id_rsa Encrypted 2be98d6a71fd58285b80abe78cc93bcdf stdOut stdErr 0 Encrypted

96 288 Y Write std_err from archive files to S3 WriteToLog Y 1 none log_level_basic Y N 0 Command that caused the error ---------------------------- aws s3 mv ${P_ARCHIVE_FOLDER_PATH} s3://${aws.s3bucket.name}/${P_ARCHIVE_FOLDER_NAME}/ --recursive; stdErr

240 384 Y archive files to S3 log 2 WriteToLog Y 1 none log_level_basic Y N 0 Moving files to S3 ---------------------------- moving ${P_ARCHIVE_FOLDER_PATH} to s3://${aws.s3bucket.name}/${P_ARCHIVE_FOLDER_NAME}/

368 288 Y filter std_err FilterRows Y 1 none archive files to S3 log 2 Write std_err from archive files to S3 N stdErr <> constant Boolean Y -1 -1 N

240 288 Y N explorerState