What is the best way to list the files in an AWS S3 bucket using a Job or Transformation? <a data-tag-text="Kettle" data-sign="#" class="user-content-hashtag" href="https://hitachi.connectedcommunity.org/search?s=tags%3A%22Kettle%22&executesearch=true" data-tag-key="e58f5353-0c00-4c3e-854c-4b082a5ea7b6">#Kettle</a> <a data-tag-text="Pentaho" data-sign="#" class="user-content-hashtag" href="https://hitachi.connectedcommunity.org/search?s=tags%3A%22Pentaho%22&executesearch=true" data-tag-key="70b4436b-4549-4ce3-aa64-4566c7b665c9">#Pentaho</a> <a data-tag-text="PentahoDataIntegrationPDI" data-sign="#" class="user-content-hashtag" href="https://hitachi.connectedcommunity.org/search?s=tags%3A%22Pentaho Data Integration PDI%22&executesearch=true" data-tag-key="b01e3d75-3411-4f9a-a674-a7e8d0c9c995">#PentahoDataIntegrationPDI</a>

Pentaho

View Only

Rob Burgess posted 05-18-2018 18:14

What is the best way to list the files in an AWS S3 bucket using a Job or Transformation?

Steven Brown posted 05-24-2018 21:18

Hi Rob,

I would use the AWS CLI to list, output to a text file and use that as an input text file source. Something like:

aws s3 ls s3://{bucket-name}/{path} >result.txt

of if you wanted to get the entire contents:

aws s3 ls s3://{bucket-name} --recursive >result.txt

And then you could use the regular expression /(.*)\/ to parse the path.

HTH,

Steven

Kiran Rajendran posted 05-24-2018 21:20

Get File Names with a path in this format:

s3://<AWS Access Key>:<AWS Secret Access Key>@s3/MyBucket/path

pastedimage_0