I need to extract a very large number of smallish (100KB to 10MB) objects (10's of million per namespace) from an HCP (v6.0.1). The files need to be written to a file system and the object's metadata written to a csv file. The only metadata I need to capture is file name, file size and hash value. And I need to break the job up into 5 or 10TB chunks.
I can see how copy out the objects and write them to a file using the data migrator tool. And I can see how using the metadata query API could return the metadata I need (and with some judicious python I could manipulate the results into a csv file).
But I don't see a way to do the job all in one go. Given the very large numbers of objects I need some way to reliably run the job, to restart the job, to log failures, etc..
Thanks in advance