We have a DR setup that mirrors our main file system to an object replication target. A couple of weeks ago, the node that serves out the EVS that this filesystem lives on inexplicably rebooted itself. After the filesystems failed over, the object replication stopped going through. The first error reported is that the filesystem was unmounted during the backup:
2016-05-19 01:00:12.977-07:00 Snapshot index 3043689889793 has 18569417 indirection object entries
2016-05-19 01:00:12.977-07:00 Starting object replication session 786a231c-b1a4-11d1-973c-2f7c01c81f22
2016-05-19 22:22:00.393-07:00 Aborting: Source file system unmounting
The time stamp marries up to when we had to reboot the nodes, due to a heap overload that stopped the main filesystem being served out via NFS.
The next time the replication was supposed to run, it gave us this error:
2016-05-20 01:00:30.449-07:00 Using latest common snapshot ID 46389582 (index 3040187645955) to get the old checkpoint number
2016-05-20 01:00:30.449-07:00 Found incomplete object replication of snapshot ID 46443022 on target, rolling back to snapshot ID 46389582
2016-05-20 01:01:04.260-07:00 Object replication failed: File system not mounted
This was probably because we hadn't migrated all the filesystems over to their preferred EVS's. From then on, the object replication fails with the error
Failed. No baseline snapshot could be established between the source and replication target file systems. Verify that the replication policy specifies the correct target file system
2016-05-23 01:00:12.205-07:00 Using replication features 0x7 = TrueClonesSupport | ObjTouchMarkAsSparseAndFree | DeclonedSnapshotFilesSupport
2016-05-23 01:00:12.221-07:00 Target has 9 persona entries
2016-05-23 01:00:12.307-07:00 Aborting: Latest object replication completed on the target has snapshot ID 46389582 which is not available on the source
For the OR to work correctly, it needs two days worth of snapshots to essentially run an incremental backup. Unfortunately, I didn't set this up and the sysadmin who did has now left the company. I think at this stage, I need to start the whole thing again from scratch, though I suppose it may be possible to run an incremental backup from a recently created snapshot. Any ideas?