Comprehensive data protection for all workloads
Post Reply
nickf
Influencer
Posts: 19
Liked: never
Joined: Jun 22, 2010 3:11 pm
Full Name: Nick Ferrar
Contact:

What happens if a DR event occurs mid-replication?

Post by nickf »

We've successfully done a couple of DR tests using Veeam replicas but each time we do a test we disable the Veeam replication jobs in advance so that when we cut the WAN link between sites we don't risk corruption on the replica. However it occurs to me what would happen during a real DR if the WAN link or source site died mid-replication job, Would the replica be usable and is there any special procedure to use when bringing it up? We do keep 3 restore points per replica but I'm not sure if that would help if the main VMDK is corrupt do to only partially updating?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by Vitaliy S. »

Hello Nick,

Yes, that would be usable, just not the latest restore point (which would be incomplete obviously). You may want to follow one of the scenarios below to perform a failover:

1. Install Veeam backup server on the DR site and use it to perform both replication jobs and a failover to a previous point in time. The restore points will be still usable even if your WAN link crashes down during the replication job.

2. You may backup your Veeam SQL configration db located on the Main site using native tools such as SQL Management Studio, and then re-use this db on the DR site with a new Veeam backup server installation.

Hope it helps!
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by tsightler »

This is pretty much why I'm a fan of "pull" replication when using Veeam. With push replication there is a risk that the replica would not be usable and you'll be left in an unrecoverable state. With "pull" replication (the Veeam server is on the DR side) you can easily roll back to a previous restore point.

If you want to use "push" replication and still be able to revert to a previous restore point then there are really two option:

1. You can install Veeam on the DR site, leave it in a "stopped" state, and use SQL replication to keep the Veeam DB at both sites in sync. In the event of a failure you can simply start Veeam at the DR site and use this instance to revert to a previous restore point. I'm not such a big fan of this because of the complication and dependencies it creates (you need working Veeam and vCenter at the DR site before you can power on VM's), but it works.

2. We "cheat" by taking storage array snapshots on our DR side prior to the start of replication. That way, if a replication cycle is interrupted by an outage we just revert the storage array snapshot and we can power on the VM's. We find this simpler, and it has no reliance on any of the rest of the infrastructure (like vCenter or Veeam).
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by lobo519 »

Sorry to bring this back from the dead but it beats trying to explain the situation again.

Is there a performance hit if I choose the "Pull" replication VS. "Push"?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by Vitaliy S. »

There shouldn't be any noticeable performance impact.

You may have a look at some Pros and Cons related to installing VBR on-site or off-site to have a clear picture:
http://www.veeam.com/forums/viewtopic.p ... 8738#p8738
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by lobo519 »

great - additional question
Looking into the Replicate sql server db to remote site for replication failures idea...

If I try to hook up the DR site Veeam install to the replicated Veeam database I get the follow error

Configuration database is in use by the Veeam Backup console Installed on "Local installation" To reconnect database to this console please run Veeam backup setup program.

I reinstalled but I am getting the same error. Am I missing something?

I can submit a support ticket if needed.

Thanks!
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by Vitaliy S. »

No need to submit a support ticket, here you go the resolution for your problem: Replicating Veeam Database :wink:
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by lobo519 »

Vitaliy S. wrote:No need to submit a support ticket, here you go the resolution for your problem: Replicating Veeam Database :wink:
just found this - works like a charm!

Thanks!!! :mrgreen:
RumataRus
Enthusiast
Posts: 78
Liked: 2 times
Joined: Jun 21, 2010 5:30 am
Full Name: Dmitry Prokudin

Possible corruption of replica

Post by RumataRus »

[merged]

if a replication job runs and due to a malfunction it is unexpectedly interrupted, will the replica roll back to previous "good" state?
In other words, is it possible that the replica will be corrupted in case of the replication job unexpectedly fails?
RumataRus
Enthusiast
Posts: 78
Liked: 2 times
Joined: Jun 21, 2010 5:30 am
Full Name: Dmitry Prokudin

Re: What happens if a DR event occurs mid-replication?

Post by RumataRus »

Sorry, but I have a different situation.
I have no DR site. I do local replication and I have Veeam Backup installed in a VM.
All my hosts are connected to one stack of switches.
I modeled the situation of failure of one of the switches (I just turned it off. :) )
In any case, there is a timeout until the stack is restored.
If at this time replication job was running, damage of the replica occurs.
Also I have experienced that if host isolation occurred in HA Cluster (host with Veeam Backup) while replication job was running, the replica has been damaged too.
Are there ways to avoid this?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by Vitaliy S. »

Well...this topic does cover your first question. Just read back through my first post and the first sentence, in particular.

Veeam will try to repair this replica automatically on the next incremental job run, though please keep in mind that you can also failover to a previous rollback with Veeam failover wizard even before the repair happens.
RumataRus
Enthusiast
Posts: 78
Liked: 2 times
Joined: Jun 21, 2010 5:30 am
Full Name: Dmitry Prokudin

Re: What happens if a DR event occurs mid-replication?

Post by RumataRus »

Mm...
I have the feeling that before I can answer you, I need to do some experiments. :?
dinger76
Enthusiast
Posts: 29
Liked: 1 time
Joined: Nov 26, 2009 11:46 am
Full Name: Chris Bell
Contact:

Veeam Replication & Failure during Replication

Post by dinger76 »

[merged]

Hi Guys,

I've not gone and tried this but just wondered what you believe would happen and the best way to avoid the situation.

If you are performing replication of your virtual machines between two sites, once a day overnight. In our scenario, we keep two rollbacks at our destination location to keep the storage requirement low. Suddenly there is an issue at your source site, during the replication process and therefore the replication task fails.
Would this have deleted the full replication that would be required to perform a failover to the destination site, or can you still failover to one of the other rollbacks?

Is there a way to make sure that we always have a safe full vm at our destination site to avoid issues like this occurring?

Thanks

Chris
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by J1mbo »

See above...
please keep in mind that you can also failover to a previous rollback with Veeam failover wizard even before the repair happens
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by Gostev »

v6 is going to be even nicer: no restore point will be left inconsistent in case of failure in the midst of replication.
darren.wardle50
Novice
Posts: 4
Liked: never
Joined: Feb 08, 2010 5:58 pm
Full Name: Darren wardle-Simmons
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by darren.wardle50 »

Gostev

Can you give us a little more detail on how replication has changed in v6 in regards to the handling of mid replication failure and the reliance of the SQL database for recovering our servers in this scenario?

Our replication server is at the source site with full ESX at the destination due to the need to run an initial replication to USB drive as our 10Mb link can't handle the initial replication of the 1TB+ of VMs we are replicating.

We do currently replicate the Veeam database using SQL replication but we have concerns that the recovery process using the database copy adds extra steps and has a much higher likelihood of failure resulting in servers that will not boot correctly.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: What happens if a DR event occurs mid-replication?

Post by Gostev »

Hi Darren, we will provide extensive details on the upcoming webinar (see sticky topic on this forum). In fact, the very next webinar is dedicated to replication enhancements. Be sure to register and attend. If you cannot attend, you will get a link to recording within 48 hours after the webinar. Thanks!
Post Reply

Who is online

Users browsing this forum: bct44 and 201 guests