We are one of the "little guys" who run DT, based in the UK, so most of this shit happened when we were drinking beer, eating dinner and getting ready for the night shift.
We're still waiting to hear from DT that there was/is a problem and still waiting to hear what caused it and still waiting to hear what they are doing to fix it, and what they intend to do to compensate us for the poor service. As has been pointed out, whatever the cause, it could have been prevented, communication has been crap, and absolutely no disrespect to Jon, but if WF is the oracle on whether mission critical systems that run hundreds of millions of dollars of business through them are up or down then I really need to look at my SLA's in the contract. Hats off to Jon for breaking this and saving me a lot of money, next time you are in London Jon let's meet up for dinner on me and get Konrad along, will be worth it. This and the Facebook issue big +reps.
Although certain functions were restored for affiliates, we still, even now, can't access certain back end functions, no doubt as a result of tweaks they have made to the security settings, or that is what I think.
The issue of monitoring for uptime is a thorny one, you pay someone for a service, if you then have to pay someone else to monitor the people you are paying to provide the service it can quickly snowball out of proportion. We've looked at industrial strength url monitoring and the cost was double what we pay DT on a monthly basis.
Any company should have a disaster recovery plan. Clearly, ours sucked. We'll review it, change bits, add bits, put in some extra layers of checking and dig our hands in our pockets to compensate affiliates that clearly were affected by it, lick the wounds and move on. We extract the affiliate data from DT every day, so we could have got something out, but we didn't know what was happening, and if it's a 5 minute blip on our DNS we'd all lose a lot more money. Next time we might not wait, but we're screwed either way.
The issue of networks who don't run DT is this. Unless they have 100% exclusive offers and don't work with any of the top networks they will have been affected by association. Networks run each others offers.
Advertisers sites go down, affiliate sites go down, networks go down, but as long as we learn and we make good we should be OK right?
I'm frightened to go to bed now.
I should probably also let the thread die.........
We're still waiting to hear from DT that there was/is a problem and still waiting to hear what caused it and still waiting to hear what they are doing to fix it, and what they intend to do to compensate us for the poor service. As has been pointed out, whatever the cause, it could have been prevented, communication has been crap, and absolutely no disrespect to Jon, but if WF is the oracle on whether mission critical systems that run hundreds of millions of dollars of business through them are up or down then I really need to look at my SLA's in the contract. Hats off to Jon for breaking this and saving me a lot of money, next time you are in London Jon let's meet up for dinner on me and get Konrad along, will be worth it. This and the Facebook issue big +reps.
Although certain functions were restored for affiliates, we still, even now, can't access certain back end functions, no doubt as a result of tweaks they have made to the security settings, or that is what I think.
The issue of monitoring for uptime is a thorny one, you pay someone for a service, if you then have to pay someone else to monitor the people you are paying to provide the service it can quickly snowball out of proportion. We've looked at industrial strength url monitoring and the cost was double what we pay DT on a monthly basis.
Any company should have a disaster recovery plan. Clearly, ours sucked. We'll review it, change bits, add bits, put in some extra layers of checking and dig our hands in our pockets to compensate affiliates that clearly were affected by it, lick the wounds and move on. We extract the affiliate data from DT every day, so we could have got something out, but we didn't know what was happening, and if it's a 5 minute blip on our DNS we'd all lose a lot more money. Next time we might not wait, but we're screwed either way.
The issue of networks who don't run DT is this. Unless they have 100% exclusive offers and don't work with any of the top networks they will have been affected by association. Networks run each others offers.
Advertisers sites go down, affiliate sites go down, networks go down, but as long as we learn and we make good we should be OK right?
I'm frightened to go to bed now.
I should probably also let the thread die.........
