Directory Anvil

2008-12-15 17:29:17 by  Support to Derek

Hello,

The directoryanvil.com domain name currently loads up a directory listing, containing of a single OpenAds directory. Please, advise whether you want us to restore a backup from a previous date?

Best Regards,

2008-12-15 17:31:50 by Derek to  Support

2008-12-15 17:42:34 by Derek to Support

Your missing the issue.

I made no changes TO ANY SITES ON THAT ACCOUNT over then entire weekend.

The open ads folder was there, but the main folder is MISSING.

AS ARE SEVERAL OTHER MAIN SITE FOLDERS>

19 sites are hosted. Tehre should be a FOLDER FOR EACH SITE.

Shadowdragonunlimited is completely missing,

Directoryanvil had a PHPLink Directory site. NOW MISSING

There needs to be a restore for several sites. Not just one, and if you restore, that does not address why they are missing to begin with.

These two above are paramount in nature to be restored. I will look up the others.

2008-12-15 17:47:26 by Support to Derek

Hello,

I have contacted our System Administrators regarding the problem and will have them look thoroughly on the matter. As soon as we have more sufficient information regarding the issue, you will be notified via this ticket.

Best Regards,

2008-12-15 17:48:24 by Derek to  Support

*ADDITIONAL INFO

of the 19 sites that are listed for being hosted on this account (my main account) there are 10 missing folders entirely, some of which were empty, but many had full sites. listed below:

10 fre***********line.com 1 1 -
11 li*******il.com 1 0 -
12 pl******du.com 1 1 -
13 re*********************ss.com 1 1 -
14 shad*****************ted.com 1 2 -
15 sh**************g.com 1 0 -
16 s***************dia.com 1 0 -
17 sh******************s.com 1 2 -
18 tra**********n.net 1 0 -
19 u*******ne.com 1 1

2008-12-15 18:18:03 by  Support to Derek

Hello,

I have sent a request to our administrators in order to restore the backup. You will be notified as soon as we have reply from them.

Best Regards,

2008-12-15 20:35:36 by Derek to Support

2008-12-15 20:38:43 by Derek to  Support

Another interesting note, the Openads folder you mentioned, has no files. And I am showing that it had a modification date of "Today"

Openads/ 15-Dec-2008 10:26

I am in the US, EST time zone, and was no where near anyplace at that time frame. So the question still becomes, who, what or how were the files updates at that time, from what IP address?

I have already changed any pw's for this account.

2008-12-15 21:17:40 by  Support to Derek

Hello,

I believe that the issue is caused by the server maintenance from yesterday, however our administrators will have the final word.
The respond time for ticket referred to administrators is 24 hours, but I think that we will receive a reply sooner.

Best Regards,

2008-12-16 14:44:48 by Support to Derek

Hello,

After checking for backup, our administrators found that there were no changes in the account's file in the last 7 days, thus there is no backup available other than the current one, I am afraid. We only keep weekly backup(up to 7 days old), and it is generated once modifications in the account's content are made.

Let us know if there is anything else we can assist with.

Best Regards,

2008-12-16 16:22:30 by Derek to Support

So your telling me you lost 11 of my sites, and there is nothing you can do about it?

These sites WERE NOT CHANGED BY ME.

You guys made changes to the servers. on Friday, the Directory anvil site had a complete website.Including content.

On Friday, ShadowDragonUnlimited had a complete site. Now on monday they do not. Someone had better look again for those two site backups. 

2008-12-16 16:38:23 by Derek to  Support

 2008-12-16 16:52:57 by  Support to Derek

Hello,

Thank you for your reply and please accept our apologies for the caused inconvenience.

I have forwarded once again your request to our higher-level technicians so they can double-check for existing backup of the domains you requested. We will let you know as soon as they provide us with a response.

Should there be anything else we can assist you with meanwhile, do not hesitate to let us know.

Best Regards,

2008-12-16 16:54:05 by Derek to Support

Sure, you can answer me how your uptime guarantee applies? Am I to be refunded for this outage, as this was not due to fault of mine?

 2008-12-16 17:56:59 by Support to Derek

Hello,

Thank you for your reply.

Since the maintenance of *************.com and the downtime were announced in advance, and as all maintenance is with the purpose of improving our server stability, I believe you do understand this downtime does not ask for compensation, as it was done for better service quality.

As for our uptime guarantee, you can view it in our Terms of Services(http://www.**************.com/terms/web_hosting_terms_of_service.html):

----
4. SERVICE GUARANTEES
***********. will guarantee 99.9% *************. network uptime and 99.9% network stability uptime. This, however, does not refer to problems stemming from:

* Server Hardware Breakdown
* User error(s) or purposeful interruption(s) of the user service (e.g. if the client shuts his/her own server down, ***********. is not responsible for the downtime)
* Failures due to software that is not explicitly supported by ***********.. Consequently, if a hardware crash provoked by the customer happens, **************. is beyond the control and responsibility for the resulting downtime.

A data backup is available on a daily basis. All the contents of a client's /www/ folder will be added to the backup, as well as all the databases that belong to the customer. The maximum space provided to a client for his backup is 5 GB of disk storage.
----

If you have further questions while we await for our administrators to double-check for any available backup, we are always available.

Best Regards,

2008-12-16 18:26:10 by Derek to  Support

Please re-read the entire ticket before quoting a service policy to me, as I am getting very tired of repeating myself.

A timely outage was expected. However you all lost my sites.

Regardless of your guaranteed uptime, this by far exceeds a "normal" or scheduled outage, since my 11 missing sites have now been "down" for almost 2 full days.

I have 11 sites missing due to a mistake on your end. One of these sites was a link directory, which drives traffic to 38 other sites I manage.

No one seems to want to do the "right thing" it seems.

first you lose my sites. Tell me there are no backups. If you had replaced my files properly there would have been no problem. Now you tell me, suck it up.

I will give you until the end of the day for a better resolution then that. 

2008-12-16 18:54:26 by  Support to Derek

Hello,

We received a reply from our administrators, there were no changes in this account the past 7 days and the backups that we have these folders do not exist. Also there were no modifications made on this account the last 7 days.

2008-12-16 19:44:05 by Derek to  Support

I have not made changes over the past seven days. However, the restore does not represent the files that were in existence on Friday.

This is simply math people. Ask teh admins if there are 19 listed domians within the WWW folder for the "*********" account, If so Restore it.

I know there were no changes, however if the restore is not correct, then it needs to be FULLY RESTORED.

Can someone compare the backup files with the account??? OR can someone upload the last several days backups for this account (in zip or tar files) so that I can do the comparisons.

If you say the files are no different, let me take a look at them. Stop simply telling me that everything is "OK"

2008-12-16 20:52:30 by  Support to Derek

Hello,

This ticket was forwarded twice to our administrators and the two times the answer was that these folders do not exist in backups.
There is no option to move the backup to the FTP space of this account.
If you insist they can restore the backup but these folders do not exist there so the final result will be the same.

Best Regards,

2008-12-16 21:39:43 by Derek to Support

I appreciate your help in this matter but let me assure you of a couple of things.

Regardless of what they say is not there. I know for a fact and can prove that on Friday night all 11 of those sites existed on the server.

I have traffic and page loads that prove pages were loaded on those days. I myself visit these sites every weekday (I take weekends off).

So I do know for fact that these sites existed, did load, and that you all caused this problem, whether you all will admit it or not.

Now I do not know if someone is not taking the time or effort to fully research the missing files, at this point I do not care who did what.

While you passed the request on twice, though appreciated does not matter either, as this is still unresolved.

If you all do not have the files, as you claim to make backups for seven days, and I know seven days ago the sites were fine, how do you explain there were no files?

Assume for a minute I am right. Assume that I know wtf I am talking about. Check the logs. You will see files loads for those days.

you will note that I had traffic and stats on Friday.

How would you react if you were 100% absolutely sure files existed? Because I am.

You can restore files, but cannot load a copy of the "restorable" files to my account? That seems odd. I am getting extremely tired of this ticket and simply want to resolve it.

2008-12-16 22:54:25 by Support to Derek

Hello,

I have checked the traffic stats and these sites are with 5-10 visits for December (only www.rea****************ss.com is an exception with 178 hits for this month) , most of the visits are from crawlers/bots of search engines. Also there are traffic stats from 16/Dec for example sha***********dia.com.
In addition I double checked with our administrators and there are no such folders in the backups that we have.

Best Regards,

2008-12-16 23:03:03 by Derek to Support

Regardless of your backup file conditions, and even though the "stats were not many", it is still proof that these sites did have content on them. I even have cache files on my pc as well specific to those days.

I know the files were there. I know I did not delete them. I know the truth and the truth is that you guys screwed up and are not willing to admit it or able to fix your screw up.

It appears then I will need to take my business elsewhere. I will work on finding a new reseller account in the future. Until such time I will work on rebuilding the lost sites.

In the meantime, I still expect to be compensated in some manner for this downtime.

2008-12-16 23:34:33 by Support to Derek

Hello,

If there are stats does not mean that there was a content. Please review the traffic stats for www.sh****************ia.com or check the daily statistics for re********************s.com you will see that there are statistics from 14/15/16 Dec.

Best Regards,

2008-12-17 06:47:17 by Derek to Support

2008-12-17 06:57:10 by Derek to Support

Additional notes, the google cache contains text version of teh files http://64.233.169.132/search?hl=en&rls=GGLJ%2CGGLJ%3A2006-42%2CGGLJ%3Aen&q=cache%3Adirectoryanvil.com

But since you all think I must have deleted 11 sites, (even when you all show no server changes to that effect), I still must be FOS.

2008-12-17 08:01:03 by  Support to Derek

Hello,

I understand your point completely on this case.
Please note that we are not trying to prove that the you are lying and that we are not trying to help.
We as well have no benefit of loosing clients .
The problem in its roots is that that the backups do not contain such files.
Backups of the accounts are created only if changes are done to the files in the account.
Also the backups are kept for 7 days and this are the oldest backups that we posses as it is stated in our Terms Of Service.
Unfortunately without the missing files in the backups there is not much that we can do.
The google cache for http://64.233.169.132/search?hl=en&rls=GGLJ%2CGGLJ%3A2006-42%2CGGLJ%3Aen&q=cache%3Adirectoryanvil.com is from 4th of December.
We are really sorry for the development of the issue in this case and if you wish so, we will send the problem to our administrators once again, so they can try to find any reason for the issue.

Looking forward for your reply.

Best Regards,
2008-12-17 15:00:45 by Derek to Support

You are still welcome to send it to the admins. Although I no longer expect any resolution.

While I know the cache is from dec 4th, there would be no reason for me to simply eliminate 11 sites, with the one in question (directoryanvil) being of importance to a dozen other sites as a link directory.

I would have no reason to want to, try to or need to have removed the files.

If you kept 7 days backups, should not there be 7 possible backups to check? or 4 now that this has gone on fr three days?

2008-12-17 15:35:09 by Support to Derek

Hello,

Thank you for your reply.

Our administrators have checked the availability for 7 days back(the first time backup was requested) and there was no backup for the folders you requested. The logs also showed that no actual changes were made to files within the folders for the last 7 days, which is also the reason why no backup was generated.

A more accurate explanation would be an example of this type:

6 days ago you uploaded a file called something.html. On the morning after you uploaded the file, daily backup is generated and something.html is saved on the backup server.
5 days ago you upload a file called jumpy.png, the next day the backup script makes a backup of the account and adds jumpy.png to the backup of your account.
3 days ago you decided to delete the file something.html, and 1 day ago you deleted jumpy.png, but their backup still remains on the backup server:
1) today you request the backup restored and we do it, jumpy.png and something.html are once again within your account
2) tomorrow (7 days go by) you request backup restored, only jumpy.png is once again within your account, something.html is already gone

I hope this information can be helpful in understanding the situation.

Best Regards,

2008-12-17 16:28:37 by Derek to Support

Look, I understand how backup work. I have been working as a "hosting provider" and tech support for the last 10 years. I also am a computer programmer / DB manager.

I simply use your server as I could not afford to spend several hundred a month on a dedicated platform. Up until now I had no problem what so ever with your company performance.

However it seems that your company is not doing its best effort to get me to leave your service.

I have received nothing but grief on this incident. You are all acting as if I am a moron. You're all assuming I did something. But you agree that "no changes were made, hence no backup changes". But yet then you make it seem like I must have changed something.

I can assure you all of the following statements are facts:

1: the files existed on the system on friday.

2: the files did not exist on Monday.

3: I made no logins, changes or did anything to affect the sites over the weekend.

4: You all somehow lost the files

5: you all somehow have either a faulty, corrupt, or incomplete backup

6: You all must think I am a complete idiot

7: You do not believe that I am telling the truth

8: that even though the logs show that "files" were downloaded, you still claim no files existed

9: That the DB data was updated by a "user" from a separate IP address who would have had to use a "submissiong form", yet you still claim there were no files.

10: that your company does not want to admit that it is at fault somewhere

11: Your doing everything possible to prove me wrong

12: your company seems more willing to lose my business (50 plus sites) and all future business my company might generate, simply to avoid accepting blame

13: your company is more willing to lose me then reimburse me for hosting even though the outage is clearly an exception to the policy

14: that whoever did the server upgrade should have made a full backup of the server first

15: that your very next reply had better come from someone who does give a damm

16: I will gladly lookup and forward this incident to whoever controls the company

17: I have now taken a loss of three days worth of my time

18:I have lost money on site promotions that I obviously would not have done unless the site existed.

19:Now I need to take time to get all the site files restructured, uploaded, and working again, whether it is on your site or someone else.

20: lastly that the very next reply I get will likely determine how quickly it will take me to decide once and for all if I am staying with your company.

2008-12-17 17:32:56 by Support to Derek

Hello,

Thank you for your reply.

I can assure you that we do not wish to chase you away and we are not pleased about the situation either, however the checks done by our administrators have the result we already mentioned. Also please accept my apologies if my previous explanation has offended you in any way, it was by no means meant in such a way - I was simply attempting to clarify the situation and how our backup system works, after I had spoken personally to our administrators to confirm and clarify the situation with your account.

Since your dissatisfaction is evident and our support team is at this moment powerless to actually assist you, I have forwarded your case to our CTO(Chief Technical Officer and technical support supervisor).

Best Regards,

2008-12-17 18:32:18 by Support to Derek

Hello Derek,

I completely disagree with you. If any in our replies you were left with the impression that we are thinking that you are the cause of this problem please accept my sincere apologies but that is not we are thinking or implying in any manner or fashion.

We offer backup on a daily basis as stated in our terms of service. We backup the sites each day. You upload your site and in 24 hours at worst the backup is generated. We use an incremental backup (since it is faster, causes less load it is more efficient) meaning that the backup is created on what-was-changed basis. If files are deleted the same files are removed from the backup. If files are changed the same change is applied in the backup.

Files can be deleted by:
- the customer - which clearly is not the case
- through an unauthorized access which can be gained by:
1) guessing/obtaining the username/password for either the account, the root FTP account or SSH account
2) through a php remote inclusion vulnerability -

Since you hosted many sites there my best guess is that this is exactly what happened. Those type of attacks one of the most common ways to hack a site. After the attacker deleted the content the backup script simply removed the files that were not present in the account.

What I will do now is to try to confirm this. I will forward the ticket so the administrators can send us any activity logs that we have left and present in the system.

Please note that we will not be able to restore any files since they are not in the backup. The deletion is irreversible in this case. The only thing we can do is to see if there are any traces regarding the deletion.

Best Regards,
Support Supervisor

 2008-12-17 19:46:49 by Derek to Support

While I understand that a potential php issue could be at risk, the sites I am using do test for such vulnerabilities.

Of these sites, the ones in question were a wordpress site, phplinkdirectory, and one site that was basic html and a subfolder with php openproject.

Also, in the event someone did hack the site, why not delete everything? Not just some? They could have deleted the folder with actual backups.

If there was a hack, then I would like to help find it and fix it.

It also seems quite "coincidental" to have a xss issue on a site that has been there for months on the exact same weekend that your company worked on the exact server it is on...
 

2008-12-17 20:23:28 by  Support to Derek

Hello,

There is a request sent to our administrators to retrieve and provide all server logs for that account. We will contact you as soon as we have reply from them.

Best Regards,

#########################################################################

NOTE:  While I was in the process of uploading files, I noticed that the site litterally changed in front of my eyes. Upon checking I noticed that suddenly, and mysteriously all 11 of the missing sites were not fully restore. see next entry.

#########################################################################

2008-12-18 18:20:51 by Derek to Support

I would like to commend whomever found my file backups.

I was working on the sites when I noticed it had changed right in front of me. I have proceeded to download all site files.

As many of these sites do not change directly (mostly DB driven) and only requiring db backups. I do not normally need to update the files, hence the site structure does not change often.

I do not know if this would cause your backups to not trigger for a while, but I will make sure I have updated backup routines for both the files and the DBS

Once again, thank you to whomever did the restore. I appreciate the effort put forth and in doing so you may have saved yourself a customer. Thank you.

Derek Wood

2008-12-18 18:24:57 by Support to Derek

Hello,

I am writing to inform you that our administrators were able to retrieve the missing data, it is located in the /home/www_current folder. Please note that you can manipulate the data there, however if you want to delete that particular folder you will need to contact us.

Best Regards,

2008-12-18 18:27:39 by Support to Derek

Hello,

You are welcome sir.
Please accept our sincere apologies for the inconvenience caused.

Best Regards,

###########################################################################

Final Notes: Notice I still neither received compensation for the downtime or an explanation of the downtime cause. To me this stall makes it a dissappointing experience.

While they obviously did have a copy of my site files somewhere, it took three days before the ticket was escalated to someone who could help. So either their teir 1 techs do not know all the places a backup can be stored,  or do not know when to escalate an issue in a timely manner.

Also, it begs to know who were the "admins" that said there were no files to restore?? Obvisouly those admins were either wrong or did not care enough to look. Perhaps they could not be bothered with helping a customer? Either way the site files were found.

What if I had simply given in to circumstances on day one. It appears to me that they would have gladly let it go at the "no files, so sorry" type of attitude.

Me personally, I used to work directly for another hosting company. We had employees that did not care, were rude to the customers and did as little work as possible. People like that are the reason behind me starting my own company in the first place. Now I still seem to have to deal with those people.

For all those whose sites I host that were affected, a refund of my service charges for this outage will be in effect. While they did not refund me, I believe in accepting responsibility for my services as it was my choice to host with this provider.

I will be looking into moving sites to a new host provider as I can. All site owners will be notified in advance for the DNS changes once this has been decided. All sites will also recieve additiona credits for the outages that will be incurred during the dns tranfers.

My sincerest Apologies for this issue.

Derek Wood -

Anvil Group Owner and Administrator

DirectoryAnvil.com - Hostanvil.com - ArticleAnvil.com