10G 2008 7search active adwords alter client configure database dedicated defragment directory disable download ecommerce email error FTP FTP7 Funny google HELP home hosting IIS7 IisManagerAuth imap issue keyword Login managed Manager miva oracle performance position price protocol QUIT restrict retention searchfeed server shrink SMTP telnet user web write yahoo

The funniest support issue this year so far

It happened just few weeks ago and it was surreal. A client of mine that hosts numerous database driven application servers for his clients skypes me with what sounded like possible breach or compromized account or server. Couple of hours later I could not stop laughing. While our chat log does not fully capture the timing and all extreneous information it still manages to be funny enough to share here. Obviously reference to certain items were edited as were our skype handles down to first letter. In the chat below I'm Z and my client is L.

L: u there?
L: somebody hacked into *SOFTWARE* / *DOMAIN* and deleted all users and partners (including files)
L: can you please restore db and files for *DOMAIN*.com? thanks
Z: what?

Z: the partners were not there last night either... last time I backed up partners was at the end of saturday
L: there are few directories missing
Z: ok let me see
L: all the users and partners were deleted yesterday between 6 and 10pm
Z: i'm running restore it will take a while

Z: i seriously doubt that anyone would gain access to the database without at least once failing to logon due to bad password and then proceed to cascade delete data from provider table while not using cascade delete (since your *SOFTWARE* doesn't have constraints) but suceeding at deleting dependent values from dependent tables for those records...

Z: and not doing anything else malicious - i think something went awry there...
L: somebody gained access to *SOFTWARE* and deleted users and partners one by one
Z: from where?
L: i could not capture the ip... at this moment i am looking in the registry for the ip
Z: useless
L: but that returns nothing sometime
Z: on server - the client IP is set in web server variables so you can get it from headers and send it back to your program

Z: also i can see the api call logs i guess...
L: can you take a look at api logs yesterday after 6 pm?
Z: started at 3:53 btw... the IP was hitting server almost daily in the past
L: i only care about yesterday - can you give me the ip?
Z: IP.IP.IP.IP (the ip is located in vegas)
L: are you sure???
Z: hell yeah
Z: someone affiliated with *VERY_SIMILAR_DOMAIN_NAME*.com possibly - and someone with Funwebproducts spyware on their machine
L: what? how do you know?
Z: and someone who apparently edits *DOMAIN* a lot
L: the guy who owns the website stays in las vegas...
Z: i've seen extensive edits over past few weeks... well it was from his machine
L: is he on drugs?

Z: looks like he visits the website about 20x a day... or more - a little obsessive
L: how do you know about the spyware?
Z: maybe he left his computer somewhere
Z: because his client string "Mozilla/4.0+(compatible....FunWebProducts...." has funwebproducts in there in the header...
Z: so that's xp with IE7
L: rofl
L: u good

Scroll down for additional info labeled you won't believe what the problem actually was

More info: you won't believe what the problem actually was

permanent link to article http://freecash.hogger.net/the_funniest_support_issue_this_year_so_far

Z: someone else was editing as well recently .. let me see about this ip
Z: IP.IP.IP.IP - that's not you is it?
Z: yeah that's you
L: i did some changes last week
Z: 3/24 is not last week - that's today
Z: but between 3/22 and today only some guy with funwebproducts was editing - no one else...
L: well, i went in today... thanks man
L: i do not pay you enough...

Z: so what is he smoking?
L: i am trying to find out...
Z: has to be good
Z: ok just to make sure do you know exact time?
L: i had it in the database... the user login arround 6pm and started deliting arround 7, why?
Z: ok because i see that a lot of calls happened 3:37PM, then bunch at 3:41PM and then 2 calls at 6:05PM and then bunch at 9:23PM
L: 2 calls at 6:05PM and then bunch at 9:23PM - these did the deletes
Z: 9:23PM through 11:25PM quite a long time
L: i know
Z: well still from the same computer
L: there where hundreds of users and partners... did the database restore finished?
Z: database a while ago - you can search directory - it's not empty anymore

L: about 350 partners and 300 users
Z: then that would explain why it took 2 hours of clicking delete before it all went away
L: :-)
Z: we need to have per incident charge if something like user's own sabotage is the reason for data loss...
L: rofl
L: and what would the charge be?
L: a bullet?
Z: actually i'm sort of serious - it took us few hours to get to the bottom of this and it's all because when you're high it's fun to click YES I'M SURE I WANT TO DELETE THIS

Z: 600 times in a row

L: rofl ROFL

L: i'll get to the bottom of this tomorrow...
Z: i mean normally it would stop being funny after maybe 10 or 20 clicks
L: lol

Z: I still can't believe what we just had to deal with - this has to go on internet as some kind of joke and if it was not real it still would be pretty funny
L: rofl
L: i know!