Last week I helped one of our clients solve their web server being inserted with iframe problem, which started a few weeks back when they told me that their customers complained accessing to their product order status page resulting in some unexpected pop-up windows. As a result this client’s website is marked unsafe by anti-virus programs such as mcfee. My initial investication showed that accessing the problematic page results in an iframe line being inserted to the top of the page in the html source code:
<iframe src=”h t t p://w w w.h a c k i n g s.c n”></iframe>
(I intentionally add some spaces in the url as I don’t want you to accidentally hit that link going to the hacker’s site.)
A bit more detail about the setup of the webserver:
1) ASP.NET + IIS
2) runs behind a Linux firewall accessible through DNAT
My investigation continued and my accessing the same page from the firewall resulted in the same result — the hacking line being inserted. But when I accessed the page from any other workstation including the webserver itself, I didn’t see the line, which led to my wrong conclusion that there must be some virus in the webserver that is smart enough to tell where the traffic is from — when it’s from the gateway, the request is most likely made from the Internet; when it’s from the LAN, it stops inserting the line to fool the web administrator that the web server is fine. My conclusion is not coincident: I also used a Linux laptop to set up a temporary web server providing the exact same urls to replace the original webserver and the hacking line disappeared. I recommended the client to bring down the IIS webserver immediately and replace it with a Linux server. I thought the problem was solved.
A few days later, my client reported that the problem came back, with a Linux server being the web server. My hunting for the real cause of the problem went on and somehow arp cache poisoning got into my head. I heard it a lot before but never seen it in action. To get started I installed and run arpwatch on the firewall and before long I saw a lot of flip-flops scrolling up on the screen (mac addresses change back-and-forth) for a bunch of ip addresses in the client’s LAN. The client doesn’t use dhcp therefore there shouldn’t be flip-flops — At least that tells me that I was on the right direction to solve the problem. From the arpwatch log I saw something like this
... Jul 29 13:21:18 linux-gw arpwatch: report: pausing (cdepth 3) Jul 29 13:21:18 linux-gw arpwatch: flip flop 172.16.0.1 18:8b:41:72:b1 (0:13:20:3d:5e:53) Jul 29 13:21:18 linux-gw arpwatch: report: pausing (cdepth 3) Jul 29 13:21:18 linux-gw arpwatch: changed MAC address 172.16.0.199 0:50:da:7b:c6:60 (0:13:20:3d:5e:53) ...
I could see from the log entries that those flip flop lines share the same pattern — the mac address for a specific ip gets changed to the same mac address 0:13:20:3d:5e:53 then back to its original mac address. At that point I concluded whichever machine that has 0:13:20:3d:5e:53 as the mac address is the one that initializes arp poisoning attack. As the client didn’t have a ip to mac address mapping, the hunting for that machine took a bit while but luckily it was found on one of the workstations. I instructed the client to disconnect that workstation from the LAN and I conducted the test again, the hacking line was gone — either from the Internet or the gateway. What a relief! arpwatch continued to report a lot of flip-flops, but they are just for changing back to their original mac addresses from the poisoned mac address. When arpwatch stopped reporting flip-flops I generated an IP-Mac address map for the client so it will a piece of cake to ID which machine generates arp cache poisoning attack when problem like this happens again.