Posted: Thu Jan 10, 2013 11:14 pm Post subject: Need some serious DNS expertise to assist with issue
ISPgeeks, I am at my limit of DNS expertise and need some help.
Situation - ISP is ViaSat on their new 140 Gbps ViaSat-1 satellite. Throughput is good (over 15 Mbps) and latency is bad, but better than most satellites. The service is plagued with intermittent DNS issues. Symptoms manifest as browser pages not loading, accompanied by various error messages depending upon your browser flavor. Customers all over the US are reporting these issues and ViaSat (Exede, Wildblue) have admitted no issues. Most of the focus has been on the satellite modem, client OS config, and other on-premises issues. Plus there's been a lot of talk about how to remediate with host file entries, tweaking the DNS client, etc. But nobody has really looked at the network.
So ran a trace off the router and hooked directly to the modem. Here are the results:
The two DNS servers are 126.96.36.199 and .69. When browsing works, DNS trace looks good, BUT DNS responses from both name servers at different IPv4 addresses have the same source Ethernet address (00-A0-BC-35-BD-2F).
When pages fail to load with browser DNS errors, both name server respond to DNS queries with ResultCode = 2, "Server Failure". RFC1035 defines ResultsCode ServFail (RCODE = 2) as
Server failure - The name server was unable to process this query due to a problem with the name server.
Additionally, when the .68 and .69 name servers respond to requets with ServFail, the source Ethernet address for both of them is different (00-A0-BC-22-A0-6E). Even more interesting is that 00-A0-BC is assigned to ViaSat themselves.
I have an idea of what might be happening.
But I'd like some other folks that might know more about DNS than I do to chime in.
Assuming it's really a DNS issue can you change the DNS servers you point to...use OpenDNS or Google DNS servers?
No, good question. That's the first thing I tried and based on OpenDNS tools, the ViaSat system is hijacking all DNS requests. This appears to be part of their scheme to decrease overall latency by reducing the number of satellite transits required for normal DNS protocol.
The harder part for me to understand is why both name servers with different IPv4 source addresses have the same Ethernet source address. Unless, the DNS servers are onboard the satellite and the two IPv4 addresses are aliased to a single NI. That is consistent with the prefix 00-0A-BC being assigned to them. But I've found a few posts saying that their DNS systems are ground-based. So not sure there...
Then the ServFail responses *always* come from a different Ethernet source address (also the same for both IPv4 name server addresses). That's gotta be significant, but I don't know what the ServFail message means here. Does it mean another DNS server downstream failed and these guys are reporting that back? Or does it mean that the source server sending the DNS reply failed itself...?
Here's a summary of a ten minute scan showing counts for the two different Ethernet source addresses:
records from 00-A0-BC-35-BD-2F ---> 2605
number of ServFails --------------> 0
number from 184...68 -------------> 1126
number from 184...69 -------------> 1479
records from 00-A0-BC-22-A0-6E ---> 19
number of ServFails --------------> 19
number from 184...68 -------------> 10
number from 184...69 -------------> 9
It has been years since I was involved in satellite based internet but it still applies today....they Mickey Mouse around with so much to get it to work its hard to tell exactly what is happening. They are probably hijacking at the gateway...I don't know if this will work or not but its worth a try. Why not setup a local DNS server on your network, point your PC's to that instead (that way nothing gets hijacked) because the translation will happen locally and all you are going to be pointing to outbound is an IP address therefor their DNS server isn't in the picture (in theory).
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum