Monthly Archives: June 2008

Slow Internet with Leopard

How to fix Slow Web Browsing and Slow Internet in Leopard (10.5.x)

Symptoms

  • Web pages load slowly in Safari or Firefox in Leopard.
  • Web sites won’t load, only load partially, stop loading after a few hours.
  • Slow DNS (domain name) lookup in Leopard. First load of web site is slow with “looking up domain” in browser status bar.
  • Once website is loaded, browsing to that site is fast.
  • AirPort wireless strength drops, then Internet connection is lost (see related post).
  • Email programs are slow in connecting to servers.
  • SSH sessions are slow to connect to remote servers.

Possible Causes of Slow Internet under Leopard

  • Your ISP’s DNS servers are (sometimes) slow to respond due to high traffic.
  • Firefox, Camino, Safari is requesting domain name lookups in IPv6 format (2001:db8::1428:57ab), but your DSL router/cable modem answers with IPv4 addresses (192.0.2.235) (references: mozillazine.org, mozilla.org bug, arstechnica.com). Safari may not be affected by this as WebKit is said to use IPv4 domain lookups first, then uses IPv6 if IPv4 fails.
  • Your router, acting as a DNS Proxy, doesn’t recognize nor forward IPv6 domain name lookup requests.
  • Leopard is now requesting SRV (service) records for domain name lookups. Your router does not recognize nor forward to SRV requests.
  • Your ISP’s DNS servers don’t recognize or doesn’t respond to SRV queries or respond with NXDOMAIN.
  • [Added 080618] Poor wireless router performance in general (references: entropy.ch). To test this, try connecting directly to your DSL router/modem if you are using an intermediate router such as an Apple AirPort Base Station, or NetGear/Linksys wireless router and seeing if web and internet speeds increase.

Fixes/Solutions/Workarounds

Details

After upgrading to Leopard, plenty of Mac OS X users have complained of “slow internet” when browsing the web, yet Windows PCs or Macs with Tiger (10.4) on the same network are much faster.

DNS Lookups

A domain name lookup or DNS lookup is done every time you visit a web page, say “apple.com”, as you’re actually visiting “17.149.160.49”. A DNS Resolver on your computer sends a request to a DNS Server that handles this lookup or translation from names (easy to remember) to numbers (hard to remember). Once your browser has this numerical IP address it can start loading the web pages at that server location.

Domain Name System Lookups in Leopard

With Leopard, a major change occurred in DNS lookups. Any program in Leopard that can use version 6 IP addresses (IPv6 explained below) will send out a new type of DNS lookup request – the SRV Record. In Tiger and previous OS X versions, DNS lookups were A record requests.

SRV records are new (sadly, 8 years old is new in the DNS world), provide more information than A records, but have terrible support in terms of hardware (your DSL router or cable modem) and DNS servers that answer with SRV information. For every SRV request that Leopard sends it must wait for a valid reply. If the request fails, Leopard must try again. If it fails again, Leopard will finally ask for an A record. This is one reason why Mac users are experiencing slow Internet on new Macs with Leopard or after upgrading to Leopard from Tiger.

Domain Name Lookup Chain

Diagnosing slow Internet problems under Leopard is difficult due to the many different slowdowns that can occur along the domain name lookup chain when connecting to the Internet in OS X. For an application like Firefox or Safari to find a domain name, this is roughly what happens:

  1. Firefox/Safari is asked to load a web page at a domain name (example: “apple.com”).
  2. Browser starts work on getting an IP address for that domain (a domain name lookup).
  3. Browser checks for recently translated domain names in its own internal “cache” and thus already has the IP address.
  4. If “apple.com” is not found in cache, Firefox/Safari then asks Directory Services (an OS X program that does DNS lookups) for the answer.
  5. Directory Services (DS) searches for the domain in its own DS cache (view the DS cache using Terminal: dscacheutil -cachedump -entries).
  6. If domain is not found in cache, DS checks flat (text) files such as /etc/hosts for the domain name (see the file using Terminal: cat /etc/hosts).
  7. If domain is still not found then DS sends a domain name lookup request to the first DNS server listed for your AirPort wireless card or your Ethernet card (your network interfaces). The first (and usually only) DNS server is often your router (often listed as 192.168.1.1 in System Preferences => Network => Advanced => DNS tab).
  8. If the router doesn’t recognize the name lookup request (SRV/IPv6), the request will be either ignored, returned without result, returned with error. If the router does recognize the DNS request, it checks its own DNS cache for a matching domain lookup.
  9. If domain name is not found in cache, the router forwards the request to the ISP’s DNS server.
  10. If the first ISP DNS server doesn’t respond or doesn’t have the record, the router sends a second lookup request to the next DNS server listed in its configuration. Continue until all DNS servers are exhausted.
  11. When name lookup result is received by router, it saves the result to cache, then forwards the domain name record back to the requesting computer.
  12. Directory Services on Leopard, receives the answer, places it in cache, then returns the results to the requesting application: your browser.
  13. Firefox/Safari receives the DNS record, with IP address, stores it in cache, then starts to retrieve the web page at that location.

(Illustration by Lion Kimbro on Wikipedia – Domain Name Systems article)

Any one of the links in the chain can be a potential source of slow Internet speeds when browsing or retrieving mail, etc. The difficulty lies is finding out where the problem exists and how it can be fix. Compound this complexity with the number of different DSL routers in use in homes, the number of different firmware (software inside the router), number of different ISP DNS servers

Caches

Caches store recent domain name lookup results in order to save time when the domain is requested again. Each time a domain name lookup is made, caches are checked to see if the lookup has occurred recently and if so, use the cache result. If no result is found in cache, the domain name lookup has failed and the DNS lookup request continues down the chain. A domain lookup may fail all the way down the chain until it’s finally resolved with the second or third DNS server listed, taking maybe 15 seconds to finally succeed. But, once domain lookup has been successfully performed, this domain request “answer” is cached all the way back up the chain, for varying amounts of time. Browsers like Safari and Firefox normally cache domain name lookups for 1 minute (30 minutes if you’re Internet Explorer in Vista). Leopard’s Directory Services program caches lookups for one hour (3600 seconds) by default.

Once a successful domain lookup has occurred, web pages from the same site will load very quickly, since the domain and its IP address are known and cached in memory. When the cached domain lookup result expires, the vicious cycle of slow domain lookups restarts. This often leads to the confusing pattern of fast Internet / slow Internet performance that can be seen sporadically throughout a browsing session.

IPv6

IPv6, the new way of addressing all things on the Internet, is important and necessary as we’ll eventually run out of IPv4 addresses (like 17.149.160.49). But part of the issue with slow browsing and slow Internet on Leopard is the combination of how IPv6 is used in Mac OS X and the current state of DSL routers and cable modems.

Whenever a program on Leopard can use IPv6 addressing, such as Firefox, it will request IP addresses for domains in IPv6 and if that fails, Firefox will then try IPv4 domain lookups. The reason this adds to the slow Internet problem is that many routers and DSL or cable modems in peoples homes are not capable of handling/routing IPv6 domain name queries (properly). This can cause repeated, failed DNS queries in IPv6 format, with the requesting application eventually falling back to sending IPv4 domain lookup requests that are successfully answered. The unfortunate problem with this “IPv6 then IPv4” order of domain lookups is users end up with delays of 5 to 10 seconds “looking up” a domain name, which is not a very long time to wait, but suffering short delays every time you visit a different website can be extremely frustrating.

SRV (Service Record) Requests

Part of the issue may be related to Apple’s decision to follow the Internet Engineering Task Force‘s recommendation of using SRV queries instead of “A record” queries when looking up domain names in Leopard.

The problem with Leopard asking for SRV records from DNS servers is that many DNS servers still don’t recognize or respond to SRV type DNS requests, or respond with a non-existent domain (NXDOMAIN) error code. This is not exactly Apple’s fault for asking, it’s actually the fault of DNS server owners who are not updating their servers to the latest standards. Regardless, whenever a program like a web browser requests a DNS record and gets failed responses, or no response at all, the program retries its requests, but only after a certain delay. Each failed SRV request and subsequent retry adds time the user must wait before the browser or application eventually gives up on the SRV requests and tries an old-school basic A record request in an attempt to get the IP address of the domain name. And all DNS servers answer to A record requests, even the old dingy ones not following the latest IETF standards. You, the user, sees this request — no response — retry dance as the browser taking a long time “Looking up domain.com….”, often seen as such on the browser status bar at the bottom left hand corner of the window. Only when the browser or application has received a valid IP address from a domain lookup can it contact the web server and start to download the HTML and display the page.

Timeouts

The delay between lookup retries is important to prevent overloading DNS servers, DNS resolvers (like Directory Services on your Mac) and simply makes sense. It’s similar to walking up to someone’s house and knocking on the door: Normally you wait a few moments for a response before trying again. If you don’t wait, you don’t know whether no one’s home, or whether they’re just taking a few seconds to respond. Continued knocking doesn’t help you. (And perhaps will earn you a stern look if not make you the target of a hissy fit).

Hammering a DNS server with domain lookups without pause is not very productive since the DNS server will simply drop (not answer) requests that it cannot handle within a timely fashion, based on its current load and worse, may get you blocked from the DNS server.

Next we’ll see how we can solve or workaround the issues discussed above that could be slowing down Leopard’s Internet speed.

Solutions

Direct DNS / Better DNS

Update 080606: Leopard 10.5.3 may have changed the order in which DNS Servers are used.

Update 080606: DNS servers entered on a DHCP configured setup are used in reverse order. I.e. the last server entered is the first to be used. If you’ve manually configured a network location, DNS servers are used in the order that you’ve entered them/see them.

New 080606: If you wish to save your current network setup and have the option of returning to it easily, follow the instructions for Creating a New Network Location. Otherwise, follow the instructions immediately below to quickly add new DNS servers.

Add DNS servers to Current Network Configuration

This is the quickest & easiest way to use new DNS servers, which is to simply add them to the DNS tab found in System Preferences => Network => Advanced => click on DNS tab.

Click on the + sign at the bottom left hand corner near IPv6 or IPv6 addresses and type in the addresses of the DNS servers you wish, in reverse priority order. (Recommended: OpenDNS servers at 208.67.220.220 and 208.67.222.222). I.e. the server that you want to use first, enter it last. Afterwards, click Ok. Then in the Network pane, click Apply to make your changes active. If you’re using an AirPort wireless connection, wait a few moments for the connection to be re-established

Creating a New Network Location

The advantage of creating a new network location is the ease of which you can move back and forth between different network setups. By creating and using a new network location, you can always revert your changes by simply selecting your original (Automatic) network location from the Location drop down list.

In Leopard, open System Preferences => Network => click the Advanced button (bottom right corner)


Click TCP/IP tab (top left).
Write down on a piece of paper (or in TextEdit) the IPv4 Address, Subnet Mask (255.255.255.0), Router, and Configure IPv6 setting. Click Cancel.

Find the Location drop down at top of the Network preferences pane. Click it and choose Edit Locations.


Highlight “Automatic” if not already
Click the Gear icon on the bottom center, choose Duplicate Location


Choose a name, I used “Home”.
Change the Location drop down box by clicking on “Automatic” and then switch it to “Home” (or the name you chose in the last step)
You’ll see the following:


Select Airport on the left (or Ethernet if you’re not using a wireless connection).
Click Advanced at the bottom right.
Click on the TCP/IP tab-button.
Change the Configure IPv4 drop down box to “Manually”.
Here’s where you use the values you saved in Step 2. Fill out IPv4 address, subnet mask, router, configure IPv6 settings. Do not click OK, instead click on DNS near the top.
Click the + button, bottom left hand corner. This creates a blue outline under DNS Servers on the left half of this window.

Enter in the DNS server of your choice. I recommend OpenDNS at 208.67.222.222. (Don’t include a period at the end). Add a second OpenDNS server by clicking again on the + button and entering 208.67.220.220. These DNS servers will automatically redirect you to the closest / best server for you, regardless of whether you’re in France (like me) or in North America. Click OK. You should be returned to the Network preferences pane and see something like the following:

At this point you’ve created a new Location called “Home”, having setup AirPort or Ethernet with the correct settings and “Services” (i.e. DNS), but none of these changes have been made active. Let’s make a backup of the configuration file that will be updated before you apply your changes. In Finder, click on the hard disk icon at the top left corner (usually Macintosh HD), then navigate to this directory: /Library/Preferences/SystemConfiguration and find this file: preferences.plist. Simply copy the file to your Documents folder or to a spot of your choice. If you have to rollback the applied changes, you can copy this file back to the above location. If you’re using Time Machine, this file should be backed up already. Now you know where this file is, so replacing it with a Time Machine version should be straightforward.

Before we make our changes effective, we’re going to check how DNS requests are handled now, before the changes, and after to make sure we’ve changed our Network Settings properly.

Leave the Network window open as is and open up a Terminal window. We’re going to be using the tcpdump program to listen to DNS traffic between your computer and your DNS server.

Type this command and hit Enter: sudo tcpdump -i en1 -s 128 port 53

(If you’re using Ethernet with a cable, use en0 instead of en1, which is the AirPort wireless interface).

Supply your password when asked to do so.

You should see something like the following:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en1, link-type EN10MB (Ethernet), capture size 96 bytes

tcpdump should now be running.

Open up another Terminal window and type the following command: curl http://www.csu.edu

This uses the curl program to read the web page located at www.csu.edu.

Going back to your tcpdump window you should see something similar to this:

00:31:37.026520 IP 192.168.1.132.56645 > WANADOO-D310.domain: 19279+ SRV? _http._tcp.www.csu.edu. (40)
00:31:37.029352 IP WANADOO-D310.domain > 192.168.1.132.56645: 19279* 0/0/0 (40)
00:31:37.029849 IP 192.168.1.132.56646 > WANADOO-D310.domain: 49549+ SRV? _http._tcp.www.csu.edu. (40)
00:31:37.032657 IP WANADOO-D310.domain > 192.168.1.132.56646: 49549* 0/0/0 (40)
00:31:37.034345 IP 192.168.1.132.56647 > WANADOO-D310.domain: 46004+ A? www.csu.edu. (29)
00:31:37.279043 IP WANADOO-D310.domain > 192.168.1.132.56647: 46004 1/0/0 A www.csu.edu (45)

Notice 192.168.1.132. That’s me, or really, my MacBook Pro’s AirPort wireless card. Then there’s a greater than sign (>) showing the direction of DNS traffic. WANADOO-D310 is my DNS server, which is actually the DSL modem/router, a.k.a. 192.168.1.1, which is passing domain name lookups to the real DNS servers at my Internet Service Provider (WANADOO, yeah I know goofy name). Remember the network settings we wrote down before starting all this? You’ll notice that the DNS server is 192.168.1.1.

OK, we’ve got a baseline of what our Mac is doing when looking up domain names, let’s apply our new network location “Home” that we created and see the difference.

Back on the Network preference pane, notice the Apply button on the bottom right hand corner. Once you apply your changes, your Mac will begin using the new Location you’ve created.

Take the plunge and click on Apply.

For AirPort wireless connections, you may have to click the Turn AirPort Off button, wait fifteen seconds, then click Turn AirPort On again in order for the new DNS settings to be used.

Going back to the Terminal window where we executed the curl command, and with our changes set, let’s execute another: curl http://www.unc.edu

Results will look like the following:

00:32:33.562589 IP 192.168.1.132.56663 > resolver1.opendns.com.domain: 39356+ SRV? _http._tcp.www.unc.edu. (40)
00:32:33.767237 IP resolver1.opendns.com.domain > 192.168.1.132.56663: 39356 NXDomain 0/0/0 (40)
00:32:33.767856 IP 192.168.1.132.56664 > resolver1.opendns.com.domain: 62833+ SRV? _http._tcp.www.unc.edu. (40)
00:32:33.809161 IP resolver1.opendns.com.domain > 192.168.1.132.56664: 62833 NXDomain 0/0/0 (40)
00:32:33.811130 IP 192.168.1.132.56665 > resolver1.opendns.com.domain: 45293+ A? www.unc.edu. (29)
00:32:33.853070 IP resolver1.opendns.com.domain > 192.168.1.132.56665: 45293 1/0/0 A www.unc.edu (45)

Notice what’s changed? WANADOO-D310.doman has changed to resolver1.opendns.com.domain. This is OpenDNS’ name for the DNS server we started using, 208.67.222.222, which we entered as our DNS for the “Home” location. Also, note how instead of just getting a 0/0/0 response, we’re getting NXDOMAIN 0/0/0? That’s at least the DNS server responding saying: that domain doesn’t exist (not exactly true, since the domain does exist, but it just doesn’t have an SRV record), rather than the DNS server sending back nothing, not even an error code. Also, notice how our Mac tried twice on asking for SRV records, and the DNS server responded twice, that no record exists for that domain, and then finally our Mac asks for an A record (A?) and gets one answer record back (1/0/0 A ww.unc.edu).

If you want to see a domain that actually has a proper SRV record, try this in the curl terminal window: curl http://s3.amazonaws.com

Results should be something like this:

09:36:56.440037 IP 192.168.1.132.61010 > resolver1.opendns.com.domain: 34536+ SRV? _http._tcp.s3.amazonaws.com. (45)
09:36:56.671881 IP resolver1.opendns.com.domain > 192.168.1.132.61010: 34536 2/0/0 CNAME s3-directional-w.amazonaws.com., (97)
09:36:56.673894 IP 192.168.1.132.61011 > resolver1.opendns.com.domain: 18143+ A? s3.amazonaws.com. (34)
09:36:56.715913 IP resolver1.opendns.com.domain > 192.168.1.132.61011: 18143 2/0/0 CNAME s3-1.amazonaws.com., A s3.amazonaws.com (69)
09:36:57.263186 IP 192.168.1.132.61012 > resolver1.opendns.com.domain: 32069+ PTR? 171.206.21.72.in-addr.arpa. (44)
09:36:57.306060 IP resolver1.opendns.com.domain > 192.168.1.132.61012: 32069 1/0/0 PTR s3.amazonaws.com. (74)

Here we’re getting 2 answer records (the “2” in 2/0/0) on the SRV requests, which are CNAME records, first being s3-directional-w.amazonaws.com, second being s3-1.amazonaws.com. CNAME records are “nickname” records, which point to true name, or A Record. Right after that our Mac asks for an A record on the first CNAME that was returned to us (s3-directional-w.amazonaws.com) to get back the actual IP address (72.21.207.246), which you can verify by using the dig program.

This fix alone has made my Internet connection much faster since my ISP’s DNS servers were sometimes under heavy load and slow to respond to DNS queries. Most of the time, I’d get name requests done in 200-400ms. Not noticeably slow. But, on occasion domain name lookups would timeout after 7 seconds, multiple times, resulting in up to 21 seconds of waiting for a single name lookup request to occur. This is excruciatingly long when I often open up multiple different websites one right after another when starting a browsing session. To make matters worse, many websites are getting into the practice of placing different parts of the web page on different domain names. Let’s take CNN.com for example. To load this single page of President Obama… oh, I mean senator Obama, waving to the crowd, tcpdump showed name lookups for the following domains:

  1. www.cnn.com
  2. edition.cnn.com
  3. i.cdn.turner.com
  4. i2.cdn.turner.com
  5. svcs.cnn.com
  6. ads.cnn.com
  7. i.cnn.net
  8. ad.doubleclick.net
  9. metrics.cnn.com
  10. m1.2mdn.net

One Page. Ten domains. Ten DNS lookups. Ouch. And I’m not including PTR/Reverse Lookups for each domain, making it really 20 DNS queries.

And does anyone wonder why problematic DNS performance in Leopard would slow web browsing to a crawl?

Disable IPv6 DNS Lookups

Firefox and Camino by default do DNS lookups using IPv6 addresses by default, reverting to IPv4 if that fails. This can be a problem when the router that we are using to connect to the Internet doesn’t work with IPv6 DNS requests properly, if at all.

To disable IPv6 DNS lookups in Firefox and Camino, type the following into the browser address bar:
about:config
If you see a large “Be Careful” warning, simply click on “I understand and I wish to continue”. Next, you will see a long list of Preference Name, Status, Type and Value columns. Above all that is a bar in which you can filter which preferences to view. In the Filter bar type: ipv6
You should see something like the following:

To change the value for this preference simply double-click the name “network.dns.disableIPv6”. The value you want is “true”, which means that IPv6 DNS requests are disabled. If this value is already “true”, don’t double-click this preference.

To make the preference change active, close the browser and Quit Firefox completely (Apple Key + Q), then restart Firefox. You may have to repeat this Quitting and Restarting to have the change take effect.

After making this change, Firefox (or Camino if that’s what you’re using) will use IPv4 only when performing DNS requests.

Update DNS Servers on Router

If you have access to your router’s administration web page, you may be able to set its DNS servers manually, avoiding the buggy DNS servers located at your ISP. Refer the manual that came with your router, or speak with your service provider about how to access the router’s administration page. Often this page can be accessed at http://192.168.1.1, so simply type that address into your browser’s address bar and press Enter. With any luck you’ll have access to the Administration login page. Many router administration sites don’t have passwords, don’t have usernames, or use very simple standard passwords such as “admin”, leaving it up to the owner to change it to something more secure. Visit the router manufacturer’s web site for more information about accessing the administration features of the router.

Keep in mind that updating the router’s DNS servers will not avoid problems you may be encountering with the router’s poor DNS Proxying/Forwarding support. If your router can’t handle IPv6 or SRV requests coming from your Mac, these DNS requests will stop here at the router and will not be forwarded onto the new DNS servers you’ve just specified, making this fix completely ineffective. DNS requests that your router cannot understand will likely be ignored or returned without answer results. DNS Proxy/Forwarding issues are discussed further in the next section.

Update Router Firmware

For those who need to continue making DNS requests through their router, rather than directly against DNS servers, due to VPN or tunneling requirements, your fix may lie in upgrading your router’s firmware. Routers are in effect “the” DNS server for the majority of home broadband Internet connections since it acts as the DNS Proxy, taking domain name lookup requests from your computer, passes them to the ISP’s DNS servers for resolution, receives the results, and finally passes the name lookup results back to your Mac, all transparently in the background. This is why your DNS server address is the same as your “Gateway” which is a fancy name for your router, since all traffic passes through this “gate” of sorts. Thus the Gateway address is often 192.168.1.1, which in turn is also the address of the DNS server for the “Automatic” network Location in Leopard.

Be aware that DNS Proxying is a common failure point in the domain name resolution chain. If the router is not compliant with the latest Internet Task Force standards, it may not know what to do with SRV requests (which Leopard now uses) and may simply ignore them, return empty results, or return NXDOMAIN (non-existent) errors. Again, a firmware update may bring your router up to the latest standards for DNS servers.

If the router is a DSL Cable modem/router, contact your ISP and ask whether there is updated firmware for the model of router you’re using. If you’re more of a do-it-yourself person you can attempt to find the manufacturer of the router/modem and find the latest firmware from their website, if available. Disclaimer: updating the firmware of your router with the wrong firmware, or not completing the firmware update due to power loss, will render your router useless. Do not attempt to update the firmware if you are not confident of what you’re doing.

New Information


Update 080606: As per a discussion on Macosxhints forums, Apple may have changed the order in which DNS servers are used. In the screenshot, the listed DNS servers are used in the order they are seen, under Leopard 10.5.2. This is true for a manually configured Network Location. In 10.5.3, users are seeing the opposite order, i.e. In a DCHP configured Network Location (automatically done by your DSL router and ISP), the DNS servers listed are used in reverse order. (Bottom server is used first, then moves up the list as needed). Thus adding a new Network Location to use a given DNS server would be unnecessary.

Update 080614: Airport Wireless Connection Drops – This is a common problem for Leopard users after upgrading to 10.5.2. This isn’t exactly a slow Internet problem, but rather, a “no Internet” problem. See this related post on wireless problems on Apple AirPort connections.

The Beginning

This is not the end, but rather, the beginning of an article that I hope will continue to grow in scope to cover more problems and offer more solutions to slow Internet problems in Leopard. Please leave a comment if you’re experiencing a problem not discussed here and we’ll get working on diagnosing the issue and searching for a cure.

If you’re having troubles implementing a fix listed above, leave a comment and I’ll try to expand on the topic or reword it so that it is understandable to you and to everyone else I’ve confused.

Keep in the Loop

– Ben

1.0.0.127.dnsbugtest. 1.0.0.127.in-addr-arpa

Update: Stuart kindly dropped me a note regarding Apple’s fix for this.  See the end of the article for his response.

This evening while randomly working on articles for Installing Cats, I was watching DNS requests from tcpdump running in a terminal window and noticed something quite odd: a DNS request every three seconds for a PTR record as follows:
PTR? 1.0.0.127.dnsbugtest.1.0.0.127.in-addr.arpa

The first question was: “What the heck is making this DNS query?” Second: “Why is it so persistent?” Thirdly: “What the hell is with that bizarre address?”

  1. mdnsResponder service is the origin of these repeated reverse-lookup requests.
  2. mdns (multicast dns) / Bonjour is trying to see if you’re using a buggy router/adsl/cable modem by sending out these ill-formed reverse-lookup requests and seeing what response it gets back. Unforunately, many routers crash and do not respond to this request. Thus, mdns will repeat the request over and over again incessantly.
  3. Explained in the last point. Address is meant to “diagnose” whether the router will respond to the “test” query in a poor or proper manner.

How did I stop mdns from continue to repeat its “test” dns request? I turned off AirPort for a few moments, waited for the mdnsResponder_Helper service to die off (in Activity Monitor), and then turned my WiFi card back on. mDns was kind enough to quit sending out these repeated queries.

The interesting thing is that I was using OpenDNS as my only DNS server. OpenDNS does handle mdns’ dnsbugtest queries fine, so I’m not sure what happened and why OpenDNS stopped responding to the requests. Perhaps they were just too fast? And instead of responding with a simple NXDOMAIN, it decided I was doing something malicious and deserved no response at all.

Either way, as a fallback plan, I’ve added my router’s dns’ ip into the DNS servers list within Network Preferences (under a new, custom Location where I’m specifying all settings, most of which are copied from the default Automatic location). If OpenDNS once again decides to not respond to the dnsbugtest queries, my router and ISP’s DNS servers should provide a second chance at mDns getting the response it’s looking for.

Read more about the technology of multicast DNS and Apple’s Bonjour service, created by the brilliant Stuart Cheshire.

From Stuart Cheshire on Sept. 16:

“[The repeated dnsbugtest requests] was a just bug, fixed and checked in back in March, and finally delivered to customers yesterday in 10.5.5.

The bug was that when mDNSResponder sent its DNS request to IP address X, and the response was sent back with source IP address Y, mDNSResponder would ignore the response as suspect and try again. Why people think it’s okay to reply with the wrong source IP address I don’t know, but they do, so now we accept those packets.”

This is good news for those who keep their Mac’s patched with the latest updates.  Thanks Stuart.

Debugging Mac OS X DNS – tcpdump

Need to debug a slow Internet connection in OS X? Or simply have a desire to watch incoming and outgoing DNS (domain name service) queries in Leopard? Then tcpdump is your friend.

Open a terminal window and use the following command:

sudo tcpdump -i en1 -s 128 port 53

-i Sets which interface to listen to. en1 for me is the AirPort wifi card. en0 would be the ethernet card.
-s sets the number of bytes to “sniff” or “snarf” per call that goes through this interface. 128 gives us a bit better coverage than the default 68 bytes. If you find that tcpdump requests are showing up simply as [|domain], that means that the request is longer than 68 bytes and is truncated. To prevent truncation, increase -s.
port 53 is simply the network port for DNS communcation

Learn more about tcpdump at developer.apple.com

Automountd trying to connect to Backups.backupdb

An explanation of why automountd is trying to find Backups.backupdb on the Internet…

I woke up this morning with a warning from Little Snitch outbound firewall that automountd wants to connect to Backups.backupdb on port 111.

Here’s what I’ve discovered since then.

automountd is a system service which mounts and unmounts network file systems (NFS) and lists contents of directories when requested (i.e. makes them accessible for use, like double clicking a .dmg file on your desktop, after that you can access the disk image).

Backups.backupdb is the Time Machine directory which contains your backups, usually on an external USB drive connected to your Mac.

When Time Machine is scheduled to do a backup, it tries to make a connection to Backups.backupdb to read its contents, which is automountd’s job to handle.
automountd pokes around, doesn’t find the directory within its network file system maps (when the external backup drive is not connected) and asks Open Directory/Directory Services “Yo, where’s Backups.backupdb?”

Directory Services stares at automountd blankly for a few moments and decides to check with DNS.

Directory Services asks the DNS server, “hey, you know where I can find Backups.backupdb”, to which your DNS server (located at your ISP or OpenDNS) will answer “Dood… that’s a nxdomain (non-existent domain) BUT, I’m gonna return you the address of a website with a bunch of search results and advertising”.

Here-in lies the rub: normally you should get a straight NXDOMAIN response from DNS meaning, there is no IP address for that domain. Instead, a lot of ISP’s (and OpenDNS) have capitalized on this and are returning an IP address to a web server dishing out search results and advertising, rather than a simple NXDOMAIN response. The result of which is applications such as Firefox or Safari, and services such as Time Machine , through automountd, are thinking that they’ve found the right address and therefore use it when handling requests.

The upside of this “service” is that instead of getting a “Website Not Found Error” in a browser, you get a list of possibly helpful search results of what you were really looking for.

The downside of course is that services such as Time Machine, have no idea that the address is not really the location of Backups.backupdb, but is in fact, a location of a website with search results and pay-per-click ads.

So, automountd attempts to read the contents of the directory called “Backups.backupdb” at the address returned by the DNS server, in my case “hit-nxdomain.opendns.com” located at 208.69.34.132, using a remote procedure call (rpc) on port 111. Of course, this remote procedure call will fail since 208.69.34.132 / hit-nxdomain.opendns.com is not a Network File System which accepts requests to mount drives, it’s a website meant for humans to see search results and click on ads.

Solutions to stop automountd from trying to connect to Backups.backupdb over the Internet?

  • Leave your USB/firewire Time Machine backup drive attached to your Mac so that automountd can find it without having to ask DNS.
  • Add a hosts file entry that maps “Backups.backupdb” to a local address, say 127.0.0.1. A rather crude, but possibly effective solution. I haven’t tried nor tested this solution, so I won’t elaborate on how that’s done.
  • Added 080602: If you’re using OpenDNS, they offer a way to exclude certain non-existent domains from being subject to the “search results” page response of hit-nxdomain.opendns.com. Thus, you can add the domain name of “Backups.backupdb” to the Typo Exceptions list and OpenDNS will return a straight NXDOMAIN response when queried for that domain. See the following screenshot for an example. Before adding frankie_valens to the Typo Exceptions list, an A record query to OpenDNS resulted in this response: 1/0/0 A hit-nxdomain.opendns.com (48) which is OpenDNS’ search results page address. After adding the fake frankie_valens domain and retrying the same query the answer is now NXDomain 0/0/0 (32) which is a proper non-existent domain response.

Although I know the first solution works for me, I’d like to call on some autofs experts for advice on how to handle this situation, with a more graceful solution.

Which is what I’m going to do right now and we’ll see what we can work out.

Updates and links to follow.

Update 2008-06-01

I think I’ve found just the right Apple autofs expert, Rajeev Karamchedu, that could help us figure out how to prevent automountd from connecting to spurious websites of search results due to a non-existent domain (NXDOMAIN) response from our DNS service provider, in this case, OpenDNS. Rajeev! Master of all things autofs… care to lend us some expertise on solutions to the above issue?