It turns out that the well-known security patch for DNS last summer breaks WDS, but only if DNS and WDS are running on the same box. This is also valid for Windows Server 2008 and Windows Server 2008 R2 which has the patch built-in.
In brief, the issue with DNS and WDS on the same machine is that DNS will bind to 2500 ports in the ephemeral port range (49152-65535). Since the 2500 ports are randomly chosen, there’s a chance that those 2500 ports will completely cover WDS’s port range (which is 6400 to 65000 by default). Since the DNS ports are randomly chosen each time the DNS service starts, it’s possible that changing the WDS port range will fix the problem now, but next time the machine restarts they will conflict again.
For this reason, the proper fix is to make the WDS port range larger than the DNS port range. There’s also an option in Windows Server 2008 R2 to have WinSock dynamically allocate ports to WDS that aren’t in use, which is controlled by a registry key as indicated below. However, this fix may not work if you are using a third party firewall that requires a rule for WDS since the ports chosen will be random.
Workaround #1: Don't install DNS on the same server as WDS :)
Workaround #2: If you are running Windows Server 2003 or Windows Server 2008, on the WDS Server properties, in the network settings tab, change the range from say 50000 to 65000 (assuming you don't have any other apps using these ports). You can also set this from the command line using WDSUTIL:
wdsutil /set-Server /Transport /StartPort:50000 /EndPort:65000
wdsutil /set-Server /Transport /StartPort:50000 /EndPort:65000
If you are running Windows Server 2008 R2 you can also configure WDS to dynamically query WinSock for available ports instead of using a port range. This is done by setting the UdpPortPolicy value in the registry to 0. You will find it in the HKLM\System\CurrentControlSet\Services\WDSServer\Parameters key. You need to restart WDS after changing the value.
For more info, see: http://support.microsoft.com/kb/977512