As the message is suggesting, the DNS server has received an invalid domain name. By invalid it means that it contains invalid characters. MS DNS only supports 0-9, a-z, A-Z, . (dot), and - (hyphen) as part of a domain name. Some other DNS servers may not strictly enforce RFC 952 (DOD INTERNET HOST TABLE SPECIFICATION) so invalid names reach the DNS server and the 5504 message is recorded. Usually this happens when Forwarders are used by the DNS server. Microsoft suggested to one user to turn off the forwarder in order to eliminate these messages. There used to be a Knowledge Base article "Q246797 - DNS EVENT IDS 5504, 9999, AND 5000 FILL EVENT VIEWER" but is no longer available.
Another condition that may generated these messages is when the Internet connection is saturated or not working properly (losing packets). Because of the poor Internet connection, the DNS may receive incomplete or corrupted data and 5504 is generated.
Article Q154554 (not available anymore) stated that Windows NT 4.0 DNS server does not enforce the name restrictions, and will do WINS lookup for host names containing invalid characters. It is not recommended to use invalid host names. Other DNS server may have problems with names containing invalid characters."
