ABRIDGED VERSION (To see the unabridged version, see this posting.)
Problem scenario
A Puppet manifest is not working, but there are no obvious error messages. When running the puppet agent command, you use the -d flag for debugging. In your manifest, you use logoutput => true stanza. But still, you cannot figure out why your manifest is not working.
You tried this command: puppet parser validate nameOfManifest.pp
The above command had no output. On the Puppet Master server, you looked in /var/log/ to find clues. When the Puppet Agent runs and communicates with the Puppet Master server, on the back end of the Puppet Master, the activity should be somewhat logged to /var/log/puppet/puppetserver/puppetserver.log. You see no error messages anywhere that correspond to a puppet agent service running. You want to fix the manifest that is not producing the desired changes and/or effects that you want. Debugging can be difficult when Puppet appears to work perfectly. It seems hard to solve a manifest problem with no runtime errors and no errors in the logs.
How do you debug or otherwise troubleshoot a silently failing manifest that will not apply correctly?
Five Possible Solutions
#1. Are you using the ENC (External Node Classifier)? If you forget or fail to classify the Puppet agent and Puppet manifest correctly, you can observe a Puppet manifest not work with no error messages. For more details on how to remedy this issue, see "Possible solutions" #5 and #7 of this posting
#2. Are the modules directories or files subdirectories associated with the manifest not in their correct location? Or have such directories been given permissions that do not allow for files to be read from their locations? Puppet Master migrations or other systems administration operations on the Puppet Master server could cause dependent files to not be retrieved properly causing manifests to fail without warning or notice.
#3. Look at the Puppet manifest closely. Is the logic correct? Imperfect logic can be a big cause of errorless Puppet manifests not doing what is intended. For example, a node stanza could signify an incorrect server from what you expected. A class declaration could have prematurely terminating braces that could exclude an exec statement. This scenario (with the exec statement never running on a Puppet agent server) could be free from errors, even if you use the "puppet parser validate" feature. Be sure you did not forget to uncomment out a line in the manifest.
#4. Temporarily re-write a copy of the manifest purely for the sake of debugging. To diagnose what happens with the errorless run of a manifest, you may want to note the time at different stages as well as the user security context throughout the progression/failure of the manifest.
To determine the exact problem you could run exec commands that are specific to the OS receiving the configuration. For example, in Linux you could use date, whoami, or env. In Windows you could use date or whoami. These statements could be redirected (with the ">" symbol" to a file for viewing afterward.
Another way to learn more about a complex yet quietly failing manifest is to use Puppet's two DSL reserved words notify or notice. These can help you obtain more detailed messages without using regular Bash or PowerShell commands. To learn more about adding debugging details, see this external link. While the requires and require keywords can be confused, they can be helpful for ensuring the explicit execution order of a manifest that is failing in an elusive way.
#5. Make sure at least one Puppet manifest will work properly on a given Puppet agent server. Incorrectly configured Puppet master servers can cause an errorless run of a Puppet agent to actually not work. Restart the Puppet Master service. If that does not work, try rebooting the servers involved. If rebooting the servers involved does not work and you have time to do a great deal more, try to redeploy Puppet Master and Puppet Agent. When you try to apply the manifest again, it may work or there may be an error message.