Security issues with Web Publishing
General FileMaker plugin security
You should exercise care when using any FileMaker plugin from within the Web Publishing Engine. This is allowing remote users to execute code on the server machine, which can potentially be used maliciously if you do not guard against that possibility. For example, let's say that you have a file manipulation plugin installed that can read the contents of a file and display it in a FileMaker field. If you create a web publishing interface that allows the user to enter any path for the file to read, they could read any file on the server's hard drive and view the result of that in the web published database.
This does not mean that you should never use plugins with the web publishing - just make sure that you access them through scripts, and that the inputs to these scripts cannot be maliciously manipulated by users accessing your site.
Security Dialog when accessing WPE pages or Zulu
Sometimes there can be permissions issues or misconfigured security settings if FileMaker Server has been deployed multiple times. Your web server has its own settings which may not be removed when FMS is redeployed or uninstalled. FileMaker has posted instructions at http://help.filemaker.com/app/answers/detail/a_id/6454/kw/IIS%20Authentication/session/L3RpbWUvMTMwMzgyNzY0MC9zaWQvQjcxTGxzc2s%3D which can help you clear settings from a previous FileMaker Server deployment which can help resolve these issues.
On IIS you should also make sure that "Integrated Windows authentication" is not checked in Default Site -> Properties -> Directory Security -> Edit.