Super Condensed: Install Orca, get someone else involved to help and go through the raw tables in sequence and then any custom action code.
All of this is obvious - if this has happened to you and you have sensitive information in the wild: all you can do is pull back the MSI (from download hopefully - it was even worse in the days of optical media), change any passwords or whatever else was revealed - and then make sure you don't experience it again. Now for the important part, how to avoid it for the future.
In addition to the information below on sensitive information, please also remember that some files you want to include in your setup may not be redistributable legally. Typical examples would be debugging tools from Microsoft or debugging tools from third party SDK toolkit. Please read documentation thoroughly and avoid use of such "hacky tools" in your custom actions.
Short Version
UPDATE: Let me jot down, before I forget, that you should eliminate the "downloaded file blocking flag" from all setup files (and generally read-only flags too).
All that is being suggested below is essentially to 1) scan your finalized MSI with Orca, 2) look over installed settings files as well as any template installation scripts delivered with your MSI. Further, 3) review your compiled custom action sources very well and maybe improve release build configuration practice (#ifdef _DEBUG for example, see below). 4) review your script custom actions by checking what is actually in your MSI (extract them). And crucially: to 5) get some help from other people for all manual testing - get some accomplices :-). Seriously: the setup is as important as the application - to make your solution succeed, it is your duty to get QA-personnel and other people involved testing - and also to tell them what and how to test.
I would avoid trying to automate such checking yourself. There is no substitute for real eyeballs on the data. Perhaps a community solution would help long term. It could become part of a validation suite? Semi-automatic help might work, but fully auto-magic: forget it. There are too many ways to use all the rope you've got to shoot yourself in the foot.
Sensitive data may be the wrong term, maybe "invalid content" is more appropriate. Problems can result by your application pointing to your test server rather than production server on launch. Unexpected message boxes may pop up from custom actions (sometimes revealing sensitive data), and similar release blunders beyond pure sensitive data being exposed.
QA - Bug Quest
Checking for sensitive data included by accident obviously ties into general QA of your package. It should be done simultaneously with general testing. QA people are so busy with application testing that you really have to push this deployment testing, and make a test plan. Nothing fancy, but do test all installation modes (install, reinstall, repair, self-repair, upgrade, patching, uninstall, administrative install, resume / suspended install (setup reboot issues) and you should also do publishing and advertisement - if you have the equipment and network to test this) and test all custom action functionality (thoroughly). Realistically and minimally you must test install, reinstall, uninstall and upgrade, but please test all modes.
And if you are localizing, test in all core regions in all editions. Also run English in German locations, and vice versa just for smoke testing. In fact, test English in all regions - obvious I guess. Custom actions could easily fail on localized machines triggered by a random state on that machine (CA trying to access a hard coded English path for example and an exception results), or show some forgotten message box in English inside your exception handler code or similar that was never triggered on an English box. Bad, oh yes, and I have seen it often enough that it should be chalked up as an issue.
And I guess it should be mentioned the words of an experienced developer: "...don't hit too many people with testing until every bug found is a genuine surprise". And also - his funnier advice - for pre-releases leave in a couple of known bugs and tell the QA guys that there are such and such number of bugs to find - just for some motivation to focus the mind :-). P.S: I like to refer to this experienced developer as "The Elder Grasshopper", or as he is more commonly known "Veggie Boy". Confucius says: "Never trust a man who can be bribed with (organic) carrots!"
A large digression, back to the real topic: erroneous inclusion of sensitive data.
Checking MSI Files
I keep it simple when it comes to checking my MSI files for sensitive information.
- First a quick once-over of the source files (WiX, Installshield, Advanced Installer, or whichever tool you use) for hard coded dev-box sins.
- Then significant attention checking the finished release-candidate MSI file itself. The real McCoy. All tables, and also extraction of some embedded content for verification (scripts, custom action dlls, etc...).
- Actual installations in all installation modes, as described above. Sensitive content can be revealed, but so can a lot of other issues - for example unexpected message boxes popping up - sometimes with sensitive debugging info (custom action test focus).
How to check? Some scripted checks could be useful, but from experience I don't get fancy about it. I prefer a second pair of eyes over fancy script checking if I am honest. Just my two cents from real release work.
- Install Orca
- Orca is as down to the wire as you get with MSI - other tools often show bogus, proprietary tables. Orca is a straight up view of the file content.
- Search for "Orca" here to find the installer quickly if you have Visual Studio installed - or tell someone with Visual studio installed to send you the installer MSI.
- You can also try "Super Orca" - but Orca is recommended.
- Now just open your release-candidate MSI with Orca - and skim through the tables.
- And just to say the obvious:
- Enforce any changes in the real source, don't hotfix the finished MSI.
- If you ask me, no in-situ hotfixing at all - you need a fix at the source and a full MSI file rebuild in my opinion. Then you label your source code (if you got proper, old-fashioned source control with delicious revisions and labels).
- Most vulnerable tables are probably:
Registry, Property, IniFile- but there could be something in several other locations.
- If you actually use the MSI GUI:
tables relating to GUI are also vulnerable.
- Many people just use the standard GUI without modifications. This should eliminate most risks.
- If you have a custom GUI, then there are quite a few tables involved in the MSI GUI declaration. I would eyeball them all.
- Perhaps particular focus on:
ListBox, ComboBox, UIText, Dialog
- Obviously with extra focus on your own, custom dialogs - if any.
- Third party tools feature vulnerable custom tables for things such as XML file updates. Eyeball these as well.
- Anything that looks like XMLFile, SQLUpdates,etc...
- There are more and more such custom tables from different vendors. They relate to all kinds of things now, not just config files (firewall rules, SQL scripts, etc...)
- Check any included scripts.
- Check in source control, but also...
CustomAction table or Binary table - the latter requiring you to stream out any scripts - or check them in their source locations).
- Check any settings files the application installs (via MSI's File table).
- INI files with hard coded settings could be installed via the File table and hence not have their values visible in Orca (as opposed to the INIFile table which shows all fields of an INI to write).
- The difference here is essentially whether the file is handled as a file or as a set of group-value pairs to write to an INI for example. The latter approach is the "correct" one.
- Note that some INI files may need to be installed as files, if they have non-standard formatting (extra fields and various strangeness contrary to normal key-value pair formatting), or even more common: INI files may have huge comments sections with help information (often for developer tools) that you want to preserve - and you can't via the INI file table. Then the option is to install it as a file.
- Other settings files such as XML files could be installed the same way. Very often in fact.
- As stated above third party tools often feature support for writing updates from a custom table viewable in Orca.
- There can be many such different tables (Encrypted fields? What is in there?)
- Files included like this are generally maintained by developers, but it is still a release responsibility to check. Make an administrative installation (further links in linked answer) of your MSI and check the extracted settings files in the created network image.
msiexec.exe /a "Your.msi", or setup.exe /a (Installshield), or setup.exe /extract (Advanced Installer). Some setup.exe info.
- Check supporting batch installation scripts, Powershell scripts, or other forms of scripts delivered with your setup - with the intent to do the actual installation of the software.
- Sometimes you see ready-made scripts delivered with some setups to help automate deployment, often some form of hard coded information can sneak in here (UNC paths, even IP-addresses, or other kinds of test data").
- These scripts, sometimes delivered as a separate download, may be an afterthought that ends up being neglected for QA in my experience.
- Use these scripts actively during QA and testing if available, or better yet: document large scale deployment in a single page PDF instead (much more generic and less error prone).
- Warning: compiled custom actions can still contain sensitive stuff even if n
