First of all: if you use ASP.NET 3.5 and controls from AJAX Control Tookit then you must use the ToolkitScriptManager, rather than the ASP.NET ScriptManager. This limitation according to the fact that toolkit script manager adds updated Ajax scripts and without that most of the controls from Ajax Control toolkit library will not work.
If you using ASP.NET 4.0 (4.5) than you have choice to use ScriptManager or ToolkitScriptManager.
Basically the main feature of the ToolkitScriptManager is that it can combine js resource added to page using ScriptReference collection. These js files should be embedded to assembly and for this Assembly ScriptCombine assembly attribute should be added. The main problem here is that you can't control how scripts are combined and after you will try to combine your own scripts you can have a lot of problems (I had experience using this feature and as a result we rejected combining of our scripts using this approach).
The main point here is that currently standard script manager has ability to combine scripts using composite script collection. This feature is more flexible and it is more easy to control how scripts are combined on your pages. The more information can be found using the following link: http://weblogs.asp.net/infinitiesloop/archive/2009/11/23/asp-net-4-0-scriptmanager-improvements.aspx.
Also, note that as far as I remember if you using ToolkitScriptManager then you will not be able to use CompositeScript feature, even to take in account that the ToolkitScriptManager derives from ScriptManager.
So, in ASP.NET 4.0 the difference is not so big, and it is better to avoid using ToolkitScriptManager. The main reason is that using standard CompositeScript feature you have more control on how scripts are combined and can optimized rendering of the page.
For ASP.NET 3.5 you don't have choice if you want to use controls from Ajax Control Toolkit library.
Edit
With the latest changes in ajax control toolkit library they proceed to update MS Ajax scripts and that's why some of the controls can't work without adding ToolkitScriptManager.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…