non sono un esperto, ma ho fatto un paio di prove ed è la documentclass amsbook che produce questo risultato. O almeno con book non lo fa. Forse c’è qualche parametro di amsbook da impostare
EDIT: dal testo della documentazione di amsbook (http://www.ctan.org/tex-archive/macros/latex/required/amslatex/amscls/doc/amsclass.pdf)
In amsart.sty \@startsection, \@sect, and a couple of other things are rede- fined to fix a few hard-coded things that would interfere with the desired style. The most noteworthy difference is that all section headings will go into the table of contents (governed by secnumdepth as usual), EVEN WHEN THE * FORM IS USED. The only section heading not listed in the table of contents is the heading for the toc itself.
A second major departure from standard LATEX is that when a short form of a section title is given, it is used only for the running heads; the table of contents still gets the full version of the title. This is correct for AMS editorial practice. However if one wants to get a line break into the table of contents it means that the standard LATEX method cannot be used. See the \except and \for commands.