I haven't spent a lot of time looking at this but I think the problem may be related to the cross-thread relationship created between the windows. If you change the code to use the desktop window as the owner of the color dialog (not recommended as a general practice) the hang seems to be resolved.
void CBounceWnd::OnCustomColor()
{
CColorDialog dlgColor(m_clrBall, 0, GetDesktopWindow());
if (dlgColor.DoModal() == IDOK)
{
m_clrBall = dlgColor.GetColor();
m_nIDColor = IDM_CUSTOM;
MakeNewBall();
Invalidate();
}
}