“Can a hands-on math museum generate the same kind of excitement? Glen Whitney, a former algorithm manager at hedge fund giant Renaissance Technologies, thinks it can.”