“ ‘We want to send a message that it’s cool to love math,’ says Mr. Whitney, and the museum succeeds brilliantly.”