Brett Brown jokes that he hasn’t gone out as much as he used to because his team is losing the series 3-0