Outgoing Australian Public Service Commissioner shed a few tears as he thanked his colleagues and senators in a public statement.