The French president has vowed to push on despite polls showing him losing support among working-class voters.