Mark Wahlberg expecting a girl

It's an equal-opportunity family for Mark Wahlberg, who's expecting his fourth child with wife Rhea Durham.

'I'm having another girl,' Wahlberg told PEOPLE at GQ's The Gentleman's Ball at New York's Edison Ballroom on Wednesday. 'So I'll have two girls and two boys.'

But four is probably enough for the Boston native, 38.

'I'm the youngest of nine, and my dad has a few others. No, four is enough,' he said.

'We were all raised with a lot of love. But we didn't get the kind of attention I think that we all needed. My parents had to provide to put food on the table. My mission in life is to raise my kids right. With all the success I've had in the world, if I fail at that, my life means nothing.'

Still, Wahlberg says he'll leave family planning up to Durham, a former model.

'My wife is allowed to pick the magic number ... as many as she wants to have,' he said. 'What I want to do is I want to raise my kids right. I want to give them the world, but I also want them to appreciate it.'

As it is, brother Jim Wahlberg said he suspects Mark's clan will continue to grow. 'He's good to go! He ain't stoppin',' said the sibling. 'He's got beautiful kids.'

Wahlberg and Durham are parents to Ella Rae, 6, Michael, 3 ½, and Brendan Joseph, 13 months.