were not part of the cast in season 3?

Although Spence appeared in like six or seven episodes (maybe more) Richie only appeared in one episode and then never returned to the show. I can understand Richie not being a castmember, but I don't get why Patton Oswalt wasn't an official cast member then.

Anybody have any info?