This is from Wikipedia, for everyone wanting to know why they left:

"In the show's final season, on CBS, George and Dorothy depart for the Middle East in conjunction with George's work (DeFore and Blake were dropped from the cast) while Harold and Hazel move in with George's younger brother, Steve (Ray Fulmer), a real estate agent, Steve's wife Barbara (Lynn Borden), and their daughter Susie (Julia Benjamin). Hazel provides housekeeping services for her revamped family. The new Baxters reflected a desire for younger demographics. (CBS said that Ms. Blake was not available after NBC's cancellation, although Don DeFore noted that he found out about the change while reading the newspaper.) The most notable new arrival was the teenage actress who played Millie Ballard, Steve Baxter's receptionist. Ann Jillian went on to star in her own series, It's a Living, and numerous television movies."