I'm binge watching and I'm on the bank robber episode and notice Barbara Pepper looking sickly, like she's lost a ton of weight and her face looks very weak. I'm wondering is this is close to when she leaves.

Yes, that was one of her last episodes. She was out for most of the third season due to her sickness.

They brought her back for a handful of episodes hoping she could continue the role but sadly her speech was slow and lethargic. She just wasn't Mrs. Ziffel anymore. She passed away about a year later. :(

Pepper lost her husband in 1949, she was left with two small children to raise and unfortunately she turned to alcohol. She was under consideration for the Ethel Mertz role but with heavy drinker Bill Frawley already on board Ball and Arnaz didn't want to chance it. Pepper was born in 1915, she was only 50 when Green Acres started (27 years younger than Hank Peterson), 9 years younger than Eddie Albert and only 4 years older than Eva but the drinking really took a toll on her and her looks. I recently wrote a piece on the interesting connection between the two Doris Ziffel's:

Barbara Pepper, Doris Ziffel I, and Fran Ryan Doris Ziffel II have a rather strange thing in common. In their personal lives the two women were married a total of three times, Pepper once in 1943 to a man named Craig Reynolds and Ryan twice to a Walter Wayne and later to a Howard Schafer. Reynolds died in a motorcycle accident in 1949, Wayne and Ryan wed in 1949 and two years later he was killed in a private plane crash (he was the pilot). In January of 1953 Ryan wed again to Howard Schafer, 5 months later he died in a private plane crash (he was the pilot).