Beside the father, don't think James ever mentioned or talked about any of his family member like mother, brother or sister. Most of relative and family visits seem to be from Florida side of the family. I think the only time James's family was mentioned was when his childhood friend Ernie (the gambler) showed up and they talked about the old times.

James Evans' family background is rarely mentioned outside of his father and childhood friend. Here is a breakdown of what was revealed:

Henry Evans (Father): James' long-lost father, Henry, is the primary family member from his side depicted in the series. He first appeared in the Season 3 episode "The Family Tree," where it was revealed James had lied for years, telling Florida and the children his father was dead because he had abandoned the family 35 years prior.

Other Immediate Family (Mother/Siblings): James never explicitly talks about a mother, brother, or sister on-screen. During Henry’s visit, Henry mentions he walked out on his "family and children," implying James may have had siblings, but they are never named or seen. James' lack of family contact is often contrasted with Florida’s side, which includes her brother Wilbert and various cousins like Cleatus.

Childhood Friend Ernie: As you noted, his boyhood friend Ernie Harris appears in the Season 2 episode "The Houseguest". Their interactions provided rare insight into James' upbringing in Mississippi, including his childhood nickname "Patches" because of his extreme poverty.

Wider Evans Family: In later seasons (after James’ death), Henry Evans returns with his common-law wife, Lena, whom the family eventually embraces as part of the Evans clan.