Is it just me or do you people out these agree that the first 2 seasons were just not up to what the series became starting with season 3 in the new house?

Nope
many excellent episodes in 1 & 2

To me Seasons 2 and 3 were the peak of the series. In Season 1 things were still warming up, yet there were many classic and iconic episodes in that season. In Season 4 Beaver is a shade less endearing, but here again there were a good many classic episodes. My general feeling is that in Seasons 4 thru' 6 there was a mix of good and weak elements, whereas Seasons 2 and 3 were pure gold, the best the show ever got.

I would add that for me the question is not limited to which house they were in, although I do agree that the second house was nicer.

To me Seasons 2 and 3 were the peak of the series. In Season 1 things were still warming up, yet there were many classic and iconic episodes in that season. In Season 4 Beaver is a shade less endearing, but here again there were a good many classic episodes. My general feeling is that in Seasons 4 thru' 6 there was a mix of good and weak elements, whereas Seasons 2 and 3 were pure gold, the best the show ever got.

I agree with this.

There is a sort-of dividing line between seasons 3 and 4. Season 3 is very strong, and Beaver is still young. Although he doesn't still have some of those "Beaverisms" like 'tricity and other abbreviations.

Season 4 sees the loss of a very strong actor in the show, Rusty Stevens. And Beaver's cuteness and naiveté give way to more Wally-centered scripts. Steve Talbot is elevated to Beaver's foil. Rusty somehow made Larry a likable character, even though he caused Beaver no end of trouble. Steve's Gilbert--there's nothing likable about him at all.

To me the order of "best to OK" seasons is 2, 3 (tie), 1, 6, and a tie with 4 and 5. 6 gets kudos for its updated music score.

I do agree that the newer sets starting in season 3 give a more modern feel to the show. But, wow, season 2 had a lot of strong scripts.