He went because the market used to a regular supermarket and it was the only place to buy groceries anywhere near where they lived. There were no Trader Joes, Whole Foods, Walmarts, Acme's, Superfreshes ... you get the idea.
The market back then was called the Penn Fruit. In the past, parents didn't live in constant fear of having their children snatched away and tortured to death by perverts if they didn't hover over them every second of the day, and as a little boy the loser would roam free in the market, small by today's standards, until his mother retrieved him and drove home.
Once, when he was five or so, he saw his mother perusing the produce and went up to her to signal that he was there. He tugged on her skirt and she turned and looked at him but ... it wasn't her!
Terrifying. The skirt and other clothes were close if not the same. Not that the Loser went berserk with fear or anything and the incident lasted for only a few seconds. But they're memorable seconds even now, and the event happened in 1962, maybe early 1963.
It was right around where these people are standing that the Loser had an inkling of what it would be like to lose his mother.