Andrew Burton / Reuters

Christmas decorations light up the outside of Macy's department store in Herald Square, Higher-end retailers, such as Macy's, did well this holiday season, as did the deep discounters. It was the middle that suffered.

Luxury brands and stores known for rock-bottom prices don't usually have much in common, but these polar opposites both enjoyed what many mass-market retailers missed during the holidays: positive sales figures.

Analysts say consumers are still shopping with recessionary mindsets and hunting for the cheapest prices on basic goods. While they'll pay more for unique products they deem splurge-worthy, three years of cautious spending have made them choosier about what they buy. This year, shoppers told once-sterling brand names like Sears and Gap, "We're just not that into you."

Earnings at upscale department stores including Nordstrom, Macy's and Saks all beat analysts' estimates. Affluent consumers, who pulled back when the stock market took a dive, are feeling more confident today, said Alison Jatlow Levy, a retail strategist at consulting firm Kurt Salmon. Luxury retailers also sharply pared their inventories in response, and they've retained those lower levels. Customers today are more willing to pay full price because goods are likely to sell out before they have to be discounted. 

On the other side of the spectrum, the parent company of discount chains T.J. Maxx and Marshall's beat estimates, as did Ross Stores. Warehouse club Costco Wholesale had a 7 percent increase — slightly below expectations but still healthy. (Wal-Mart, retail's 800-pound gorilla, doesn't release monthly sales figures.) Even dollar stores are looking forward to 2012. Dollar General announced this week it plans to open 625 new stores this year and relocate or renovate another 550. 

The middle of the road didn't fare as well. These bigger companies can't rely on scarcity to keep prices high the way luxury brands can. When the recession froze spending, they slashed prices to compete with discount chains as their customers traded down. The economy has improved somewhat, but shoppers still have a siege mentality. 

"While the economy shows mild signs of recovery, consumers haven't changed their behavior," Levy said. "The mindset of the middle market customer has markedly shifted" to focus more on value, she said. 

These stores have a challenge getting shoppers to trade back up, said Samantha Panella, vice president of softline retail at Raymond James Financial. "People will shop where they can get the lowest prices. In the middle, you really need to differentiate to get the customer through the door," she said.

 J.C. Penney and Kohl's both slugged it out with deep discounts, but both only ended the season bruised. Each lowered its earnings outlook after missing expectations. Sears Holdings didn't even wait until the new year to announce it would close up to 100 Sears and Kmart stores due to dismal holiday sales, in spite of huge markdowns. 

Deep discounts are a vicious cycle for stores, Levy said. They may drive sales, but it comes at the expense of profits. It also teaches shoppers that if they just hold out, they won't have to pay full price for anything. 

Gap Inc., once the standard-bearer for mass-market apparel, capped off a bad year with a 4 percent drop in same-store sales. "Gap stands out as a laggard for most of this year," Panella said. Shoppers just aren't excited by what's on the racks, and the "basics" — jeans, t-shirts, chinos — aren't distinctive enough in consumers' minds to merit the price premium. Gap's ubiquity, once an asset, is now a hindrance. "They have a lot of stores, and that's a challenge," she said. 

Mid-tier retailers also are having a tough time because vying for a big chunk of the population means stocking clothes everybody wants to wear, Levy said. "You want to be everything to everybody. What happens when you're in that kind of position is overassorting — a lot of styles, a lot of inventory." The invariable duds have to be deeply discounted before consumers will bite, which compounds the problem. "It's almost like the retailers were nervous to take a bet on what the big styles would be," she said.