That's Landa's trophy for his work: holding people in a state of uncertainty that he's figured them out, while they can't escape the situation because of rules of decorum. He does it to the the Frenchman. He does it to Shoshanah with the Strudel and saying there's something else I wanted to ask you. He does it to the whole crew laughing at the mountain climbing and asking what mountain in Paris she claimed and then switching subjects saying he's just teasing and then going through Italian accents. Every time he knows the truth, but ratchets up the tension by making it vague he knows. That's why he reacts differently with von Hammersmark at the end and strangles her in rage. She says "oops" and won't continue the charade dialogue.
I read on reddit someone's theory that Hellstrom DID know the whole time. Bridget says the 3 finger thing gave them away, but he knew earlier. (the reddit post gives some decent evidence of it, including what was written on the card being an obscure German actress in movies that had been embargoed during the war which would expose Hicock's non-German nature, the fact that he immediately tries to get them in a drinking came to have an advantage in the fight to come by ordering scotch for everyone else but immediately saying he doesn't drink it, his insistence on sitting next to Hicock, Hugo's presence) The reaction to the 3 finger thing wasn't that he now knows, but because it puts an end to the "monkeyshines" because now he cannot feign not knowing -- even the bartender would notice he was not behaving German-ly. So it forces the issue.