Answer by wipet for Check for group as argument (expandable)

I don't understand what you exactly want, but maybe the following code can help you. (Edit: I added new version of the code which is full expandable)

\def\MyIfGroup#1{\expandafter\MyIfGroupA\romannumeral-`\.#1.{}\end}\def\MyIfGroupA#1#{\ifx\end#1\end\expandafter\MyIfGroupB   \else\expandafter\MyIfGroupE\fi}\def\MyIfGroupB#1#2#3\end{\ifx\end#3\end\expandafter\MyIfGroupT   \else\expandafter\MyIfGroupF\fi}\def\MyIfGroupE#1\end{\MyIfGroupF}\def\MyIfGroupF#1#2{#2}\def\MyIfGroupT#1#2{#1}\tt\MyIfGroup{{}}{true}{false}   -> true\MyIfGroup{ {}}{true}{false}  -> true\MyIfGroup{{} }{true}{false}  -> true\MyIfGroup{ {} }{true}{false} -> true\MyIfGroup{ }{true}{false}    -> false\MyIfGroup{a}{true}{false}    -> false\MyIfGroup{{}a}{true}{false}  -> false\MyIfGroup{ {a} }{true}{false} -> true\MyIfGroup{ {a}{} }{true}{false} -> false\MyIfGroup{{}{}}{true}{false} -> false    \bye

Edit: How it works: The potential first space is removed by \romannumeral-``\. The rest is read by \MyIfGroupA macro which reads a parameter #1 to the first occurrence of { (without it). If there is no such brace then the original parameter followed by dot is read because we added .{}\end to the input stream. This means that the #1 of \MyIfGroupA is not empty, so \MyIfGroupE is processed (this macro removes the rest from input stream and goes to the false branch. If thee exist { but something is before it then the result is the same. Else #1 of \MyIfGroupA is empty. Then the \MyIfGroupB is processed. It reads the (first) braced parameter to #1 then reads one token to #2 and the rest of input stream to \end in #3. The reading of #2 ignores potentially spaces and if there is nothing else the . is read to #2 and nothing is in #3 because outer braces are removed when #3 is read. And this is exactly the situation where we need to go to the true branch. Else there is something else after first braced parameter and we go to the false branch.