Just watched the TSN footage again. You can clearly see Adams point at the ref as he is walking down the line of scrimmage towards the sideline. The ref pointed back confirming he was on the line. After this though Adams did appear to change his lead foot and in doing so did end up taking a half step back from the line. As soon as the QB sneak was whistled dead the sideline ref ran right to the pile where he was called over by the head referee. After a brief conversation the sideline ref begins to throw his flag just as the TSN camera angle changes.
So it was indeed the sideline ref on Adams' side that threw the flag but it seems like it was the head ref that told him to do it and that was why it was so late.