It took me a while, but not that long

I don't remember ever not realizing this

I thought that was the whole point?
He was showing up to case the place? He even mentions it later in the movie