It was because of the doll with buttons. Wybie gives the doll to Coraline, and through it, the Other mother knows everything that happened to her. How did she knew that Coraline was there in the first place? We don´t really know, but maybe Wybie told his grandma about a blue haired girl wearing a yellow jumper on the rosa, -de-rosa Palace and the Other mother heard about it through another doll in Wybie´s house and made another doll that looked like Coraline. Nevertheless it´s just a theory
Any children who mover into the rosa, -de-rosa Palace are instantly found out about por Beldam. That's why Wybie isn't allowed inside. His Grandmother doesn't want to loose him too
