Make Sims stay in their houses when loading a lot

Every time I start a new save file, inactive sims spawn near the loaded lot to act as walk-bys, making them spawn outside their lot when I go in Manage Worlds. It's pretty annoying when I want to create a custom starter save file, is there any way or mods to make them stay on their home lot?