Originally, it consisted of two solidly built houses, possibly dating from the 16th century, which the Knight of Alcántara, Juan de Ortega Lara y Angulo, used to construct, from 1702 onwards, a single, larger and more imposing building.

After passing through various owners, including the Castro family (from whom it takes its name), it housed the Segovia Recruitment Office and Mobilisation Zone, whilst also serving as premises for a private academy for officers stationed in Segovia.

The building is home to the Segovia Provincial Historical Archive, whose collections are also classified as Assets of Cultural Interest.