In the Middle Ages, stages were "raked" -- this means that the stage was slanted at an angle toward the audience to improve visibility. An actor walking away from the audience was literally walking up an incline. Hence, the stage direction "up" stage. Even though stages are no longer raked, we still say an actor is moving "upstage" when he walks away from the audience toward the back of the stage. When he crosses towards the audience, he moves downstage.
When the director instructs the actor to move left, or "stage left" she means for the actor to move to the actor's own left as he is facing the audience. "Stage right" would be the actor's right when facing the audience. There's another way to indicate right and left of the stage, however. "House right" and "house left" are meant to designate the right and left side from the audience members' (the "house's") perspective.
The middle of the stage is called "center stage." The actor in a scene who is blocked to be center stage will usually be the focus of attention.
Upstage and downstage, stage left and stage right, would be all the stage directions required if actors moved in perfect vertical or horizontal lines as if on a grid. Of course staging a play requires greater range of movement for the characters. So if the director requires the actor to move to a telephone to the right of the actor and near to the audience, the director would say "cross downstage right."
Actors have a convenient shorthand for writing down a director's blocking instructions. If the director says cross stage right, it's abbrevieated "XSR" or "XR." If the actor is asked to cross downstage right, it's "XDSR."