The English language, especially where it concerns American expressions, always finds a way to throw me off just a little.
If a male refers to someone as "my old man," he is talking about his father.
If a woman refers to someone as "my old man," she is talking about her boyfriend or husband.
If a man refers to someone as "my old lady," he means his girlfriend or wife.
A woman never refers to anyone as "my old lady," or at least no straight women do that I know of. I wonder if a lesbian says it, does she mean her girlfriend or her mother?