Hard to say. Both the folder and the file are completely dependent upon which SP theme you are using and whether you have created a custom or child theme. Only you will know that. And dependent upon this it might be a .php or a ,css file. Again we do not know.
When you find it though – there should be a rule for .ui-dialog already in place for you to edit..