In the below example code I have created a CustomDialog Fragment. Here, first I have created custom_dialog.xml layout , On this xml layout I have added EditText and  Button. In next step, in activity_main.xml layout I have added a Button and TextView. Now See Programming area, here first I have created MyCustomDialog class and extended DialogFragment class then I have created a onSubmitListner interface , here I have also used Dialog property  and OnClickListener() method.In MainActivity I have implemented onSubmitListner interface and also used OnClickListener() method.
You can see below example code it clearly describe you "How to make CustomDialog Fragment in android".

Step(1) I have created a custom_dialog.xml layout-

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:orientation="vertical"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:background="#eee2f7f1"
    android:gravity="center">
    <EditText
        android:id="@+id/etDialog"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:ems="10">
        <requestFocus/>
        </EditText>
    <Button
        android:layout_width="fill_parent"
        android:layout_height="wrap_content"
        android:id="@+id/btnDialog"
        android:text="Submit"
        android:background="#ee6dcad9"
        android:textStyle="bold"/>

</LinearLayout>

Step(2)- activity_main.xml layout-

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent">

    <Button
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginLeft="120dp"
        android:layout_marginTop="40dp"
        android:text="Click"
        android:id="@+id/btn"
        android:paddingLeft="10dp"
        android:textSize="15dp"
        android:paddingRight="10dp"
        android:background="#eef2f0e5"
        />
    <TextView
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:id="@+id/tv"
        android:text=""
        android:textSize="25dp"
         android:layout_marginLeft="10dp"
        android:textColor="#cf2a2a"
        android:layout_marginTop="50dp"/>

    </LinearLayout>

    Step(3)-Created a new MyCustomDialog class -
   

public class MyCustomDialog extends DialogFragment {
    Button mButton;
    EditText mEditText;
    onSubmitListner mListener;
    String text ="";

    interface onSubmitListner{
       void setOnSubmitListener(String arg);
    }
    @Override
    public Dialog onCreateDialog(Bundle savedInstanceState){
        final Dialog dialog = new Dialog (getActivity());
        dialog.getWindow().requestFeature(Window.FEATURE_NO_TITLE);
        dialog.getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN, WindowManager.LayoutParams.FLAG_FULLSCREEN);
        dialog.setContentView(R.layout.custom_dialog);
        dialog.getWindow().setBackgroundDrawable(new ColorDrawable(Color.TRANSPARENT));
        dialog.show();
        mButton =(Button)dialog.findViewById(R.id.btnDialog);
        mEditText=(EditText)dialog.findViewById(R.id.etDialog);
        mEditText.setText(text);
        mButton.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {

                mListener.setOnSubmitListener(mEditText.getText().toString());
                dismiss();
            }
        });

return dialog;

    }
}

Step(4)-MainActivity-
 

public class MainActivity extends AppCompatActivity  implements onSubmitListner {
    TextView mTextView;
    Button mButton;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        mTextView=(TextView)findViewById(R.id.tv);
        mButton=(Button)findViewById(R.id.btn);
        mButton.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
                MyCustomDialog fragment = new MyCustomDialog();
                fragment.mListener=MainActivity.this;
                fragment.text=mTextView.getText().toString();
                fragment.show(getFragmentManager(),"");
            }
        });

    }

    @Override
    public void setOnSubmitListener(String arg) {
      mTextView.setText(arg);
    }
}