For a database using the full or bulk-logged recovery model, in most
cases you must back up the tail of the log before restoring the
database. Restoring a database without first backing up the tail of
the log results in an error, unless the RESTORE DATABASE statement
contains either the WITH REPLACE or the WITH STOPAT clause, which must
specify a time or transaction that occurred after the end of the data
backup. For more information about tail-log backups, see Tail-Log
Backups (SQL Server).
MSDN Source on Replace
A tail-log backup captures any log records that have not yet been
backed up (the tail of the log) to prevent work loss and to keep the
log chain intact. Before you can recover a SQL Server database to its
latest point in time, you must back up the tail of its transaction
log. The tail-log backup will be the last backup of interest in the
recovery plan for the database.
MSDN Source on Tail-Log Backup
What is the difference when using or not using "With Replace" when
If you have a database called DatabaseA and you have another database called DatabaseB, let's say you want A to be a copy of B. You would take a full backup of B and then you can restore over A with B. It would have an issue without replace. In one circumstance you may run into it saying: The backup set holds a backup of a database other than the existing 'DatabaseA' database. In another circumstance you may run into it stating you need the tail-log. More on those scenarios here.
I know that With Replace, I would get identical database as backup
file even log file would be replaced/overwritten and current content /
transactions would be broken.
This is correct, you are overwriting the existing database with the backup file. The chain of transactions is now based on the backup restored.
But, what is actual difference; I read somewhere that some
configuration for database like recovery model is stored in Master
database; With Replace - it can be overwritten! What else?
Basically, if you did a backup restore of DatabaseB over DatabaseA, it becomes DatabaseB but named as DatabaseA.
I cannot find much on what is the use of With Replace? Why it is used
a lot, most of the developer restore databases using "With Replace".
It sounds exactly like it is, it is used when you want to replace the existing DB with the DB restored from the backup.