Having built databases since before there were efficient database engines, I would recommend you create FOREIGN KEY constraints where required.
- Presence or absence of ON DELETE and ON UPDATE does not avoid the INSERT requirement on the parent table. They will influence what happens when the parent key is changed or removed. The parent record will be required if the child table's key is not nullable.
- The foreign key will enforce referential integrity and this will ensure that joins will work. Without the foreign key it will be possible to enter values in the child table which do not exist in the parent table. Outer joins to the parent table will fail for those records.
- Query speed will not slow down due to the FOREIGN KEY. There will be a minor impact on INSERT as the parent table's index will need to be check for presence of the parent record. This is not a reason to sacrifice referential integrity. (The cost to continually verify it by other means will usually outweigh the costs of the constraint.) The cost of cleaning up the referential integrity after the fact will greatly exceed the savings.
I avoid the ON DELETE and ON UPDATE actions in FOREIGN KEY definitions. I haven't yet designed a database where they are necessary. These two actions control what happens to the child records when the parent changes. ON DELETE can be extremely destructive if a frequently referenced parent is unintentionally deleted. ON UPDATE is only useful where the parent key can change. Use of surrogate keys eliminates the need for ON UPDATE actions.
If the reverence to the parent table is always from the child it might be appropriate not to index the foreign key column in the child table.