Like the documentation says:
[...] changing the type of an existing column will require the entire table and its indexes to be rewritten. As an exception, when changing the type of an existing column, if the
USING clause does not change the column contents and the old type is either binary coercible to the new type or an unconstrained domain over the new type, a table rewrite is not needed; but any indexes on the affected columns must still be rebuilt.
So the key is that the type conversion is binary coercible, in other words, you need a type cast between these types that is defined
This is not the case for the data types you mention in your question, but if you are a superuser, you can create such a cast and then use it with
CREATE CAST (date AS int4) WITHOUT FUNCTION;
ALTER TABLE mytable ALTER datecol TYPE int4 USING (datecol::int4);
DROP CAST (date AS int4);
Of course you have to understand the internal representation of
date for that to make sense.
It is a good idea to remove the cast after you are done, because adding type casts to PostgreSQL can easily lead to ambiguities during type resolution, which will cause errors.