intarray Extension
PostgreSQL ships with an extension that provides this and it's even faster than core array functionality on ints. You can use string_to_array and then intarray's - operator. Click here for the docs on intarray in 9.2
Before we install intarray.
test=# SELECT ARRAY[3,4,5,13,14] - ARRAY[4,5];
ERROR: operator does not exist: integer[] - integer[]
LINE 1: SELECT ARRAY[3,4,5,13,14] - ARRAY[4,5];
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
Install intarray and try again
test=# CREATE EXTENSION intarray;
CREATE EXTENSION
test=# SELECT ARRAY[3,4,5,13,14] - ARRAY[4,5];
?column?
-----------
{3,13,14}
(1 row)
So from varchar.
SELECT string_to_array(allowed_types, ',')::int[] - ARRAY[4,5] AS after_operation
FROM ( VALUES ('3,4,5,13,14') ) AS t(allowed_types);
Other notes
I agree with @Erwin when he says you should probably use a normalized 1:n design. But even if you can't, it's very clear you do not want a csv in your column. Consider at least upgrading the type to int[]. This requires no change in schema and can be done very easily using
ALTER TABLE table
ALTER COLUMN allowed_types
SET DATA TYPE int[]
USING string_to_array(allowed_types, ',')::int[];