I opened a ticket with AWS and here is what they said:
Please allow me to inform you that unfortunately, "plpythonu"
extension is not on the list of supported extensions provided by
Amazon RDS. In order to check the available supported extensions on
your instance you can run the below queries or check the following
=> SHOW rds.extensions;
The extension "plpythonu" in currently not be supported by RDS/Aurora
as it would need access to the underlying filesystem. RDS as a managed
service does not grant access to its underlying filesystem as it would
break the ability to manage a database.
I regret any inconvenience caused to you due to the limitation of the
service. As I believe this is an important feature for you so after
researching internally, I can confirm that there is already an
existing feature request in place for supporting "plpythonu"
So ,I went ahead and added your case to this feature request to have
this functionality added. The AWS service teams use these feature
requests, in conjunction with other avenues of customer feedback, to
plan and prioritize the features they deliver.
As you might understand, any new feature has to go through a review process and once approved it will go through a development cycle and rigorous testing before it can be rolled out to the public. And hence, I don't have an exact ETA on when this feature will be available.