I've seen replying for how to enable the filestream feature by a sql statement. but when the feature is opened, how to restart the sqlserver service instance to made it effects?
You can shutdown SQL Server from T-SQL, but not restart the service. You'll need to do the start using PowerShell (ex: Restart-Service cmdlet) or other command line command like sc.exe.
Or, you could use dbatools PowerShell module do to the whole thing Example for default instance on server1 (error handling to be added):
Enable-DbaFilestream -SqlInstance server1 -FileStreamLevel [1, 2 or 3] Restart-DbaService -ComputerName server1 -Type Engine
If the only remaining step is to restart the SQL Server service, simply logon to the Windows Server Core machine via Remote Desktop or through the Hypervisor client, and type the following into the console:
net stop "SQL Server (<instanceName>)" net start "SQL Server (<instanceName>)"
Replace <instanceName> with the name of the SQL Server instance. If the SQL Server is not configured as a Named Instance, then use these commands:
net stop "SQL Server (MSSQLSERVER)" net start "SQL Server (MSSQLSERVER)"
Be aware that while the SQL Server service is stopped, no clients will be able to connect - you probably want to ensure this change happens during a planned outage window.