You do not need the SQL Server Agent. It has nothing to do with database connectivity.
http://msdn.microsoft.com/en-us/library/ms189237.aspx
SQL Server Agent is a Microsoft
Windows service that executes
scheduled administrative tasks, which
are called jobs. SQL Server Agent uses
SQL Server to store job information.
Jobs contain one or more job steps.
Each step contains its own task, for
example, backing up a database. SQL
Server Agent can run a job on a
schedule, in response to a specific
event, or on demand. For example, if
you want to back up all the company
servers every weekday after hours, you
can automate this task. Schedule the
backup to run after 22:00 Monday
through Friday; if the backup
encounters a problem, SQL Server Agent
can record the event and notify you.