Slow remote SELECT statement due to long "client processing time", but fast locally

Question

While connected to our production server (SQL Server 2008, very powerful machine), this SELECT statement takes 2 seconds, spitting back all fields (4 MB of data in total).

SELECT TOP (30000) *
FROM person
WITH(NOLOCK);

From any other box on the same network (connecting using SQL authentication or Windows Authentication), the same query takes 1 minute, 8 seconds.

I am testing with this very simple statement to illustrate that it's not an indexing problem or query-related problem. (We have performance issues with all queries at the moment...)

The rows come in chunks, and not all at once. I get my first rows instantly, and then wait for over 1 minute for the batches of rows to come in.

Here are the Client Statistics of the query, when it is ran from the remote box:

Query Profile Statistics
  Number of INSERT, DELETE and UPDATE statements 0
  Rows affected by INSERT, DELETE, or UPDATE statements 0
  Number of SELECT statements  2
  Rows returned by SELECT statements 30001
  Number of transactions 0

Network Statistics
  Number of server roundtrips 3
  TDS packets sent from client        3
  TDS packets received from server 1216
  Bytes sent from client         266
  Bytes received from server 4019800

Time Statistics
  Client processing time 72441 ms (72 seconds)
  Total execution time   72441 ms
  Wait time on server replies 0

We can see that the "Client Processing Time" is equal to the total execution time.

Does anyone know what steps I can take to diagnose why the transfer of the actual data is taking a long time?

Is there an SQL configuration parameter that restricts or limits data transfer speed between machines?

Answer 1

Your problem is definitely network related, based on your info. As such, it has to be dealt with with network professionals (I am not the one).

Things that might help:

• Faster NIC cards (on SQL server).
• Adding of allocated/specific NIC card/subnet between the servers (web-server and SQL Server).

Is the web-server in the same sub-net as the SQL server?

Are there routers/bridges etc. between them?

Not many possible changes on SQL server:

• Output data is being sent by SQL Server with proprietary MS "TDS protocol".
• Default size of the TDS buffer is 4 KB. See in MSDB: "network packet size Option"
• Compressing the data (with SQL Server or an external application) - depends upon nature of data.

You are using a default size: see your stats: "TDS packets received from server 1216" (4MB/1K=4KB). Yes, size of the TDS buffer can be changed: see in google: "TDS protocol batch size"

Good discussion on the topic: "does sql's network packet size really determine round trip traffic?"

However, changing the TDS package size will (inevitably) have unpredictable effects and should only be used in production in exceptional cases.

Changing of architecture or introduction of caching of data on mid-tier would also help.

Answer 2

This issue is now resolved.

It was a network problem, and the SQL box was using a 100 MB/s NIC card, instead of a 10 GB/s NIC card...

A network configuration change to use the correct network card has fixed the problem. Now we are getting similar performance for all queries from the Production SQL box and from other boxes on the network.

Thanks everyone for your help.

Answer 3

At initial reading it sounds like you are experiencing some network latency issues. Have you looked at some of the Network Perfmon counters? Those may give you some indication of what is going on with the network.

Quote from What Perfmon counters should I monitor and what each of them mean?

NETWORK IO

To measure network I/O, you can use the following counters:

Network InterfaceBytes Total/sec

Threshold: Sustained values of more than 80 percent of network bandwidth.

Significance: This counter indicates the rate at which bytes are sent and received over each network adapter. This counter helps you know whether the traffic at your network adapter is saturated and if you need to add another network adapter. How quickly you can identify a problem depends on the type of network you have as well as whether you share bandwidth with other applications.

Network InterfaceBytes Received/sec

This counter indicates the rate at which bytes are received over each network adapter. You can calculate the rate of incoming data as a part of total bandwidth. This will help you know that you need to optimize on the incoming data from the client or that you need to add another network adapter to handle the incoming traffic.

Network InterfaceBytes Sent/sec

This counter indicates the rate at which bytes are sent over each network adapter. You can calculate the rate of incoming data as a part of total bandwidth. This will help you know that you need to optimize on the data being sent to the client or you need to add another network adapter to handle the outbound traffic.

ServerBytes Total/sec

This value should not be more than 50 percent of network capacity.

This counter indicates the number of bytes sent and received over the network. Higher values indicate network bandwidth as the bottleneck. If the sum of Bytes Total/sec for all servers is roughly equal to the maximum transfer rates of your network, you may need to segment the network.

Processor% Interrupt Time

This counter indicates the percentage of time the processor spends receiving and servicing hardware interrupts. This value is an indirect indicator of the activity of devices that generate interrupts, such as network adapters.

Network Interface(*)Output Queue Length

This counter checks to see how many threads are waiting on the network adapter. If there are a lot of threads waiting on the network adapter, then the system is most likely saturating the network I/O most likely due to network latency or network bandwidth.

Output Queue Length is the length of the output packet queue (in packets). If this is longer than two, there are delays and the bottleneck should be found and eliminated, if possible. Since the requests are queued by the Network Driver Interface Specification (NDIS) in this implementation, this will always be 0.

Answer 4

Some preliminary questions: 1) The server has a SQL client on Prod. server machine set up, right? So if you make the same query from the client located on the same machine it will be completed in 2 seconds? Did you try to do this? Is it really 2 seconds? 2) You mentioned that configuration of your production environment has been changed (or production server moved to other network/total server rebuild done), right? What was the query consumption time in old production environment?

From any other box on the same network ...the same query takes 1 minute, 8 seconds. 3) You are saying that the query returns and is consumed from the client, located on any machine in the given network (expept your specific machine) in about 70 seconds? I understood correctly? 3.1 Incidentally what is the timing for consumption of this query, acceptable by the business? 4) However, you are specifiying that for a specific client machine that you are using the query output consumption time is: Client Execution Time 15:30:48 15 minutes? (and this time is clearly not acceptible)? Correct? 5) so the problem is limited to a single client machine? Or to ANY client/mid-tier etc machine (in a new environment)? 6) what is the delay shown by ping? from client computer to the server? 7) You (or network admin) did run tracert both ways (from client to server, from server to client)? How many hops? What is the combined time? 8) Is the old production network alive? Can you compare using Ping and Traceroute - what was the time and hops between client and server there?

Out of curiousity: this is an example of the query? or exact wording of the query? The query really does NOT contain WHERE clause? Agree with me that this is very unusual.. The table has a clustered index or is a Heap? The table contains how many rows all in all? The table is heavily fragmented? Out of curiosity: have why SELECT TOP NNN? Why not SET ROWCOUNT NNN - then SELECT * ? This query is issued how many times by the client per day? 1? 100? 1MLN? Underlying data is static or is dynamic and is changed much? How much (0.01 percent per day? 1 percent per day? 10 percent per day?) The query output is processed programmatically? (not by a user?) Why is it not cached/not stored on mid-tier? thanks, Alexei