http.Client.Do() returning EOF error rarely but predictably

[Daniel Bryan]

I'm running a small monitoring service in Go that runs a series of HTTP
requests sequentially every minute, reporting on the status of various
services and comparing HTTP responses to what's expected.

We're getting a number of false alarms at the moment, which seem to come
down to client.Do() returning an EOF error roughly once every 2000-2500
requests.

This is the essence of my code:

var (
tr = &http.Transport{
TLSClientConfig: &tls.Config{
InsecureSkipVerify: true,
ServerName: HOST,
},
DisableKeepAlives: true,
}
result = newCheckResult()
client = http.Client{Transport: tr}
req *http.Request
res *http.Response
)

if req, err = http.NewRequest(method, url, nil); err != nil {
return nil, err
}

req.Host = HOST
req.Close = true

if res, err = client.Do(req); res != nil {
defer res.Body.Close()
}

if err != nil {
return nil, err
}

// Response handling logic here; does not read in the response body

EOF is being returned from that last block there. It's hard to match up
precisely, but I'm fairly certain that the Apache servers I'm hitting have
logged these requests as a response with code 200.

The roundabout way of creating the requests is to avoid problems with
hostname verification, and to ensure that SNI works by setting the
ServerName. The failing requests are all actually over plain HTTP, so I
don't think the TLS config has any effect.

I've tried digging into the code but it's difficult to see where an EOF
would come from in the RoundTripper. I found some old bugs and a
StackOverflow thread that suggest bugs or quirks with the connection
pooling, which is why I see Close to true and disable keep-alive.

Is there some way to get better visibility over why a request actually
failed, rather than just EOF? It's a peculiar error to get as I don't even
try to read in the response body.

[Caleb Doxsey]

The docs say:

An error is returned if caused by client policy (such as CheckRedirect), or

if there was an HTTP protocol error. A non-2xx response doesn't cause an
error.

So EOF shouldn't happen and would seem to imply a protocol error of some
sort. Are the servers just plain Apache? Or are they some sort of custom
code?

Often useful for debugging HTTP requests is fiddler (
http://www.telerik.com/download/fiddler), or the much lower level wireshark
(http://www.wireshark.org/).

Just taking a shot in the dark here: but maybe the Content-Length header of
the response is wrong?

I'm running a small monitoring service in Go that runs a series of HTTP
requests sequentially every minute, reporting on the status of various
services and comparing HTTP responses to what's expected.
We're getting a number of false alarms at the moment, which seem to come
down to client.Do() returning an EOF error roughly once every 2000-2500
requests.
var (
tr = &http.Transport{
TLSClientConfig: &tls.Config{
InsecureSkipVerify: true,
ServerName: HOST,
},
DisableKeepAlives: true,
}
result = newCheckResult()
client = http.Client{Transport: tr}
req *http.Request
res *http.Response
)
if req, err = http.NewRequest(method, url, nil); err != nil {
return nil, err
}
req.Host = HOST
req.Close = true
if res, err = client.Do(req); res != nil {
defer res.Body.Close()
}
if err != nil {
return nil, err
}
// Response handling logic here; does not read in the response body
EOF is being returned from that last block there. It's hard to match up
precisely, but I'm fairly certain that the Apache servers I'm hitting have
logged these requests as a response with code 200.
The roundabout way of creating the requests is to avoid problems with
hostname verification, and to ensure that SNI works by setting the
ServerName. The failing requests are all actually over plain HTTP, so I
don't think the TLS config has any effect.
I've tried digging into the code but it's difficult to see where an EOF
would come from in the RoundTripper. I found some old bugs and a
StackOverflow thread that suggest bugs or quirks with the connection
pooling, which is why I see Close to true and disable keep-alive.
Is there some way to get better visibility over why a request actually
failed, rather than just EOF? It's a peculiar error to get as I don't even
try to read in the response body.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.

[James Bardin]

Post by Caleb Doxsey
Just taking a shot in the dark here: but maybe the Content-Length header
of the response is wrong?

The content-length isn't wrong, since the go client wouldn't get to that
until it tries to read the resp.Body.

You can get an EOF if there is no double newline at the end of the headers
though.

[Andy Balholm]

EOF means that the socket was closed before that was expected. It often
signals a dropped network connection. When I get it, it usually means that
the http.Transport was reusing an old connection, and that connection got
closed by the server without the Transport's connection-reuse logic
noticing it.

[Daniel Bryan]

I added some instrumentation last night and verified that client.Do() is
what's returning EOF.

Post by Andy Balholm
EOF means that the socket was closed before that was expected. It often
signals a dropped network connection. When I get it, it usually means that
the http.Transport was reusing an old connection, and that connection got
closed by the server without the Transport's connection-reuse logic
noticing it.

Interesting. As mentioned, I've done req.Close = True, as well as DisableKeepAlives:
True, so it's odd that it'd be reusing a TCP connection. I might try a
retry loop with a loop to see what affect that has - though it's an awful
solution.

[James Bardin]

Post by Daniel Bryan
True, so it's odd that it'd be reusing a TCP connection. I might try a
retry loop with a loop to see what affect that has - though it's an awful
solution.

If you can instrument it further, I'm willing to bet that you're not
getting a complete header from some server, probably missing the blank line
to terminate the headers. I know some libraries like curl/python don't
report this as an error, but go wants that extra '\r\n' no matter what.

[Brad Fitzpatrick]

Post by James Bardin

Post by Daniel Bryan
True, so it's odd that it'd be reusing a TCP connection. I might try a
retry loop with a loop to see what affect that has - though it's an awful
solution.

If you can instrument it further, I'm willing to bet that you're not
getting a complete header from some server, probably missing the blank line
to terminate the headers. I know some libraries like curl/python don't
report this as an error, but go wants that extra '\r\n' no matter what.

Except we don't consider it "extra". ;-)

I haven't seen that in a long time, though. Most servers are pretty
compliant.

I would be interested in seeing a packet capture, though. Or at least an
analysis of one, if it contains secrets.

[Dave Cheney]

My money is apache closed a connection which the client has kept in a pool.

This is a tricky problem to solve, most clients retry gets.

EOF means that the socket was closed before that was expected. It often signals a dropped network connection. When I get it, it usually means that the http.Transport was reusing an old connection, and that connection got closed by the server without the Transport's connection-reuse logic noticing it.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
For more options, visit https://groups.google.com/groups/opt_out.