issue with running tomcat behind apache where apache throws periodic 503 errors

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

issue with running tomcat behind apache where apache throws periodic 503 errors

Tyler Danstrom-2
Hi,

We are having an issue with our DSpace install where Tomcat appears to idle out during any period of sustained inactivity and then Apache throws a 503 error the next time any user attempts to connect to our dspace instance.

Dspace version: 5.1
UI type: xmlui
OS: Redhat Enterprise Linux release 6.7
database: oracle
Tomcat version: 7.0.59
Apache version:2.2.15

Tomcat config:

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"

URIEncoding="UTF-8" connectionTimeout="20000"/>

 

Apache config:


ProxyPass /jspui ajp://localhost:8009/jspui/ timeout=10 retry=0

ProxyPass /xmlui ajp://localhost:8009/xmlui/ timeout=10 retry=0



We are running Tomcat behind Apache with Shibboleth and accessing dspace with the following path:

/xmlui 

after the host name

The connection is not encrypted over ssl by default.

Narrative of the problem

Any time that there is a period of inactivity (inactivity is defined as no traffic) Tomcat appears to cease to respond to Apache and throws a 503 error to the end-user.

The solution to this problem seems to be to upon first receiving the 503 error immediately reload the page (this may only be required once but has been witnessed to require three times) within seconds of the unsuccessful attempt.

Eventually, the xmlui url resolves correctly and returns a successful connection to the dspace application and from then on until there is another sustand period of inactivity everything works fine for one or multiple simultaneous connections.

Has anyone else experienced this issue? Does anyone know of a long-term solution to this problem?

Any and all help is immensely appreciated.

Sincerely,
Tyler Danstrom
Programmer/Analyst
University of Chicago Library

--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: issue with running tomcat behind apache where apache throws periodic 503 errors

helix84
Never experienced this problem. 503 only ever correctly appears only when tomcat is down. Here's my configuration:

Apache 2.2, 2.4:

        ProxyPass / ajp://localhost:8009/ retry=2
        ProxyPassReverse / ajp://localhost:8009/

Tomcat 7, 8:
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" />

I probably wouldn't set retry to 0. Try to turn on the keepalive parameter:


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: issue with running tomcat behind apache where apache throws periodic 503 errors

Tyler Danstrom-2
Hi,

So, after more investigation we've figured out that the problem is with the ajp ilink

The following appears in our logs

[Wed Sep 16 14:51:35 2015] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

[Wed Sep 16 14:51:35 2015] [error] ajp_read_header: ajp_ilink_receive failed

[Wed Sep 16 14:51:35 2015] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (localhost)

 
Does anyone know what DSpace recommends to set on timeout with an AJP link?
 

On Thursday, September 17, 2015 at 11:00:00 AM UTC-5, helix84 wrote:
Never experienced this problem. 503 only ever correctly appears only when tomcat is down. Here's my configuration:

Apache 2.2, 2.4:

        ProxyPass / ajp://localhost:8009/ retry=2
        ProxyPassReverse / ajp://localhost:8009/

Tomcat 7, 8:
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" />

I probably wouldn't set retry to 0. Try to turn on the keepalive parameter:
<a href="http://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fhttpd.apache.org%2Fdocs%2F2.4%2Fmod%2Fmod_proxy.html%23proxypass\46sa\75D\46sntz\0751\46usg\75AFQjCNF60kkZvV1Y3YomU9ioi4sFUEBi6A&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fhttpd.apache.org%2Fdocs%2F2.4%2Fmod%2Fmod_proxy.html%23proxypass\46sa\75D\46sntz\0751\46usg\75AFQjCNF60kkZvV1Y3YomU9ioi4sFUEBi6A&#39;;return true;">http://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
<a href="https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fwiki.duraspace.org%2Fdisplay%2FDSPACE%2FMailing%2BList%2BEtiquette\46sa\75D\46sntz\0751\46usg\75AFQjCNFD59wScPCIGn2ZtnOG38LzYhO--Q&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fwiki.duraspace.org%2Fdisplay%2FDSPACE%2FMailing%2BList%2BEtiquette\46sa\75D\46sntz\0751\46usg\75AFQjCNFD59wScPCIGn2ZtnOG38LzYhO--Q&#39;;return true;">https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

RE: issue with running tomcat behind apache where apache throws periodic 503 errors

Pottinger, Hardy J.-2
Hi, we recently encountered a similar situation, and in researching default timeouts, I discovered the default timeout for AJP sessions is 300 seconds. Not knowing exactly *why* our sessions need to last a bit longer, I decided to try a really big number in that timeout:

# let's let proxy connections live a whole 24 hours (i.e. 86400 seconds) the default is 300 seconds
ProxyTimeout 86400

So far, I haven't heard any more complaints about 503 errors.

Now, you may not be comfortable using such a large number for your timeout. I'd urge caution setting this number. If anyone has a suggestion for a better/more-reasonable number for the AJP timeout, I'd love to hear it.

Thanks!

--Hardy


From: [hidden email] [[hidden email]] on behalf of Tyler Danstrom [[hidden email]]
Sent: Tuesday, September 22, 2015 9:38 AM
To: DSpace Technical Support
Cc: [hidden email]; [hidden email]
Subject: Re: [dspace-tech] issue with running tomcat behind apache where apache throws periodic 503 errors

Hi,

So, after more investigation we've figured out that the problem is with the ajp ilink

The following appears in our logs

[Wed Sep 16 14:51:35 2015] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

[Wed Sep 16 14:51:35 2015] [error] ajp_read_header: ajp_ilink_receive failed

[Wed Sep 16 14:51:35 2015] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (localhost)

 
Does anyone know what DSpace recommends to set on timeout with an AJP link?
 

On Thursday, September 17, 2015 at 11:00:00 AM UTC-5, helix84 wrote:
Never experienced this problem. 503 only ever correctly appears only when tomcat is down. Here's my configuration:

Apache 2.2, 2.4:

        ProxyPass / ajp://localhost:8009/ retry=2
        ProxyPassReverse / ajp://localhost:8009/

Tomcat 7, 8:
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" />

I probably wouldn't set retry to 0. Try to turn on the keepalive parameter:


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
<a href="https://wiki.duraspace.org/display/DSPACE/Mailing&#43;List&#43;Etiquette" rel="nofollow" target="_blank">https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: issue with running tomcat behind apache where apache throws periodic 503 errors

Tim Donohue
Administrator
In reply to this post by Tyler Danstrom-2
Hi Tyler,

In case it's of any help..

For sites I've managed in the past, I've always had the following
general settings in Apache (with mod_proxy and mod_proxy_ajp):

## Proxy / Forwarding Settings ##
<Proxy *>
AddDefaultCharset Off
Order deny,allow
Allow from all
</Proxy>

# Pass all requests to Tomcat's AJP Connector
ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/

In Tomcat 7, I just have:

<Connector port="8009" protocol="AJP/1.3" URIEncoding="UTF-8"/>

I also setup the / (ROOT) path to just load my DSpace UI of choice.

Since you are using Shibboleth, you may also wish to review your
settings for mod_shib. I doubt it's the root issue, but it's possible
that Shibboleth could be a factor as well. We do have recommended
mod_shib settings documented at:
https://wiki.duraspace.org/display/DSDOC5x/Authentication+Plugins#AuthenticationPlugins-ShibbolethAuthentication

This StackOverflow answer regarding the same error message also points
to using either "timeout" (in ProxyPass) or ProxyTimeout (which Hardy
recommended):
http://serverfault.com/questions/655362/mod-proxy-ajp-70007the-timeout-specified-has-expired-ajp-ilink-receive-can

Good luck! Let us know if you find a resolution, so that we can add it
to the wiki/documentation if needed.

- Tim

On 9/22/2015 9:38 AM, Tyler Danstrom wrote:

> Hi,
>
> So, after more investigation we've figured out that the problem is with
> the ajp ilink
>
> The following appears in our logs
>
> [Wed Sep 16 14:51:35 2015] [error] (70007)The timeout specified has
> expired: ajp_ilink_receive() can't receive header
>
> [Wed Sep 16 14:51:35 2015] [error] ajp_read_header: ajp_ilink_receive failed
>
> [Wed Sep 16 14:51:35 2015] [error] (70007)The timeout specified has
> expired: proxy: read response failed from (null) (localhost)
>
> Does anyone know what DSpace recommends to set on timeout with an AJP link?
>
>
> On Thursday, September 17, 2015 at 11:00:00 AM UTC-5, helix84 wrote:
>
>     Never experienced this problem. 503 only ever correctly appears only
>     when tomcat is down. Here's my configuration:
>
>     Apache 2.2, 2.4:
>
>              ProxyPass / ajp://localhost:8009/ retry=2
>              ProxyPassReverse / ajp://localhost:8009/
>
>     Tomcat 7, 8:
>          <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
>     URIEncoding="UTF-8" />
>
>     I probably wouldn't set retry to 0. Try to turn on the keepalive
>     parameter:
>     http://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass
>     <http://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass>
>
>
>     Regards,
>     ~~helix84
>
>     Compulsory reading: DSpace Mailing List Etiquette
>     https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>     <https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette>
>
> --
> You received this message because you are subscribed to the Google
> Groups "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email]
> <mailto:[hidden email]>.
> To post to this group, send email to [hidden email]
> <mailto:[hidden email]>.
> Visit this group at http://groups.google.com/group/dspace-tech.
> For more options, visit https://groups.google.com/d/optout.

--
Tim Donohue
Technical Lead for DSpace & DSpaceDirect
DuraSpace.org | DSpace.org | DSpaceDirect.org

--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: issue with running tomcat behind apache where apache throws periodic 503 errors

Graham Triggs-2
In reply to this post by Tyler Danstrom-2
Hi,

There are a couple of distinct issues that could be going on:

1) The Tomcat connector may be accepting fewer connections than Apache - so Apache requests have to wait for Tomcat to finish processing some requests.

Note - be careful in looking at Tomcat being "idle" - a lack of processor activity is not the same as the threads being idle. They may be sleeping, but allocated to a request.

2) You may be running out of permitted open ports, so Apache can't open another network connection until the OS times out some of the connections, even if Tomcat isn't doing anything.

G

On Thursday, 17 September 2015 16:47:12 UTC+1, Tyler Danstrom wrote:
Hi,

We are having an issue with our DSpace install where Tomcat appears to idle out during any period of sustained inactivity and then Apache throws a 503 error the next time any user attempts to connect to our dspace instance.

Dspace version: 5.1
UI type: xmlui
OS: Redhat Enterprise Linux release 6.7
database: oracle
Tomcat version: 7.0.59
Apache version:2.2.15

Tomcat config:

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"

URIEncoding="UTF-8" connectionTimeout="20000"/>

 

Apache config:


ProxyPass /jspui ajp://localhost:8009/jspui/ timeout=10 retry=0

ProxyPass /xmlui ajp://localhost:8009/xmlui/ timeout=10 retry=0



We are running Tomcat behind Apache with Shibboleth and accessing dspace with the following path:

/xmlui 

after the host name

The connection is not encrypted over ssl by default.

Narrative of the problem

Any time that there is a period of inactivity (inactivity is defined as no traffic) Tomcat appears to cease to respond to Apache and throws a 503 error to the end-user.

The solution to this problem seems to be to upon first receiving the 503 error immediately reload the page (this may only be required once but has been witnessed to require three times) within seconds of the unsuccessful attempt.

Eventually, the xmlui url resolves correctly and returns a successful connection to the dspace application and from then on until there is another sustand period of inactivity everything works fine for one or multiple simultaneous connections.

Has anyone else experienced this issue? Does anyone know of a long-term solution to this problem?

Any and all help is immensely appreciated.

Sincerely,
Tyler Danstrom
Programmer/Analyst
University of Chicago Library

--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.