[BBLISA] Odd Latency issues over VPN

[Nick Cammorato]

Hi everyone,

I thought someone here might have some ideas, because I'm currently
stumped.  For some background: I recently consolidated all of our "inside"
layer 3 onto our Juniper SRX 1400. Prior to this everything was scattered
across a few different devices with some point to point links.  For the
most part, everything works as expected - pretty well.  The exception being
why I'm mailing the list - VPN connections(via our ASA) to our internal
instances of atlassian confluence are suddenly excruciatingly slow.

We have 2 confluence instances: a development/test instance and a
production instance, each of which live on a different VLAN/has a different
gateway.  The exhbited behavior is: page loads of up to 30-40 seconds,
almost all most of which is a single batched ajax JS load - which is about
300 -> 500kb or so and loads at a rate of 10kbps.  This is new behavior.

Traffic not over VPN is perfectly normal.

Current topography looks as follows:
ASA(inside) --> SRX (ge-x/x/x.0)
Clients -(Ge)-> Client Distribution Switch --(2XGe VPC)--> Nexus Switches
--(2XGe VPC)--> SRX(ae0.1)
Confluence1 -(Ge)->  Distribution Switch --(2XGe VPC)--> Nexus Switches
--(2XGe VPC)--> SRX(ae0.2)
Confluence2 -(Ge)->  Distribution Switch --(2XGe VPC)--> Nexus Switches
--(2XGe VPC)--> SRX(ae0.3)

And I've tested the following:
- The ASA was at one point cabled off the Client Distribution Switch with
the vlan dwelling on the agg interface, moving it had no effect.
- I've monitored traffic via an inline tap, tcpdumps at both ends, and a
tcpdump on the router itself looking for fragmentation, out of sequence
packets, etc. and seen nothing.
- I've done the above looking for DNS traffic to see if maybe there is an
nslookup issue somewhere, and nada.
- iperf shows normal bandwidth to the confluence servers themselves -
10mbps or so from home.
- There don't appear to be any autonegotiation issues.
- No errors on any involved interface.
- No errors in apache, confluence or tomcat logs, regardless of log level.
- Software version of confluence has no effect.

Now here's an odd thing, if I do a curl on one of the slowly loading
scripts, in isolation it loads at 10kbps or so - this is repeatable too,
daisy chain 10 loads of the same script and they will all load at 10kbps.
 If I fork and run the curl twice or more in parallel, however, it loads
instantly.

Anyone have any ideas before I start opening TAC/JTAC cases?

Thanks,
--Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.bblisa.org/pipermail/bblisa/attachments/20140124/781d3684/attachment.html>