[BBLISA] Odd Latency issues over VPN

[John Stoffel]

Nick,

>From looking at your description, it really sounds like you've got
some sort of caching in the middle which is slowing things down.  But
you don't explain the other side of the VPN well enough to know.  

Can the client using the VPN got a simple FTP from either of your
Confluence servers at full speed?  Or can they pull http data from
other internal hosts over the VPN at full speed?  

The fact that serial access is slow, while parallel access is fast
is... surprising.  Does each access when done in parallel stay at
10kbps, or do they all speed up to whatever the max the pipe to their
end supports?


Nick> I thought someone here might have some ideas, because I'm currently
Nick> stumped.  For some background: I recently consolidated all of our "inside"
Nick> layer 3 onto our Juniper SRX 1400. Prior to this everything was scattered
Nick> across a few different devices with some point to point links.  For the
Nick> most part, everything works as expected - pretty well.  The exception being
Nick> why I'm mailing the list - VPN connections(via our ASA) to our internal
Nick> instances of atlassian confluence are suddenly excruciatingly slow.

Nick> We have 2 confluence instances: a development/test instance and a
Nick> production instance, each of which live on a different VLAN/has a different
Nick> gateway.  The exhbited behavior is: page loads of up to 30-40 seconds,
Nick> almost all most of which is a single batched ajax JS load - which is about
Nick> 300 -> 500kb or so and loads at a rate of 10kbps.  This is new behavior.

Nick> Traffic not over VPN is perfectly normal.

Nick> Current topography looks as follows:
Nick> ASA(inside) --> SRX (ge-x/x/x.0)
Nick> Clients -(Ge)-> Client Distribution Switch --(2XGe VPC)--> Nexus Switches
Nick> --(2XGe VPC)--> SRX(ae0.1)
Nick> Confluence1 -(Ge)->  Distribution Switch --(2XGe VPC)--> Nexus Switches
Nick> --(2XGe VPC)--> SRX(ae0.2)
Nick> Confluence2 -(Ge)->  Distribution Switch --(2XGe VPC)--> Nexus Switches
Nick> --(2XGe VPC)--> SRX(ae0.3)

Nick> And I've tested the following:
Nick> - The ASA was at one point cabled off the Client Distribution Switch with
Nick> the vlan dwelling on the agg interface, moving it had no effect.
Nick> - I've monitored traffic via an inline tap, tcpdumps at both ends, and a
Nick> tcpdump on the router itself looking for fragmentation, out of sequence
Nick> packets, etc. and seen nothing.
Nick> - I've done the above looking for DNS traffic to see if maybe there is an
Nick> nslookup issue somewhere, and nada.
Nick> - iperf shows normal bandwidth to the confluence servers themselves -
Nick> 10mbps or so from home.
Nick> - There don't appear to be any autonegotiation issues.
Nick> - No errors on any involved interface.
Nick> - No errors in apache, confluence or tomcat logs, regardless of log level.
Nick> - Software version of confluence has no effect.

Nick> Now here's an odd thing, if I do a curl on one of the slowly loading
Nick> scripts, in isolation it loads at 10kbps or so - this is repeatable too,
Nick> daisy chain 10 loads of the same script and they will all load at 10kbps.
Nick>  If I fork and run the curl twice or more in parallel, however, it loads
Nick> instantly.

Nick> Anyone have any ideas before I start opening TAC/JTAC cases?

Nick> Thanks,
Nick> --Nick
Nick> _______________________________________________
Nick> bblisa mailing list
Nick> bblisa at bblisa.org
Nick> http://www.bblisa.org/mailman/listinfo/bblisa