dbcp连接池不合理的锁导致连接耗尽解决方案--688IT编程网

dbcp连接池不合理的锁导致连接耗尽解决⽅案

dbcp 连接池不合理的锁导致连接耗尽解决⽅案

应⽤报错，表象来看是连接池爆满了。

ansaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.apachemons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:241) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]

at com.ansport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) [dubbo-2.5.3.jar:2.5.3]

at urrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_33]

at urrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_33]

at java.lang.Thread.run(Thread.java:662) [na:1.6.0_33]

Caused by: org.apachemons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted

at org.apachemons.Connection(PoolingDataSource.java:103) ~[commons-dbcp.jar:1.2.1]

at org.apachemons.Connection(BasicDataSource.java:540) ~[commons-dbcp.jar:1.2.1]

at pub.jdbc.Connection(MultiDataSource.java:74) ~[crmpub-jdbc-1.0.jar:1.0]

at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]

java dubbo

... 32 common frames omitted

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

at org.apachemons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174) ~[commons-pool-1.6.jar:1.6]

at org.apachemons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) ~[commons-dbcp.jar:1.2.1]

at org.apachemons.Connection(PoolingDataSource.java:95) ~[commons-dbcp.jar:1.2.1]

... 35 common frames omitted

应⽤堆栈，⼏⼗条线程都阻塞到dbcp池的PoolableConnectionFactory.makeObject⽅法了，此⽅法在等待0x0000000709a638a0对象锁，但0x0000000709a638a0对象⼀直被其中⼀条线程

执⾏PoolableConnectionFactory.makeObject时锁了，⽽且这条线程获取到锁后还阻塞住了，所以导致后⾯⼏⼗条线程都BLOCKED了。

"DubboServerHandler-132.121.95.136:28101-thread-300" daemon prio=10 tid=0x00007f02b4136800 nid=0x4954 waiting for monitor entry [0x00007f028*******]

java.lang.Thread.State: BLOCKED (on object monitor)

at org.apachemons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)

- waiting to lock <0x0000000709a638a0> (a org.apachemons.dbcp.PoolableConnectionFactory)

at org.apachemons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)

at org.apachemons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)

at org.apachemons.Connection(PoolingDataSource.java:95)

at org.apachemons.Connection(BasicDataSource.java:540)

at pub.jdbc.Connection(MultiDataSource.java:74)

at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)

at ansaction.Transaction(AbstractPlatformTransactionManager.java:372)

at ansaction.ateTransactionIfNecessary(TransactionAspectSupport.java:417)

at ansaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)

at ansaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)

at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)

at servicemon.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)

flect.GeneratedMethodAccessor253.invoke(Unknown Source)

flect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at flect.Method.invoke(Method.java:597)

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)

at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)

at service.sysmgr.sca.impl.SysMgrServiceImpl$$EnhancerByCGLIB$$bfd71326.qryBulletinList(<generated>)

at com.alibaba.dubbomon.bytecode.Wrapper101.invokeMethod(Wrapper101.java)

at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)

at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)

at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)

at pub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)

at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)

at com.alibaba.dubbo.rpc.protocol.dubbo.ply(DubboProtocol.java:108)

at com.hange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)

at com.hange.support.ived(HeaderExchangeHandler.java:170)

at com.ived(DecodeHandler.java:52)

at com.ansport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)

at urrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at urrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:

- <0x000000070a120f00> (a urrent.locks.ReentrantLock$NonfairSync)

"DubboServerHandler-132.121.95.136:28101-thread-290" daemon prio=10 tid=0x00007f03203da800 nid=0x4948 runnable [0x00007f0286a22000]

java.lang.Thread.State: RUNNABLE

at java.SocketInputStream.socketRead0(Native Method)

at ad(SocketInputStream.java:129)

sql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)

sql.jdbc.adFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)

sql.jdbc.ad(ReadAheadInputStream.java:187)

- locked <0x0000000715a9ce70> (sql.jdbc.util.ReadAheadInputStream)

sql.adFully(MysqlIO.java:3140)

sql.adPacket(MysqlIO.java:597)

sql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085)

sql.Connect(ConnectionImpl.java:2494)

sql.tOneTryOnly(ConnectionImpl.java:2527)

sql.ateNewIO(ConnectionImpl.java:2309)

- locked <0x00000007159d1898> (sql.jdbc.JDBC4Connection)

sql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)

sql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)

wInstance(Unknown Source)

wInstance(DelegatingConstructorAccessorImpl.java:27)

at wInstance(Constructor.java:513)

sql.jdbc.Util.handleNewInstance(Util.java:408)

sql.Instance(ConnectionImpl.java:419)

sql.t(NonRegisteringDriver.java:344)

at org.apachemons.ateConnection(DriverConnectionFactory.java:37)

at org.apachemons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)

- locked <0x0000000709a638a0> (a org.apachemons.dbcp.PoolableConnectionFactory)