Wednesday, June 24, 2020

java.net.SocketException: No buffer space available (maximum connections reached?)

I have come across an issue that Tableau server degraded suddenly. We had to restart the services to bring Tableau server online.

We see below errors from the log file and after research, we found that Tableau Server Port Exhaustion caused the issue

Errors in Server event log
TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.

Error in Tableau log files
2020-07-05 07:10:42.800 -0700 (,,,) StatusServerThread_dataserver_2 : INFO com.tableausoftware.service.thrift.win32.NamedPipeServerTransport - Awaiting client connection on \\.\pipe\dataserver_2-status, handle native@0x1a3c.
2020-07-05 07:10:49.990 -0700 (,,,) FilestoreHostLocatorThread : ERROR org.apache.curator.framework.imps.CuratorFrameworkImpl - Background exception was not retry-able or retry gave up
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1845)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:452)
at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
at org.apache.curator.ConnectionState.reset(ConnectionState.java:262)
at org.apache.curator.ConnectionState.start(ConnectionState.java:109)
at org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:191)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:259)
at com.tableausoftware.tdfs.common.ZkConnection.<init>(ZkConnection.java:67)
at com.tableausoftware.tdfs.common.ZkConnectionProvider.getConnection(ZkConnectionProvider.java:50)
at com.tableausoftware.tdfs.common.HostLocator.lambda$initialize$1(HostLocator.java:146)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor105.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1843)
... 19 more
Caused by: java.io.IOException: Unable to establish loopback connection at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:94)
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.PipeImpl.<init>(PipeImpl.java:171)
at sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50)
at java.nio.channels.Pipe.open(Pipe.java:155)
at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:127)
at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44)
at java.nio.channels.Selector.open(Selector.java:227)
at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
... 23 more

Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): bind
1. To Identify the dynamic port range

Open a command prompt as an administrator, and run the following command to determine the dynamic port range:

netsh int ipv4 show dynamicportrange tcp
or
netsh int ipv6 show dynamicportrange tcp

The command will display the port number of the first dynamic port, and the number of ports. You need to add the number of ports to the start port to determine the port range.

2. Run below command to view ports currently in use
netstat –n 

3. Run the following command to increase the dynamic port range:

netsh int <ipv4|ipv6> set dynamicportrange <tcp|udp> start=number num=range store=<active/persistent>

Note that port 65535 is the maximum port number . You can start by setting the range to include ports 32767 through 65535

Also refer below article To determine port exhaustion caused the issue.
https://kb.tableau.com/articles/issue/tableau-server-port-exhaustion-problems#:~:text=Increase%20the%20dynamic%20port%20range,is%20the%20maximum%20port%20number.

Refer below article identify port-exhaustion at server side.
https://support.microsoft.com/en-us/help/3123245/update-improves-port-exhaustion-identification-in-windows-server-2012
https://social.technet.microsoft.com/Forums/ie/en-US/b632acdc-a546-4014-a299-4c27781e6c5a/tcpip-failed-to-establish-an-outgoing-connection-event-id-4227?forum=winserverPN
https://technet.microsoft.com/en-us/library/cc938217.aspx 

Thanks & Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

23 comments:

  1. Bet Mr. Santorum's kids just loved finding out what the Left think their name means. Did you dance a little Hitlerian jig when his daughter's hospitalization was announced as the reason for his withdrawal?
    Sattamatka

    ReplyDelete
  2. Thanks for sharing this ...
    Leanpitch provides online training in Scrum during this lockdown period everyone can use it wisely.
    Scrum Values

    ReplyDelete
  3. Thanks for sharing this ...
    Leanpitch provides online training in DevOps during this lockdown period everyone can use it wisely.
    DevOps Online Training

    ReplyDelete
  4. Hello there, simply was aware of your blog through Google, and located that it’s really informative. win915

    ReplyDelete
  5. I have just found this website while searching over the internet, you have posted valuable information which i like reading. Bandar Slot

    ReplyDelete
  6. Products seems to be original. I can feel the originality form photos. Lsm99

    ReplyDelete
  7. first-rate knowledge, restructure everybody mind merely reference assign assist to to it Situs IDN Poker

    ReplyDelete
  8. I have just found this website while searching over the internet, you have posted valuable information which i like reading. 토토

    ReplyDelete
  9. افضل شركات نقل العفش في منطقة المدينة المنورة تلك الشركات التي تقدم افضل الخدمات الخاصة بالمنزل كما في افضل شركة نقل اثاث بالمدينة المنورة المضمونة وتلك الخدمات تكون بمنتهى الدقة والاحترافية وتمتلك سيارات مجهزة خصيصا لنقل العفش من مكان الى مكان آخر بعد تنظيف المكان الجديد من قبل ارخص شركة تنظيف شقق بالمدينة المنورة محترفة وذلك قبل نقل العفش اليها لكي يتم ترتيب عفشك على نظافة وقد تقوم في الحال اقوى شركة تنظيف خزانات بالمدينه المنوره بتنظيف وتعقيم لخزان المياه خاصتك لضمان سلامة ونقاء مياهك وايضا تعمل افضل شركة تنظيف كنب بالمدينة المنورة بتنظيف وغسيل المفروشات والكنب والموكيت والستائر وتعقيم المنزل كاملا وتقدم شركه مكافحه حشرات بالمدينه المنوره خدمات رش المبيدات الحشرية لكي يتم تعقيم المنزل كاملا من البكتيريا والجراثيم الضارة

    ReplyDelete
  10. exciting publicize. i have Been questioning more or less this count number, as a end result thanks for posting. pretty cold publication.It 's in fact very fine and useful say.thank you 먹튀검증 안전놀이터 스포츠중계 꽁머니 먹튀검증 토토사이트 안전놀이터 꽁머니

    ReplyDelete
  11. I have just found this website while searching over the internet, you have posted valuable information which i like reading. Click Here
    Click Here
    Click Here

    ReplyDelete
  12. dog supplement, Are you a cat or dog owner looking to maximize nourishment for your furry family members? As a pet parent, you naturally strive to provide your pets with the best lifestyle options available

    ReplyDelete
  13. Gogoanime

    Given by the Council should not deceive us. Other sources of Chia Anime and even Asian Drama the street. And particularly touching episode drama Chia-Anime can be downloaded video in MP4 just about any forms q.e.

    ReplyDelete
  14. I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article... joker123

    ReplyDelete
  15. I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article먹튀검증업체

    ReplyDelete
  16. If your looking for Online Illinois license plate sticker renewals then you have need to come to the right place.We offer the fastest Illinois license plate sticker renewals in the state. click here

    ReplyDelete
  17. Thank you so much for providing good quality information.I think it would not have been easy to collect this information and organize it in an easy to understand way.igtok

    ReplyDelete
  18. This comment has been removed by the author.

    ReplyDelete