Wednesday, June 24, 2020

java.net.SocketException: No buffer space available (maximum connections reached?)

I have come across an issue that Tableau server degraded suddenly. We had to restart the services to bring Tableau server online.

We see below errors from the log file and after research, we found that Tableau Server Port Exhaustion caused the issue

Errors in Server event log
TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.

Error in Tableau log files
2020-07-05 07:10:42.800 -0700 (,,,) StatusServerThread_dataserver_2 : INFO com.tableausoftware.service.thrift.win32.NamedPipeServerTransport - Awaiting client connection on \\.\pipe\dataserver_2-status, handle native@0x1a3c.
2020-07-05 07:10:49.990 -0700 (,,,) FilestoreHostLocatorThread : ERROR org.apache.curator.framework.imps.CuratorFrameworkImpl - Background exception was not retry-able or retry gave up
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1845)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:452)
at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
at org.apache.curator.ConnectionState.reset(ConnectionState.java:262)
at org.apache.curator.ConnectionState.start(ConnectionState.java:109)
at org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:191)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:259)
at com.tableausoftware.tdfs.common.ZkConnection.<init>(ZkConnection.java:67)
at com.tableausoftware.tdfs.common.ZkConnectionProvider.getConnection(ZkConnectionProvider.java:50)
at com.tableausoftware.tdfs.common.HostLocator.lambda$initialize$1(HostLocator.java:146)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor105.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1843)
... 19 more
Caused by: java.io.IOException: Unable to establish loopback connection at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:94)
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.PipeImpl.<init>(PipeImpl.java:171)
at sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50)
at java.nio.channels.Pipe.open(Pipe.java:155)
at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:127)
at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44)
at java.nio.channels.Selector.open(Selector.java:227)
at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
... 23 more

Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): bind
1. To Identify the dynamic port range

Open a command prompt as an administrator, and run the following command to determine the dynamic port range:

netsh int ipv4 show dynamicportrange tcp
or
netsh int ipv6 show dynamicportrange tcp

The command will display the port number of the first dynamic port, and the number of ports. You need to add the number of ports to the start port to determine the port range.

2. Run below command to view ports currently in use
netstat –n 

3. Run the following command to increase the dynamic port range:

netsh int <ipv4|ipv6> set dynamicportrange <tcp|udp> start=number num=range store=<active/persistent>

Note that port 65535 is the maximum port number . You can start by setting the range to include ports 32767 through 65535

Also refer below article To determine port exhaustion caused the issue.
https://kb.tableau.com/articles/issue/tableau-server-port-exhaustion-problems#:~:text=Increase%20the%20dynamic%20port%20range,is%20the%20maximum%20port%20number.

Refer below article identify port-exhaustion at server side.
https://support.microsoft.com/en-us/help/3123245/update-improves-port-exhaustion-identification-in-windows-server-2012
https://social.technet.microsoft.com/Forums/ie/en-US/b632acdc-a546-4014-a299-4c27781e6c5a/tcpip-failed-to-establish-an-outgoing-connection-event-id-4227?forum=winserverPN
https://technet.microsoft.com/en-us/library/cc938217.aspx 

Thanks & Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

19 comments:

  1. Bet Mr. Santorum's kids just loved finding out what the Left think their name means. Did you dance a little Hitlerian jig when his daughter's hospitalization was announced as the reason for his withdrawal?
    Sattamatka

    ReplyDelete

  2. Thank you very much for writing such an interesting article on this topic. This has really made me think and I hope to read more. @ Packers and Movers mumbai

    ReplyDelete
  3. Get Packers and Movers Jaipur List of Top Reliable, 100% Affordable, Verified and Secured Service Provider. Get Free ###Packers and Movers Jaipur Price Quotation instantly and Save Cost and Time. Packers and Movers Jaipur ✔ ✔ ✔ Reviews and Compare Charges for household Shifting, Home/Office Relocation, ***Car Transportation, Pet Relocation, Bike SHifting @ Packers And Movers Jaipur

    ReplyDelete
  4. Thanks for sharing this ...
    Leanpitch provides online training in Scrum during this lockdown period everyone can use it wisely.
    Scrum Values

    ReplyDelete
  5. Thanks for sharing this ...
    Leanpitch provides online training in DevOps during this lockdown period everyone can use it wisely.
    DevOps Online Training

    ReplyDelete
  6. Hello there, simply was aware of your blog through Google, and located that it’s really informative. win915

    ReplyDelete
  7. I have just found this website while searching over the internet, you have posted valuable information which i like reading. Bandar Slot

    ReplyDelete
  8. Products seems to be original. I can feel the originality form photos. Lsm99

    ReplyDelete
  9. first-rate knowledge, restructure everybody mind merely reference assign assist to to it Situs IDN Poker

    ReplyDelete
  10. I have just found this website while searching over the internet, you have posted valuable information which i like reading. design merchandise

    ReplyDelete
  11. This will open a window containing the HTML source code of the page. Inspect an HTML Element: Right-click on an element basic HTML CODE

    ReplyDelete
  12. I have just found this website while searching over the internet, you have posted valuable information which i like reading. 토토

    ReplyDelete
  13. افضل الخدمات الخاصة باعمال نقل العفش مع الفك والتركيب في الطائف تقدمها افضل شركه نقل عفش بالطائف وهي تعتبر من افضل الشركات المتخصصة في اعمال نقل الاثاث بواسطة فريق عمل مميز ومحترف وسيارات نقل عفش حديثة ومجهزة لأعمال شركات نقل الاثاث بالطائف مقفلة ومبطنة من الداخل وستحصل على افضل خدمات نقل الاثاث بالطائف باحترافية عالية وبالضمان من افضل شركات نقل عفش الطائف المتميزة والمؤهلة

    ReplyDelete
  14. افضل شركات نقل العفش في منطقة المدينة المنورة تلك الشركات التي تقدم افضل الخدمات الخاصة بالمنزل كما في افضل شركة نقل اثاث بالمدينة المنورة المضمونة وتلك الخدمات تكون بمنتهى الدقة والاحترافية وتمتلك سيارات مجهزة خصيصا لنقل العفش من مكان الى مكان آخر بعد تنظيف المكان الجديد من قبل ارخص شركة تنظيف شقق بالمدينة المنورة محترفة وذلك قبل نقل العفش اليها لكي يتم ترتيب عفشك على نظافة وقد تقوم في الحال اقوى شركة تنظيف خزانات بالمدينه المنوره بتنظيف وتعقيم لخزان المياه خاصتك لضمان سلامة ونقاء مياهك وايضا تعمل افضل شركة تنظيف كنب بالمدينة المنورة بتنظيف وغسيل المفروشات والكنب والموكيت والستائر وتعقيم المنزل كاملا وتقدم شركه مكافحه حشرات بالمدينه المنوره خدمات رش المبيدات الحشرية لكي يتم تعقيم المنزل كاملا من البكتيريا والجراثيم الضارة

    ReplyDelete
  15. exciting publicize. i have Been questioning more or less this count number, as a end result thanks for posting. pretty cold publication.It 's in fact very fine and useful say.thank you 먹튀검증 안전놀이터 스포츠중계 꽁머니 먹튀검증 토토사이트 안전놀이터 꽁머니

    ReplyDelete
  16. I have just found this website while searching over the internet, you have posted valuable information which i like reading. Click Here
    Click Here
    Click Here

    ReplyDelete
  17. Your website is really cool and this is a great inspiring article. ผลบอล

    ReplyDelete