[JDK-8208690] network tests failing due to "JavaTest Message: Problem cleaning up the following threads" Created: 2018-08-02  Updated: 2018-10-02  Resolved: 2018-10-02

Status: Resolved
Project: JDK
Component/s: core-libs
Affects Version/s: 12
Fix Version/s: None

Type: Bug Priority: P3
Reporter: Daniel Daugherty Assignee: Chris Hegarty
Resolution: Fixed Votes: 0
Labels: ProblemList, tier2, tier3
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relates
relates to CODETOOLS-7902259 AgentVMThreadgroup.cleanup needs to a... Resolved
relates to JDK-8211092 test/jdk/sun/net/www/http/HttpClient/... Resolved
relates to JDK-8211325 test/jdk/java/net/Socket/LingerTest.j... Resolved
Sub-Tasks:
Key
Summary
Type
Status
Assignee
JDK-8209018 ProblemList tests affected by JDK-820... Sub-task Resolved Daniel Daugherty  
JDK-8209019 Remove tests affected by JDK-8208690 ... Sub-task Resolved Daniel Daugherty  
JDK-8209023 fix 2 compiler tests to avoid JDK-820... Sub-task Resolved Vladimir Kozlov  
JDK-8211099 ProblemList two networking tests unti... Sub-task Resolved Chris Hegarty  
Subcomponent: java.net
CPU:
generic
OS:
generic

 Description   
The following tests failed in the JDK12 CI at tier2:

sun/net/www/http/HttpClient/MultiThreadTest.java (4 instances)
java/net/Socket/LingerTest.java (2 instances)

Here's a snippet of on of the log files:

----------System.err:(55/3247)----------

JavaTest Message: Test complete.


JavaTest Message: Problem cleaning up the following threads:
Thread-251
  at java.base@12-internal/java.net.SocketInputStream.socketRead0(Native Method)
  at java.base@12-internal/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
  at java.base@12-internal/java.net.SocketInputStream.read(SocketInputStream.java:168)
  at java.base@12-internal/java.net.SocketInputStream.read(SocketInputStream.java:140)
  at java.base@12-internal/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
  at java.base@12-internal/java.io.BufferedInputStream.read(BufferedInputStream.java:271)
  at Worker.readUntil(MultiThreadTest.java:255)
  at Worker.run(MultiThreadTest.java:285)

Thread-252
  at java.base@12-internal/java.net.SocketInputStream.socketRead0(Native Method)
  at java.base@12-internal/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
  at java.base@12-internal/java.net.SocketInputStream.read(SocketInputStream.java:168)
  at java.base@12-internal/java.net.SocketInputStream.read(SocketInputStream.java:140)
  at java.base@12-internal/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
  at java.base@12-internal/java.io.BufferedInputStream.read(BufferedInputStream.java:271)
  at Worker.readUntil(MultiThreadTest.java:255)
  at Worker.run(MultiThreadTest.java:285)

Thread-253
  at java.base@12-internal/java.net.SocketInputStream.socketRead0(Native Method)
  at java.base@12-internal/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
  at java.base@12-internal/java.net.SocketInputStream.read(SocketInputStream.java:168)
  at java.base@12-internal/java.net.SocketInputStream.read(SocketInputStream.java:140)
  at java.base@12-internal/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
  at java.base@12-internal/java.io.BufferedInputStream.read(BufferedInputStream.java:271)
  at Worker.readUntil(MultiThreadTest.java:255)
  at Worker.run(MultiThreadTest.java:285)

Thread-254
  at java.base@12-internal/java.net.SocketInputStream.socketRead0(Native Method)
  at java.base@12-internal/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
  at java.base@12-internal/java.net.SocketInputStream.read(SocketInputStream.java:168)
  at java.base@12-internal/java.net.SocketInputStream.read(SocketInputStream.java:140)
  at java.base@12-internal/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
  at java.base@12-internal/java.io.BufferedInputStream.read(BufferedInputStream.java:271)
  at Worker.readUntil(MultiThreadTest.java:255)
  at Worker.run(MultiThreadTest.java:285)

Thread-255
  at java.base@12-internal/java.net.SocketInputStream.socketRead0(Native Method)
  at java.base@12-internal/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
  at java.base@12-internal/java.net.SocketInputStream.read(SocketInputStream.java:168)
  at java.base@12-internal/java.net.SocketInputStream.read(SocketInputStream.java:140)
  at java.base@12-internal/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
  at java.base@12-internal/java.io.BufferedInputStream.read(BufferedInputStream.java:271)
  at Worker.readUntil(MultiThreadTest.java:255)
  at Worker.run(MultiThreadTest.java:285)

----------System.out:(1/41)----------
Connection count = 5 Request count = 100
result: Error. Error while cleaning up threads after test


test result: Error. Error while cleaning up threads after test

 Comments   
Comment by Daniel Daugherty [ 2018-08-02 ]
[~jjg] - Might be a coincidence, but this is the first JDK12 tier2 CI job that
contains the fix for:

JDK-8208647 switch jtreg to 4.2b13

and the error message is coming from JavaTest. Can you take a look?
Comment by Chris Hegarty [ 2018-08-02 ]
Assigning to Jon for now, until jtreg is either identified as the culprit or eliminated.
Comment by David Holmes [ 2018-08-03 ]
I independently rediscovered this before seeing this bug report. The issue is in jtreg I'm pretty sure, and I filed:

https://bugs.openjdk.java.net/browse/CODETOOLS-7902259

The socket threads now factor in timeoutFactor in their SO_TIMEOUT calculation, but the cleanup code still waits the same amount of time before deciding there' been an error in cleanup.
Comment by Jonathan Gibbons [ 2018-09-21 ]
With my jtreg hat on, we've addressed timeouts when cleaning up tests.

I don't have a core-libs/java.net hat, so removing myself as Assignee.
Comment by Chris Hegarty [ 2018-09-24 ]
[~dcubed] what's left in the java.net subcategory that can be tracked by this bug now? The tier2 networking tests have been reasonably stable in the jdk/jdk CI job of recent.
Comment by Daniel Daugherty [ 2018-09-24 ]
Looks like both tests are still on the ProblemList:

$ grep 8208690 open/test/jdk/ProblemList.txt
java/net/Socket/LingerTest.java 8208690 generic-all
sun/net/www/http/HttpClient/MultiThreadTest.java 8208690 generic-all

There's a subtask of this bug to fix that. Once that fix has been
pushed, we can run the re-enabled tests in the CI for 3-5 days
and if this failure mode doesn't appear, then I would say the
issue is resolved.

It looks like Jon used this bug to fix JTREG:

CODETOOLS-7902259 AgentVMThreadgroup.cleanup needs to adjust the cleanup timeout based on timeoutFactor
https://bugs.openjdk.java.net/browse/CODETOOLS-7902259

so this bug should be resolved as either a duplicate of CODETOOLS-7902259
or resolved with a reason of "external".
Comment by Chris Hegarty [ 2018-09-25 ]
When jtreg b14 is promoted and in use, these tests should be removed from the ProblemList.txt ( once verified ).
Comment by Chris Hegarty [ 2018-10-01 ]
Note: JDK-8211092 changes the termination of MultiThreadTest, so that it does not complete until its worker threads have completed. This avoids the cleanup issue in jtreg, and therefore MultiThreadTest has been taken off the ProblemList ( regardless of the jtreg version ).
Comment by Chris Hegarty [ 2018-10-02 ]
Note: JDK-8211325 changes the termination of LingerTest.java so that it does not complete until its worker threads have completed. This avoids the cleanup issue in jtreg, and therefore LingerTest.java has been taken off the ProblemList ( regardless of the jtreg version ).
Comment by Chris Hegarty [ 2018-10-02 ]
Both tests in the networking area, identified as problematic in this bug, have been resolved ( see above comments ). There are no more networking related actions required. Closing
Generated at Sat Aug 24 03:28:19 UTC 2019 using Jira 7.13.5#713005-sha1:8d78f1047b9cca7d35d4d13f706b37e27d869e07.