This is by design. Clip shapes are not antialiased since the accumulated
blending operations along their outline would erode those pixels until
none of the original color remained, thus defeating the purpose of
antialiasing the edge in the first place.
Clipping actions should, in general, be reproducible so that if you
render with two non-overlapping clip shapes, you can perform the
operations in either order and achieve the same effect. Antialiased
clipping violates this rule since the edge operations would depend
on the order in which the immediate mode operations were performed.
If you want a rendering to be composited on top of another with blended
edges then the best way to achieve that effect would be to render the
drawing to a BufferedImage and then blend that image onto the destination
in one of two ways.
1. Install the BufferedImage as a Texture and perform an
antialiased fill of the desired shape.
2. Use a BufferedImage with a full alpha channel and then
render the desired shape antialiased onto the BufferedImage
with an AlphaComposite.DST_IN operation. Finally, composite
the drawing onto the destination with an AlphaComposite.SRC_OVER
It also bears mentioning that hints are just hints and can be ignored by
any compliant implementation so this is not a bug even if antialiased
clipping were feasible or desireable.
I will change this to an RFE since there are at least 2 ways in which it
might be useful:
1. The developer may acknowledge that edge operations accumulate
errors and still want the effect regardless, most likely for
single operations which only perform a single edge operation
and don't have time to accumulate any errors over multiple
operations. Developers would have to use the capability with
those caveats in mind though.
2. We could, with a great deal of effort, employ a sub-pixel
mask record for each pixel to calculate the edge effects more
exactly and remember the history of colors and coverages for
each pixel as we proceed. This mechanism would enable us
to do much more intuitively accurate soft clipping, but we
have no plans to ever enhance the implementation in this way.
In either case, the capability would be controlled through a separate
hint since the default case has behavior implications that would create
incorrect results if care was not taken to minimize the operations.