Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8201533

Enhanced javadoc support for code samples (snippets)

    XMLWordPrintable

    Details

    • Type: JEP
    • Status: Draft
    • Priority: P3
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: tools
    • JEP Type:
      Feature
    • Exposure:
      Open
    • Subcomponent:
    • Scope:
      JDK

      Description

      Summary

      Provide enhanced support for including fragments of source code ("snippets") in API documentation generated by javadoc's standard doclet.

      Goals

      • Facilitate checking source code fragments for accuracy. Although the ultimate responsibility for accuracy lies with the author, enhanced support in javadoc can make it easier to achieve.

      • Enable modern styling, such as syntax highlighting and automatically linking names to declarations when appropriate.

      • Enable better IDE support for creating and editing snippets.

      Non-Goals

      • It is not a goal for the javadoc tool to be able to validate, compile or run any source code fragments: that task is left to external tools.

      • It is not a goal to support interactive code examples at this time: although we do not rule out such support in future, any such support will require external infrastructure that is beyond the scope of this proposal.

      Success Metrics

      Replace most if not all use of <pre>{@code ...}</pre> blocks in key modules in JDK API documentation with instances of a new tag.

      Motivation

      Authors of API documentation frequently want to include fragments of source code in documentation comments. Although {@code ...} can be used by itself for very small fragments of code, for anything non-trivial, fragments are typically represented in documentation comments with the following compound pattern:

      <pre>{@code
          lines of source code
      }</pre>

      There are various shortcomings to this approach:

      • There is no way for tools to reliably detect code fragments, in order to check their validity. Moreover, the fragments are often incomplete with placeholder comments and ellipses for the reader to "fill in the blanks". With no way to check each fragment, errors can easily occur and have been seen in practice.

      • Fragments using this pattern are not amenable to being presented with "syntax highlighting", which is nowadays a common expectation for code fragments in documentation. There is no formal indication of the kind of content in the fragment, which is necessary if the fragment is to be validated or displayed with syntax highlighting.

      • Fragments using this pattern are not amenable to being edited in an IDE except as plain text in the comment. Furthermore, not all code constructs can be included in comments: the most notable exception being traditional /* ... */ comments, because the fragment as a whole is presented in a Java comment, and /* ... */ comments cannot be nested. Also, the character sequence */ cannot be used in string constants, as may be useful for "glob" patterns and regular expressions.

      A better methodology to address all these concerns is to provide a new tag with metadata that allows the author to implicitly or explicitly specify the kind of content, in order that it can be validated, and presented in the desired manner. In addition, it is desirable to allow the fragments to be placed in separate files that can be directly manipulated in an appropriate manner in the author's preferred editor.

      Description

      A new inline tag {@snippet ...} is introduced, to declare code fragments to appear in the generated documentation. It can be used to provide inline snippets, where the code fragment is included within the tag itself, and external snippets, where the code fragment is read from a separate, external source file.

      For example,

          /**
           * The following code shows how to use Optional.isPresent:
           * {@snippet :
           *     if (v.isPresent()) {
           *         System.out.println("v: " + v.get());
           *     }
           * }
           */

      or,

          /**
           * The following code shows how to use Optional.isPresent:
           * {@snippet file="ShowOptional.java" region="example"}
           */

      where ShowOptional.java is a file that can be read by the standard doclet, and which contains a region of text identified by the name example. For example,

      public class ShowOptional {
          void show(Optional<String> v) {
              // @start example
              if (v.isPresent()) {
                  System.out.println("v: " + v.get());
              }
              // @end
          }
      }

      A hybrid form is also supported where the code fragment is in an external file, but an equivalent copy of the text can also be provided within the tag, for the convenience of anyone reading the source code for the class being documented.

      The complete form of the new tag is as follows:

      {@snippet [name=value ...] :
          lines of source code
      }

      The sequence colon newline and the following lines of source code can be omitted if not required: that is, for an external snippet.

      The name=value pairs can be used to specify properties of the code fragment. Values can be quoted with either ' or " if they contain whitespace. No other escapes are supported. The initial set of supported names is as follows:

      • class=classname
        Specifies the name of the class containing the content of the snippet. The class should be in a package hierarchy rooted in one of the following locations:

        • the same package hierarchy as the code containing the snippet tag, or
        • the snippet-files subdirectory of the immediately enclosing package, in a manner similar to the doc-files subdirectory, or
        • in a separate source file hierarchy identified by a new --snippet-path option.

        The name snippet-files is deliberately not a valid Java identifier, so that source files in or under this directory should not be construed as part of the enclosing package hierarchy. Only one of class and file may be specified.

      • file=filename
        Specifies the relative URI for a file containing the content of the snippet. The URI is evaluated relative to the same set of locations as for the class attribute.

        Only one of class and file may be specified.

      • hide=regex
        Specifies that any text in the content matching the regular expression should be replaced by an ellipsis in the generated output.

      • id=name
        Specifies an identifier for the snippet. The name will be used in the generated output, so that the snippet can be the target of links from elsewhere. To avoid any conflict with other names that might be generated by the standard doclet, it is recommended that the name should not be a simple Java identifier.

      • lang=name
        Specifies the type of content. Valid names are java, properties and text. The default is java if the class attribute is specified or the content is in a .java source file, properties if the content is in a .properties file, and text otherwise. The type of content is used to determine the kind of syntax highlighting, if any.

      • region=name
        Specifies the name of a region within the content to be included in the generated output. The region itself is identified by meta-comments in the content (see below).

      An external file may contain more than one snippet, to be included at different places within the documentation.

      Indentation

      When the source code fragment is included in the snippet tag, leading whitespace is stripped from the code fragment using String.stripLeading. This addresses an annoying shortcoming of <pre>{@code ...}</pre> blocks, where the text to be displayed starts immediately after any leading space and asterisk characters.

      Markup

      It is often desirable to include pseudo-code in code fragments in documentation, or to highlight specific parts of the fragment, even though it is not possible to do so directly in the underlying language of the fragment. To that end, the source code defined by the snippet tag may be marked up with "markup tags" in comments to be interpreted by javadoc when processing the tag, in order to affect the presentation of the text in the generated documentation. These comments use "end-of-line"-style comments appropriate for the kind of content and can be used equally within inline and external snippets. The tags can be used to define subsequences of the text and actions to be performed on those subsequences.

      Each subsequence may be either a part of a line, defined by a literal string or regular expression, or a group of lines, defined by @start and @end tags.

      The basic set of actions includes:

      • highlight a subsequence, by using a different style for the text
      • replace a subsequence with some pseudo-code, to be displayed in a suitable style
      • insert some additional code, perhaps to give the reader additional context

      Markup tags appear in markup comments, which are end-of-line comments beginning with a valid markup tag. A markup comment may contain additional markup tags after the initial tag. Markup comments need not appear on a line by themselves, and may appear after any preceding text on the same line. This allows markup comments to be placed in the source code without affecting line or column numbers of the primary content in the source code.

      Markup comments are not part of the generated documentation.

      For example, to highlight an entire region, combine the @start [name] and @highlight tag on the same line:

      class HelloWorld {
          // @start @highlight
          public static void main(String ... args) {
              System.out.println("Hello World!");
          }
          // @end
      }

      To highlight part of a line, even in a comment:

      class HelloWorld {
          public static void main(String ... args) {
              // write the standard output           // @highlight "standard"
              System.out.println("Hello World!");    // @highlight /".*"/
          }
      }

      The initial set of markup tags is as follows:

      • // @start [ <em>name</em> ]
        Defines the beginning of a region within the content. The region begins after the terminating newline. The text to be included in the generated output can be restricted to a named region by using the region attribute in the snippet tag. The name can be omitted if the comment also contains an action tag, such as @highlight or @replace, that applies to the region.

      • // @end [ <em>name</em> ]
        Defines the end of a region within the content. The region ends immediately before the initial //. The text to be included in the generated output can be restricted to a named region by using the region attribute in the snippet tag. The name can be omitted if it would be the same as that of the immediately preceding @start tag.

      • // @insert text
        Provides text that to be included in the generated output that is not otherwise part of the content to be analyzed by checking tools.

      • // @highlight [ string_or_regex ]
        Highlights either a region of text, when specified after @start, or a fragment of text on the line that is commented. The text can be specified by providing a string, enclosed in quotes, or by a regular expression, enclosed with / characters.

      • // @replace [ string_or_regex ] replace-text
        Replaces either a region of text, when specified after @start, or a fragment of text on the line that is commented. This can be used to substitute "template text" into the source code.

        • If specified after @start and no string or regular expression is given, the replacement matches the region defined by @start and the corresponding @end.
        • If a string is specified, enclosed in quotes, the replacement will be a literal replacement.
        • If a regular expression is specified, enclosed in / characters, dollar signs can be used in the replacement text to make references to captured groups in the regular expression, and backslashes can be used to escape literal characters in the replacement string.

      API

      The Compiler Tree API will be extended to provide support for the new snippet tag. This will allow external tools to scan the documentation comments in a library for uses of snippet tags, in order for those tools to validate the content of the snippet.

      Validating snippets

      By providing an API to access the structured content of snippets, we do not constrain the concept of validation to support within the javadoc tool.

      A significant advantage of using external snippets is that it is expected that such files will be compilable, in some suitable compilation context. It will be up to the test infrastructure for a library to locate these files, and to verify that they can be suitably compiled, perhaps using the Java Compiler API, and possibly executed as well, perhaps in some appropriate test infrastructure.

      For inline snippets, especially those that are not a full compilation unit, it will be up to the test infrastructure to "wrap" the code fragment in a full compilation unit, such that it can be compiled and possibly executed.

      For validating uses of the snippet tag in the JDK API documentation in particular, it is envisaged that we can provide library support within the jtreg framework.

      Other kinds of snippet content

      Although it is expected that the primary use of the snippet tag will be for Java source code, it is also possible to use snippets for other kinds of content, such as properties files, or plain text such as the output from running a command. The javadoc tool may provide basic highlighting for some of these kinds of additional content.

      Generated HTML

      The HTML that is generated to present a snippet is deliberately unspecified. However, the generated HTML for each snippet will declare an id such that it can be the target of a link from elsewhere in the documentation. The value for the id will either use the value of the id attribute declared in the snippet tag, if there is such a value available, or a default value will be used.

      Alternatives

      There are third-party JavaScript solutions to provide syntax highlighting. However, a noteworthy characteristic of JDK API documentation is the desire to provide examples involving new language features, which may not be correctly handled by such solutions in a timely manner. In addition, such solutions are typically based on the use of regular expressions, which can be very fragile, and cannot leverage any additional knowledge that might be available when the documentation is generated.

      The use of block comments to specify markup in the snippet content was considered. However, block comments for markup are visually intrusive in the source code, and can only be used in external snippets.

      Testing

      The feature can be tested using the standard test infrastructure for javadoc features: this includes jtreg tests, and related tools to check for the correctness of the generated documentation.

      Risks and Assumptions

      It is assumed that there will be a parallel effort to provide tests to validate the code fragments in the existing JDK API documentation. However, that is not a requirement for the success of this feature.

      Dependencies

      There are no external dependencies.

        Attachments

          Activity

            People

            Assignee:
            prappo Pavel Rappo
            Reporter:
            aaprameya Arvind Aprameya
            Owner:
            Pavel Rappo Pavel Rappo
            Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

              Dates

              Created:
              Updated: