| 1 | // Copyright 2018 The Go Authors. All rights reserved. |
|---|---|
| 2 | // Use of this source code is governed by a BSD-style |
| 3 | // license that can be found in the LICENSE file. |
| 4 | |
| 5 | /* |
| 6 | Package packages loads Go packages for inspection and analysis. |
| 7 | |
| 8 | The Load function takes as input a list of patterns and return a list of Package |
| 9 | structs describing individual packages matched by those patterns. |
| 10 | The LoadMode controls the amount of detail in the loaded packages. |
| 11 | |
| 12 | Load passes most patterns directly to the underlying build tool, |
| 13 | but all patterns with the prefix "query=", where query is a |
| 14 | non-empty string of letters from [a-z], are reserved and may be |
| 15 | interpreted as query operators. |
| 16 | |
| 17 | Two query operators are currently supported: "file" and "pattern". |
| 18 | |
| 19 | The query "file=path/to/file.go" matches the package or packages enclosing |
| 20 | the Go source file path/to/file.go. For example "file=~/go/src/fmt/print.go" |
| 21 | might return the packages "fmt" and "fmt [fmt.test]". |
| 22 | |
| 23 | The query "pattern=string" causes "string" to be passed directly to |
| 24 | the underlying build tool. In most cases this is unnecessary, |
| 25 | but an application can use Load("pattern=" + x) as an escaping mechanism |
| 26 | to ensure that x is not interpreted as a query operator if it contains '='. |
| 27 | |
| 28 | All other query operators are reserved for future use and currently |
| 29 | cause Load to report an error. |
| 30 | |
| 31 | The Package struct provides basic information about the package, including |
| 32 | |
| 33 | - ID, a unique identifier for the package in the returned set; |
| 34 | - GoFiles, the names of the package's Go source files; |
| 35 | - Imports, a map from source import strings to the Packages they name; |
| 36 | - Types, the type information for the package's exported symbols; |
| 37 | - Syntax, the parsed syntax trees for the package's source code; and |
| 38 | - TypeInfo, the result of a complete type-check of the package syntax trees. |
| 39 | |
| 40 | (See the documentation for type Package for the complete list of fields |
| 41 | and more detailed descriptions.) |
| 42 | |
| 43 | For example, |
| 44 | |
| 45 | Load(nil, "bytes", "unicode...") |
| 46 | |
| 47 | returns four Package structs describing the standard library packages |
| 48 | bytes, unicode, unicode/utf16, and unicode/utf8. Note that one pattern |
| 49 | can match multiple packages and that a package might be matched by |
| 50 | multiple patterns: in general it is not possible to determine which |
| 51 | packages correspond to which patterns. |
| 52 | |
| 53 | Note that the list returned by Load contains only the packages matched |
| 54 | by the patterns. Their dependencies can be found by walking the import |
| 55 | graph using the Imports fields. |
| 56 | |
| 57 | The Load function can be configured by passing a pointer to a Config as |
| 58 | the first argument. A nil Config is equivalent to the zero Config, which |
| 59 | causes Load to run in LoadFiles mode, collecting minimal information. |
| 60 | See the documentation for type Config for details. |
| 61 | |
| 62 | As noted earlier, the Config.Mode controls the amount of detail |
| 63 | reported about the loaded packages. See the documentation for type LoadMode |
| 64 | for details. |
| 65 | |
| 66 | Most tools should pass their command-line arguments (after any flags) |
| 67 | uninterpreted to the loader, so that the loader can interpret them |
| 68 | according to the conventions of the underlying build system. |
| 69 | See the Example function for typical usage. |
| 70 | */ |
| 71 | package packages // import "golang.org/x/tools/go/packages" |
| 72 | |
| 73 | /* |
| 74 | |
| 75 | Motivation and design considerations |
| 76 | |
| 77 | The new package's design solves problems addressed by two existing |
| 78 | packages: go/build, which locates and describes packages, and |
| 79 | golang.org/x/tools/go/loader, which loads, parses and type-checks them. |
| 80 | The go/build.Package structure encodes too much of the 'go build' way |
| 81 | of organizing projects, leaving us in need of a data type that describes a |
| 82 | package of Go source code independent of the underlying build system. |
| 83 | We wanted something that works equally well with go build and vgo, and |
| 84 | also other build systems such as Bazel and Blaze, making it possible to |
| 85 | construct analysis tools that work in all these environments. |
| 86 | Tools such as errcheck and staticcheck were essentially unavailable to |
| 87 | the Go community at Google, and some of Google's internal tools for Go |
| 88 | are unavailable externally. |
| 89 | This new package provides a uniform way to obtain package metadata by |
| 90 | querying each of these build systems, optionally supporting their |
| 91 | preferred command-line notations for packages, so that tools integrate |
| 92 | neatly with users' build environments. The Metadata query function |
| 93 | executes an external query tool appropriate to the current workspace. |
| 94 | |
| 95 | Loading packages always returns the complete import graph "all the way down", |
| 96 | even if all you want is information about a single package, because the query |
| 97 | mechanisms of all the build systems we currently support ({go,vgo} list, and |
| 98 | blaze/bazel aspect-based query) cannot provide detailed information |
| 99 | about one package without visiting all its dependencies too, so there is |
| 100 | no additional asymptotic cost to providing transitive information. |
| 101 | (This property might not be true of a hypothetical 5th build system.) |
| 102 | |
| 103 | In calls to TypeCheck, all initial packages, and any package that |
| 104 | transitively depends on one of them, must be loaded from source. |
| 105 | Consider A->B->C->D->E: if A,C are initial, A,B,C must be loaded from |
| 106 | source; D may be loaded from export data, and E may not be loaded at all |
| 107 | (though it's possible that D's export data mentions it, so a |
| 108 | types.Package may be created for it and exposed.) |
| 109 | |
| 110 | The old loader had a feature to suppress type-checking of function |
| 111 | bodies on a per-package basis, primarily intended to reduce the work of |
| 112 | obtaining type information for imported packages. Now that imports are |
| 113 | satisfied by export data, the optimization no longer seems necessary. |
| 114 | |
| 115 | Despite some early attempts, the old loader did not exploit export data, |
| 116 | instead always using the equivalent of WholeProgram mode. This was due |
| 117 | to the complexity of mixing source and export data packages (now |
| 118 | resolved by the upward traversal mentioned above), and because export data |
| 119 | files were nearly always missing or stale. Now that 'go build' supports |
| 120 | caching, all the underlying build systems can guarantee to produce |
| 121 | export data in a reasonable (amortized) time. |
| 122 | |
| 123 | Test "main" packages synthesized by the build system are now reported as |
| 124 | first-class packages, avoiding the need for clients (such as go/ssa) to |
| 125 | reinvent this generation logic. |
| 126 | |
| 127 | One way in which go/packages is simpler than the old loader is in its |
| 128 | treatment of in-package tests. In-package tests are packages that |
| 129 | consist of all the files of the library under test, plus the test files. |
| 130 | The old loader constructed in-package tests by a two-phase process of |
| 131 | mutation called "augmentation": first it would construct and type check |
| 132 | all the ordinary library packages and type-check the packages that |
| 133 | depend on them; then it would add more (test) files to the package and |
| 134 | type-check again. This two-phase approach had four major problems: |
| 135 | 1) in processing the tests, the loader modified the library package, |
| 136 | leaving no way for a client application to see both the test |
| 137 | package and the library package; one would mutate into the other. |
| 138 | 2) because test files can declare additional methods on types defined in |
| 139 | the library portion of the package, the dispatch of method calls in |
| 140 | the library portion was affected by the presence of the test files. |
| 141 | This should have been a clue that the packages were logically |
| 142 | different. |
| 143 | 3) this model of "augmentation" assumed at most one in-package test |
| 144 | per library package, which is true of projects using 'go build', |
| 145 | but not other build systems. |
| 146 | 4) because of the two-phase nature of test processing, all packages that |
| 147 | import the library package had to be processed before augmentation, |
| 148 | forcing a "one-shot" API and preventing the client from calling Load |
| 149 | in several times in sequence as is now possible in WholeProgram mode. |
| 150 | (TypeCheck mode has a similar one-shot restriction for a different reason.) |
| 151 | |
| 152 | Early drafts of this package supported "multi-shot" operation. |
| 153 | Although it allowed clients to make a sequence of calls (or concurrent |
| 154 | calls) to Load, building up the graph of Packages incrementally, |
| 155 | it was of marginal value: it complicated the API |
| 156 | (since it allowed some options to vary across calls but not others), |
| 157 | it complicated the implementation, |
| 158 | it cannot be made to work in Types mode, as explained above, |
| 159 | and it was less efficient than making one combined call (when this is possible). |
| 160 | Among the clients we have inspected, none made multiple calls to load |
| 161 | but could not be easily and satisfactorily modified to make only a single call. |
| 162 | However, applications changes may be required. |
| 163 | For example, the ssadump command loads the user-specified packages |
| 164 | and in addition the runtime package. It is tempting to simply append |
| 165 | "runtime" to the user-provided list, but that does not work if the user |
| 166 | specified an ad-hoc package such as [a.go b.go]. |
| 167 | Instead, ssadump no longer requests the runtime package, |
| 168 | but seeks it among the dependencies of the user-specified packages, |
| 169 | and emits an error if it is not found. |
| 170 | |
| 171 | Overlays: The Overlay field in the Config allows providing alternate contents |
| 172 | for Go source files, by providing a mapping from file path to contents. |
| 173 | go/packages will pull in new imports added in overlay files when go/packages |
| 174 | is run in LoadImports mode or greater. |
| 175 | Overlay support for the go list driver isn't complete yet: if the file doesn't |
| 176 | exist on disk, it will only be recognized in an overlay if it is a non-test file |
| 177 | and the package would be reported even without the overlay. |
| 178 | |
| 179 | Questions & Tasks |
| 180 | |
| 181 | - Add GOARCH/GOOS? |
| 182 | They are not portable concepts, but could be made portable. |
| 183 | Our goal has been to allow users to express themselves using the conventions |
| 184 | of the underlying build system: if the build system honors GOARCH |
| 185 | during a build and during a metadata query, then so should |
| 186 | applications built atop that query mechanism. |
| 187 | Conversely, if the target architecture of the build is determined by |
| 188 | command-line flags, the application can pass the relevant |
| 189 | flags through to the build system using a command such as: |
| 190 | myapp -query_flag="--cpu=amd64" -query_flag="--os=darwin" |
| 191 | However, this approach is low-level, unwieldy, and non-portable. |
| 192 | GOOS and GOARCH seem important enough to warrant a dedicated option. |
| 193 | |
| 194 | - How should we handle partial failures such as a mixture of good and |
| 195 | malformed patterns, existing and non-existent packages, successful and |
| 196 | failed builds, import failures, import cycles, and so on, in a call to |
| 197 | Load? |
| 198 | |
| 199 | - Support bazel, blaze, and go1.10 list, not just go1.11 list. |
| 200 | |
| 201 | - Handle (and test) various partial success cases, e.g. |
| 202 | a mixture of good packages and: |
| 203 | invalid patterns |
| 204 | nonexistent packages |
| 205 | empty packages |
| 206 | packages with malformed package or import declarations |
| 207 | unreadable files |
| 208 | import cycles |
| 209 | other parse errors |
| 210 | type errors |
| 211 | Make sure we record errors at the correct place in the graph. |
| 212 | |
| 213 | - Missing packages among initial arguments are not reported. |
| 214 | Return bogus packages for them, like golist does. |
| 215 | |
| 216 | - "undeclared name" errors (for example) are reported out of source file |
| 217 | order. I suspect this is due to the breadth-first resolution now used |
| 218 | by go/types. Is that a bug? Discuss with gri. |
| 219 | |
| 220 | */ |
| 221 |
Members