Upgrading Seeq’s Swagger tooling to OpenAPI Specification 3 has been a long-standing task that was always important but never urgent. However, as we begin to think about the future direction of our API and its documentation, it became more and more pressing to build on an up-to-date foundation that we could leverage for the next several years. With that I mind, I sat down over the holidays to push it through, and though it took a lot longer than I expected, R57 will ship later this year with upgraded tooling for fully backwards compatible SDKs and a new, more secure API reference page.
Below is an overview of what it took to upgrade to OAS 3. The functional code changes didn’t turn out to be as extensive as they felt, but a lot of time went into troubleshooting behavior changes from Swagger 2 and figuring out workarounds to maintain as much backwards compatibility as possible. The various fixes outlined below were often interleaved with each other and full of dead end approaches, and the hardest part of the upgrade was never knowing how close I was to completion, as each fix exposed new problems. If you’re facing a Swagger upgrade of your own, hopefully this post will give you a high-level roadmap of the challenges you might encounter, as well as provide some tips for how to solve them in your own situation.
Changing annotations
Seeq uses code-first Swagger definitions, so Java and Kotlin annotations throughout our API layer define the behavior of endpoints and serialization classes. Though OAS 3 encourages spec-first definitions, in which those behaviors are defined in a JSON or YAML file, several of us on the development team appreciate the co-location of the API’s logical code with its contractual behavior, and while we haven’t decided yet whether to eventually migrate to spec-first definitions, we did decide that it would be easier to migrate the technology to OAS 3 than to migrate both the technology and the dev team’s conventions.
To upgrade all those annotations, we had to make the following changes:
@ApiModeland@ApiModelPropertyhad to become equivalent@Schemaand@ArraySchemaannotations@ApiOperationhad to become@Operation@ApiParamhad to become@Parameter@ApiResponsehad to become a different@ApiResponse
Across the API layer, those changes touched 322 files and more than 4,000 lines of code. The brunt of the work was done by a script I wrote that parsed annotations from Java and Kotlin code, changed the annotation name and keyword argument names to the new ones, and rewrote the file’s imports statements to remove the old annotations and import the new. The script provided a good baseline for making manual changes.
Some manual changes were needed because of ambiguities in the original annotations that OAS 3 now required to stated explicitly, such as needing @ArraySchema on list fields:
@Schema(description = "..."))
private List<ParameterInputV1> parameters = List.of();@ArraySchema(schema = @Schema(description = "...", type = "string"))
private List<ParameterInputV1> parameters = List.of();This was especially necessary for cases like the one above, where ParameterInputV1 is a class that boxes up a string. Swagger 2 coerced a list strings to a list of ParameterInputV1s, but without an array schema stating type = "string", OAS 3 expected to be given actual instances of ParameterInputV1. Other tweaks were less common, such as providing accessMode arguments to configure read-only and read-write statuses, as well as supporting input stream parameters on endpoints using @Parameter(schema = @Schema(type = "string", format = "byte")).
The most common change I had to make was adding defaultValue = "false" to schemas on boolean fields. Without an explicit default value, any boolean field (output as a Java Boolean) would be set to null, which broke several tests. After hunting down a few of these through labyrinths of code, I found the rest by comparing the field properties in our new JSON schema with the properties in our old schema. In a handful of cases, I had to add @Schema annotations to fields that hadn’t had @ApiModelProperty just so I could specify a default value.
The other big change from the rewrite script’s outputs was to the Kotlin syntax for property annotations in constructors. Since instance properties can be declared as constructor arguments, all the @Schema annotations in constructors decorated the arguments, not the properties. To annotate the fields themselves, I had to use Kotlin’s special syntax @field:Schema.
After updating the annotations across the API layer, the differences in annotation name and argument name length caused our auto-formatter to break multi-line strings in unattractive ways, and I ended up writing a second script to rewrap documentation.
Updating SDK templates
We also anticipated having to make lots of changes our SDK templates, but it turned out to be relatively straight-forward. We have Mustache templates, which under OAS 3 appear to be interpreted as Handlebars templates, but the migration tips described here allowed us to make the templates compatible with Handlebars with only minor changes to the syntax. Specifically I had to change -first and -last to @first and @last, and whenever templating syntax was used immediately before or after a curly brace, I had to use {{braces "left"}} or {{braces "right"}} to avoid triple curly braces.
I did have to do some troubleshooting on specific templates. The only variable I came across that changed was one in a JavaScript template, vendorExtensions.x-codegen-argList, which is now vendorExtensions.x-codegen-arg-list. Other times I had to add a template file, which usually I copied verbatim from the current version in the swagger-codegen repo. A few times when debugging the generated SDKs I ran into behavior that seemed it may have changed, in which case I diffed our template against the current swagger-codegen version to see what had been added or removed that I might need to copy over into the template with our custom changes.
Extracting Swagger schema as JSON
Since we use code-first spec generation, we have a class that reads Swagger annotations and generates a JSON file. The original changes to it only required updating static types from io.swagger.models.Swagger to io.swagger.v3.oas.models.OpenAPI. I also ended up folding in some customizations to the generated JSON, such as auto-generating some error response codes, as well as renaming an operation that OAS 3 was de-conflicting unnecessarily and would require SDK users to update their code. Later, however, I ended up hooking in more extensive logic in order to handle MIME types; more details in the “MIME type handling” section below.
Custom codegen classes
While OAS 3 changed some behaviors we needed to keep the same for backwards compatibility, it also granted us numerous hooks by which we could customize its behavior. The main mechanism is to create a custom codegen configuration class, of which we already had one for working around a bug in one SDK under Swagger 2. Due to behavior changes with OAS 3, we now have a custom code config class for each SDK.
The first thing I needed to work around was the ordering of arguments to operations. In our existing Java and C# SDKs, POST body arguments usually come last, but OAS 3’s codegen by default makes it first, as discussed in this open issue. Fortunately, overriding fromOperation allows changing parameter order. I also needed to mark all parameters before the last as having another parameter after them, and the last parameter as not having a parameter after it, so templates would put the proper commas between arguments:
class SeeqJavaClientCodegen : JavaClientCodegen() {
override fun fromOperation(path: String?, httpMethod: String?, operation: Operation?,
schemas: MutableMap<String, Schema<Any>>?, openAPI: OpenAPI?): CodegenOperation {
return super.fromOperation(path, httpMethod, operation, schemas, openAPI).also {
it.moveBodyParamLast()
}
}
}
fun CodegenOperation.moveBodyParamLast() {
if (this.bodyParam != null) {
val nonBodyParams = this.allParams.filter { it.paramName != "body" }.map { it.hasMore(); it }
this.allParams = nonBodyParams + this.bodyParam.also { it.hasNoMore() }
}
}
fun CodegenParameter.hasMore() =
this.setVendorExtensions(this.vendorExtensions + (CodegenConstants.HAS_MORE_EXT_NAME to "true"))
fun CodegenParameter.hasNoMore() =
this.setVendorExtensions(this.vendorExtensions - CodegenConstants.HAS_MORE_EXT_NAME)Similarly, I customized our Typescript codegen to remove the body parameter from the list of parameters, since it’s passed separately. I also adapted it to use a namespace on our model types so the Typescript compiler could find the names we were referring to, and added the built-in JavaScript types Date and Blob. There were some other adjustments to restore the existing conventions for class names and paths, and to allow underscores in operation IDs as we currently do.
For the Python SDK I had to replace periods in module names with the file system’s separator character:
open class SeeqPythonClientCodegen : PythonClientCodegen() {
override fun apiFileFolder() = "$outputFolder${File.separatorChar}seeq_sdk${File.separatorChar}${
apiPackage().replace('.', File.separatorChar)
}"
override fun modelFileFolder() = "$outputFolder${File.separatorChar}seeq_sdk${File.separatorChar}${
modelPackage().replace('.', File.separatorChar)
}"
}Beyond that, several generators needed some special handling to rename operations’ IDs so they matched the current names.
Custom template loader
I did run into mishandling of custom template paths on Windows. Since I didn’t find a direct hook for modifying template resolution, I provided codegen configs with a custom template engine that overrode the getHandlebars method with the same code used by swagger-codegen, only changing the template loader to call a custom class. The custom template loader ended up looking like this:
class CustomTemplateLoader(private val templateDir: String, suffix: String) : CodegenTemplateLoader(templateDir,
suffix) {
override fun sourceAt(uri: String?): TemplateSource {
Validate.notEmpty(uri, "The uri is required.")
// Rather than always prepending the template dir, first check whether the uri with the suffix already exists
val location = uri + suffix
val resource = getResource(location)
if (resource != null) {
return URLTemplateSource(location, resource)
}
val uri2 = uri?.replace(Regex("^/"), "")
val location2 = "$templateDir/$uri2$suffix"
val resource2 = getResource(location2)
if (resource2 != null) {
return URLTemplateSource(location2, resource2)
}
// Fallback to what Swagger wants to do
return super.sourceAt(uri)
}
}MIME type handling
The biggest set of changes I had to make upgrading to OAS 3 was to fix MIME type handling. The default JSON serialization of the Swagger schema didn’t include the consumes or produces fields, so I had to wrap operations in a CustomOperation class that had those as public properties. To get the correct values specified by our Java annotations, I created a custom reader that overloaded the parseMethod method and captured the arguments classConsumes, methodConsumes, classProduces, and methodProduces:
class CustomReader(openApi: OpenAPI) : Reader(openApi) {
override fun parseMethod(method: Method?, globalParameters: MutableList<Parameter>?, methodProduces: Produces?,
classProduces: Produces?, methodConsumes: Consumes?, classConsumes: Consumes?,
classSecurityRequirements: MutableList<SecurityRequirement>?,
classExternalDocs: Optional<ExternalDocumentation>?, classTags: MutableSet<String>?,
classServers: MutableList<Server>?, isSubresource: Boolean, parentRequestBody: RequestBody?,
parentResponses: ApiResponses?, jsonViewAnnotation: JsonView?,
classResponses: Array<out io.swagger.v3.oas.annotations.responses.ApiResponse>?,
annotatedMethod: AnnotatedMethod?): Operation {
val operation = super.parseMethod(method, globalParameters, methodProduces, classProduces, methodConsumes,
classConsumes, classSecurityRequirements, classExternalDocs, classTags, classServers, isSubresource,
parentRequestBody, parentResponses, jsonViewAnnotation, classResponses, annotatedMethod)
return CustomOperation(operation,
consumes = unionMimeTypes(classConsumes?.value, methodConsumes?.value),
produces = unionMimeTypes(classProduces?.value, methodProduces?.value))
}
}
fun unionMimeTypes(a: Array<out String>?, b: Array<out String>?): Set<String> {
val result = mutableSetOf<String>()
a?.forEach { result.add(it) }
b?.forEach { result.add(it) }
return result
}To work properly, operations also needed to have request bodies whose content types matched the consumes content types:
fun ensureRequestBodyContentType() {
if (operation.requestBody == null) {
operation.requestBody = RequestBody().apply { content = Content() }
}
val content = operation.requestBody.content
consumes?.forEach { mimeType ->
if (content[mimeType] == null) {
content[mimeType] = MediaType()
}
}
}And it needed response bodies that matched the produces content types:
fun ensureResponseContentTypes() {
if (operation.responses["200"] == null
&& operation.responses["201"] == null
&& operation.responses["202"] == null) {
operation.responses["200"] = ApiResponse()
}
val success = operation.responses["200"] ?: operation.responses["201"] ?: operation.responses["202"]!!
if (success.content == null) {
success.content = Content()
}
produces?.forEach { mimeType ->
if (success.content[mimeType] == null) {
success.content[mimeType] = MediaType()
}
}
}Some of our custom code generators either picked up these MIME types automatically or didn’t need them in templates. Where they were needed and not picked up, I had to add them manually:
override fun fromOperation(path: String?, httpMethod: String?, operation: Operation?,
schemas: MutableMap<String, Schema<Any>>?, openAPI: OpenAPI?): CodegenOperation {
return super.fromOperation(path, httpMethod, operation, schemas, openAPI).also { op ->
val consumesMimeTypes = operation?.requestBody?.content?.keys ?: listOf("application/vnd.seeq.v1+json")
op.consumes = consumesMimeTypes.mapIndexed { i, mimeType ->
mapOf("mediaType" to mimeType, "hasMore" to (if (i < consumesMimeTypes.size - 1) "true" else "false"))
}
op.hasConsumes()
}
}
fun CodegenOperation.hasConsumes() =
this.setVendorExtensions(this.vendorExtensions + (CodegenConstants.HAS_CONSUMES_EXT_NAME to "true"))Conclusion
OAS 3’s support for code-first spec generation seems to be an afterthought, as do certain aspects of code generation. Swagger did provide sufficient hooks and other means of customizing its behavior, but with as many workarounds as I added I often wondered whether I was still leveraging Swagger or slowly undoing it. While I don’t think the time I spent troubleshooting and implementing fixes would have been better spent implementing SDK generation from scratch, as the development team discusses the future needs of our SDKs, I have less confidence that Swagger can meet them.