Response description support, especially useful for swagger

Author: elasti-ron

PROPOSAL_FOR_RESPONSE_DESCRIPTIONS.md

@@ -0,0 +1,244 @@
+# Proposal for supporting response descriptions in Apipie
+
+## Rationale
+
+Swagger allows API authors to describe the structure of objects returned by REST API calls.
+Client authors and code generators can use such descriptions for various purposes, such as verification, 
+autocompletion, and so forth.
+
+The current Apipie DSL allows API authors to indicate returned error codes (using the `error` keyword), 
+but does not support descriptions of returned data objects. As such, swagger files
+generated from the DSL do not include those, and are somewhat limited in their value.
+
+This document proposes a minimalistic approach to extending the Apipie DSL to allow description of response
+objects, and including those descriptions in generated swagger files. 
+
+## Design Objectives
+
+* Full backward compatibility with the existing DSL
+* Minimal implementation effort
+* Enough expressiveness to support common use cases
+* Optional integration of the DSL with advanced JSON generators (such as Grape-Entity)   
+* Allowing developers to easily verify that actual responses match the response declarations   
+
+## Approach
+
+#### Add a `returns` keyword to the DSL, based on the existing `error` keyword
+
+Currently, returned error codes are indicated using the `error` keyword, for example:
+```ruby
+api :GET, "/users/:id", "Show user profile"
+error :code => 401, :desc => "Unauthorized"
+```
+
+The proposed approach is to add a `returns` keyword, that has the following syntax:
+```ruby
+returns <type-identifier> [, :code => <number>] [, :desc => <response-description>]
+```
+
+For example:
+```ruby
+api :GET, "/users/:id", "Show user profile"
+error :code => 401, :desc => "Unauthorized"
+returns :SomeTypeIdentifier  # :code is not specified, so it is assumed to be 200
+```
+
+
+#### Leverage `param_group` for response object description
+
+Apipie currently has a mechanism for describing complex objects using the `param_group` keyword.
+It seems reasonable to leverage this mechanism as the basis of the response object description mechanism,
+so that the `<type-identifier>` in the `returns` keyword will be the name of a param_group.
+
+For example:
+```ruby
+  def_param_group :user do
+    param :user, Hash, :desc => "User info", :required => true, :action_aware => true do
+      param_group :credentials
+      param :membership, ["standard","premium"], :desc => "User membership", :allow_nil => false
+    end
+  end
+
+  api :GET, "/users/:id", "Get user record"
+  returns :user, "the requested record"
+  error :code => 404, :desc => "no user with the specified id"
+```
+
+Implementation of this DSL extension would involve - as part of the implementation of the `returns` keyword - 
+the generation of a Apipie::ParamDescription object that has a Hash validator pointing to the param_group block.
+
+#### Extend action-aware functionality to include 'response-only' parameters
+
+In CRUD operations, it is common for `param_group` input definitions to be very similar to the 
+output of the API, with the exception of a very small number of fields (such as the `:id` field
+which usually appears in the response, but is not described in the `param_group` because it is passed as a 
+path parameter).
+
+To allow reuse of the `param_group`, it would be useful to its definition to describe parameters that are not passed 
+in the request but are returned in the response.  This would be implementing by extending the DSL to 
+support a `:only_in => :response` option on `param` definitions.  Similarly, params could be defined to be 
+`:only_in => :request` to indicate that they will not be included in the response.    
+
+For example:
+```ruby
+  # in the following group, the :id param is ignored in requests, but included in responses
+  def_param_group :user do
+    param :user, Hash, :desc => "User info", :required => true, :action_aware => true do
+      param :id, Integer, :only_in => :response
+      param :requested_id, Integer, :only_in => :request
+      param_group :credentials
+      param :membership, ["standard","premium"], :desc => "User membership", :allow_nil => false
+    end
+  end
+
+  api :GET, "/users/:id", "Get user record"
+  returns :user, :desc => "the requested record"  # includes the :id field, because this is a response
+  error :code => 404, :desc => "no user with the specified id"
+```
+
+
+#### Support `:array_of => <param_group-name>` in the `returns` keyword 
+
+Very often, a REST API call returns an array of some previously-defined object 
+(the most common example an `index` operation that returns an array of the same entity returned by a `show` request), 
+and it would be tedious to have to define a separate `param_group` for each one.
+
+For added convenience, the `returns` keyword will also support an `:array_of =>` construct
+to specify that an API call returns an array of some object type.
+
+For example:
+```ruby
+  api :GET, "/users", "Get all user records"
+  returns :array_of => :user, :desc => "the requested user records"
+
+  api :GET, "/user/:id", "Get a single user record"
+  returns :user, :desc => "the requested user record"
+```
+
+#### Integration with advanced JSON generators using an [adapter](https://en.wikipedia.org/wiki/Adapter_pattern) to `param_group`
+
+While it makes sense for the sake of simplicity to leverage the `param_group` construct to describe 
+returned objects, it is likely that many developers will prefer to unify the 
+description of the response with the actual generation of the JSON.
+
+Some JSON-generation libraries, such as [Grape-Entity](https://github.com/ruby-grape/grape-entity),
+provide a declarative interface for describing an object model, allowing both runtime
+generation of the response, as well as the ability to traverse the description to auto-generate
+documentation.
+
+Such libraries could be integrated with Apipie using adapters that wrap the library-specific
+object description and expose an API that includes a `params_ordered` method that behaves in a 
+similar manner to [`Apipie::HashValidator.params_ordered`](https://github.com/Apipie/apipie-rails/blob/cfb42198bc39b5b30d953ba5a8b523bafdb4f897/lib/apipie/validator.rb#L315).
+Such an adapter would make it possible to pass an externally-defined entity to the `returns` keyword
+as if it were a `param_group`.
+
+Such adapters can be created easily by having a class respond to `#describe_own_properties` 
+with an array of property description objects.  When such a class is specified as the 
+parameter to a `returns` declaration, Apipie would query the class for its properties
+by calling `<Class>#describe_own_properties`. 
+
+For example:
+```ruby
+# here is a class that can describe itself to Apipie
+class Animal
+  def self.describe_own_properties
+    [
+        Apipie::prop(:id, Integer, {:description => 'Name of pet', :required => false}),
+        Apipie::prop(:animal_type, 'string', {:description => 'Type of pet', :values => ["dog", "cat", "iguana", "kangaroo"]}),
+        Apipie::additional_properties(false)
+    ]
+  end
+  
+  attr_accessor :id
+  attr_accessor :animal_type  
+end
+
+# Here is an API defined as returning Animal objects.
+# Apipie creates an internal adapter by querying Animal#describe_own_properties
+api :GET, "/animals", "Get all records"
+returns :array_of => Animal, :desc => "the requested records"
+```
+
+The `#describe_own_properties` mechanism can also be used with reflection so that a
+class would query its own properties and populate the response to `#describe_own_properties`
+automatically.  See [this gist](https://gist.github.com/elasti-ron/ac145b2c85547487ca33e5216a69f527)  
+for an example of how Grape::Entity classes can automatically describe itself to Apipie
+
+#### Response validation
+
+The swagger definitions created by Apipie can be used to auto-generate clients that access the
+described APIs.  Those clients will break if the responses returned from the API do not match
+the declarations.  As such, it is very important to include unit tests that validate the actual
+responses against the swagger definitions.
+
+The ~~proposed~~ implemented mechanism provides two ways to include such validations in RSpec unit tests:
+manual (using an RSpec matcher) and automated (by injecting a test into the http operations 'get', 'post', 
+raising an error if there is no match).
+
+Example of the manual mechanism:
+
+```ruby
+require 'apipie/rspec/response_validation_helper'
+
+RSpec.describe MyController, :type => :controller, :show_in_doc => true do
+
+describe "GET stuff with response validation" do
+  render_views   # this makes sure the 'get' operation will actually
+                 # return the rendered view even though this is a Controller spec
+
+  it "does something" do
+    response = get :index, {format: :json}
+
+    # the following expectation will fail if the returned object
+    # does not match the 'returns' declaration in the Controller,
+    # or if there is no 'returns' declaration for the returned
+    # HTTP status code
+    expect(response).to match_declared_responses
+  end
+end
+```
+
+
+Example of the automated mechanism:
+```ruby
+require 'apipie/rspec/response_validation_helper'
+
+RSpec.describe MyController, :type => :controller, :show_in_doc => true do
+
+describe "GET stuff with response validation" do
+  render_views
+  auto_validate_rendered_views
+
+  it "does something" do
+    get :index, {format: :json}
+  end
+  it "does something else" do
+    get :another_index, {format: :json}
+  end
+end
+
+describe "GET stuff without response validation" do
+  it "does something" do
+    get :index, {format: :json}
+  end
+  it "does something else" do
+    get :another_index, {format: :json}
+  end
+end
+```
+
+Explanation of the implementation approach:
+
+The Apipie Swagger Generator is enhanced to allow extraction of the JSON schema of the response object
+for any controller#action[http-status].  When validation is required, the validator receives the 
+actual response object (along with information about the controller, action and http status code),
+queries the swagger generator to get the schema, and uses the json-schema validator (gem) to validate
+one against the other.
+
+Note that there is a slight complication here:  while supported by JSON-shema, the Swagger 2.0 
+specification does not support a mechanism to declare that fields in the response could be null.  
+As such, for a response that contains `null` fields, if the exact same schema used in the swagger def 
+is passed to the json-schema validator, the validation fails.  To work around this issue, when asked 
+to provide the schema for the purpose of response validation (i.e., not for inclusion in the swagger),
+the Apipie Swagger Generator creates a slightly modified schema which declares null values to be valid.
+