Thesis Master

UMA NOVA ESPECIFICAÇÃO PARA
CONSULTAR DADOS EM APIS BASEADAS EM
GRAFOS
GLEISON BRITO BATISTA
UMA NOVA ESPECIFICAÇÃO PARA
CONSULTAR DADOS EM APIS BASEADAS EM
GRAFOS
Projeto de tese apresentado ao Programa de

Pós-Graduação em Ciência da Computação
do Instituto de Ciências Exatas da Univer-
sidade Federal de Minas Gerais como req-
uisito parcial para a obtenção do grau de
Doutor em Ciência da Computação.
Orientador: Marco Túlio de Oliveira Valente
Belo Horizonte
Dezembro de 2018
GLEISON BRITO BATISTA
A NOVEL SPECIFICATION TO QUERYING DATA
IN GRAPH APIS
Thesis project presented to the Graduate

Program in Computer Science of the Fed-
eral University of Minas Gerais in partial
fulfillment of the requirements for the de-
gree of Doctor in Computer Science.
Advisor: Marco Túlio de Oliveira Valente
Belo Horizonte
December 2018
Abstract
The successful implementation of complex software systems heavily depends on the

quality and expertise of development teams. For this reason, there is currently a trend
of specialization in the skills and roles of software developers. For instance, medium-
to-large development teams normally include developers who are experts on specific
technologies and programming tasks. As a consequence, companies are struggling to
find qualified candidates to fulfil their open positions. In this thesis proposal, we first
define a set of models to predict and assess developers technical skills by mining their
activities on public source code repositories. Particularly, we propose an unsupervised
model to classify developers expertise on popular JavaScript frameworks and libraries.
Next, we investigate a machine learning approach to classify developers in four key
expertise areas: front-end, back-end, full-stack, and mobile development.
vii
List of Figures
2.1 Grey literature articles by year of appearance on Hacker News . . . . . . . 8

2.2 Number of comments and upvotes on Hacker News (for the articles included
in the grey literature review) . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Summary of grey literature findings . . . . . . . . . . . . . . . . . . . . . . 12
2.4 RQ3 results: number of API calls (REST vs GraphQL) per function . . . . 17
2.5 RQ4: Number of fields returned by API calls (REST vs GraphQL) per
function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.6 Number of fields returned by REST and GraphQL calls . . . . . . . . . . . 20
2.7 RQ5: Size of JSON documents returned by API calls (REST vs GraphQL)
per query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.8 Size of JSON documents returned by REST and GraphQL calls . . . . . . 23
3.1 Response to a REST query performed using a web browser . . . . . . . . . 39

3.2 Response to a GraphQL query performed using GraphiQL . . . . . . . . . 39
3.3 Time to conclude the tasks (REST vs GraphQL) . . . . . . . . . . . . . . 40
3.4 Time to implement the proposed tasks for querying GitHub data (REST vs
GraphQL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5 Warning message about missing type parameter . . . . . . . . . . . . . . . 42
3.6 Time to conclude the tasks, grouped by query type . . . . . . . . . . . . . 43
3.7 Time to conclude the tasks grouped by academic level . . . . . . . . . . . 46
3.8 Time to conclude the tasks (REST vs GraphQL) grouped by previous ex-
perience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
ix
List of Tables
2.1 Selected Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.2 Stats of Selected Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 REST Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4 Papers and queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.1 Experiment Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.2 Subjects’ experience in REST and GraphQL . . . . . . . . . . . . . . . . . 36
3.3 Tasks allocation among participants (Groups A and B) and between treat-
ments (REST and GraphQL) . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4 Group Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.5 Answer’s Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
xi
Contents
Abstract vii
List of Figures ix
List of Tables xi
1 Introduction 1
1.1 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 Migrating to GraphQL: A Practical Assessment 3

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 GraphQL in a Nutshell . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Grey Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3.1 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.4 Migration Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.4.1 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.5 Runtime Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.5.1 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.6 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.7 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3 REST vs GraphQL: A Controlled Experiment 29

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2.1 REpresentational State Transfer (REST) . . . . . . . . . . . . . 31
xiii
3.2.2 GraphQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3 Research Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3.1 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.3.2 Subjects Selection . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3.3 Experiment Design . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3.4 Experimental Procedure . . . . . . . . . . . . . . . . . . . . . . 38
3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.5.1 Why GraphQL requires less effort? . . . . . . . . . . . . . . . . 49
3.5.2 Can we improve REST results? . . . . . . . . . . . . . . . . . . 49
3.6 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.6.1 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.6.2 Internal Validity . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.6.3 External Validity . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.6.4 Conclusion Validity . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.7 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.7.1 Controlled Experiments . . . . . . . . . . . . . . . . . . . . . . 51
3.7.2 GraphQL Studies . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4 Next Steps 55
Bibliography 57
xiv
Chapter 1
Introduction
1.1 The Problem
1
Chapter 2
Migrating to GraphQL: A Practical

Assessment
2.1 Introduction
GraphQL is a novel query language for implementing Web-based APIs [Facebook Inc.,
2015]. Proposed by Facebook in 2016, the language represents an alternative to popu-
lar REST-based APIs [Fielding and Taylor, 2002, 2000; Fielding, 2000], shifting from
servers to clients the decision on the precise data returned by API calls. To illustrate
the usage of the language, suppose the REST API currently implemented by arXiv, the
popular preprint service maintained by Cornell University. This API includes a search
endpoint that allows clients to retrieve metadata about preprints with a given title.
The result is a complex and large JSON document, with at least 33 fields. However,
clients might need only a few ones (e.g., only the paper’s URL). Despite that, the men-
tioned endpoint returns all fields in a JSON document, which should be parsed by the
clients. After that, the unneeded fields are discarded, although they have consumed
server, client, and network resources. By contrast, suppose arXiv decides to support
GraphQL. Using the language, clients formulate a simple call like this one:
1 search ( title : " A Solution of the P versus NP Problem ") {
2 pdfUrl
3 }
By means of this query, the client request a single field (pdfUrl) of a preprint
entitled “A Solution of the P versus NP Problem”. The result is a JSON file with just
this URL. Therefore, instead of receiving a full document, with 33 fields, the client
receives exactly the single field it needs to process.
GraphQL is gaining momentum and it is now supported by important web ser-
3
4 Chapter 2. Migrating to GraphQL: A Practical Assessment
vices, as the ones provided by GitHub and Pinterest Hartig and Pérez [2018]. Despite
that, we have few studies investigating the real benefits of using GraphQL for im-
plementing Web-based APIs. Therefore, in this paper we ask the following research
questions: (RQ1) What are the key characteristics and benefits of GraphQL? (RQ2)
What are the main disadvantages of GraphQL? (RQ3) When using GraphQL, what is
the reduction in the number of API calls performed by clients? (RQ4) When using
GraphQL, what is the reduction in the number of fields of the documents returned
by servers? (RQ5) When using GraphQL, what is the reduction in the size of the
documents returned by servers? To answer RQ1 and RQ2, we conduct a grey liter-
ature review, covering 28 popular Web articles (mostly blog posts) about GraphQL.
Since the query language has just two years, we focus on grey literature, instead of
analysing scientific papers, as usually recommended for emerging technologies [Ogawa
and Malen, 1991; Garousi et al., 2016, 2017]. As a result, we confirmed two key charac-
teristics of GraphQL: (a) support to an hierarchical data model, which can contribute
to reduce the number of endpoints accessed by clients; (b) support to client-specific
queries, i.e., queries where clients only ask for the precise data they need to perform
a given task. Motivated by these findings, we also assess the benefits achieved by
GraphQL in terms of a reduction in the number of API calls (RQ3) and in the number
of fields returned by service providers (RQ4). To answer these questions, we manually
migrated five clients of the GitHub REST API to use the new GraphQL API provided
by GitHub. We also implemented a GraphQL wrapper for two endpoints of arXiv’s
REST API and migrated two open source clients to use this wrapper. Finally, to an-
swer RQ5, we reimplemented in GraphQL 14 queries used in seven recent empirical
software engineering papers, published at two major software engineering conferences
(ICSE and MSR).
Our contributions are twofold: (1) we reveal that GraphQL does not lead to a
reduction in the number of queries performed by API clients in order to perform a given
task, when compared to the number of required REST endpoints. For example, in our
migration study, we migrated 29 API calls that access REST endpoints (distributed
over seven clients) to 24 GraphQL queries, which therefore does not represent a major
reduction; (2) we reveal that client-specific queries can lead to a drastic reduction in
the size of JSON responses returned by API providers. On the median, in our study,
JSON responses have 93.5 fields, against only 5.5 fields after migration to GraphQL,
which represents a reduction of 94%. In terms of bytes, we also measure an impres-
sive reduction: from 9.8 MB (REST) to 86 KB (GraphQL). Altogether, our findings
suggest that API providers should seriously consider the adoption of GraphQL. We
also see space for tool builders and researchers, with interest on providing support and
2.2. GraphQL in a Nutshell 5
improving the state-of-the-practice on GraphQL-based API development.

The rest of this paper has seven sections. Section 2.2 provides a brief introduction
to GraphQL. Section 2.3 describes a grey literature review, covering popular Web
articles about GraphQL. Section 2.4 presents the migration study conducted to answer
RQ3 and RQ4. Section 2.5 describes a study to evaluate the runtime gains achieved by
GraphQL (and therefore answer RQ5). Threats to validity are discussed in Section 3.6;
and related work is discussed in Section 2.7. Section 3.8 concludes the paper and
summarizes lessons learned.
2.2 GraphQL in a Nutshell

This section introduces the key concepts of GraphQL. The goal is to make this paper
self-contained; for a detailed and throughout presentation of the language we refer
the reader to its documentation [Facebook Inc., 2015]. Essentially, GraphQL allows
clients to query a database represented by a schema. Therefore, this design represents
a major shift from REST APIs: in REST, server applications implement a list of
endpoints (operations) that can be called by clients; by contrast, in GraphQL servers
export a database, which can be queried by clients. A GraphQL database is defined
by a schema, which is a multi-graph [Hartig and Pérez, 2017]. In this multi-graph,
nodes are objects, which define a type and include a list of fields; each field has a name
and also a type. Edges appear when objects define fields whose types are other object
types.1 GraphQL proposes a simple DSL for defining schemas. To illustrate the usage
of this language, we use a simple blogging system, with two object types: Post and
Author. As presented in Listing 2.1, object types are defined using the keyword type.
In this example, Post has four fields: id, author, title, and body. The first field is a
non-null String (the ! after the type discards null values). The second field (author)
references another object type in the schema, called Author (lines 8-12). The remaining
two fields in Post have a String type. Finally, schemas usually include a predefined
type, called Query, which represents the entry point of GraphQL APIs (lines 14-16). A
Query object exposes the object types that can be queried by clients and the arguments
that must be provided by them. For example, post (line 15) is a query that accepts a
non-null string as argument and returns the Post object having this string as id.
1 type Post {
2 id : String !
3 author : Author
1
Since an object T1 can have multiple fields of type T2 , multiple edges can connect T1 to T2 ,
leading to a multi-graph.
4 title : String
5 body : String
6 }
7
8 type Author {
9 id : String !
10 name : String
11 email : String
12 }
13
14 type Query {
15 post ( id : String !) : Post
16 }
Listing 2.1: Schema with two types (Post and Author) and a Query end-point
GraphQL also defines a query language, used by clients. Listing 2.2 shows three
examples of queries in this language. In the first query (PostByTitle), the client asks
for the Post object with id equals to 1000; specifically, the client only requests the
title field of this object. The second query (PostByTitleAndBody) is similar, but
in this case the client asks for two fields, title and body. Finally, the last query
(PostByTitleAndAuthor) asks for the title and author of the same Post. Since
author is another object, we have to specify its queried fields, in this case only name.
The result of this third query is presented in Listing 2.3. As we can see, the result is
a JSON object, which should be parsed and possibly deserialized by clients.
1 query PostByTitle {
2 post ( id :"1000") {
3 title
4 }
5 }
6 query PostByTitleAndBody {
7 post ( id :"1000") {
8 title
9 body
10 }
11 }
12 query P ostByTitleAndAuthor {
13 post ( id :"1000") {
14 title
15 author {
16 name
17 }
18 }
19 }
Listing 2.2: Querying distinct data from a given Post object
1 { " data ": {

2 " post ":{
2.3. Grey Literature Review 7
3 " title ": " GraphQL : A data query language "

4 " author ":{
5 " name ": " Lee Byron "
6 }
7 }
8 }
9 }
Listing 2.3: JSON object returned by PostByTitleAndAuthor query
To respond to queries, the developer of a GraphQL server must implement a re-

solver function for each query declared in the Query type. These functions are called
each time the GraphQL server engine needs to retrieve an object type specified in a
query. Typically, these functions retrieve these objects from an underlying data struc-
ture, which can be any kind of database (relational, non-relational, in-memory, etc).
Finally, it is also possible to define another predefined type in schemas, called Mutation,
which is used to insert new objects on the server’s database or modify existing ones.
Listing 2.4 shows an example that defines an addPost mutation, which receives a Post
object (and returns the object, just to confirm the operation has been successfully ex-
ecuted). Each endpoint (operation) in a Mutation type must have a resolver function,
which implements the operation. In our running example, this function must insert
the Post object received as argument in the underlying database.
1 type Mutation {
2 addPost ( post : Post ) : Post
3 }
Listing 2.4: Mutation operation for Post objects
2.3 Grey Literature Review

This section reports the results of a systematic analysis of the grey literature about
GraphQL, covering documents and discussions on blogs, tutorials, and similar Web
articles. Our goal is to better understand the key characteristics, benefits, and short-
comings appointed by practitioners who had a real experience with the language. Since
it is a new technology, papers about GraphQL are not common in the scientific lit-
erature. Therefore, for such emerging technologies, a grey literature tends to provide
a better coverage of relevant documents than a traditional literature review [Garousi
et al., 2016; Barik et al., 2015].2
2
Nevertheless, peer-reviewed articles are also discussed in this paper, but in Section 2.7 (Related
Work).
2.3.1 Study Design

To retrieve an initial list of Web articles considered in this review, we used Hacker
News,3 which is a news aggregator site widely used by practitioners [Aniche et al.,
2018]. Recently, other similar reviews have used Hacker News as data source, e.g., a
grey literature review on clouding computing services [Leitner et al., 2018]. To retrieve
Hacker News documents, we used the Algolia search engine,4 querying for posts con-
taining graphql in their titles, as in September, 2018. We found 1,242 articles. We
then sequentially removed articles that do not include a valid URL (286 articles), that
do not contain comments (760 articles), or that are just promoting a tool or project
(168 articles). After this filtering step, we selected 28 articles for analysis (1,242 - 286
- 760 - 168), which we refer as A1 to A28.5 Figure 2.1 shows the year when these
articles appeared on Hacker News. We can see an increase in the interest on GraphQL
in the last four years. Interestingly, we found four articles in 2015, i.e; before GraphQL
official release. Figure 2.2 shows violion plots with the distribution of the number of
comments and upvotes of the 28 articles. The median number of comments is 9.5; and
the median number of upvotes is 53 (which is usually enough to put the article in the
front page of Hacker News).
2018 9
2017 10
2016 5
2015 4
0 2 4 6 8 10
Articles
Figure 2.1: Grey literature articles by year of appearance on Hacker News
After collecting the articles, the first author of this paper carefully read them and
followed an open coding protocol to provide answers to the first two research questions:
RQ1: What are the characteristics and benefits of GraphQL?
RQ2: What are the main disadvantages of GraphQL?
3
https://news.ycombinator.com
4
https://hn.algolia.com
5
Detailed information at https://github.com/gleisonbt/migrating-to-graphql
2.3. Grey Literature Review 9
9.5 53
0 30 60 90 0 100 200
Comments Upvotes
Figure 2.2: Number of comments and upvotes on Hacker News (for the articles included
in the grey literature review)
2.3.2 Results
RQ1: Key Characteristics and Benefits
GraphQL is strongly typed, since all objects and fields have types (as mentioned in A1,
A2, A5, A6, A10, and A28). This contributes to better tooling support, as reported in
this article:
GraphQL is strongly-typed. Given a query, tooling can ensure that the query is syntacti-
cally correct and valid within the GraphQL type system before execution. (A5)
A related benefit is the possibility of having better error messages, e.g., [types]
allow GraphQL to provide descriptive error messages before executing a query (A28).
GraphQL enables client-specified queries, as mentioned by almost half of the articles

(A1, A2, A5, A6, A10, A14, A17, A19, A21, A25, A26, and A27). The following article
nicely describes this characteristic:
In GraphQL, the specification for queries are encoded in the client rather than the server.
These queries are specified at field-level granularity. In the vast majority of applications
written without GraphQL, the server determines the data returned in its various scripted
endpoints. A GraphQL query, on the other hand, returns exactly what a client asks for
and no more. (A5)
This characteristic makes GraphQL particularly interesting for mobile applica-

tions, which often face limited bandwidth and speed (A4, A5, A14, and A26). It also
moves the focus of development to client apps, where designers and developers spend
their time and attention (A1). Finally, client-specific queries allow servers to better
understand the needs of clients (A9, A12) and therefore improve the quality of their
service:
It’s great for service operators too, because its explicitness allows them to get a better
understanding of exactly what their users are trying to do. (A9)
GraphQL data model is hierarchical, as mentioned in five articles (A1, A2, A3, A5, and
A8) and defined as:
As its name would suggest, GraphQL models objects as a graph. Technically, the graph
starts with a root node that branches into query and mutation nodes, which then descend
into API-specific resources. (A2).
This characteristic allows clients to retrieve data from multiple sources (or end-
points) in a single request, therefore acting as gateways for different APIs (A3, A4,
A14, and A20):
GraphQL makes it easy to combine multiple APIs into one, so you can implement different
parts of your schema as independent services. (A20)
Introspection, which allows clients to inspect the types and fields defined in a schema,
at runtime (A1, A3, A9, A16, and A28). Combined with a static type system,
introspection allows clients to learn and explore an API quickly without grepping the
codebase or wrangling with cURL. (A28). It also frees servers to support an interface
description language, which are usually not available for REST; and when available
they are often not completely accurate because the description is not tied directly to
the implementation (A28).
Deprecation: As common in mainstream programming languages, it is possible to

deprecate fields, using a @deprecated annotation (A1, A2 and A19). However, in
GraphQL, new fields added to a type do not lead to breaking changes (as in standard
APIs [Brito et al., 2018; Xavier et al., 2017]); and deprecations can be restricted to
deleted fields. As a result, the pressure for versioning is less frequent, as mentioned in
this article:
This process removes the need for incrementing version numbers. We still support three
years of released Facebook applications on the same version of our GraphQL API. (A1)
2.4. Migration Study 11
RQ2: Disadvantages
GraphQL does not support information hiding. GraphQL does not support private
fields, i.e., all fields are visible to client applications (A8, A11, A18, A20 and A24).
Furthermore, according to A18, GraphQL queries tend to be more complex to imple-
ment, since they require a detailed understanding of the data schema, which can be a
time-consumming task in large APIs:
By design, a developer who integrates against GraphQL needs to know the names of the
fields to access, their relation to other objects and when to retrieve them. (A18)
Complex caching: In GraphQL, each query can be different, even though operating on
the same type. This demands more sophisticated server-side caching, as mentioned in
this article:
GraphQL does not follow the HTTP specification for caching and instead uses a single end-
point. Thus, it’s up to the developer to ensure caching is implemented correctly . . . (A20)
Performance: GraphQL servers can have to process complex queries (e.g., queries
with deep nesting) that can consume server resources (A8, A11, A20, A23, and A25),
as mentioned in the following article:
Great care has to be taken to ensure GraphQL queries don’t result in expensive join queries
that can bring down server performance or even DDoS the server. (A20)
Figure 3 summarizes the grey literature review results, by presenting the key
characteristics, benefits, and disadvantages of GraphQL, and the number of articles
mentioning them.
2.4 Migration Study

With this second study, we aim to quantitatively evaluate two key characteristics
associated to GraphQL in the grey literature: (1) clients can precisely request the
data they need from servers (due to the support to client-specific queries) (2) clients
rely on a single endpoint to retrieve the data they need (due to a hierarchical data
model). In the study, we migrate seven client applications based on REST APIs to
Key Characteristics
Client−specified
12
queries
Strongly
6
typed
Hierarquical
5
data model
Introspection 5
Deprecation 3
Articles
Benefits
Recommended to
4
mobile applications
Multiple sources in
4
a single request
Tooling
1
support
Less preassure
1
to versioning
Better error
1
messages
Articles
Disadvantages
Performance 5
No Information
5
hiding
Complex
1
caching
0 2 4 6 8 10 12
Articles
Figure 2.3: Summary of grey literature findings
GraphQL. Then, we assess the gains achieved by the GraphQL version. Specifically,
we answer two research questions:
RQ3: When using GraphQL, what is the reduction in the number of API calls
performed by clients? GraphQL clients normally implement a single query to retrieve
all data they need to perform a given task; by contrast, when using REST, clients
frequently have to access multiple endpoints. Therefore, in this RQ, we compare the
number of endpoints accessed by REST clients with the number of endpoints accessed
by the same clients after refactored to use GraphQL.
RQ4: When using GraphQL, what is the reduction in the number of fields of the JSON
documents returned by servers? In GraphQL, client-specific queries allow developers
to inform precisely the fields they need from servers. Therefore, in this RQ, we com-
pare the number of fields in the following JSON documents: (a) returned by servers
when responding to requests performed by REST clients; (b) returned by servers when
responding to queries performed by the same clients but after being migrated to use
GraphQL.
2.4.1 Study Design

Selected APIs: First, to answer the proposed research questions, we selected the
APIs provided by two widely popular services: GitHub and arXiv.6 GitHub is an
interesting case study because the system provides both REST and GraphQL APIs
(the latter since 2016). Moreover, GitHub’s GraphQL API is quite complete and
large, including 120 object types, 21 queries, and 62 mutations. By contrast, arXiv
only provides a REST API. Therefore, we implemented ourselves a small GraphQL
API for the system, in the form of a wrapper for the original API. This wrapper
supports two queries (getPreprint and search), as presented in Listing 2.5. The first
query (getPreprint) returns metadata about a preprint, given its ID. This metadata
includes the paper’s title, authors, DOI, summary, URL, etc (see Listing 2.6). The
second query searches for preprints whose title match a given string; it is also possible
to define the maximal number of results the query should return, the first result that
should be returned and the sort order. The implemented GraphQL wrapper was
installed in a private server, in our research lab.
1 type Query {
2 getPreprint ( id : ID !) : Preprint
3 search ( query : String ! , maxResults : Int ! ,
4 start : Int ! , sortBy : String ,
5 sortOrder : String ) : [ Preprint ]
6 }
Listing 2.5: arXiv Query
1 type Preprint {
2 id : ID
3 pdfUrl : String
4 published : String
5 arxivComment : String
6 title : String
7 authors : [ String ]
8 arxivUrl : String
9 doi : String
10 tags : [ Tag ]
11 a rxivPrimaryCategory : ArxivPrimaryCategory
12 updated : String
13 summary : String
6
https://arxiv.org
14 }
Listing 2.6: Preprint type
Selected Clients: When searching for GitHub API clients, we first found that they
usually have the tag (or topic) GitHub. Therefore, we selected five projects with
this tag and that have at least 100 stars, as described in Table 2.1. In the case
of arXiv, we selected two clients mentioned in the project’s page7 and that have
their source code publicly available on GitHub (see also their names in Table 2.1).
Table 2.2 shows information about the programming language, number of stars, size
(in lines of code), and contributors of the selected systems. The smallest project
is bibcure/arxivcheck (131 LOC, one contributor, and five stars); the largest
projects are vdaubry/github-awards (35,153 LOC, 15 contributors, and 1,296
stars) and donnemartin/gitsome (17,273 LOC, 24 contributors and 5,913 stars).
Table 2.1: Selected Projects
Project Description
donnemartin/viz Visualization of GitHub repositories
donnemartin/gitsome Command line interface for GitHub
csurfer/gitsuggest A tool to suggest GitHub repositories
guyzmo/git-repo Command line interface to manage Git services
vdaubry/github-awards Ranking of GitHub repositories
bibcure/arxivcheck A tool to generate BibTex of arXiv preprints
karpathy/
Web interface for searching arXiv submissions
arxiv-sanity-preserver
Table 2.2: Stats of Selected Projects
Project Lang. Stars LOC Contrib

donnemartin/viz Python 627 9,556 1
donnemartin/gitsome Python 5,913 17,273 24
csurfer/gitsuggest Python 613 389 2
guyzmo/git-repo Python 764 5,602 17
vdaubry/github-awards Ruby 1,296 35,153 15
bibcure/arxivcheck Python 5 131 1
karpathy/
Python 2,322 2,431 19
7
https://arxiv.org/help/api/index
Migration Step: After selecting the APIs and client projects, the paper’s first author
exhaustively searched the code of each client looking for REST calls. He then mi-
grated each one to use GraphQL. Just to show one example of migration, in csurfer-
/gitsuggest the following REST endpoint is used to search GitHub for repositories
matching a given string:
1 GET / search / repositories
This endpoint requires three parameters: q (a string with the search keywords),
sort (the sort field, e.g., stars), and the order (asc or desc).8 The request returns
a JSON document with 94 fields, containing data about a repository. However, only
three fields are used by csurfer/gitsuggest: owner’s login, description, and stargaz-
ers_count. Therefore, we changed the function that implements the search call to
use the following GraphQL query, which retrieves exactly the three fields used by
csurfer/gitsuggest:
1 query searchRepos {
2 search ( query : $query , type : REPOSITORY , first : 100) {
3 nodes {
4 ... on Repository {
5 nameWithOwner
6 description
7 stargazers {
8 totalCount
9 }
10 }
11 }
12 }
13 }
Listing 2.7: Example of GraphQL query (csurfer/gitsuggest)
In Listing 2.7, the search query returns an union type, which might be either
a Repository, User, or an Issue type, depending on the type argument. We use a
feature of GraphQL called inline fragments to access only the fields of the Repository
variant type. This variant is labeled as ...onRepository (line 4). Therefore, in this
case one REST endpoint is replaced by one GraphQL query (RQ3’s answer) and 91
fields (= 94 − 3) are retrieved but not used by the REST code (RQ4’s answer).
In total, the first author migrated 29 REST endpoint calls—distributed over the
seven projects (see Table 2.3)—to use GraphQL queries. For the sake of legibility,
we use labels F1 to F22 to refer to the functions including these REST calls (instead
of the functions’ original names). This migration effort consumed around 60 work-
ing hours (of the paper’s first author), including the time to understand the clients code.
8
https://developer.github.com/v3/search/
Table 2.3: REST Calls
Project Func REST endpoints

GET /users/:user
GET /users/:user/starred
F1
csurfer/gitsuggest GET /users/:user/following
GET /users/:user/starred
F2 GET /search/repositories
F3 GET /users/:user/followers
F4 GET /users/:user/following
F5 GET /repos/:owner/:repo/issues
GET /users/:user/repos
F6
GET /repos/:owner/:repo/pulls
donnemartin/gitsome
F7 GET /users/:user/repos
F8 GET /search/issues
F10 GET /users/:user/starred
GET /users/:user
F11
F12 GET /users/:user/repos
F13 GET /users/:user/gists
GET /repos/:owner/:repo
guyzmo/git-repo F14
GET /repos/:owner/:repo/pulls
F15 GET /repos/:owner/:repo
F17 GET /users/:user
donnemartin/viz
vdaubry/github-awards
GET /users/:user
F20
bibcure/arxivcheck F21 GET /query/:search_query
karpathy/
F22 GET /query/:search_query
Number of JSON fields: To answer RQ4, we have to compute the number of fields
returned by the original API calls (performed using REST) and by the migrated calls
REST GraphQL
4
4
3
Calls
2 2 2 2 2 2
2
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1
0
F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14 F15 F16 F17 F18 F19 F20 F21 F22
Functions
Figure 2.4: RQ3 results: number of API calls (REST vs GraphQL) per function
(using GraphQL). First, it is important to highlight that we only count root nodes,
i.e., the ones that have a primitive value associated to them, instead of referring to
another JSON entry. Second, when the returned fields are lists, we only consider a
single list element. For example, Listing 2.8 shows a JSON object that contains a
list of users followed by a given GitHub user. The list contains three nodes elements,
delimited by square brackets (lines 5-7). Each node contains only one root field called
name. Therefore, we consider that the JSON document in Listing 2.8 has only one field
(which appears three times). Essentially, we followed this strategy to allow computing
the number of fields in each document without having to define a synthetic load for
executing the systems, which is not a simple task. Instead, we executed the systems
with a trivial load and input, which is sufficient for counting the number of unique root
nodes, without considering their number. We leave a more detailed evaluation of the
runtime gains achieved with GraphQL to Section 2.5.
1 { " data ": {

2 " user ": {
3 " following ": {
4 " nodes ": [
5 { " name ": " user_1 " , } ,
6 { " name ": " user_2 " , } ,
7 { " name ": " user_3 " , }
8 ]
9 }
10 }
11 }
12 }
Listing 2.8: JSON document with a single root field (name)

REST GraphQL
419
399
400
300
Fields
235
200
124 113 113 113 124
100 94 93 94 93 93 94
49 41 41 31 33 33
8 18 18 9 9 9 9 9 16 12 5 5 8 6 10
3 1 1 3 2 3 2 1 3
0
F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14 F15 F16 F17 F18 F19 F20 F21 F22
Functions
Figure 2.5: RQ4: Number of fields returned by API calls (REST vs GraphQL) per
function
2.4.2 Results
RQ3: What is the reduction in the number of API calls?
The 29 REST calls migrated in the study are implemented in 22 functions (see
Table 2.3). For each function (identified by F1 to F22), Figure 2.4 shows the number
of REST calls performed in the original code and the number of GraphQL queries
implemented in the migrated code. As we can see, in 17 functions (77%), there is a
single REST call, which was therefore migrated to a single GraphQL query. In another
function (F20), the two existing REST calls were migrated to two GraphQL calls. In
only four functions (F1, F6, F11 and F14), there is a reduction in the number of REST
calls. The highest reduction was observed in F1, where four REST calls were replaced
by a single GraphQL query. In this case, the REST calls retrieve the repositories
starred by a user and by the users he/she follows; they were replaced by the following
semantically equivalent GraphQL query:
1 query interestingRepos ( $username : String !) {
2 user ( login : $username ) {
3 starredRepositories {
4 nodes { ... }
5 }
6 following ( first : 100) {
7 nodes {
8 starredRepositories {
9 nodes { ...}
10 }
11 }
12 }
13 }
14 }
Listing 2.9: Query that returns the repositories starred by an user (lines
3-5) and by the users he/she follows (lines 6-12)
Interestingly, in a single function (F17), there is an increase in the number of

API calls after migrating to GraphQL. This function retrieves data about GitHub
users; however, the required data depends on whether the user has an individual or
an organizational account. In the REST API, there is a single endpoint that returns
the whole set of fields about GitHub users, despite their account type. By contrast, in
the GraphQL API, data about users is spread over three types: User, Organization,
and Actor. The migrated code first queries Actor to retrieve the user’s category.
Depending on the result, a second query targets User or Organization.
RQ3’s summary: The support to an hierarchical data model is a key characteristic

of GraphQL, since it allows clients to retrieve data from multiple endpoints in a
single request. However, in our migration study, we found very few opportunities
to implement such queries. The reason is that most client functions access a single
REST endpoint; the straightforward migration strategy is therefore to replace such
calls by a single GraphQL query. Typically, client functions perform simple tasks,
which reduces the demand for queries returning complex and nested data structures.
RQ4: What is the reduction in the number of JSON fields?
Figure 2.5 shows the number of unique root fields in the JSON documents
returned by the original REST calls and by the same calls migrated to GraphQL. As
we can see, in almost all calls there is a major decrease in the number of returned
fields when using GraphQL (and therefore client-specific queries). This reduction
ranges from 17 fields (F3 and F4) to 416 fields (F14). Particularly, F14 is a function
that returns data about the pull requests of a given repository. In the original code,
the function relies on two REST endpoints to perform this task. The first endpoint
returns all fields about the repository of interest. However, F14 consumes only the
pulls_url field. Then, for each pull request returned by the second endpoint, F14
uses only three fields (number, title, and html_url ); these are precisely the fields
returned by the GraphQL query. Figure 2.6 shows violin plots with the distribution
of the number JSON fields returned by REST and GraphQL. The REST calls return
93.5 fields (median values), against only 5.5 after migration to GraphQL. The 1st
quartile measures are 41 (REST) and 3 (GraphQL); the 3rd quartiles are 113 (REST)
and 9 (GraphQL).
REST 93.5
GraphQL 5.5
1 10 100
Fields (log10)
Figure 2.6: Number of fields returned by REST and GraphQL calls
REST GraphQL
517361.1 542941.4 402692.1

97609.5
105
Size (KB) (log10)
49660.2
27083
9712.3 14302.3
7794.4 5449.5 8009 5370.8
4626.4
863.5
103 501.8
205.2 135.6
94.9 122.4
77.8 62.2
43.3
26.8
101 3.9
1.7 1.2
0.5 1.1
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14

Queries
Figure 2.7: RQ5: Size of JSON documents returned by API calls (REST vs GraphQL)
per query
RQ4’s summary: When using REST, clients need to process large JSON documents
to consume just a few fields, which is often called over-fetch (A8 and A5). By contrast,
when using GraphQL, clients specify exactly the fields they need from servers. In our
study, there is a reduction from 93.5 to 5.5 in the number of JSON fields returned by
REST endpoints when compared to equivalent GraphQL queries.
2.5 Runtime Evaluation

In the previous section, we evaluated the number of unique fields returned by
GraphQL-based APIs. However, the results were based on executing the systems with
trivial input data. A more realistic execution requires the definition of a representative
sample of users, repositories, preprints, etc; which is not trivial. For example, some
systems retrieve the list of followers of a given user. The size of this list depends on
the selected users (most GitHub users have few followers, but others have thousands
2.5. Runtime Evaluation 21
of followers). Therefore, we postponed an evaluation of the runtime gains achieved by

GraphQL to this section, where we ask this research question:
RQ5: When using GraphQL, what is the reduction in the size (in bytes) of the JSON
documents returned by servers?
2.5.1 Study Design

To answer RQ5, we abandoned the idea of defining a realistic sample of input data
for the migrated systems. Instead, we rely on real and precisely defined queries used
on recent empirical software engineering papers. Typically, these papers depend on a
dataset to evaluate their object of study. Therefore, we first retrieve a list of papers
published in two recent and relevant software engineering conferences: International
Conference on Software Engineering (ICSE, 2017 edition) and Mining Software Repos-
itories Conference (MSR, 2017 edition). Then, we selected three papers from ICSE
and four papers from MSR that rely on GitHub to create a dataset with data about
open source projects. The advantage is that these papers carefully describe the cri-
teria they use to select the projects and the data (fields) they collect for each one.
For example, a paper by Floyd et. al Floyd et al. [2017] selects the top 100 projects
whose main language is C. For each project, they collect the full name and the 1,000
most recent pull requests. For each pull request, they retrieve the number of edited
files and the comments (we considered the pull requests . . . with at most two edited files
. . . [and] non-empty developer comments Floyd et al. [2017]). In other words, the paper
precisely specifies the amount of data retrieved from GitHub (100 C projects, 1,000
PRs per project, etc).
After selecting the papers, we carefully implemented queries to collect the
datasets, first using GitHub’s REST API and then using the GraphQL API. The
GraphQL queries retrieve only the data used in the papers. In total, we reimplemented
14 queries (denoted by Q1 to Q14, see Table 2.4), which are used by seven papers. Fi-
nally, we executed the queries and computed the size in bytes of the returned JSON
documents.
2.5.2 Results
Figure 2.7 shows the size of the JSON documents returned by the selected queries,
when implemented in REST and GraphQL. In almost all queries, there is a drastic
difference after migrating to GraphQL. For example, when using REST, Q11 returns
JSON documents that add up to almost 400 MB; when running the same query in
Table 2.4: Papers and queries
Paper Query Data

Q1 Name of the top-100 C projects by stars
Floyd et al. [2017] For each project: total number and body of the 1K most
Q2
recent PRs
Q3 For each PR: body of comments
Name and URL of the top-5 projects by stars (in any
Xiong et al. [2017] Q4
programming language)
For seven projects: number of commits, branches, bugs,
Q5
releases and contributors
Ma et al. [2017]
Q6 For each project: title, body of closed bugs
Q7 For each closed bug: body of comments
Name and URL of Java projects created before Jan,
Osman et al. [2017] Q8
2012, with 10+ stars, and 1+ commits
Zampetti et al. [2017] Q9 Number of stars of specific projects
Q10 Name of repositories with at least 1K stars
Macho et al. [2017]
Q11 Number of commits in a repository
For eight projects: number of releases, stars, and lan-
Q12
guage
Wan et al. [2017]
Title, body, date and project name of open issues tagged
Q13
with a bug tag
Q14 For each issue: body of comments
GraphQL the size of the answer drops to 77 KB. This query counts the number of
commits in a repository. In GraphQL, lists have a field called totalCount that returns
their size (this field is similar to a COUNT function in SQL, for example). Therefore,
it is straightforward to recover the total number of commits in the master branch of a
given repository, using GraphQL, as presented in the following listing:
1 query totalCountCommits ( $owner : String ! , $name : String !) {
2 repository ( owner : $owner , name : $name ) {
3 ref ( qualifiedName : " master ") {
4 target {
5 ... on Commit {
6 history {
7 totalCount
8 }
9 }
10 }
11 }
12 }
13 }
2.5. Runtime Evaluation 23
REST 9875.9
GraphQL 86.35
101 103 105

Size (KB) (log10)
Figure 2.8: Size of JSON documents returned by REST and GraphQL calls
By contrast, using the REST API, the client needs to receive data about all
commits and then locally compute the number of commits. The totalCount field also
explains the reduction in the size of the JSON responses in queries Q5 (from 5.4 MB
to 1.7 KB), Q12 (from 62.2 to 1.1 KB), and Q13 (from 14 MB to 205 KB). In the
remaining queries, the papers only need a small subset of the fields in the returned
documents. For example, in Q1 only the repositories’ names are needed; the remaining
fields are discarded. Figure 2.8 shows violin plots with the distribution of the size of
the JSON documents returned by REST and GraphQL. The REST responses have
around 9.8 MB (median values), against only 86 KB after moving to GraphQL. The
1st quartile measures are 1.5 MB (REST) and 2.2 KB (GraphQL); the 3rd quartiles
are 85 MB (REST) and 699 KB (GraphQL)
RQ5’s summary: When comparing the size of the JSON documents returned by
REST and GraphQL calls—implemented to reproduce queries performed in recent
empirical software engineering papers—we observed a major difference, from 9.8 MB
(REST) to 86 KB (GraphQL), on the median; which represents a reduction of 99%.
As in RQ4, this difference happens due to the over-fetching problem typical of REST
clients, which receive several fields they do not need at all. This problem is amplified
in queries that only need to compute the number of elements in lists of commits,
releases, and branches.
2.6 Threats to Validity
In this section, we present threats to validity, separated by the three studies conducted
in this paper.
Grey Literature Review: In this first study, we only review articles discussed on
Hacker News. Although it is a popular technology news aggregator, we might have
missed interesting articles that did not appear on this site. Further, the open coding
protocol to identify key characteristics, advantages, and disadvantages of GraphQL
was performed by a single paper’s author. Therefore, he might have missed important
codes or incorrectly classified some of the articles discussions. However, this threat is
minimized by two facts. First, because the number of reviewed articles is not high (28
articles). Second, because the classification was partially reviewed and checked by the
paper’s third author.
Migration Study: First, the study is based on seven clients, of two APIs, which
therefore should be considered before generalizing the presented results. Second, the
GraphQL wrapper for arXiv’s API cover only two endpoints. Finally, the migration
from REST to GraphQL was manually performed by one of the paper’s author and
it is therefore error-prone. To minimize this threat, we performed functional tests
in all systems, after migration to guarantee their behavior. We are also making the
source code publicly available, to allow inspection, replication, and testing by other
researchers and by practitioners.
Runtime Evaluation: In this study, we consider two software engineering conferences:

a general conference (ICSE) and a topic-specific conference (MSR), whose papers nor-
mally depend on large datasets. Furthermore, the queries documented in these papers
might not be representative of real data retrieved by software applications. In fact,
since the studied papers depend on large datasets, the amount of data consumed by
them would probably compare with the data retrieved by an application over days or
weeks. Finally, we reimplemented (and not reused the code) of the queries, which is
an error-prone task. Particularly, in the case of the GraphQL queries, we had to define
exactly the data (fields) used in the papers, which is also subjected to errors and (in
some cases) subjective interpretation. To reduce this threat, this task was performed
by two authors, who read the papers independently and them discussed together the
data effectively consumed by them.
2.7. Related Work 25
2.7 Related Work

Research on simple and easy-to-use programmatic interfaces to computer systems dates
from the 70s. For example, Query by Example (QBE) [Zloof, 1977] was proposed in
mid-1970s to facilitate writing queries to database systems. QBE allows users to spec-
ify the fields they want to recover from a relational database, by filling a template form
and therefore without having to write SQL code. To some extent, GraphQL shares the
same goals of QBE, but putting less emphasis on the presence of a graphical interface
to formulate the queries. Tuple spaces—as proposed by Gelernter [1985]; Carriero and
Gelernter [1989], in the 80s—is another data structure to facilitate the access to a com-
puter system by distributed and parallel clients. When using Linda, clients communi-
cate by inserting (out), reading (rd), or removing (in) ordered sequences of data, called
tuples, from a centralized data structured (the tuple space). Clients perform queries
(in or rd) by means of a template, where wild cards designate any value. However,
unlike supported by GraphQL, all fields are returned when a matching tuple is found
in the server. In the early 2000s, REST (REpresentational State Transfer) Fielding
and Taylor [2002, 2000]; Fielding [2000] was proposed as a set of principles and archi-
tectural styles for implementing APIs based on Web standards and protocols, such as
HTTP and URIs. For example, in REST-based architectures, all resources have URIs
and communication is fully stateless. Due to its flexibility, robustness, and scalability,
REST is largely used by major Internet companies to implement Web-based APIs.
However, REST interfaces—in order to reduce the need of frequent access by clients—
tend to rapidly become coarse-grained services. As a result, clients tend to receive
superfluous data as a result of REST calls. This problem—called over-fetching—was
the main motivation for GraphQL design.
Despite its recent popularity, GraphQL is an understudied topic in the scientific
literature. In a workshop paper, Hartig and Perez were one of the first to study and
provide a formal definition for GraphQL [Hartig and Pérez, 2017]. Later, they comple-
mented and finished this formalization in a conference paper [Hartig and Pérez, 2018].
In this second paper, the authors also prove that evaluating the complexity of GraphQL
queries is a NL-problem (i.e., a decision problem that can be solved by a nondetermin-
istic Turing machine under a logarithmic amount of memory). In practical terms, this
result shows that it is possible to implement efficient algorithms to estimate the com-
plexity of GraphQL queries before their execution; which is important for example to
handle the performance problems normally associated to GraphQL (as reported in our
grey literature review) and particularly to avoid denial-of-service attacks. Vogel et al.
[2017] present a case study on migrating to GraphQL part of the API provided by a
smart home management system. They report the runtime performance of two end-
points after migration to GraphQL. For the first endpoint, the gain was not relevant;
but in the second one GraphQL required 46% of the time required by the original
REST API. The authors also comment that parallel operation of REST and GraphQL
services is possible without restrictions. Romano et al. [2014] propose a genetic algo-
rithm for refactoring “fat interfaces”, i.e., coarse-grained interfaces whose clients rely
on different subsets of their methods. The authors argue that such interfaces should
be refactored into fine-grained interfaces, containing only methods effectively called by
groups of clients. Therefore, they focus on superfluous methods, while GraphQL focus
on superfluous data returned by REST-based APIs. Wittern et al. [2018] propose a
tool to generate GraphQL wrappers from REST-like APIs with OpenAPI Specifica-
tion (OAS). Their tool takes as input an specification that describes a REST API and
generates a GraphQL wrapper. They evaluate the proposed tool with 959 publicly
available REST APIs; and it was able to generate GraphQL wrappers for 89.5% of
these APIs, but with limitation in some cases.
2.8 Conclusion
As our key finding, we show that there is a drastic reduction in the number of fields
and size of the returned JSON documents when using GraphQL, instead of REST.
Probably to avoid frequent client/server interactions Baker and Dobson [2005] (or to
avoid the implementation of slightly different endpoints), REST-based interfaces are
usually coarse-grained components, designed to provide at once all possible data needed
by clients. However, specific clients require only a small subset of the data provided
by such interfaces; and therefore simply discard the unneeded information. Our results
show that the proportion of data received but discarded by clients is outstanding:
GraphQL can reduce the size of the JSON documents returned by REST-based APIs
in 94% (measured in number of fields) and in 99% (measured in bytes); both measures
are median values. To our knowledge, we are the first to reveal such numbers, by means
of a study involving 24 queries performed by seven open source clients of two popular
REST APIs (GitHub and arXiv) and 14 queries performed by seven recent empirical
papers published in two software engineering conferences.
As our secondary finding, we show that it is not straightforward to refactor API
clients to use complex GraphQL queries. The reason is that developers tend to organize
their code around small functions that consume small amounts of data. Refactoring
these programs to request at once large graph structures is probably a complex reengi-
2.8. Conclusion 27
neering task.
Our work paper can be extended as follows: (a) by evaluating the runtime perfor-
mance of GraphQL queries, particularly the ones used in Section 2.5; (b) by interview-
ing developers to reveal their views and experience with GraphQL; (c) by migrating
more systems to GraphQL and studying the logs they produce during normal opera-
tion; (d) by investigating the benefits of GraphQL in specific domains, such as mobile
applications and microservices orchestration Jamshidi et al. [2018].
The dataset used in this paper—including the articles of the grey literature, the
source code of the migrated systems, and the queries used in the runtime evaluation—is
publicly available at https://github.com/gleisonbt/migrating-to-graphql.
Chapter 3
REST vs GraphQL: A Controlled

Experiment
3.1 Introduction
GraphQL is a query language for implementing and consuming Web-based APIs [Face-
book Inc., 2015]. The language was internally developed at Facebook, as a solution
to several API problems faced by them when building a mobile client for the popular
social network. In 2015, Facebook open-sourced the definition and implementation of
GraphQL. As a result, the language started to gain momentum and it is now supported
by major Web APIs, including the ones provided by GitHub, Airbnb, Netflix, Twitter,
and The New York Times. In December 2018, Facebook decided to transfer GraphQL
to a non-profit organization, called GraphQL Foundation.
GraphQL is as an alternative to REST-based APIs [Fielding and Taylor, 2002,
2000; Fielding, 2000]. To understand GraphQL’s differences from REST, we must first
remember that endpoints are the key abstraction provided by REST. In REST, an
endpoint is defined by an URL and a list of parameters. For example, in the GitHub’s
REST API
GET / search / repositories ? q = stars : >100
is an endpoint that returns data about GitHub repositories with more than 100
stars. Since REST relies on HTTP abstractions to support queries (URLs, endpoints,
GET/PUT parameters, etc), it can be considered as a low-level technology. By con-
trast, GraphQL is a full data query language to implement Web APIs, centered on
high-level abstractions, such as schemas, types, queries, and mutations. For example,
the previous REST query is implemented in GraphQL as follows:
29
30 Chapter 3. REST vs GraphQL: A Controlled Experiment
1 query searchRepos {
2 search ( query :" stars : >100" , first :100 , type : REPOSITORY ) {
3 nodes {
4 ... on Repository {
5 nameWithOwner
6 }
7 }
8 }
9 }
When using GraphQL, API clients can define exactly the data they require from
the service. In our previous REST example, the API returns a JSON document with 94
fields, although the client only needs one field (the repository’s name). This problem is
often called over-fetching Schrock [2015]; Buna [2015]. On the other hand, in GraphQL,
clients can precisely specify the fields they require from the server (in our example, just
nameWithOwner, line 5).
Previous studies compared REST and GraphQL, but mostly under a quantita-
tive perspective. For example, Brito et al. [2019] investigated the gains of migrating
to GraphQL queries performed by seven GitHub API clients; and Wittern et al. [2018]
performed a study to evaluate the gains achieved with a tool that automatically gener-
ates GraphQL wrappers from REST APIs. Therefore, to our knowledge, we still lack
studies that contrast the effort and the perceptions of developers when implementing
queries for accessing REST and GraphQL APIs. As a contribution to close this gap,
in this paper we present the results of a controlled experiment where we asked 22 stu-
dents to implement a set of queries for accessing the GitHub API. We anchored the
experiment on this particular API because GitHub supports a REST implementation
and more recently a GraphQL-based version of this API. Therefore, we instructed the
students to implement half of the proposed API queries in REST; and the other half
in GraphQL.
More specifically, we ask two questions in this paper:
RQ1: How much time do developers spend when implementing queries in REST and
GraphQL? Our intention is to investigate possible gains achieved by GraphQL; not
in terms of transferring less data to clients, i.e., avoiding over-fetching, but possibly
on requiring less effort to implement the queries. To provide a more solid result, we
expanded this first RQ by investigating three related sub-questions: (RQ1.1) How
does this time vary between the types of queries? (RQ1.2) How does this time vary
among undergraduate and graduate students? and (RQ1.3) How does this time vary
depending on the participants’ experience in the studied technologies?
RQ2: What are the participants’ perceptions about REST and GraphQL? With this
3.2. Background 31
second question, our intention is to provide qualitative data on the experience of the
experiment’s participants, when implementing the proposed GitHub queries. Basi-
cally, after the experiment, we surveyed the participants about their perceptions on
GraphQL, REST, and our experiment, in general.
In summary, our results show that GraphQL requires less effort to implememt
API queries, when compared to REST (9 vs 6 minutes, median times). We also report
that these gains increase when REST queries demand more complex endpoints, with
several parameters. Interestingly, GraphQL outperforms REST even among experi-
enced participants (as is the case of graduate students) and among participants with
previous experience in REST, and no previous experience in GraphQL. Finally, when
surveyed, the participants mentioned two key benefits of using GraphQL: (1) tool-
ing support when building and testing the queries (particularly the help provided by
auto complete features); (2) a syntax and semantics closed to standard programming
languages, based on concepts such as schemas, types, queries, interfaces and objects.
The rest of this paper contains seven sections. Section 3.2 provides a brief intro-
duction to REST and GraphQL using GitHub APIs as example. Section 3.3 describes
the research design of the proposed study. Section 3.4 presents the results of the two
proposed research questions. Section 3.5 discuss the main findings of the study. Threats
to validity are discussed in Section 3.6; and related work is discussed in Section 3.7.
Finally, Section 3.8 concludes the paper.
3.2 Background
This section presents a short overview of REST and GraphQL. For a detailed presen-
tation of GraphQL, we refer the reader to its documentation Facebook Inc. [2015]. For
REST, we recommend the doctoral thesis that proposed this concept Fielding [2000].
3.2.1 REpresentational State Transfer (REST)

REST is an architectural style for implementing distributed systems. The style defines
a set of constraints intended to improve performance, availability, and scalability of
systems and it is based on the traditional client-server paradigm Fielding and Taylor
[2002, 2000]; Fielding [2000]. REST-based APIs are the ones that follow the constraints
defined by the REST style. REST also defines an uniform interface for system com-
ponents based on resource identification and dynamic data provision. In REST-based
APIs, data is exposed by endpoints. Each endpoint returns data about one resource
and each resource has a predefined set of fields.
For example, GitHub’s REST API provides 366 endpoints. An example of end-
point is
GET / users / torvalds / repos
This endpoint returns a list of public repositories of a given user, i.e., torvalds,
in this example. The following listing shows a fragment of the returned JSON. It
contains 93 fields, e.g., full_name (line 3), owner (line 5–8), created_at (line 10),
among others.
1 [
2 {
3 " full_name ": " torvalds / libdc - for - dirk " ,
4 " private ": false ,
5 " owner ": {
6 " login ": " torvalds " ,
7 ...
8 },
9 ...
10 " created_at ": "2017 -01 -17 T00 :25:49 Z " ,
11 ...
12 },
13 ...
14 ]
3.2.2 GraphQL
In GraphQL, the API data is exposed as a graph [Hartig and Pérez, 2017], defined
by means of a schema. Each node of this graph/schema represents object types and
contains fields. Each field has a name and a type. Edges appear when a field’s type
references another object type. Clients access the API through a single endpoint, which
is used to perform queries.
GraphQL also provides a domain specific language for defining schemas, including
object types and queries. For example, GitHub’s GraphQL API provides a schema with
types representing Repositories and Users, among other entities.1 The following listing
shows a fragment of Repository and Language types, provided in this schema.
1 interface Node {
2 id : ID !
3 }
4 ...
5 type Repository implements Node {
6 ...
1
Available at https://github.com/octokit/graphql-schema
3.2. Background 33
7 nameWithOwner : String !
8 primaryLanguage : Language !
9 ...
10 }
11
12 type Language implements Node {
13 id : ID !
14 name : String !
15 color : String
16 }
17 ...
Like many type systems, GraphQL also supports interfaces. An interface is an

abstract type that includes fields that a type must define when implementing the
interface. Most types from GitHub’s schema—including Repository and Language—
implements the Node interface (lines 1–3). This interface has only one field called id,
that represents a unique identifier. The Repository type contains 71 fields. However,
to the sake of clarity, our example only shows two fields: nameWithOwner (line 7) and
primaryLanguage (line 8). The primaryLanguage field is of type Language (lines 12–
16), which contains three fields: id, name, and color (which is the color defined for
the language on GitHub’s web interface). The ! symbol means that a field must not be
null.
In a GraphQL schema, queries are defined in a special type, called Query. The
following listing shows a fragment of the Query type, with only one query, called
repository (line 4), which has two parameters: name and owner. Both parameters
are of the type String. This query returns an object of a Repository type.
1 ...
2 type Query {
3 ...
4 repository ( name : String ! , owner : String !) : Repository
5 ...
6 }
Finally, GraphQL defines a query language, used by clients to perform queries.

The following listing shows two examples of repository queries. The first query
(exampleRepository01) returns only the full name (nameWithOwner, line 3) of the
repository facebook/react. The second query (exampleRepository02) is similar,
but in this case it requests two fields: description and primaryLanguage. Since
primaryLanguage is another object, we have to specify its queried fields, in this case
just name (line 11).
1 query exampleRepository01 {
2 repository ( owner :" facebook " , name :" react ") {
3 nameWithOwner
4 }
5 }
6
7 query e xampleRepository02 {
8 repository ( owner :" facebook " , name :" react ") {
9 description
10 primaryLanguage {
11 name
12 }
13 }
14 }
The result of exampleRepository02 query is presented in the following listing.

As we can see, this result is a JSON object, which resembles the structure of the query.
1 {
2 " data ": {
3 " repository ": {
4 " description ": " A declarative , efficient , and flexible JavaScript
library for building user interfaces ." ,
5 " primaryLanguage ": {
6 " name ": " JavaScript "
7 }
8 }
9 }
10 }
3.3 Research Design

In this paper, we describe a controlled experiment to compare two technologies for im-
plementing Web APIs: REST and GraphQL. We aim to understand which technology
requires less effort to implement queries to Web APIs. Specifically, we ask the following
research questions:
• RQ1: How much time do developers spend implementing queries in REST and
GraphQL? In fact, to provide an in-depth understanding of this first question,
we also investigate three related questions:
– RQ1.1: How does this time vary among the types of queries?
– RQ1.2: How does this time vary among undergraduate and graduate stu-
dents?
– RQ1.3: How does this time vary depending on the participants’ experience
in the studied technologies?
• RQ2: What are the participants’ perceptions about REST and GraphQL? With
this second RQ, our goal is to provide qualitative data about the effort to im-
3.3. Research Design 35
Table 3.1: Experiment Tasks
Type Task Description

Implement a query that returns the full name (owner/name)
T1 and the description of the top-10 most starred Python repositories,
sorted in descending order.
Search Implement a query that returns the number of stars and the num-
Repositories T2 ber of forks of the top-10 most starred repositories, sorted in de-
scending order.
Implement a query that returns the URL and owner login of the
T3 top-10 most starred Java repositories, created after Jan-01-2018,
sorted in descending order.
Implement a query that returns the URL of 10 users with more
T4
Search than 10,000 followers, sorted in descending order.
Users Implement a query that returns the login of 10 individual users
T5
(i.e., non-organizations) with more than 10,000 repositories.
Implement a query that returns the primary language, the de-
Repository T6
scription, and URL of facebook/graphql repository.
Implement a query that returns the number of followers and the
T7
number of repositories of the user torvalds.
User Implement a query that returns the number of watchers and the
T8 number of stars of the first 10 repositories owned by facebook,
sorted by creation date, in descending order.
plement API queries using REST and GraphQL, based on the perceptions and
views of the participants.
Before performing the controlled study, we also performed a pilot study with two
graduate students. Both participants had previous experience only in REST; for this
reason, the first author of this paper presented a short talk on GraphQL (one hour).
They implemented eight queries, one using REST and the other using GraphQL. We
used their comments and observations to help us plan and calibrate our study.
3.3.1 Tasks
To answer the proposed research questions, we rely on a controlled experiment in-
volving four types of queries to GitHub: search repositories, search users, repository,
and user. More specifically, we elaborated three tasks requiring the implementation of
search repositories queries, two requiring search users queries, one requiring a repos-
itory query, and two demanding user queries. These eight queries are described in
Table 3.1. Search repositories and search users are generic queries that return data
about repositories and users, respectively, using parameters to filter the results. Repos-
itory and User queries are specific queries that return data about only one repository
or user, respectively.
3.3.2 Subjects Selection

We performed our controlled experiment with 22 subjects, including 10 undergraduate
students and 12 graduate students. All subjects have at least one year of programming
experience. Additionally, as we can see in Table 3.2, 11 subjects have previous experi-
ence with REST, and 4 have experience with both REST and GraphQL. We also have
7 subjects without experience in any of these technologies. It is also worth noting that
no participants have experience only in GraphQL.
Table 3.2: Subjects’ experience in REST and GraphQL
REST GraphQL REST and GraphQL None

11 (50%) 0 (0%) 4 (18.2%) 7 (31.8%)
3.3.3 Experiment Design

The treatment in our experiment is the technology used to implement the queries, i.e.,
REST or GraphQL. The dependent variable is the time the subjects take to implement
the proposed tasks. We also analyze the results under three dimensions: (i) types
of queries (search repositories, search users, repository, and user ), (ii) students level
(undergraduate and graduate), and (iii) previous experience with REST and GraphQL.
The experiment followed a within-subject design [Seltman, 2012], where all par-
ticipants are exposed to every treatment and condition. In other words, all participants
implemented tasks using REST and GraphQL (four tasks in each technology). How-
ever, it is well-known that the order in which the treatments are given affects the
subjects’ performance. To counteract this fact, we also used a counterbalanced de-
sign [Seltman, 2012]. Basically, we elaborated two task description documents (A and
B) with the tasks alternating between REST and GraphQL, as described in Table 3.3.
We also divided the subjects into two groups; one group received the document A,
and the other received document B. Finally, Table 3.4 shows that both groups are bal-
anced, regarding their general programming experience and academic level. However,
Table 3.3: Tasks allocation among participants (Groups A and B) and between treat-
ments (REST and GraphQL)
Group
Tasks
A B
T1 REST GraphQL
T2 GraphQL REST
T3 REST GraphQL
T4 GraphQL REST
T5 REST GraphQL
T6 GraphQL REST
T7 REST GraphQL
T8 GraphQL REST
Table 3.4: Group Profiles
Experience in REST and GraphQL

Group
REST GraphQL REST and GraphQL None
A 4 0 2 5
B 7 0 2 2
General Programming Experience (years)
Group
<1 1..3 >3
A 0 2 9
B 0 2 9
Academic Level
Group
Undergraduate Graduate
A 5 6
B 5 6
in terms of experience in REST and GraphQL, Group A has more participants with
experience in REST. We could not achieve an uniformed distribution in this case due
to last changes in the list of participants (some confirmed participants did not appear
and others appeared, despite having not answered our initial invitation).2
2
However, this fact seems to have not impacted our findings; indeed, as reported in Section 3.4,
GraphQL usually demands less effort than REST.
3.3.4 Experimental Procedure

Before asking the participants to perform the proposed tasks, the first author of this
paper presented a short talk (about one hour) on using REST and GraphQL to query
GitHub data. After this talk, the subjects completed a pre-experiment form, where
they informed their experience with the studied technologies. During the execution of
the tasks, the subjects had access to the slides of this initial talk, as well as to the
GitHub documentation of both APIs.3,4
The experiment was conducted using IDLE, which is a simple IDE for program-
ming in Python.5 IDLE is suitable for beginners, especially in an educational environ-
ment. As presented in Listing 3.1, we provided to the participants a single source code
file, containing the description of the tasks (in the form of comments, see lines 1–8)
and specific string variables to store the queries (line 13, for example). Furthermore,
after executing the provided code, it automatically informs if the query is correctly
implemented or not. If it is correct, we instructed the subjects to move to the next
task/query. Otherwise, he/she was instructed to revise and change the implementa-
tion and try again. Additionally, each execution generates a log, containing information
about the queries (code, result, time, etc). We used this log to analyze the participant’s
solutions and attempts, in order to better answer the proposed RQs.
1 #======================================================#
2 # Task 1: Write a query that returns the full name #
3 #( owner / name ) and the description of the top -10 most #
4 # starred Python repositories , #
5 # sorted in descending order . #
6 # ** First , remove the comment (#) from the used API #
7 # ** After , implement the query instead of xxx #
8 #======================================================#
9
10 # API = ’ rest ’
11 # API = ’ graphql ’
12
13 query_1 = """ xxx """
Listing 3.1: Fragment of source code, used to implement Task #1
During the experiment, the subjects were allowed to use a web browser to test the
queries, particularly the ones implemented in REST. For example, Figure 3.1 shows an
example of testing a REST query in a web browser. The query is provided in the address
bar and the resulting JSON documented is showed in the browser. Finally, GitHub
provides a web app, called GraphiQL, to test GraphQL queries (see an screenshot
3
https://developer.github.com/v4/query/
4
https://developer.github.com/v3/
5
https://docs.python.org/3/library/idle.html
Figure 3.1: Response to a REST query performed using a web browser
Figure 3.2: Response to a GraphQL query performed using GraphiQL
in Figure 3.2). This app leverages GraphQL features to support for example auto
complete. We claim that allowing the participants to use this IDE does not represent
a bias towards GraphQL, since it is used by practitioners in their daily experience with
this query language (just to reinforce, GraphiQL is an official application, supported
by GitHub).
Finally, it is also important to mention that all participants concluded the pro-
posed tasks, i.e., no participants had to leave during the experiment or were not able
to implement some of the queries.
● ● ●●
● ● ● ● ● ●● ●● ● ●
REST ● ● ●
● ● ● ● ●●● ●●
● ● ●
●
● ●
●
●
●
● ● ● ● ● ● ● ● ●●
● ● ●
● ●● ●● ● ●
●● ●● ● ●
● ● ●
● ● ●
● ● ●
● ● ● ●
● ●
● ●●
● ● ●●
GraphQL ● ●
●
●
●
●
●
●
●
● ● ●
● ●●
● ● ● ●
●
●
●●
●
● ● ●
● ●●● ● ●●
● ●
●
100 101 102

duration (min) (log10)
Figure 3.3: Time to conclude the tasks (REST vs GraphQL)
T1 T2 T3 T4 T5 T6 T7 T8
102
●
● ● ●
●
●●
●●
●● ●
● ● ●
● ● ●
● ● ● ● ●
● ●
● ● ●● ●●
●● ●
●
● ●
●
● ● ● ●
● ● ●
●
● ● ● ●
● ● ● ● ● ●
1 ● ● ●
10 ● ●
●●● ● ● ● ● ●● ● ● ●
●
●●
●● ● ● ● ● ● ● ●● ● ●
● ●
● ● ● ●● ● ● ● ● ●● ● ● ●
● ● ● ● ● ● ● ● ● ● ●
● ●● ● ● ●● ● ●
● ● ● ● ● ● ● ●
● ●● ● ● ● ●●
● ● ● ● ●●
● ● ●● ●●● ● ●
● ● ● ●
● ● ● ● ●● ● ● ●
100 ●
GraphQL REST GraphQL REST GraphQL REST GraphQL REST GraphQL REST GraphQL REST GraphQL REST GraphQL REST
Figure 3.4: Time to implement the proposed tasks for querying GitHub data (REST
vs GraphQL)
3.4 Results
RQ1: How much time do developers spend implementing queries in REST and
GraphQL?
Figure 3.3 shows violin plots with the time in minutes to implement the proposed
tasks using REST and GraphQL. The points inside the violins represent a pair (task,
participant), i.e., the time spent by a participant to conclude a task. As we can see,
the subjects spent nine minutes to implement each REST query, against six minutes to
implement the GraphQL ones (median values). We check these differences by applying
a Mann-Whitney test. The p-value is 0.00034, which allows us to conclude that the
time for implementing the tasks using GraphQL is statistically different than using
REST.
To shed light on these results, we first inspected the performance of the partic-
3.4. Results 41
ipants in each task. Figure 3.4 shows violin plots with the execution times for each
proposed task (as in the previous figure, each point represents the time spent by a
participant in a given task ). As can be observed, only two tasks have significant differ-
ences between REST and GraphQL distributions: T5 and T8. When we compare the
distributions of these tasks using a Mann-Whitney test the result shows they are indeed
statistically different. Therefore, we present next a detailed analysis of the results for
T5 and T8.
Task T5. This task requires the implementation of a search users query to retrieve the
top-10 GitHub users with more than 10,000 followers, in decreasing order. In this case,
the subjects spent eight minutes (median values) to perform the REST implementation,
against only three minutes in GraphQL.
By analyzing the log files of the five participants that spent more time than the
median when implementing T5 in REST, we found that they all initially implemented
the following query:
1 GET / search / users ? q = repos : >10000& page =1& per_page =10
In this code, the type qualifier is missing. As a result, the query returns data
about both personal (e.g., torvalds) and organizational (e.g., facebook ) accounts. The
following listing shows the correct REST query, where the type qualifier (in bold) is
used to return information just about individual (i.e., non-organizational) accounts.
1 GET / search / users ? q = repos : >10000+ type : user & page =1& per_page =10
By contrast, for GraphQL, all 11 subjects implemented T5 correctly, using the

following query;
1 query t5 {
2 search ( query :" repos : >10000" , type : USER , first :10 ) {
3 nodes {
4 ... on User {
5 login
6 }
7 }
8 }
9 }
As we can see in this listing, the type parameter is mandatory in GraphQL queries
(line 2). When this parameter is missing, GraphiQL (the IDE used by the participants)
does not execute the query and shows a warning, as showed in Figure 3.5.
Task T8. This task requires the implementation of a user query to retrieve the number
of watchers and stars of the ten first repositories from the facebook organization.
Figure 3.5: Warning message about missing type parameter
The subjects spent nine minutes to perform the implementation in REST, against only
four minutes in GraphQL (median values).
The correct implementation for task T8 in REST is:
1 GET users / facebook / repos ? sort = created & direction = desc & page =1& per_page =10
By analyzing the log of the participants with implementation times greater than
the median for REST, we did not identify a common pattern of problems. For example,
one subject used the type qualifier, as showed in the following listing.
1 GET / users / facebook / repos ? type = organization & per_page =10& page =1& sort = created
& direction = desc
However, it is not necessary to use this qualifier (as in T5), because the endpoint
/users/{user}/repos/ already returns data about users or organizations.
In another case, one subject did not use the complete endpoint, as in the following
listing, where /repos is missing:
1 GET users / facebook ? sort = created & direction = desc & page =1& per_page =10
Finally, in two cases, the subjects relied on the repos and search endpoints to
implement T8 (i.e., an incorrect endpoint), as in the following listings:
1 GET / repos / facebook ? sort = created & order = desc & page =1& per_page =10
1 GET / search / repositories ? q = user = facebook & sort = created & order = desc & page =1&
per_page =10
Essentialy, these observations confirm that a REST endpoint is a low-level ab-

straction. As a result, it allows a variety of coding and specification errors, such as
using the incorrect endpoints, using incorrect parameters, not passing mandatory pa-
rameters, etc.
For GraphQL, all subjects implemented T8 in at most 8 minutes. The following
listing shows the correct solution using GraphQL.
1 query task8 {
2 organization ( login :" facebook ") {
3 repositories ( orderBy :{ field : CREATED_AT , direction : DESC } , first :10) {
3.4. Results 43
4 watchers { totalCount }
5 stargazers { totalCount }
6 }
7 }
8 }
It is important to highlight that in GraphQL we do need to include an endpoint

in the query. Instead, we can assume that there is a single and unique endpoint, which
gives access to the GraphQL’s schema. By contrast, as discussed before, selecting
and specifying the correct endpoint was a relevant challenge faced by the participants
when implementing their queries in REST.
RQ1’s summary: When comparing the implementation time of all tasks, there
is a difference of 3 minutes favoring GraphQL (9 minutes vs 6 minutes; median
times). Specifying the correct endpoints and parameters was the main challenge
faced by the REST participants.
RQ1.1: How does the implementation time vary among the types of queries?
In this RQ, we compare the tasks grouped by queries types (search repositories,
search users, repository, and user ). Figure 3.6 shows violin plots with the results. Each
point in the plots represent the time spent by a participant using REST or GraphQL.
search repositories
● ●
● ● ● ● ●● ●● ● ●●
REST ●
● ● ● ●
●
● ●
●● ● ● ●● ● ●
● ● ●
● ● ●● ●
●● ● ●● ●
GraphQL ● ● ● ● ● ●
● ● ● ● ● ●● ● ●
●
search users
● ● ● ● ●
REST ● ● ● ● ● ● ●●
●● ●● ● ● ●
● ● ●
● ● ● ●
GraphQL ● ● ● ● ● ●
● ● ● ●
●
repository
● ● ● ● ● ●
REST ● ● ● ●
● ● ● ●
GraphQL ●
● ● ● ● ●
user
● ● ● ●
REST ●
●
●
●
●
● ● ●●
●●
●
●●
●
●
● ● ●
GraphQL ● ●
● ●
●
● ● ● ● ●
● ●
● ● ● ●
100 101 102

Figure 3.6: Time to conclude the tasks, grouped by query type

As we can see, for three query types (search repositories, search users, and user ) the
median implementation times was higher when the tasks were implemented in REST.
Only in the case of repository tasks, we can observe higher implementation times for
GraphQL, on the median. In such tasks, the REST endpoint /repos/{owner}/{repo}
is used to implement the proposed task (T6). This endpoint demands only two param-
eters to obtain the required data, as we can see in the following listing.
1 GET / repos / facebook / graphql
On other hand, to implement T6 in GraphQL, besides the two parameters owner

and repository name (line in the following listing), it is necessary to specify filters to
return only the fields mentioned specification of T6, i.e., primary language, description,
and URL.
1 {
2 repository ( owner :" facebook " , name :" graphql ") {
3 primaryLanguage {
4 name
5 }
6 description
7 url
8 }
9 }
However, even REST presenting better results for T6, the difference is only one
minute (5 minutes in REST, against 6 minutes in GraphQL). Indeed, by applying
Mann-Whitney test, we did not find a statistical difference in this case.
By contrast, search users and users presented statistical difference. We hypoth-
esize that queries that return user elements require more effort to be implemented in
REST than in GraphQL, because they demand several parameters. In GraphQL, these
parameters can be provided using the auto complete feature of the GraphiQL IDE. For
instance, the implementation of task T1 by participant S16 is a clear case of problems
associated to the use of REST parameters. This participant spent 80 minutes to im-
plement T1 (in REST). In total, S16 spent 125 minutes to conclude all eight queries.
Therefore, only in T1—his first query—he spent 64% of his overall implementation
time. By contrast, the maximum time for implementing T1 in GraphQL was 44 min-
utes, by S9. This task demands the implementation of a query returning the full name
and description of the top-10 most starred Python repositories in descending order.
The following listings shows some attempts, performed by S16:
search / repositories ? q = language : python + stars & sort = stars & order = desc
search / repositories ? q = language : python + stars : >100& sort = stars & order = desc
search / repositories ? q = language : python + stars : >10& sort = stars & order = desc
3.4. Results 45
search / repositories ? q = language : python + stars & sort = stars & order = desc & page =1&
per_page =10
In the first three attempts, S16 did not inform the page and per_page parame-
ters, which are a mandatory parameters for defining the number returned elements. In
the first and fourth tries, he did not inform the value of the stars parameters, which
is necessary to filter the most starred repositories.
It is also interesting to mention that S16 spent 73 minutes to conclude all
GraphQL tasks (i.e., 58% of his time in REST). The GraphQL task he spent more
time was in T2 (45 minutes). We emphasize that S16 had no previous experience
with REST or GraphQL. Therefore, tasks T1 and T2 were his first contact with both
technologies.
RQ1.1’s summary: GraphQL outperforms REST mainly in queries that require
several parameters. In such queries, auto complete—as provided by GraphQL’s
IDEs—is a powerful feature to help developers.
RQ1.2: How does this time vary among undergraduate and graduate students?
Figure 3.7 presents the results according to the subjects’ academic level. In both
groups, the participants who performed the tasks in REST spent more time than the
ones who implemented them in GraphQL. Therefore, even subjects which more ex-
perience, as is typically the case of graduate students, take benefit of GraphQL and
implemented the proposed tasks in less time, than when using REST. Indeed, the high-
est difference between the median times (REST − GraphQL) was observed for graduate
students, 3 minutes (graduate students), against 2.5 minutes (undergraduate students).
According to Mann-Whitney tests, both distributions (graduate and undergraduate)
are statistically different.
RQ1.2’s summary: Both undergraduate and graduate students have taken ben-
efit of GraphQL and implemented the tasks in this technology in less time.
RQ 1.3 How does this time vary depending on the participants previous experience in
the studied technologies?
Figure 3.8 presents the results according to the subjects’ previous experience
in REST or GraphQL. Each point in the violin plots represents the time spent
by a participant in the proposed implementation tasks. As we can see, in the
three groups (no previous experience in both technologies, previous experience in
REST, and previous experience in REST and GraphQL) the median implementa-
tion times is higher when the tasks were implemented in REST. Interestingly, this
graduate
● ● ● ● ● ●●●●
●● ● ● ●●
REST ●
●
●
● ●
●
● ● ●
● ● ●
● ●
●
●● ● ●
● ● ●
●
●
● ●● ●
● ●
● ● ●
● ● ● ●● ●● ● ● ● ●● ● ●
GraphQL ●
●
●
●
●
●
● ●
●
●
● ●
● ● ● ● ● ●
●
●
undergraduate
● ● ● ●
●● ● ● ●●
●● ● ●●
REST ● ●
●
● ● ● ●●
● ● ● ● ●● ●
●
● ●● ● ●
● ● ● ● ● ● ● ●
●
GraphQL ● ●
●
●
●
●
●
● ● ●
● ● ●
● ●●
●
● ● ●
100 101 102

Figure 3.7: Time to conclude the tasks grouped by academic level
happened even for the group of participants with previous experience in REST. In
other words, these participants spent more time implementing their tasks in REST
than in a completely novel technology for them, i.e., GraphQL. As in the previous
RQs, we checked the statistical differences using Mann-Whitney tests. We found
a statistical difference in the last two groups, i.e., participants with previous expe-
rience in REST and participants with previous experience in both technologies.
RQ1.3’s summary: GraphQL outperforms REST even among participants with

previous experience in REST, and no previous contact with GraphQL.
RQ2: What are the participants’ perceptions about REST and GraphQL?
After implementing the tasks, the participants were invited to answer a post-
experiment form, with their perceptions about REST, GraphQL, and the experiment.
3.4. Results 47
None
● ● ● ● ●
● ● ●● ●
REST ● ● ● ●
●●
●
● ● ● ●
● ●●
● ● ●
● ● ● ●● ●
●
● ● ●
GraphQL ●
●
● ●
● ●
● ●●
● ● ●
● ●
● ●
REST
● ● ● ● ● ●●● ●●● ●● ● ●●
REST ●
●
● ● ● ● ● ● ●● ● ● ● ●● ● ●
● ● ● ● ●
● ● ●
● ● ● ● ●●
● ●● ● ● ●● ●
GraphQL ●
● ●
●
●
● ●
● ● ● ●
●
●●
● ● ● ● ●
REST and GraphQL
● ● ● ● ● ● ●
REST ● ● ●
●
● ●
●
●
●
● ●
● ● ●
GraphQL ● ●
● ●
●
● ● ● ●
100 101 102

Figure 3.8: Time to conclude the tasks (REST vs GraphQL) grouped by previous
experience
We received responses of 11 participants (3 undergraduate and 8 graduate).

The first author of this paper carefully read these responses and grouped them
in five categories: Tool support, Syntax, Previous Experience, and Documentation.
Table 3.5 shows the participants with answers in each category. We received at least
three answers in all categories, except in Previous Experience.
Table 3.5: Answer’s Categories
Category Participants
Tool support S1, S7, S8, S17
Syntax S9, S12, S17
Previous Experience S1
Documentation S4, S11, S7
In the following paragraphs, we discuss the answers of each category.
Tool support. Two participants (S1, S17) explicitly mentioned the benefits of using
GraphiQL IDE. As examples, we have these answers: queries with GraphQL are
much more interesting to do, because of the ease of Explorer (S1) and The GraphQL
environment helped a lot during query testing (S17). Furthermore, GraphQL allows
IDEs to implement a auto complete feature, due to the avaiability of a schema for
defining types and fields. This feature was a powerful tool to assist the implementation
of queries, as mentioned by subject S7: the auto complete feature of GraphQL has
helped me a lot to put together complex queries.
Syntax. GraphQL queries follow a JSON syntax. For instance, they can be indented
for better understanding and visualization of the code, as mentioned by subject
S12: For me GraphQL is better than REST, because the query structure allows a
better visualization of the query. Furthermore, S17 mentioned that GraphQL is
a more intuitive technology because it relies on programming language concepts,
like types, functions, and queries: I think the syntax of the query looks much more
like what we are accustomed to see in programming languages (functions, types,
SQL queries). Additionally, when using GraphQL, it is easier to define the param-
eters required by the proposed queries, as mentioned by subject S17: I liked the
GraphQL filters because they allow you to better specify the characteristics of the query.
Experience. Seven participants have no previous experience in GraphLQL, but only S1

complained about this fact: My main difficulty was the lack of previous experience with
3.5. Discussion 49
GraphQL. Although S1 had mentioned his lack of experience, he spent only 41 minutes
to implement all GraphQL queries, against 146 minutes for REST. Additionally, he
also does not have experience in REST.
Documentation. During the experiment, the participants had access to REST and
GraphQL API documentation. Two participants mentioned the GitHub’s GraphQL
API documentation is limited, e.g., API documentation about GitHub’s GraphQL is
poor (S11).
RQ2’s summary: According to the subjects, the main benefit of GraphQL are
the the tool support provided by GraphiQL, e.g., auto complete feature. An-
other mentioned benefit is better syntax to understanding the code and less
effort to specify parameters. By contrast, two participants commented about
the poor quality of GitHub’s GraphQL API documentation.
3.5 Discussion
3.5.1 Why GraphQL requires less effort?

By triangulating the results of RQ1 and RQ2, it is clear that the presence of a type
system—expressed as a schema— is one of the key benefits provided by GraphQL, in
terms of reducing the effort to implement queries, when compared to REST. Essentially,
this schema allows GraphQL IDEs to check type errors before submitting the queries
and also to provide messages with a clear indication of the errors committed by devel-
opers. This happened for example in T5, when all REST participants forgot to include
an important parameter; in GraphQL, the same error was detected by the GraphiQL
IDE, which produced a clear error message. Consequently, T5 was implemented in
eight minutes in REST and in three minutes in GraphQL (median times).
3.5.2 Can we improve REST results?

Interestingly, our results suggest that REST may also benefit from introducing a type
system in endpoints. In other words, for each endpoint, this type system would describe
the required parameters, their category (mandatory or optional, for example) and the
type of the respective arguments. In this way, it might be possible to develop REST
IDEs with features similar to the ones of the GraphQL IDE used in our experiment,
including the support to auto complete. Furthermore, these IDEs could also have a
Web browser version, in order to allow in-browser type checking of REST queries.
3.6 Threats to Validity

In this section, we report threats to validity, as well as, the respective treatments, based
on the guidelines proposed by Wohlin et al. Wohlin et al. [2012].
3.6.1 Construct Validity

The construct validity in controlled studies refers to correctly measuring of the depen-
dent variables, which in our case, is the time to implement the proposed queries. A
possible threat to the experimental procedure is the possibility of dialogue between the
experimenter and the subjects, interfering in the process to implement the proposed
queries. Therefore, the first author of this paper carefully asked the students to avoid
discussing and commenting their work during the experiment.
3.6.2 Internal Validity

The internal validity is related to uncontrolled aspects that may affect the experimen-
tal results, since the subjects’ experience. To mitigate this threat, we distributed the
subjects in two groups with the tasks alternating between REST and GraphQL (coun-
terbalanced design). We also equally distributed graduate and undergraduate students
in these two groups.
3.6.3 External Validity

The external validity is related to the possibility to generalize our results. The ex-
periment was conducted with 22 subjects. Thus, this number of subjects might not
be a representative sample. However, our sample is diversified; the subjects have dif-
ferent academic levels, general programming experience, and previous experience with
REST and GraphQL. Additionally, the number of proposed tasks is another possible
threat. In our study, we used eight tasks to measure the effort to implement REST and
GraphQL queries. However, we investigate four different types of queries, with differ-
ent difficult levels. A final threat is the fact that our subjects are students. However,
according to previous studies [Salman et al., 2015; Höst et al., 2000; Runeson, 2003],
students may provide an adequate model of professional developers.
3.7. Related Work 51
3.6.4 Conclusion Validity
Conclusion validity refers to the statistical power of tests to provide significant con-
clusions. We paid special attention to the appropriate use of the such tests (i.e.,
Mann-Whitney) when reporting our results in RQ1. This reduces the possibility that
our findings are due to random events.
3.7 Related Work

We separated related work in two categories: (a) studies about controlled experiments;
and (b) studies on the query language GraphQL.
3.7.1 Controlled Experiments
Controlled experiments have been widely adopted in software engineering research as

a way to evaluate tools and technologies [Wohlin et al., 2012]. According Wohlin
et al. [2012], controlled study is an empirical strategy that manipulates one factor (or
variable) of the studied setting where different treatments are applied to one or more
variables, while other variables are kept constant. Avidan and Feitelson [2017] con-
ducted a controlled experiment where nine developers tried to understand six methods
from utility classes, either with the original variable names or with names replaced by
meaningless single letters. This study shows that parameter names are more signifi-
cant for comprehension than local variables. In another experiment, Hebig et al. [2018]
investigate the benefits of model-transformation languages adoption in the scenario
of Model-Driven Software Development. For this purpose, the authors compare two
transformation languages (ATL and QVT-O) against one modern general-purpose lan-
guage (Xtend) in common development tasks, e.g., comprehension, modification, and
creation. The results point that there is no statistically significant benefits of using
a dedicated transformation language over a modern general-purpose language. Melo
et al. [2016], perform a controlled experiment to quantify the impact of variability on
debugging of preprocessor-based programs. They measured the speed and precision for
bug finding tasks at three different degrees of variability on several real systems. As
well in these previous studies, we also apply a controlled experiment to quantify the
impact of REST and GraphQL adoption in the time to implement queries.
3.7.2 GraphQL Studies
Because it is a recent technology, there are few studies in the scientific literature on
GraphQL. One of the first paper, Hartig and Pérez [2017] provide a formal definition
for GraphQL. Recently, the authors complemented and finished this formalization by
proving that evaluating the complexity of GraphQL queries is a NL-problem (i.e., a
decision problem that can be solved by a nondeterministic Turing machine under a
logarithmic amount of memory) [Hartig and Pérez, 2018]. In practical terms, this
result shows that it is possible to implement efficient algorithms to estimate the com-
plexity of GraphQL queries before their execution; which is important to handle the
performance problems normally associated to GraphQL. Vogel et al. [2017] present
a case study on migrating to GraphQL part of the API provided by a smart home
management system. They report the runtime performance of two endpoints after
migration to GraphQL. For the first endpoint, the gain was not relevant; but for the
second, GraphQL required 46% of the time of the original REST API. The authors
also point that it is possible to perform parallel operations of REST and GraphQL
services without restrictions. Wittern et al. [2018] assess the feasibility of automati-
cally generating GraphQL wrappers for existing REST(-like) APIs. For this purpose
the authors propose a tool to generate GraphQL wrappers from REST-like APIs with
OpenAPI Specification (OAS). Their tool takes as input a specification that describes
a REST API and automatically generates a GraphQL wrapper. The proposed tool was
evaluated with 959 publicly available REST APIs and was able to generate GraphQL
wrappers for 89.5% of these APIs, with limitation in some cases. Vargas et al. [2018]
perform a study to investigate the feasibility of the adoption of a classic technique to
test generation in GraphQL schema (deviation testing). They use an implementation
of GraphQL for Pharo, and run the proposed technique in two popular GraphQL APIs
provided by Yelp and Apollo GraphQL. The authors conclude that deviation testing
is able to identify anomalies in GraphQL schemas. Finally, Brito et al. [2019] perform
a study on migrating GitHub clients from REST to GraphQL API. First, the authors
conduct a grey literature review to understand characteristics and benefits of GraphQL
adoption. After, they assess these benefits by migrating seven systems to use GraphQL
instead of REST-based APIs. Their key finding is that GraphQL can reduce the size
of the JSON documents returned by REST APIs in 94% (in number offields) and in
99% (in number of bytes).
3.8. Conclusion 53
3.8 Conclusion
This paper presented a controlled study to investigate the effort to implement API
queries in REST and GraphQL. As our key finding, we found that GraphQL requires
less effort to implement API queries, when compared with REST. Our results show
that the difference in the implementation time between the two studied technologies is
3 minutes. We also showed that (i) queries which many parameters are more difficult
to be implemented in REST than in GraphQL; (ii) we also observe that subjects
which previous experience in REST spent less time to implement GraphQL queries,
i.e., GraphQL requires less effort even for developers that have no previous experience
which this technology. In our study, we also investigate the perceptions of the subjects.
Almost of them related that the main benefit of GraphQL are the the tool support
provided by GraphiQL, e.g., auto complete feature. Another mentioned benefit is
better syntax to understanding the code and less effort to specify parameters.
As future work, we intend to extend this research by surveying and interview-
ing practitioners to reveal their views and experience with GraphQL. We also intend
to investigate possible challenges to adopt GraphQL in real systems, e.g., by migrat-
ing REST APIs to GraphQL. Another future work is to investigate the development
of REST IDEs with features similar to the ones of the GraphQL IDE used in our
experiment.
Chapter 4
Next Steps
55
Bibliography
Aniche, M., Treude, C., Steinmacher, I., Wiese, I., Pinto, G., Storey, M.-A., and
Gerosa, M. A. (2018). How modern news aggregators help development commu-
nities shape and share knowledge. In 40th International Conference on Software
Engineering (ICSE), pages 499--510.
Avidan, E. and Feitelson, D. G. (2017). Effects of variable names on comprehension

an empirical study. In 25th International Conference on Program Comprehension
(ICPC), pages 55--65.
Baker, S. and Dobson, S. (2005). Comparing service-oriented and distributed object ar-
chitectures. In 7th International Symposium on Distributed Objects and Applications
(DOA), pages 631--645.
Barik, T., Johnson, B., and Murphy-Hill, E. (2015). I heart Hacker News: expanding
qualitative research findings by analyzing social news websites. In 10th Foundations
of Software Engineering Conference (FSE), pages 882--885.
Brito, A., Xavier, L., Hora, A., and Valente, M. T. (2018). Why and how Java devel-
opers break APIs. In 25th International Conference on Software Analysis, Evolution
and Reengineering (SANER), pages 255--265.
Brito, G., Mombach, T., and Valente, M. T. (2019). Migrating to GraphQL: A practical
assessment. In 26th International Conference on Software Analysis, Evolution and
Reengineering (SANER), pages 140--150.
Buna, S. (2015). REST APIs are rest-in-peace apis. long live GraphQL. https://
reactjs.org/blog/2015/05/01/graphql-introduction.html. [accessed 02-April-
2019].
Carriero, N. and Gelernter, D. (1989). Linda in context. Communications of the ACM,

32(4):444--458.
57
58 BIBLIOGRAPHY
Facebook Inc. (2015). GraphQL specification (draft). https://facebook.github.io/

graphql/draft/. [accessed 15-October-2018].
Fielding, R. T. (2000). Architectural Styles and the Design of Network-based Software

Architectures. PhD thesis, University of California.
Fielding, R. T. and Taylor, R. N. (2000). Principled design of the modern web ar-
chitecture. In 22nd International Conference on on Software Engineering (ICSE),
pages 407--416.
Fielding, R. T. and Taylor, R. N. (2002). Principled design of the modern Web archi-
tecture. ACM Transactions on Internet Technology (TOIT), 2(2):115--150.
Floyd, B., Santander, T., and Weimer, W. (2017). Decoding the representation of code
in the brain: An fMRI study of code review and expertise. In 39th International
Conference on Software Engineering (ICSE), pages 175--186.
Garousi, V., Felderer, M., and Mäntylä, M. V. (2016). The need for multivocal liter-
ature reviews in software engineering: complementing systematic literature reviews
with grey literature. In 20th International Conference on Evaluation and Assessment
in Software Engineering (EASE), page 26.
Garousi, V., Felderer, M., and Mäntylä, M. V. (2017). Guidelines for including the
grey literature and conducting multivocal literature reviews in software engineering.
arXiv preprint arXiv:1707.02553.
Gelernter, D. (1985). Generative communication in Linda. ACM Transactions on

Programming Languages and Systems (TOPLAS), 7(1):80--112.
Hartig, O. and Pérez, J. (2017). An initial analysis of Facebook’s GraphQL language.

In 11th Alberto Mendelzon International Workshop on Foundations of Data Manage-
ment and the Web (AMW), pages 1–10.
Hartig, O. and Pérez, J. (2018). Semantics and complexity of GraphQL. In 27th World
Wide Web Conference on World Wide Web (WWW), pages 1155--1164.
Hebig, R., Seidl, C., Berger, T., Pedersen, J. K., and Wasowski, A. (2018). Model trans-
formation languages under a magnifying glass: a controlled experiment with xtend,
atl, and qvt. In 26th ACM Joint Meeting on European Software Engineering Con-
ference and Symposium on the Foundations of Software Engineering (ESEC/FSE),
pages 445--455.
BIBLIOGRAPHY 59
Höst, M., Regnell, B., and Wohlin, C. (2000). Using students as subjects—a compar-
ative study of students and professionals in lead-time impact assessment. Empirical
Software Engineering, 5(3):201--214.
Jamshidi, P., Pahl, C., Mendonça, N. C., Lewis, J., and Tilkov, S. (2018). Microser-
vices: The journey so far and challenges ahead. IEEE Software, 35(3):24--35.
Leitner, P., Wittern, E., Spillner, J., and Hummer, W. (2018). A mixed-method
empirical study of function-as-a-service software development in industrial practice.
PeerJ PrePrints, 6:1–24.
Ma, W., Chen, L., Zhang, X., Zhou, Y., and Xu, B. (2017). How do developers fix cross-
project correlated bugs? a case study on the GitHub scientific Python ecosystem.
In 39th International Conference on Software Engineering (ICSE), pages 381--392.
Macho, C., McIntosh, S., and Pinzger, M. (2017). Extracting build changes with
builddiff. In 14th International Conference on Mining Software Repositories (MSR),
pages 368--378.
Melo, J., Brabrand, C., and Wasowski, A. (2016). How does the degree of variability
affect bug finding? In 38th Proceedings of the 38th International Conference on
Software Engineering (ICSE), pages 679--690.
Ogawa, R. T. and Malen, B. (1991). Towards rigor in reviews of multivocal litera-

tures: Applying the exploratory case study method. Review of Educational Research,
61(3):265--286.
Osman, H., Chiş, A., Corrodi, C., Ghafari, M., and Nierstrasz, O. (2017). Exception
evolution in long-lived Java systems. In 14th International Conference on Mining
Software Repositories (MSR), pages 302--311.
Romano, D., Raemaekers, S., and Pinzger, M. (2014). Refactoring fat interfaces using a
genetic algorithm. In 30th IEEE International Conference on Software Maintenance
and Evolution (ICSME), pages 351--360.
Runeson, P. (2003). Using students as experiment subjects–an analysis on graduate and

freshmen student data. In 7th International Conference on Empirical Assessment in
Software Engineering (EASE), pages 95--102.
Salman, I., Misirli, A. T., and Juristo, N. (2015). Are students representatives of profes-
sionals in software engineering experiments? In 37th IEEE International Conference
on Software Engineering (ICSE), pages 666--676.
60 BIBLIOGRAPHY
Schrock, N. (2015). GraphQL introduction. https://reactjs.org/blog/2015/05/

01/graphql-introduction.html. [accessed 02-April-2019].
Seltman, H. J. (2012). Experimental design and analysis. Online at: http://www. stat.
cmu. edu/, hseltman/309/Book/Book. pdf.
Vargas, D. M., Blanco, A. F., Vidaurre, A. C., Alcocer, J. P. S., Torres, M. M., Bergel,
A., and Ducasse, S. (2018). Deviation testing: A test case generation technique for
GraphQL APIs. In 11th International Workshop on Smalltalk Technologies (IWST),
pages 1--9.
Vogel, M., Weber, S., and Zirpins, C. (2017). Experiences on migrating RESTful
Web Services to GraphQL. In 15th International Conference on Service-Oriented
Computing (ICSOC), pages 283--295.
Wan, Z., Lo, D., Xia, X., and Cai, L. (2017). Bug characteristics in blockchain systems:
a large-scale empirical study. In 14th International Conference on Mining Software
Repositories (MSR), pages 413--424.
Wittern, E., Cha, A., and Laredo, J. A. (2018). Generating GraphQL-Wrappers for
REST (-like) APIs. In International Conference on Web Engineering, pages 65--83.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2012).
Experimentation in software engineering. Springer Science & Business Media.
Xavier, L., Brito, A., Hora, A., and Valente, M. T. (2017). Historical and impact anal-
ysis of API breaking changes: A large scale study. In 24th International Conference
on Software Analysis, Evolution and Reengineering (SANER), pages 138--147.
Xiong, Y., Wang, J., Yan, R., Zhang, J., Han, S., Huang, G., and Zhang, L. (2017).
Precise condition synthesis for program repair. In 39th International Conference on
Software Engineering (ICSE), pages 416--426.
Zampetti, F., Scalabrino, S., Oliveto, R., Canfora, G., and Penta, M. D. (2017). How
open source projects use static code analysis tools in continuous integration pipelines.
In 14th International Conference on Mining Software Repositories (MSR), pages 334-
-344.
Zloof, M. M. (1977). Query-by-example: A data base language. IBM Systems Journal,

16(4):324--343.

Thesis Master

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Thesis Master

Uploaded by

Copyright:

Available Formats

UMA NOVA ESPECIFICAÇÃO PARA

CONSULTAR DADOS EM APIS BASEADAS EM

UMA NOVA ESPECIFICAÇÃO PARA

CONSULTAR DADOS EM APIS BASEADAS EM

Projeto de tese apresentado ao Programa de

Orientador: Marco Túlio de Oliveira Valente

A NOVEL SPECIFICATION TO QUERYING DATA

Thesis project presented to the Graduate

Advisor: Marco Túlio de Oliveira Valente

The successful implementation of complex software systems heavily depends on the

2.1 Grey literature articles by year of appearance on Hacker News . . . . . . . 8

3.1 Response to a REST query performed using a web browser . . . . . . . . . 39

2.1 Selected Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.1 Experiment Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

2 Migrating to GraphQL: A Practical Assessment 3

3 REST vs GraphQL: A Controlled Experiment 29

1.1 The Problem

Migrating to GraphQL: A Practical

improving the state-of-the-practice on GraphQL-based API development.

2.2 GraphQL in a Nutshell

Listing 2.2: Querying distinct data from a given Post object

1 { " data ": {

3 " title ": " GraphQL : A data query language "

Listing 2.3: JSON object returned by PostByTitleAndAuthor query

To respond to queries, the developer of a GraphQL server must implement a re-

Listing 2.4: Mutation operation for Post objects

2.3 Grey Literature Review

2.3.1 Study Design

Figure 2.1: Grey literature articles by year of appearance on Hacker News

RQ1: What are the characteristics and benefits of GraphQL?

RQ2: What are the main disadvantages of GraphQL?

GraphQL enables client-specified queries, as mentioned by almost half of the articles

This characteristic makes GraphQL particularly interesting for mobile applica-

Deprecation: As common in mainstream programming languages, it is possible to

2.4 Migration Study

Figure 2.3: Summary of grey literature findings

2.4.1 Study Design

Listing 2.5: arXiv Query

Listing 2.6: Preprint type

Table 2.1: Selected Projects

Table 2.2: Stats of Selected Projects

Project Lang. Stars LOC Contrib

Listing 2.7: Example of GraphQL query (csurfer/gitsuggest)

Table 2.3: REST Calls

Project Func REST endpoints

1 { " data ": {

Listing 2.8: JSON document with a single root field (name)

Interestingly, in a single function (F17), there is an increase in the number of

RQ3’s summary: The support to an hierarchical data model is a key characteristic

RQ4: What is the reduction in the number of JSON fields?

Figure 2.6: Number of fields returned by REST and GraphQL calls

517361.1 542941.4 402692.1

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14

2.5 Runtime Evaluation

of followers). Therefore, we postponed an evaluation of the runtime gains achieved by

2.5.1 Study Design

Table 2.4: Papers and queries

Paper Query Data

101 103 105

2.6 Threats to Validity

Runtime Evaluation: In this study, we consider two software engineering conferences:

2.7 Related Work

REST vs GraphQL: A Controlled

3.2.1 REpresentational State Transfer (REST)