​​Zulip GSoC ’18 Progress Log

​
​Overall work

​​This is a link to a list of all my commits in Zulip repositories along GSoC.

​
​Weekly report

​
​Week 1 — 2018/05/14

​​GSoC has begun! As discussed with Tim, I took care of a few tasks in the ​+Roadmap for "REST API” docs​ which I had started during the community bonding period.
​​
​​Then I began to document endpoints, starting with /users/me/<stream_id>/topics​. This involved a few things: adding support in our Python and JS bindings, and the documentation itself (using the new API docs ecosystem). This also involved some fixes here and there.
​​Also, after having a hands-on experience with the new documentation procedure, I discovered some steps that could be optimized to keep things simpler and avoid duplicating work.
​​I discussed some of my concerns with Tim and Eeshan, and we agreed that we could try OpenAPI as a format for storing our REST API docs.
​​That’s when I started to work on a system that could implement parsing OpenAPI files into the docs’ template system.

​
​Week 2 — 2018/05/14

​​Along the process I faced some issues when parsing files: mostly stuff that was specific from OpenAPI (and therefore it wasn’t already implemented in any standard YAML parser).
​​Since those OpenAPI-specific features were exactly what made worth switching to the OAS (OpenAPI Specification), I decided it was worth creating ourselves.
​​
​​The first of this OAS features I had to implement was JSON references: in order to insert content from other parts in the local file (or even external files), you can use a $ref​ tag with a JSON reference.
​​That’s how yamole was born: with the goal of resolving those references inside a YAML file. Since I considered that it might be helpful to other people, I uploaded it to a separate repo and published it as a PyPI module with the same name.
​​
​​The second feature is the allOf​ key, which takes a list of objects and merges them into a single one. For instance, in the following example:
​​
​​components:
​​  schemas:
​​    NewPet:
​​      required:
​​        - name
​​      properties:
​​        name:
​​          type: string
​​        tag:
​​          type: string
​​    Pet:
​​      allOf:
​​        - $ref: '#/components/schemas/NewPet'
​​        - required:
​​          - id
​​          properties:
​​            id:
​​              type: integer
​​              format: int64
​​
​​Pet​  should become:
​​
​​Pet:
​​  schema:
​​    items:
​​      properties:
​​        id:
​​          format: int64
​​          type: integer
​​        name:
​​          type: string
​​        tag:
​​          type: string
​​      required:
​​      - id
​​      - name
​​    type: array
​​
​​As the YAML parser identifies allOf​'s contents as a list of dicts, the goal was to merge all the dicts in the list (which could have other dicts nested), and replace the allOf​ key with the dict resulting from the merger.
​​
​​As a new Zulip release is coming (1.9), the usual translation cycle started: Tim pushed a batch of strings for us the translators to work on before making the release. I translated to Spanish the ~100 strings we needed, and made a couple small fixes of i18n-related problems I came across when translating:

​
​Week 3 — 2018/05/28

​​This week I focused on followup work, in order to get my open PRs merged. This involved making changes in PRs I worked on during the past two weeks until they were good to go.
​​
​​The most notable change consisted on extending our API testing framework, to be able to test the endpoint’s results regardless of the contents of the database. The need for having this came from the fact that a small part of the data in our dev env’s database is randomly generated, so it’s not possible to check an endpoint’s response against a static fixture in some cases.
​​
​​The new system was based in typing, by making sure that every key in the response’s object matched the type we expect it to be. The source of information on what to expect is the OpenAPI file, which supports specifying the type of every field in a schema.
​​
​​After this, I’ve started to work on documenting endpoints. The first step is migrating everything in the old format to the new one, and going on from there.

​
​Week 4 — 2018/06/04

​​Mostly documenting endpoints, by doing the migration and creating docs for those endpoints that weren’t documented. I came across some bugs in yamole while working on this, so I had to fix them to have the docs displayed properly.
​​
​​As discussed with Tim, I’ll open a PR when I have a decent amount of changes that have allowed me to make sure that both documentation and infraestructure are realiable enough to merge into master.

​
​Week 5 — 2018/06/11

​​Finished the migration of the old endpoints and started completing the actions that were missing (e.g. the POST verb in endpoint X), starting with the following list of prioritary endpoints:
​​
​​messages/​
​​messages/<message_id>​
​​fetch_api_key/​
​​dev_fetch_api_key/​
​​get_stream_id/​
​​streams/​
​​streams/<stream_id>​
​​streams/<stream_id>/members/​
​​users/​
​​users/<user_id>​
​​users/<user>/presence​
​​users/me/subscriptions​
​​user_uploads/​
​​realm/​
​​

​
​Week 6 — 2018/06/18

​​Continued with the endpoint documentation sprint. I also started filing the first pull requests to get my work from the past days merged:

​
​Week 7 — 2018/06/25

​​More work on API docs:
​​Plus a fix in /messages/<message_id>/history​’s logic, that solved the internal server error produced when fetching the history for a message that had never been edited before:

​
​Week 8 — 2018/07/02

​​After talking to Tim, we decided that the next endpoints to prioritize are those we use in our mobile and terminal clients. So here are the ones we use in the mobile app:
​​
​​messages/flags
​​messages/<message_id>/reactions
​​users/me/subscriptions/muted_topics
​​users/me/subscriptions/properties
​​users/me/alert_words
​​users/me/presence
​​users/me/android_gcm_reg_id
​​users/me/apns_device_token
​​user_groups/create
​​realm/emoji
​​realm/filters
​​settings/notifications
​​server_settings
​​mark_all_as_read
​​mark_stream_as_read
​​mark_topic_as_read
​​typing
​​dev_list_users
​​
​​This also made me notice that there are some endpoints that were not in my complete list of endpoints, which is probably because new ones have been added since a year ago (when that list was compiled). I will have to update the list soon to have an up-to-date view of our current status with the docs.
​​
​​Along the process, when hunting an example in our JavaScript bindings, I came across an important-ish issue that made many methods in the bindings modify the developer’s input without them knowing. I filed a PR fixing it:
​​And here are the rest of the PRs I filed along the week. 

​
​Week 9 — 2018/07/09

​​I’m in Allappuzha! Tim and I had a chat and decided that during this week I’d try to implement into Zulip a way to have multiple API keys per user. In parallel, I’ve kept working on REST API documentation as well:

​
​Week 10 — 2018/07/16

​​The Zulip event in Kerala ended on Monday, and I took this week off to visit a couple more countries in Asia.

​
​Week 11 — 2018/07/23

​​The trip was amazing! After returning home I kept working on the multiple API docs feature, and already have a first functional version. Most of the time spent after that was dedicated to fix all the tests that failed after adding these, which were quite a few (we make an intensive use of users’ API keys in our backend tests). I was able to file a PR by the end of the week:
​​I also talked to Tim about merging the work I’ve done so far on the API docs side, since I have a lot of local commits that should be ready to merge.

​
​Week 12 — 2018/07/30

​​This week I had to spend some time doing followups on my previous PRs: after making an excessively big commit in my multiple-API-keys PR, I had to do some Git work to split it into smaller pieces of work to make it more reasonable to review and merge. We found some things that could be improved on the docs part, as well.
​​
​​On the API keys project, the next step was creating endpoints in our REST API in order to list and revoke any keys associated to a user. The endpoints are now almost ready, I only have to write some unit tests for them before submitting the code for review.
​​
​​A few prioritary issues popped up, and we wanted to have them fixed before the upcoming 1.9 release. I claimed one of them and fixed it:
​​And these are this week’s PRs for my previous work on API docs:

​
​Week 13 — 2018/08/06

​​So the plan for this week is finishing with the API docs and get all my work in public, reviewable PRs.
​​On the process I also recalled a conversation with Tim about how would it be possible to detect changes in the OpenAPI file and automatically reload it without needing to restart the server. Here’s the PR with the implementation:
​​And some other things that popped up along the process
​​