Weekly status updates | GSoC project page | List of GSoC pull requests

Goals

• Create a markdown toolbar
• Create a freehand drawing widget
• Add the freehand drawing widget to the mobile app
• Add the freehand drawing widget to the desktop app
  • This was done via a plugin.

The image editor/freehand drawing widget

js-draw on NPM | Joplin Plugin

While creating my GSoC proposal, I decided to spend my summer working on two sub-projects: building a markdown toolbar for Joplin’s mobile app and creating an image editor for both the mobile and desktop apps.

While js-draw hasn’t yet been added to the mobile app, I’ve added it to the desktop app through a plugin.

Try js-draw below!

Features

• Zoom levels from \(2\cdot 10^{-10}\times\) to \(1 \cdot 10^{12}\times\).
• Keyboard shortcuts:
  • h, j, k, l, a, d, and arrow keys move the view.
  • w and s zoom the view in/out.
  • ctrl+z, ctrl+Shift+Z: undo and redo.
• Bézier curve fitting:
  • Strokes are converted into Bézier curves
  • This uses bezier-js for projecting points onto Bézier curves, finding normals, and Bézier curve intersection testing.
• Localization:
  • Much of the UI has a Spanish translation. If navigator.languages contains es or es-MX/es-ES (or a similar Spanish-based locale), this translation is used.
• Partially accessible to a screen reader:
  • Text objects within js-draw can be read with a screen reader.
  • Users with a screen reader can focus a textarea element to send keyboard input.
  • Actions are announced.
    • For example, after pressing w in the textarea, the screen reader announces that the view was moved left.
  • There’s still work to be done:
    • Drawing new strokes requires a mouse (or on a phone with TalkBack, touching the screen with two fingers).
    • Strokes aren’t described to screen readers.
    • It’s currently difficult/impossible to change the color of a pen with a screen reader.
      • I’m using the Coloris library for color selection. While Coloris should be accessible, perhaps due to its configuration, it’s very difficult to select and close the color picker with a screen reader.

What’s done

• Tool that allows adding text to an image.
• Pen/drawing tool
• Saving/loading from a note in Joplin
• Add to the desktop app
  • Done as a plugin
• Add to the mobile app
• Realtime collaborative editing
  • This wasn’t originally planned, but I’ve started working on it — serialization to JSON for some of the important Command subclasses (e.g. AddElementCommand, Erase, …) has been implemented. Serialization and deserialization will allow these commands to be sent over a network.

The markdown toolbar

UI pull request | Markdown commands pull request

The markdown toolbar is a feature added to Joplin’s (beta) mobile editor. It allows quickly changing the formatting of a region.

What’s done

• Find and replace tool
  • Implemented and merged
• Markdown commands (merged and implemented)
  • Bold, italicize, toggle code region, toggle \(\TeX\) region, toggle header
  • Edit/insert link
• Markdown toolbar
  • Implemented and merged

Bugs and what isn’t done

• On iOS, the keyboard for the markdown editor is always in light mode
  • As of the time of this writing, react-native-webview doesn’t support this. I tried adding support, but struggled with building react-native-webview and and the solution uses runtime type manipulation (which I’m hesitant to use as I’m very new to Objective-C).
  • Joplin issue tracker

Better markdown syntax highlighting

Relevant pull request (merged)

• Added syntax highlighting for math (\(\KaTeX\)) regions in the CodeMirror 6 editor.
  • This involved writing a custom CodeMirror parser to recognize regions surrounded by $s and $$s.
  • This functionality supports the markdown toolbar — the markdown toolbar uses CodeMirror’s syntax tree to determine the bounds of regions like math and code. As such, moving the cursor within a math region and clicking the “toggle math” button on the markdown toolbar can toggle an entire math region.
• Added syntax highlighting for fenced code regions
  • This did not require writing a custom parser.

Reflection

I accomplished my major goals for this summer’s GSoC project: I created a markdown toolbar for the mobile app and an image editor (with zoom from \(2\cdot 10^{-10}\) to \(1 \cdot 10^{12}\) times zoom!).

More time was spent revising code and responding to feedback than expected. For example, although I had a draft of the markdown toolbar created and ready (in my mind) to merge in the first few weeks of the program, it took a significant amount of refactoring, revising, and breaking my work into smaller pull requests to get the markdown toolbar merged into Joplin. If I apply for GSoC again, I will schedule more time for responding to feedback.

Additionally, I struggled with several bugs in React Native, including its vanishing KeyboardAvoidingView, improperly-sized KeyboardAvoidingView, and react-native-webview’s lack of support for dark-mode keyboards on iOS. While some of the issues related to the vanishing KeyboardAvoidingView have been fixed upstream, as of the time of this writing, many of these issues remain unsolved. Fixes may require pull requests to the upstream projects.

Even with the above challenges, I consider this summer’s GSoC project successful.