Nice. That latency is tough though. On that topic, I wrote an equivalent "magic mirror" type tool that lets you write to the mirror/oracle/VLM on the remarkable and get an answer back.

A few possible solutions I explored:

1) You can try and consume (and possibly write to) the frame buffer directly. https://github.com/ddvk/remarkable2-framebuffer was my starting point. This gets you instant updates about what's going on. I guess you could pair this with speculative decoding to get a much faster output.

2) You can use the streaming API on the device to stream the screen to a beefy server, possibly over Tailscale, letting you do everything off device.

3) You can write your own Qt app; ddvk's repos are a good starting point here.

I ultimately instructed claude to write me my own app. Which worked enough to scratch my itch, and I never use it. But this was five months ago, an eternity in vibe hobby projects, so perhaps modern tooling would let me get it in shape and be more usable. Basically, it worked, but VLMs weren't great at what I wanted -- decorating a blank page with a grimoire-style answer to a written question while leaving the original text alone -- and getting it started / stopped from the pen UI is difficult.

Where my mind goes for your project is that I think it'd be nicest to keep a sort of Jupyter notebook somewhere that's the canonical representation, that would have your handwritten blocks and an interpretation, and then the output. then a render layer to get it back onto the screen. At that point, I don't think I'd care very much if it's stored as a PDF on the device, which points back to having this be an app.

Either way it's fun to tinker! the RM line is very hackable, and I still wish it were even easier, the hardware just makes you think of so many possibilities.

This awesome! I actually had no idea I could ssh into my Remarkable and do neat stuff like this!

> Why do it? It's so impractical!

Because you can and it's fun is always a perfectly valid answer here!

I’m actually impressed by the handwritten blog - really cool concept.

Was it exported by writing on remarkable? How did he include link into the text that he wrote?

A fun experiment, and cute if not for the lag. A full 14 seconds between the pen stopping writing and the result,

What I'd find interesting is the trace of that 14 seconds. How much is the Remarkable processing, how much is the claude transcription, how much is the let-go start-up / processing, etc.

Amazing! I'd recommend using PaddlePaddle OCR models if you want to make your handwriting conversion local, it's definitely fast enough. Probably faster than the Sonnet call.

I tried the model on your handwriting and it worked great. My handwriting is bad enough that it doesn't work :)

I find getting stuff up on the rM2 screen a hassle.

The challenge comes from the way the e-paper works. To turn a pixel from white to black (or vice versa) it needs multiple actual frames. The pixel data must also be packed in a specific format. Instructions for how many frames a single operation requires are coded in a wbf[0] file, which comes included with the OEM firmware.

The most commonly used approach is hooking into xochitl, since it handles all of the user facing stuff like the notebooks but also the actual drawing. This is somewhat brittle and tends to break with software updates, because all of the actual function addresses have to be updated as well.

I was excited to find waved[1], a C++ library that allowed to drive the display directly using a sane API. Although it's not been updated for quite a while it still works and you can compile and run it yourself.

Since I was interested in driving the display myself, I tried to rewriting waved in Zig. It works - I can now get pixels up on the screen. Unfortunately the code is a mess and its only redeeming quality (stemming from being written in Zig) is that I can cross compile a statically linked binary that 'just werks' using just the Zig toolchain as the only dependency. For debugging purposes, I also implemented a SDL emulator for the display. [2]

While writing this, I stumbled upon a recent Rust implementation of the same ideas. Nice. [3]

Since the xochitl display implementation is the most optimized, that's probably the reason why it's being used most commonly*, even though it might be 'ugly'.

* Citation needed

[0] https://gitlab.com/zephray/glider#understanding-waveform

[1] https://github.com/matteodelabre/waved

[2] https://github.com/jakubvf/dazed

[3] https://github.com/yobert/swtcon

Great work.

The experience programming with handwritten code in tablets keeps being a pain, when it would be ideal as digital executable paper.

I wonder if this would be easier on the Supernote Manta with the new plugins API that they've introduced recently…

Ok this might be the coolest thing I've seen all month.

You must be a Recurser, right? This is so very Recurser.

Did foo(42) just output 44 ?

How do you handle latency between handwriting recognition and REPL evaluation is there a noticeable delay before you see the output?

Now I have a reason to get reMarkable :)

Very nice

y tho