
<p><iframe allow="monetization" class="embedly-embed" src="//cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLwqUp4Dc1mQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLwqUp4Dc1mQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FLwqUp4Dc1mQ%2Fhqdefault.jpg&amp;key=d932fa08bf1f47efbbe54cb3d746839f&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="360" scrolling="no" title="GitHub Universe MCP Server Demo video" frameborder="0" allowfullscreen="true"></iframe></p>

I was honoured to be invited to demo the GitHub MCP Server at GitHub Universe with Toby Padilla. This was a chance to showcase the work my team has been doing on agent extensibility with a live demonstration of the server in action.

The demo covered setting up Copilot agent mode to interact with GitHub from VS Code, key MCP tools, prompt engineering, and agent-to-agent capabilities that unlock powerful multi-step workflows through natural language.

## The Week Around Universe

Universe was an epic and ridiculously well organised event. There was a huge buzz around MCP and I had a lot of great conversations with developers and builders exploring agent extensibility.

I also attended <a href="https://resources.github.com/mcp-universe-2025/">MCP Universe</a>, a micro summit for agentic builders hosted alongside GitHub Universe. The event brought together open source contributors, startup founders, and enterprise builders to explore MCP and the agentic web. I attended with Carlos Becker, maintainer of the awesome GoReleaser tool.

The event featured an impressive lineup of speakers including David Soria Parra (Anthropic), Den Delimarsky (Microsoft), and Nick Cooper (OpenAI) from the MCP Steering Committee. We also heard from Jeremiah Lowin (FastMCP, Prefect.io), Shannon Williams (Obot), Toby Padilla (GitHub), Vitaly Gordon (Faros AI), Angie Jones (Block), Matvei Popov (Roboflow), and Cathy Di (Dedalus). The event was moderated by Rob Ferguson (Microsoft for Startups), Kevin Crosby (GitHub), and Simina Pasat (GitHub).

Later in the week, I attended the Figma MCP Meetup at Figma HQ. It was another great night of MCP discussion featuring talks from Jake Albaugh and Aditya Muttur from Figma, Harald Kirschner from Microsoft, Tadas Antanavicius (PulseMCP), and Gagan Bhat from Anthropic. It was a chance to connect with VS Code team members and other MCP adopters in person.

## Other Highlights

I took my first Waymo rides around San Francisco, which was an experience in itself.

I also took home a hackable badge from the event after quickly hacking it to show the GitHub MCP Server star count (which has almost hit 25k stars!).


GitHub Universe: MCP Server Demo


I've been using [reMarkable](https://remarkable.com/) tablets since 2021. They're excellent for capturing notes, annotating PDFs, and generally keeping your focus where it should be. But getting content back out has always been a friction point, and I finally got annoyed enough to do something about it.

I built an [MCP](https://modelcontextprotocol.io/) server that lets AI assistants read your reMarkable documents directly. What started as a weekend project turned into something genuinely useful.

## Why I Built This

I've been gradually building up an [Obsidian](https://obsidian.md/)-based second brain for my MCP R&D work. It's where I index ideas, draft articles, prepare conference talks, and collect academic papers. The system works well for digital content, but my reMarkable had years of handwritten notes that weren't part of it. Conference notes, research annotations, talk outlines, margin notes on PDFs, ideas and TODO lists. All that knowledge was trapped on the device, invisible to my Obsidian vault and not accessible for AI agents either.

reMarkable does offer ways to export content and get OCR through their cloud service. But somehow I never actually did it. The friction of manual export, one document at a time, meant those notes just sat there accumulating.

Two things inspired me to finally solve this. First, my day job. I build the [GitHub MCP Server](https://github.com/github/github-mcp-server), so I spend a lot of time thinking about MCP tool design. Second, [Scrybble Sync](https://scrybble.ink/), which syncs reMarkable to Obsidian. Scrybble showed me what was possible, but I wanted something that worked directly with AI assistants rather than as a sync tool.

So I built remarkable-mcp. Within a day, I had completely indexed everything from my reMarkable. Hundreds of documents spanning years of notes. Ideas I'd forgotten about resurfaced. Connections I'd missed became obvious. My handwritten notes were no longer lost; they were searchable, queryable, and integrated into my knowledge system. Sometimes you need to build the tool that finally makes you do the thing you've been putting off.

## SSH Mode is the Way

My first version used the reMarkable Cloud API. It worked, but felt sluggish. Then I pivoted to using SSH to connect directly to the tablet over USB, and the difference is dramatic (the cloud APIs were just not built to support my use-cases although the MCP Server still supports it). Fortunately, I already had developer mode enabled on my tablet so it was very straightforward to get going.

Connecting via USB and SSH gives you direct filesystem access. No cloud roundtrip, no subscription gatekeeping for basic functionality. The tablet becomes a local device you can query instantly. The [reMarkable guide](https://remarkable.guide/guide/access/ssh.html) has excellent documentation on enabling developer mode.

I should note that I actually enjoy reMarkable Connect. I have both a Paper Pro Move for when I'm out of the house and a Paper Pro for reading academic papers, and cloud sync across devices is genuinely useful. I just think basic programmatic access to your own documents should be available to all owners, not just subscribers. SSH mode makes that possible.

## Five Tools, Not Fifty

The reMarkable API has dozens of endpoints. I could have exposed them all as tools. Instead, I shipped five [MCP tools](https://modelcontextprotocol.io/docs/concepts/tools):

| Tool | Purpose |
|------|---------|
| `remarkable_read` | Extract text from any document |
| `remarkable_browse` | Navigate folders or search by name |
| `remarkable_search` | Search content across multiple docs |
| `remarkable_recent` | See what you've worked on lately |
| `remarkable_status` | Check connection health |

That's it. These five cover maybe 95% of what you'd actually want to do. The constraint forced better design decisions. Each tool is deeply capable rather than superficially complete.

This approach comes from [our research on the GitHub MCP Server](https://github.blog/engineering/copilot/how-were-making-github-copilot-smarter-with-fewer-tools/). We found that having too many tools actually decreased resolution rates by 2-5%. Models get confused when they have dozens of similar-looking options. Fewer, more capable tools perform better.

## Native Text Extraction

Something most people don't know: reMarkable tablets running v3+ software store typed text in a structured format. If you use the Type Folio keyboard or the on-screen keyboard, that text is stored as data, not pixels.

The [`rmscene`](https://github.com/ricklupton/rmscene) library can parse it directly. This means instant text extraction for Type Folio documents, typed annotations on PDFs, quick notes, and any typed content in notebooks. No OCR needed, no delay, no cost.

For handwritten content, you'll need OCR.

## Handwriting OCR

I originally used Tesseract for OCR. It's free, offline, and works great for printed text. For handwriting? Pretty terrible unless you write like a font. My handwriting is atrocious, so bad my 10-month-old daughter probably writes better than I do, so Tesseract was essentially useless for my notes.

Google Cloud Vision is a different story. Their handwriting recognition is genuinely impressive. My scrawled meeting notes become readable text. The service offers 1,000 free requests per month, then costs about $1.50 per 1,000 images. That's reasonable for actual use.

The server supports both: Google Vision for quality, Tesseract as an offline fallback.

## Response Design

David Cramer's [work on MCP tool design](https://cra.mr/subagents-with-mcp) influenced how I think about responses. Every tool response includes a `_hint` field suggesting logical next steps:

```json
{
  "documents": [
    {"name": "Meeting Notes", "path": "/Work/Meeting Notes"}
  ],
  "_hint": "Found 1 document. To read: remarkable_read('Meeting Notes')."
}
```

This matters more than you'd think. AI models work better when responses guide them toward sensible follow-up actions. Errors don't just say "failed," they explain what went wrong and suggest fixes:

```json
{
  "_error": {
    "type": "document_not_found",
    "message": "Document 'Meting Notes' not found",
    "suggestion": "Did you mean: 'Meeting Notes'?",
    "did_you_mean": ["Meeting Notes", "Meeting Notes 2"]
  }
}
```

## Root Path Filtering

Sometimes you don't want an agent accessing all your documents, so I added: `REMARKABLE_ROOT_PATH`. Set it to a folder e.g. `/Work` and the AI only sees your work documents. Set it to `/Journals` and it only accesses your journals. Whatever you need.

```json
{
  "env": {
    "REMARKABLE_ROOT_PATH": "/Work"
  }
}
```

This is useful for privacy (limiting AI access to specific folders), for focus (pinning to a project folder during deep work), and for context (different configs for different AI workflows). When configured, all paths are relative to the root. Browsing `/` shows the contents of `/Work`. Documents outside the provided root simply don't exist to the AI.

## MCP Resources

Beyond tools, MCP supports [Resources](https://modelcontextprotocol.io/docs/concepts/resources), which are data the AI can access directly without tool calls. Every document registers as a resource:

```
remarkable:///Meeting%20Notes.txt
remarkable:///Work/Project/Report.txt
```

In SSH mode, you also get raw file access (which provides the full text of the PDF or EPUB):

```
remarkableraw:///Research%20Paper.pdf
remarkableraw:///Books/Deep%20Work.epub
```

These appear in VS Code's MCP resource panel, letting you attach documents directly to conversations.

## Technical Stack

- **[FastMCP](https://github.com/jlowin/fastmcp)** for the [MCP server](https://modelcontextprotocol.io/docs/concepts/servers) framework
- **rmscene** for native .rm file parsing (typed text, no OCR)
- **PyMuPDF** for PDF text extraction
- **[Google Cloud Vision](https://cloud.google.com/vision)** for handwriting OCR
- **Paramiko** for SSH transport

The codebase is modular: separate files for tools, resources, API interactions, and extraction logic. Makes testing easier and keeps the mental model clean.

## What I Learned

**Design for intent, not capability.** Users don't want to "call the document metadata endpoint." They want to "find my notes from last week."

**SSH beats cloud for local devices.** Direct access is faster, works offline, and doesn't require subscriptions for basic functionality.

**Guide the next action.** Response hints aren't hand-holding. They're context that helps AI models make better decisions.

**Good OCR is worth paying for.** Tesseract is fine for printed text. For handwriting, Google Vision is transformatively better.

**Constraints improve design.** Limiting to five tools forced me to make each one deeply capable.

## Try It

The package is available on [PyPI](https://pypi.org/project/remarkable-mcp/).

### SSH Mode (Recommended)

See the [SSH Setup Guide](https://github.com/SamMorrowDrums/remarkable-mcp/blob/main/docs/ssh-setup.md) for enabling developer mode on your tablet.

Add to your `.vscode/mcp.json` or similar:

```json
{
  "inputs": [
    {
      "type": "promptString",
      "id": "google-vision-key",
      "description": "Google Vision API Key",
      "password": true
    }
  ],
  "servers": {
    "remarkable": {
      "command": "uvx",
      "args": ["remarkable-mcp", "--ssh"],
      "env": {
        "GOOGLE_VISION_API_KEY": "${input:google-vision-key}"
      }
    }
  }
}
```

### Cloud Mode

1. Visit [my.remarkable.com/device/desktop/connect](https://my.remarkable.com/device/desktop/connect) to get a one-time code
2. Convert it to a token:

```bash
uvx remarkable-mcp --register YOUR_CODE
```

3. Add to your `.vscode/mcp.json`:

```json
{
  "inputs": [
    {
      "type": "promptString",
      "id": "remarkable-token",
      "description": "reMarkable API Token",
      "password": true
    },
    {
      "type": "promptString",
      "id": "google-vision-key",
      "description": "Google Vision API Key",
      "password": true
    }
  ],
  "servers": {
    "remarkable": {
      "command": "uvx",
      "args": ["remarkable-mcp"],
      "env": {
        "REMARKABLE_TOKEN": "${input:remarkable-token}",
        "GOOGLE_VISION_API_KEY": "${input:google-vision-key}"
      }
    }
  }
}
```

The [Quick Install badges](https://github.com/SamMorrowDrums/remarkable-mcp) in the README handle VS Code configuration automatically.

## What's Next

The [GitHub issues](https://github.com/SamMorrowDrums/remarkable-mcp/issues) track future plans: write support, more OCR providers, semantic search, and export features. But for now, the core experience is solid. Your reMarkable as a queryable second brain. If you encounter any problems, that's the best place to file them.


Building an MCP Server for reMarkable


<p><iframe allow="monetization" class="embedly-embed" src="//cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F0d9TR91iLx0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D0d9TR91iLx0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F0d9TR91iLx0%2Fhqdefault.jpg&amp;key=d932fa08bf1f47efbbe54cb3d746839f&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="360" scrolling="no" title="LastMile AI Silsila Sessions: MCP at Hyperscale video" frameborder="0" allowfullscreen="true"></iframe></p>

I had the pleasure of joining the [LastMile AI](https://lastmileai.dev/) Silsila Sessions this week, where we went deep on what it really takes to make MCP agents production-ready. The session brought together some fantastic speakers to discuss the core challenges of building secure, scalable agent systems.

## The Panel

- **Sam Morrow (GitHub)** — I broke down what it really looks like to run an MCP server at scale, from tool bloat to evals to security surprises.
- **Tobin South (WorkOS)** — Unpacked the practical realities of OAuth, agent identity, and enterprise-grade authentication.
- **Sarmad Qadri (LastMile AI)** — Shared upcoming MCP spec updates and how they unlock more advanced agent behaviors.

## Key Takeaways

One of my main points during the session was around the tradeoffs between atomic tools and higher-level workflow tools:

> "Atomic tools give you flexibility, but too many create context bloat and tool confusion — workflow-style tools are where agents really shine."

This is something we've learned firsthand running MCP at hyperscale at GitHub. When you're operating at this level, the design decisions around tool granularity become critical. Too many fine-grained tools and you overwhelm the model with choices; too few and you lose the flexibility that makes agents powerful.

We also discussed:
- Running MCP at hyperscale and the unique challenges that come with it
- OAuth and agent identity in enterprise contexts
- Upcoming MCP spec changes that open the door to long-running workflows and smarter agents

Huge thanks to [LastMile AI](https://lastmileai.dev/) for hosting, and to everyone who joined live. These conversations are invaluable for pushing the MCP ecosystem forward.


LastMile AI Silsila Sessions: MCP at Hyperscale


Toby Padilla, Harald Kirschner and I will be live-streaming on The Context, on the MCP Developers Summit YouTube channel this coming Tuesday.

https://www.youtube.com/live/rtht8dbwe5U?feature=shared

The channel hosts the amazing content from their conference, as well as numerous videos and livestreams on all sorts of MCP and AI related topics.

I'm honoured to be invited to talk about my involvement in building the GitHub MCP Server, there have been so many legends in the field on their channel! I can't wait to share the ideas, problems and dreams we have.

The Context - Special Episode ft. GitHub & Microsoft



<p><iframe allow="monetization" class="embedly-embed" src="//cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FT44T55wvDD0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DT44T55wvDD0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FT44T55wvDD0%2Fhqdefault.jpg&amp;key=d932fa08bf1f47efbbe54cb3d746839f&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="360" scrolling="no" title="YouTube embed" frameborder="0" allowfullscreen="true"></iframe></p>

> Join us for VS Code Live: Extending Agent Mode, a livestream showing how to unlock the full potential of agent mode using MCP servers and extension tools. We'll kick things off with a look at VS Code's vision in the MCP space, followed by demos from the VS Code team. Plus, we'll spotlight powerful MCP servers and extensions that you can start using in agent mode right away - featuring special guests you won't want to miss!

I had great fun chatting with Olivia Guzzardo McVicker (@OliviaGuzzardo) on the VS Code Livestream on YouTube about the [Remote GitHub MCP Server](https://github.blog/changelog/2025-06-12-remote-github-mcp-server-is-now-available-in-public-preview/) that launched during the episode! We chat about the server, I demo some functionality live and we also chat about using and extending agents in general.

I hope she'll have me back on again in future to see what we've achieved over the next few months.

VS Code Live: Extending Agent Mode


If you're like many people, the rapid rise of AI tools might bring a mix of excitement and, perhaps, some unease. It's natural to wonder what it all means for how we work. I've been thinking a lot about this, especially in my role with GitHub Copilot (although comments and opinions on this blog are my own), and I've noticed something that feels important to share.

Will AI make our existing tasks faster and easier, or will it replace our jobs entirely? I believe the answer is neither.

I've come to a different conclusion, which I think is more honest: AI probably won't make your job easier in the ways you expect, but it *can* help you do work that's genuinely better than what you might have produced before, and I do not believe we have arrived at an employment extinction event.

---

## The Quality Shift

When powerful AI tools become widely accessible, something interesting happens to the value of work.

Standard, good-enough output becomes abundant. Whether it's code, writing, or analysis, the baseline of acceptable is easier for everyone to reach. This also means that simply meeting that baseline is no longer a differentiator, worse than that it's almost indistinguishable from the noise floor of AI slop that people are already cursing the internet with. Tasks AI can do excellently autonomously, should be offloaded to AI, so what is left?

Truly excellent work—work that demonstrates deep understanding, thoughtful judgment, or unique insight—becomes necessary to stand out from the new noise floor.

This isn't just about working harder or faster. It's about the nature and quality of what we produce, and redefining what our best work is.

> Something I told 16 yo: In the near term, AI will increase variation in the returns for work. For example, mediocre programmers are now finding it hard even to get hired, but great programmers are making more than ever.

> Something else I told him: This is a trend that has been going since the stone age. Technology always increases variation in the returns for work. The bottom end is anchored at zero. Technology lets the top end move.

*Paul Graham*

---

## Moving Beyond "Write This For Me"

I've noticed a common pattern (and I've done this myself) when we first start using AI tools:

We provide minimal input, ask the AI to generate something complete, and then make a few tweaks.

It often works, to a degree. The output is usually... fine. But it rarely feels remarkable, and sometimes there's a subtle "off-ness" that's hard to pinpoint or fix with simple edits.

I've come to believe this isn't necessarily a limitation of the tools themselves, but more about how we're engaging with them.

---

## Finding A More Collaborative Approach

What's been working better for me is treating AI less like a vending machine and more like a thinking partner:

- Using it to explore various approaches to a problem, some of which I wouldn't have considered on my own.
- Asking it to critique my work (and then, importantly, critiquing its critique in return). With current models you have to push hard for critical feedback, they behave like sycophants by default, so ignore their flattery or at least remain suspicious of it.
- Using it to help articulate complex thoughts I'm struggling to put into clear words.
- Giving it rich context: not just *what* I'm trying to do, but *why* and what the broader goals are. This includes doing real research. Data, citations and subtle nuance are all things you should provide as context. 

It's important to note that these approaches often don't save time upfront. In fact, they can sometimes take longer than just diving into the work directly. The real difference I'm seeing is in the depth and quality of the final result.

---

## A Personal Example

Recently, I was wrestling with an annual performance self-reflection. My early drafts lacked a narrative that contextualised the relevance of my work to the company as a whole, undersold things and also lacked hard numbers, citations and other additions that would give clout to my words.

Instead of prompting AI to "write my self-reflection," I tried a more layered process:

I gathered all the relevant context—project outcomes, feedback from colleagues, company values, specific challenges I'd encountered—and then worked with an AI to help me structure and articulate this information coherently.

We went back and forth multiple times. I'd challenge its framing of a point; it would prompt me to consider an angle I'd missed. We explored different ways to structure the narrative.

The final document definitely took longer to create than if I'd just pushed through on my own. But it was clearer, more balanced, and, I think, more insightful than what I would have produced in isolation. I expect the next one will be much faster too.

---

## Some Things I've Learned About Working With AI

If you're looking to explore this kind of collaborative approach, here are a few things I've found helpful:

- **Be generous with context.** AI tools provide better assistance when they understand not just *what* you want, but *why* you want it and what success looks like.

- **Use it for thinking, not just producing.** Some of the most powerful uses involve exploration, brainstorming, critique, and refining ideas, rather than just generating finished products from scratch.

- **Expect to iterate.** As screenwriting guru Robert McKee wisely said: "Secure writers don't sell first drafts. They patiently rewrite until the script is as director-ready, as actor-ready as possible. Unfinished work invites tampering, while polished, mature work seals its integrity." This principle holds true for AI collaboration; the real value often emerges through a process of refinement.

- **Give yourself permission to experiment.** The most effective ways to use these tools are still being discovered. What works best for you might be different from what works for others, and new models are released almost daily.

---

## It's Not Just About Productivity

One of the most interesting realizations for me has been that the primary value here isn't solely about boosting productivity in the traditional sense.

A thoughtful VP at Cloudflare, William Allen, put it well: "The lesson isn't 'do more with less'—it's 'do more with more.'"

The opportunity isn't just to produce the same work faster. It's about raising the ceiling on the quality and impact of what we can achieve.

---

## A Suggestion

If you're navigating the world of AI tools, perhaps consider shifting your primary expectation from "this will make my current job easier" to "this might help me do work I'm even more proud of."

With that mindset, the learning curve can feel different. The occasional frustrations or clunky outputs become part of the investment in a new skill, rather than just failures of the tool.

And the collaboration—because that's what it truly is when it works best—can lead to outcomes that genuinely surprise you, in a good way.

Not necessarily faster, not always easier. But potentially, meaningfully better.

---

## A Note on How This Came Together

If you've made it this far, thanks for reading! I wanted to share a little about the process behind this piece, as it mirrors some of the iterative collaboration I've been discussing.

This article didn't spring out fully formed. It started as a stream-of-consciousness reflection on my experiences and observations. From there, it went through quite a journey:

*   I ran initial drafts and ideas through four different AI models, using them not just for generation, but for critical discussion, feedback on tone, and to explore different ways of articulating the core message.
*   There were many rounds of manual edits and rewrites based on my own reflections and the AI's input.
*   I consciously wove in quotes from people whose ideas resonated with the points I wanted to make.
*   Crucially, I also shared drafts with human friends and colleagues to get their invaluable perspectives and feedback.
* I consciously left in em dashes (I am not pretending I didn't get help from AI, why would I? This is exactly what I wanted to write and better use of punctuation is a gift)
*   Finally, I let it sit for a while, giving myself time to reflect and ensure it felt authentic to my voice and experience before feeling ready to share it.

It was a process of layering, refining, and sometimes stepping away. I genuinely hope you enjoyed reading it, and more importantly, that you found something to consider in your own interactions with AI.

For my part, I certainly enjoyed the process of producing it and learned a lot more about my own writing style and how to collaborate effectively with these new tools along the way.

Blog

GitHub Universe: MCP Server Demo

Building an MCP Server for reMarkable

LastMile AI Silsila Sessions: MCP at Hyperscale

The Context - Special Episode ft. GitHub & Microsoft

VS Code Live: Extending Agent Mode

AI Won't Make Your Job Easier—so what will it do?