I recently made some improvements to my personal site. The updates included swapping over to TailwindCSS and some other architectural improvements using Jigsaw. One of the features I like the most is an activity feed on the home page that retrieves some of my recent GitHub activity and my highest voted Stack Overflow answers. I was able to gather the data and access it through a CDN with virtually no-cost, or infrastructure setup, using GitHub's Flat Data.

If you aren't familiar with githubocto/flat it provides a GitHub Action to easily run Git scraping tasks. Git scraping is an approach brought to the mainstream by Simon Willison that allows you to easily track changes to a data set over time. There's a bunch of neat, potentially socially impactful, things you can do like tracking how much of California is under fire or a COVID-19 tracker. I decided it would be a great way, though probably not as meaningful, to start getting dynamic data into my statically generated sites.

Gathering the Data

The first step in gathering the data is configuring your Flat Action. While I understand why GitHub Actions are configured with YAML I don't especially care for the markup language. Fortunately, the team over at githubocto/flat also published a VS Code Extension, Flat Editor. I don't normally develop with VS Code but I'll gladly use it if I can avoid writing YAML! I installed the OSS build available on my Linux distro but any build should work. After I loaded the extension I was able to easily configure the Action with a UI that looked like this in the end.

I really appreciate just how easy it was to get the data into the repository and start working with it. It's just 3 configuration values! With support for SQL statements and the recent addition of fully-configurable HTTP requests the possibilities for how you can get data with a Flat Action are pretty flexible!

Massaging the Data

If you're not familiar with GitHub's events there are quite a few of them. I didn't have the time necessary to implement a UI for all of the events so I chose to implement those that are most valuable; creating something, pushing a commit, or starring a repository. I wanted to make sure the front-end was super simple and didn't have to worry about filtering data or doing any processing that could be handled in the Action. You'll note in the screenshot of VS Code above there's a processing file specified. This is a Deno script that runs after your data has been gathered. For GitHub all I need to do is filter out the appropriate events and clean up the data structure to be a little easier for my front-end to work with.

import { readJSON, writeJSON } from 'https://deno.land/x/flat/mod.ts';

// The filename is the first invocation argument
const filename = Deno.args[0];
const data = await readJSON(filename);

const cleanData = [];

data.forEach(ghEvent => {
    if (ghEvent.type === 'PushEvent' || ghEvent.type === 'CreateEvent' || ghEvent.type === 'WatchEvent') {
        const cleanEvent = {};
        cleanEvent.id = ghEvent.id;
        cleanEvent.type = ghEvent.type;
        cleanEvent.createdAt = ghEvent.created_at;
        cleanEvent.payload = {};
        cleanEvent.payload.repoName = ghEvent.repo.name;
        cleanEvent.payload.repoUrl = `https://github.com/${ghEvent.repo.name}`;
        if (ghEvent.type === 'PushEvent') {
            cleanEvent.payload.branch = ghEvent.payload.ref.replace('refs/heads/', '');
            cleanEvent.payload.branchUrl =  `${cleanEvent.payload.repoUrl}/tree/${cleanEvent.payload.branch}`;
            cleanEvent.payload.head = ghEvent.payload.head.substring(0, 7);
            cleanEvent.payload.headUrl = `${cleanEvent.payload.repoUrl}/commit/${ghEvent.payload.head}`;
        } else if (ghEvent.type === 'CreateEvent') {
            cleanEvent.payload.refType = ghEvent.payload.ref_type;
            if (cleanEvent.payload.refType === 'branch') {
                cleanEvent.payload.branch = ghEvent.payload.ref;
            }
        } else if (ghEvent.type === 'WatchEvent') {
            cleanEvent.payload.action = ghEvent.payload.action;
        }

        cleanData.push(cleanEvent);
    }
});

await writeJSON('recent_github_activity.json', cleanData);

Overall this doesn't look too bad! My front-end now has a more normalized structure to work with and I'm sure to only get the events that my front-end can actually process. If you have more control over the response from the HTTP request you could likely skip this step completely. I really appreciate Flat Data's decision to go with Deno here instead of the more traditional choice of Node.js. Not having to configure a package.json or really even worry about dependencies at all is very nice.

Showing the Data

Now I need to actually gather the data and display it on my front-end. I've intentionally setup my blog to not require any JavaScript to read articles and any scripting I do add needs to be optional content that's ok with not being there. These feeds definitely fall into this category! Since the JavaScript has always been kept minimal by design I chose not to introduce any browser-side templating libraries and instead just create the DOM elements I need with vanilla JavaScript. Showing the data really boils down to this event handler that runs whenever the home page document content is loaded.

document.addEventListener('DOMContentLoaded', async () => {
    const githubEventsNode = document.querySelector('#github-events-feed-list');
    const githubActivityResponse = await fetch('https://raw.githubusercontent.com/cspray/blog-activity-feed/main/recent_github_activity.json');
    const githubActivityEvents = await githubActivityResponse.json();

    githubActivityEvents.slice(0, 10).forEach(activityEvent => {
        const eventHandler = `create${activityEvent.type}Node`;
        if (eventDomHandlers[eventHandler]) {
            githubEventsNode.append(eventDomHandlers[eventHandler](activityEvent));
        }
    });
});

I have a JavaScript object that maps the activity events to a function that will parse out the appropriate data and return a DOMNode that represents the UI for that event. That's all there is to it! For now each of the event handlers creates elements using document.createElement and document.createTextNode. I imagine future upgrades to my site will bring inline templating so creating and updating these types of UIs are easier to deal with. I could also imagine this function getting called in a JavaScript timeout to make it auto-refresh.

Thoughts on Flat Data

In case I haven't made it clear I think Flat Data is awesome! I was able to include dynamic, updated data served through a CDN into my static website without spinning up any servers, connecting to databases, or any operational setup or changes to the way my site works... all for effectively free. I see a ton of potential in websites that can make use of a read-only dataset where a 5~ minute delay in data updates is good enough. I'm already thinking about ways I can add other interactive content to my site using a combination of Netlify functions for writing and Flat Data for reading. This could allow me to keep content on a network designed for speed and make sure my function invocation usage stays within Netlify's free tier... though realistically I doubt my site would ever see that much traffic! ;)

I hope you check out Flat Data and build something awesome with it! If you do, let me know on Twitter @charlesspray!